Hybrid Evaluation for Occlusion-based Explanations on CNN Inference Queries

Deep CNNs are increasingly prevalent in various application domains such as image processing. To explain a CNN prediction, it is popular to employ occlusion-based explanations (OBE). OBE helps users understand which parts of an image are important to a CNN prediction. Existing systems have explored incremental evaluation to accelerate CNN inference in OBE. However, they are oblivious that incremental evaluation does not always outperform full evaluation for certain layers. To address this issue, we propose a hybrid evaluation to efficiently interleave full and incremental evaluations during the CNN inference. Ad-ditionally, it employs a cost model to compare the overhead costs of two types of evaluations and a heuristic method to determine the efficient plan combination for common CNNs. More impor-tantly, hybrid evaluation adopts a dynamic programming-based method for attention-based CNNs. In particular, the dynamic programming-based method significantly reduces the overhead of searching for the efficient plan combination on the complex DAG structure. To demonstrate the efficiency of our techniques, we implement HyInJ, a hybrid CNN inf erence system based on PyTorch. Our experiments show that HyInf reduces execution time by up to 22% on GPU and 55% on CPU in comparison to the state-of-the-art incremental evaluation.