Abstract
Classification neural networks fail to detect inputs that do not fall inside the classes they have been trained for. Runtime monitoring techniques on the neuron activation pattern can be used to detect such inputs. We present an approach for monitoring classification systems via data abstraction. Data abstraction relies on the notion of box with a resolution. Box-based abstraction consists in representing a set of values by its minimal and maximal values in each dimension. We augment boxes with a notion of resolution and define their clustering coverage, which is intuitively a quantitative metric that indicates the abstraction quality. This allows studying the effect of different clustering parameters on the constructed boxes and estimating an interval of sub-optimal parameters. Moreover, we automatically construct monitors that leverage both the correct and incorrect behaviors of a system. This allows checking the size of the monitor abstractions and analysing the separability of the network. Monitors are obtained by combining the sub-monitors of each class of the system placed at some selected layers. Our experiments demonstrate the effectiveness of our clustering coverage estimation and show how to assess the effectiveness and precision of monitors according to the selected clustering parameter and monitored layers.
Support from the European Union’s Horizon 2020 research and innovation programme under grant agreement No. 956123 - FOCETA and the French national program “Programme Investissements d’Avenir IRT Nanoelec" (ANR-10-AIRT-05).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
Recall that the k-means algorithm divides a set of N samples from a set X into k disjoint clusters \(\mathbb {C} = \left\{ C^1,\ldots , C^k \right\} \), each cluster \(C^j\) denoted by the mean \(\mu _j\) of the samples in the cluster, and aims to choose centroids that minimise inertia. .
- 2.
The starting point of searching a fine k can be optimized, details are omitted for space reasons.
- 3.
- 4.
To save computation cost (and favor reproducibility), we proceed 3 steps: 1) high-level features extraction, which can be one-time generated in seconds and used multiple times afterwards; 2) feature partition with many values of \(\tau \), during the experiment, which took the most time due to the search of the fine k. But, this can be very efficient if the options are tried out in descending order; 3) monitor creation and test, which can be done immediately.
References
Chalapathy, R., Chawla, S.: Deep learning for anomaly detection: a survey. arXiv preprint arXiv:1901.03407 (2019)
Chandola, V., Banerjee, A., Kumar, V.: Anomaly detection: a survey. ACM Comput. Surv. (CSUR) 41(3), 1–58 (2009)
Cheng, C.H.: Provably-robust runtime monitoring of neuron activation patterns. arXiv preprint arXiv:2011.11959 (2020)
Cheng, C.H., Nührenberg, G., Yasuoka, H.: Runtime monitoring neuron activation patterns (2019)
Cromwell, P.R.: Polyhedra. Cambridge University Press, Cambridge (1997)
Hendrycks, D., Gimpel, K.: A baseline for detecting misclassified and out-of-distribution examples in neural networks. arXiv preprint arXiv:1610.02136 (2016)
Henzinger, T.A., Lukina, A., Schilling, C.: Outside the box: abstraction-based monitoring of neural networks. In: ECAI 2020, pp. 2433–2440. IOS Press (2020)
Hodge, V., Austin, J.: A survey of outlier detection methodologies. Artif. Intell. Rev. 22(2), 85–126 (2004)
Huang, X., et al.: A survey of safety and trustworthiness of deep neural networks: verification, testing, adversarial attack and defence, and interpretability. Comput. Sci. Rev. 37, 100270 (2020)
Intelligence, F.C.F.A.: Research challenge ii: Dependability, finnish center for artificial intelligence (2018)
Krizhevsky, A., Hinton, G., et al.: Learning multiple layers of features from tiny images (2009)
LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)
Lloyd, S.: Least squares quantization in pcm. IEEE Trans. Inf. Theory 28(2), 129–137 (1982)
Lukina, A., Schilling, C., Henzinger, T.A.: Into the unknown: active monitoring of neural networks. arXiv preprint arXiv:2009.06429 (2020)
McMullen, P.: On zonotopes. Trans. Am. Math. Soc. 159, 91–109 (1971)
Murtagh, F., Contreras, P.: Algorithms for hierarchical clustering: an overview. Wiley Interdisc. Rev. Data Min. Knowl. Disc. 2(1), 86–97 (2012)
Pei, K., Cao, Y., Yang, J., Jana, S.: Deepxplore: automated whitebox testing of deep learning systems (2017)
Schubert, E., Sander, J., Ester, M., Kriegel, H.P., Xu, X.: DBSCAN revisited, revisited: why and how you should (still) use DBSCAN. ACM Trans. Database Syst. (TODS) 42(3), 1–21 (2017)
Sculley, D., et al.: Hidden technical debt in machine learning systems (2015)
Seshia, S.A., Sadigh, D.: Towards verified artificial intelligence. CoRR arXiv:1606.08514 (2016)
Sun, Y., Huang, X., Kroening, D., Sharp, J., Hill, M., Ashmore, R.: Testing deep neural networks. arXiv preprint arXiv:1803.04792 (2018)
Tian, Y., Pei, K., Jana, S., Ray, B.: Deeptest: automated testing of deep-neural-network-driven autonomous cars (2018)
Wicker, M., Huang, X., Kwiatkowska, M.: Feature-guided black-box safety testing of deep neural networks (2018)
Wu, C., Falcone, Y., Bensalem, S.: Customizable reference runtime monitoring of neural networks using resolution boxes. CoRR abs/2104.14435 (2021). https://arxiv.org/abs/2104.14435v2
Xiao, H., Rasul, K., Vollgraf, R.: Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747 (2017)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Wu, C., Falcone, Y., Bensalem, S. (2023). Customizable Reference Runtime Monitoring of Neural Networks Using Resolution Boxes. In: Katsaros, P., Nenzi, L. (eds) Runtime Verification. RV 2023. Lecture Notes in Computer Science, vol 14245. Springer, Cham. https://doi.org/10.1007/978-3-031-44267-4_2
Download citation
DOI: https://doi.org/10.1007/978-3-031-44267-4_2
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-44266-7
Online ISBN: 978-3-031-44267-4
eBook Packages: Computer ScienceComputer Science (R0)