More Web Proxy on the site http://driver.im/

Article

Free access

Precision and recall for time series

Authors:

Justin GottschlichAuthors Info & Claims

NIPS'18: Proceedings of the 32nd International Conference on Neural Information Processing Systems

Pages 1924 - 1934

Published: 03 December 2018 Publication History

PDF eReader Publisher Site

Abstract

Classical anomaly detection is principally concerned with point-based anomalies, those anomalies that occur at a single point in time. Yet, many real-world anomalies are range-based, meaning they occur over a period of time. Motivated by this observation, we present a new mathematical model to evaluate the accuracy of time series classification algorithms. Our model expands the well-known Precision and Recall metrics to measure ranges, while simultaneously enabling customization support for domain-specific preferences.

References

[1]

C. C. Aggarwal. Outlier Analysis. Springer, 2013.

Digital Library

[2]

S. Ahmad, A. Lavin, S. Purdy, and Z. Agha. Unsupervised Real-time Anomaly Detection for Streaming Data. Neurocomputing, 262:134-147, 2017.

[3]

A. AlEroud and G. Karabatis. A Contextual Anomaly Detection Approach to Discover Zero-Day Attacks. In International Conference on CyberSecurity (ICCS), pages 40-45, 2012.

Digital Library

[4]

S. Aminikhanghahi and D. J. Cook. A Survey of Methods for Time Series Change Point Detection. Knowledge and Information Systems, 51(2):339-367, 2017.

Digital Library

[5]

O. Anava, E. Hazan, and A. Zeevi. Online Time Series Prediction with Missing Data. In International Conference on Machine Learning (ICML), pages 2191-2199, 2015.

Digital Library

[6]

R. Baeza-Yates and B. Ribeiro-Neto. Modern Information Retrieval: The Concepts and Technology behind Search (Second Edition). Addison-Wesley, 2011.

Digital Library

[7]

P. Bailis, E. Gan, S. Madden, D. Narayanan, K. Rong, and S. Suri. MacroBase: Prioritizing Attention in Fast Data. In ACM International Conference on Management of Data (SIGMOD), pages 541-556, 2017.

Digital Library

[8]

V. Chandola, V. Mithal, and V. Kumar. Comparative Evaluation of Anomaly Detection Techniques for Sequence Data. In IEEE International Conference on Data Mining (ICDM), pages 743-748, 2008.

Digital Library

[9]

V. Chandola, A. Banerjee, and V. Kumar. Anomaly Detection: A Survey. ACM Computing Surveys, 41(3): 15:1-15:58, 2009.

Digital Library

[10]

G. H. Chen, S. Nikolov, and D. Shah. A Latent Source Model for Nonparametric Time Series Classification. In Annual Conference on Neural Information Processing Systems (NIPS), pages 1088-1096, 2013.

Digital Library

[11]

H. Cheng, P. Tan, C. Potter, and S. Klooster. Detection and Characterization of Anomalies in Multivariate Time Series. In SIAM International Conference on Data Mining (SDM), pages 413-424, 2009.

Digital Library

[12]

D. J. Cook and N. C. Krishnan. Activity Learning: Discovering, Recognizing, and Predicting Human Behavior from Sensor Data. John Wiley & Sons, 2015.

Digital Library

[13]

H. Ding, G. Trajcevski, P. Scheuermann, X. Wang, and E. Keogh. Querying and Mining of Time Series Data: Experimental Comparison of Representations and Distance Measures. Proceedings of the VLDB Endowment, 1(2):1542-1552, 2008.

Digital Library

[14]

S. Formenti and S. Demaria. Systemic Effects of Local Radiotherapy. The Lancet Oncology, pages 718-726, 2017.

[15]

J. Gottschlich. Paranom: A Parallel Anomaly Dataset Generator. https://arxiv.org/abs/1801.03164/, 2018.

[16]

S. Guha, N. Mishra, G. Roy, and O. Schrijvers. Robust Random Cut Forest Based Anomaly Detection on Streams. In International Conference on Machine Learning (ICML), pages 2712-2721, 2016.

Digital Library

[17]

M. Hermans and B. Schrauwen. Training and Analyzing Deep Recurrent Neural Networks. In Annual Conference on Neural Information Processing Systems (NIPS), pages 190-198, 2013.

Digital Library

[18]

S. A. Hofmeyr, S. Forrest, and A. Somayaji. Intrusion Detection using Sequences of System Calls. Journal of Computer Security, 6(3):151-180, 1998.

Digital Library

[19]

M. Hüsken and P. Stagge. Recurrent Neural Networks for Time Series Classification. Neurocomputing, 50: 223-235, 2003.

[20]

E. Keogh. UCR Time-Series Datasets. http://www.cs.ucr.edu/~eamonn/discords/, 2005.

[21]

E. Keogh, J. Lin, and A. Fu. HOT SAX: Efficiently Finding the Most Unusual Time Series Subsequence. In IEEE International Conference on Data Mining (ICDM), pages 226-233, 2005.

Digital Library

[22]

K. Kourou, T. P. Exarchos, K. P. Exarchos, M. V. Karamouzis, and D. I. Fotiadis. Machine Learning Applications in Cancer Prognosis and Prediction. Computational and Structural Biotechnology Journal, 13:8-17, 2015.

[23]

B. Krishnamurthy, S. Sen, Y. Zhang, and Y. Chen. Sketch-based Change Detection: Methods, Evaluation, and Applications. In ACM SIGCOMM Conference on Internet Measurement (IMC), pages 234-247, 2003.

Digital Library

[24]

N. Laptev, S. Amizadeh, and I. Flint. Generic and Scalable Framework for Automated Time-series Anomaly Detection. In ACM International Conference on Knowledge Discovery and Data Mining (KDD), pages 1939-1947, 2015.

Digital Library

[25]

A. Lavin and S. Ahmad. Evaluating Real-Time Anomaly Detection Algorithms - The Numenta Anomaly Benchmark. In IEEE International Conference on Machine Learning and Applications (ICMLA), pages 38-44, 2015.

[26]

T. J. Lee, J. Gottschlich, N. Tatbul, E. Metcalf, and S. Zdonik. Greenhouse: A Zero-Positive Machine Learning System for Time-Series Anomaly Detection. In Inaugural Conference on Systems and Machine Learning (SysML), 2018.

[27]

S. C. Li and B. Marlin. A Scalable End-to-end Gaussian Process Adapter for Irregularly Sampled Time Series Classification. In Annual Conference on Neural Information Processing Systems (NIPS), pages 1812-1820, 2016.

Digital Library

[28]

LinkedIn. LinkedIn's Anomaly Detection and Correlation Library. https://github.com/linkedin/luminol/, 2018.

[29]

Z. C. Lipton, D. C. Kale, C. Elkan, and R. C. Wetzel. Learning to Diagnose with LSTM Recurrent Neural Networks. In International Conference on Learning Representations (ICLR), 2016.

[30]

P. Malhotra, L. Vig, G. Shroff, and P. Agarwal. Long Short Term Memory Networks for Anomaly Detection in Time Series. In European Symposium on Artificial Neural Networks, Computational Intelligence, and Machine Learning (ESANN), pages 89-94, 2015.

[31]

P. Malhotra, A. Ramakrishnan, G. Anand, L. Vig, P. Agarwal, and G. Shroff. LSTM-based Encoder-Decoder for Multi-sensor Anomaly Detection. ICML Anomaly Detection Workshop, 2016.

[32]

Numenta. NAB Data Corpus. https://github.com/numenta/NAB/tree/master/data/, 2017.

[33]

S. Papadimitriou, J. Sun, and C. Faloutsos. Streaming Pattern Discovery in Multiple Time-Series. In International Conference on Very Large Data Bases (VLDB), pages 697-708, 2005.

Digital Library

[34]

A. Patcha and J. Park. An Overview of Anomaly Detection Techniques: Existing Solutions and Latest Technological Trends. Computer Networks, 51(12):3448-3470, 2007.

Digital Library

[35]

N. Singh and C. Olinsky. Demystifying Numenta Anomaly Benchmark. In International Joint Conference on Neural Networks (IJCNN), pages 1570-1577, 2017.

[36]

M. Tavallaee, N. Stakhanova, and A. A. Ghorbani. Toward Credible Evaluation of Anomaly-based Intrusion Detection Methods. IEEE Transactions on Systems, Man, and Cybernetics, 40(5):516-524, 2010.

Digital Library

[37]

Twitter. Twitter's Open-Source Anomaly Detection R Package. https://github.com/twitter/AnomalyDetection/, 2015.

[38]

C. Urmson, J. Anhalt, D. Bagnell, C. Baker, R. Bittner, M. N. Clark, J. Dolan, D. Duggins, T. Galatali, C. Geyer, M. Gittleman, S. Harbaugh, M. Hebert, T. M. Howard, S. Kolski, A. Kelly, M. Likhachev, M. McNaughton, N. Miller, K. Peterson, B. Pilnick, R. Rajkumar, P. Rybski, B. Salesky, Y.-W. Seo, S. Singh, J. Snider, A. Stentz, W. R. Whittaker, Z. Wolkowicki, J. Ziglar, H. Bae, T. Brown, D. Demitrish, B. Litkouhi, J. Nickolaou, V. Sadekar, W. Zhang, J. Struble, M. Taylor, M. Darms, and D. Ferguson. Autonomous Driving in Urban Environments: Boss and the Urban Challenge. Journal of Field Robotics, 25(8):425-466, 2008.

Digital Library

[39]

J. A. Ward, P. Lukowicz, and H. W. Gellersen. Performance Metrics for Activity Recognition. ACM Transactions on Intelligent Systems and Technology (TIST), 2(1):6:1-6:23, 2011.

Digital Library

[40]

H. Yu, N. Rao, and I. S. Dhillon. Temporal Regularized Matrix Factorization for High-dimensional Time Series Prediction. In Annual Conference on Neural Information Processing Systems (NIPS), pages 847-855, 2016.

Digital Library

[41]

S. Zhai, Y. Cheng, W. Lu, and Z. Zhang. Deep Structured Energy Based Models for Anomaly Detection. In International Conference on Machine Learning (ICML), pages 1100-1109, 2016.

Digital Library

Cited By

Freeman CMerriman JBeaver IMueen A(2022)Experimental Comparison and Survey of Twelve Time Series Anomaly Detection AlgorithmsJournal of Artificial Intelligence Research10.1613/jair.1.1269872(849-899)Online publication date: 4-Jan-2022
https://dl.acm.org/doi/10.1613/jair.1.12698
Boniol PPaparrizos JKang YPalpanas TTsay RElmore AFranklin M(2022)TheseusProceedings of the VLDB Endowment10.14778/3554821.355487915:12(3702-3705)Online publication date: 1-Aug-2022
https://dl.acm.org/doi/10.14778/3554821.3554879
Jacob VSong FStiegler ARad BDiao YTatbul N(2021)A demonstration of the exathlon benchmarking platform for explainable anomaly detectionProceedings of the VLDB Endowment10.14778/3476311.347635514:12(2827-2830)Online publication date: 28-Oct-2021
https://dl.acm.org/doi/10.14778/3476311.3476355
Show More Cited By

Recommendations

Time-Series Aware Precision and Recall for Anomaly Detection: Considering Variety of Detection Result and Addressing Ambiguous Labeling
CIKM '19: Proceedings of the 28th ACM International Conference on Information and Knowledge Management

We proposetime-series aware precision andrecall, which are appropriate for evaluating anomaly detection methods in time-series data. In time-series data, an anomaly corresponds toa series of instances. The conventional metrics, however, overlook this ...
Deep learning for anomaly detection in multivariate time series: Approaches, applications, and challenges
Abstract
Anomaly detection has recently been applied to various areas, and several techniques based on deep learning have been proposed for the analysis of multivariate time series. In this study, we classify the anomalies into three types, ...
Highlights
- The methods for anomaly detection on multivariate time series are reviewed.
- The ...
Semantic precision and recall for ontology alignment evaluation
IJCAI'07: Proceedings of the 20th international joint conference on Artifical intelligence

In order to evaluate ontology matching algorithms it is necessary to confront them with test ontologies and to compare the results with some reference. The most prominent comparison criteria are precision and recall originating from information ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Guide Proceedings

NIPS'18: Proceedings of the 32nd International Conference on Neural Information Processing Systems

December 2018

11021 pages

Publisher

Curran Associates Inc.

Red Hook, NY, United States

Publication History

Published: 03 December 2018

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

7
Total Citations
View Citations
135
Total Downloads

Downloads (Last 12 months)68
Downloads (Last 6 weeks)8

Reflects downloads up to 20 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Freeman CMerriman JBeaver IMueen A(2022)Experimental Comparison and Survey of Twelve Time Series Anomaly Detection AlgorithmsJournal of Artificial Intelligence Research10.1613/jair.1.1269872(849-899)Online publication date: 4-Jan-2022
https://dl.acm.org/doi/10.1613/jair.1.12698
Boniol PPaparrizos JKang YPalpanas TTsay RElmore AFranklin M(2022)TheseusProceedings of the VLDB Endowment10.14778/3554821.355487915:12(3702-3705)Online publication date: 1-Aug-2022
https://dl.acm.org/doi/10.14778/3554821.3554879
Jacob VSong FStiegler ARad BDiao YTatbul N(2021)A demonstration of the exathlon benchmarking platform for explainable anomaly detectionProceedings of the VLDB Endowment10.14778/3476311.347635514:12(2827-2830)Online publication date: 28-Oct-2021
https://dl.acm.org/doi/10.14778/3476311.3476355
Jacob VSong FStiegler ARad BDiao YTatbul N(2021)ExathlonProceedings of the VLDB Endowment10.14778/3476249.347630714:11(2613-2626)Online publication date: 27-Oct-2021
https://dl.acm.org/doi/10.14778/3476249.3476307
David-John BPeacock CZhang TMurdison TBenko HJonker T(2021)Towards gaze-based prediction of the intent to interact in virtual realityACM Symposium on Eye Tracking Research and Applications10.1145/3448018.3458008(1-7)Online publication date: 25-May-2021
https://dl.acm.org/doi/10.1145/3448018.3458008
Eichmann PSolleza FTatbul NZdonik SBoncz PManegold SAilamaki ADeshpande AKraska T(2019)Visual Exploration of Time Series Anomalies with Metro-VizProceedings of the 2019 International Conference on Management of Data10.1145/3299869.3320247(1901-1904)Online publication date: 25-Jun-2019
https://dl.acm.org/doi/10.1145/3299869.3320247
Eichmann PSolleza FTan JTatbul NZdonik SBrewster SFitzpatrick GCox AKostakos V(2019)Metro-VizExtended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems10.1145/3290607.3312912(1-6)Online publication date: 2-May-2019
https://dl.acm.org/doi/10.1145/3290607.3312912

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents