More Web Proxy on the site http://driver.im/

research-article

Public Access

Semi-Markov Switching Vector Autoregressive Model-Based Anomaly Detection in Aviation Systems

Authors:

Arindam Banerjee,

Bryan Matthews,

Nikunj OzaAuthors Info & Claims

KDD '16: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

Pages 1065 - 1074

https://doi.org/10.1145/2939672.2939789

Published: 13 August 2016 Publication History

Abstract

In this work we consider the problem of anomaly detection in heterogeneous, multivariate, variable-length time series datasets. Our focus is on the aviation safety domain, where data objects are flights and time series are sensor readings and pilot switches. In this context the goal is to detect anomalous flight segments, due to mechanical, environmental, or human factors in order to identifying operationally significant events and highlight potential safety risks. For this purpose, we propose a framework which represents each flight using a semi-Markov switching vector autoregressive (SMS-VAR) model. Detection of anomalies is then based on measuring dissimilarities between the model's prediction and data observation. The framework is scalable, due to the inherent parallel nature of most computations, and can be used to perform online anomaly detection. Extensive experimental results on simulated and real datasets illustrate that the framework can detect various types of anomalies along with the key parameters involved.

Supplementary Material

MP4 File (kdd2016_matthews_autoregressive_model_01-acm.mp4)

Download
291.46 MB

References

[1]

NASA Flight Dataset. Available at https://c3.nasa.gov/dashlink/projects/85/.

[2]

A. Ang and A. Timmermann. Regime changes and financial markets. Technical report, National Bureau of Economic Research, 2011.

[3]

D. Barber. Bayesian Reasoning and Machine Learning. Cambridge University Press, 2012.

[4]

S. Basu and M. Meckesheimer. Automatic outlier detection for time series: an application to sensor data. Knowledge and Information Systems, 11(2):137--154, 2007.

Digital Library

[5]

Boeing. Statistical summary of commercial jet airplane accidents. Available at www.boeing.com/ news/techissues/pdf/statsum.pdf, 2013.

[6]

S. Das, L. Li, A. N. Srivastava, and R. J. Hansman. Comparison of algorithms for anomaly detection in flight recorder data of airline operations. In AIAA Aviation Technology, Integration, and Operations Conference, 2012.

[7]

S. Das, B. Matthews, A. Srivastava, and N. Oza. Multiple kernel learning for heterogeneous anomaly detection: Algorithm and aviation safety case study. In International Conference on Knowledge Discovery and Data Mining, pages 47--56, 2010.

Digital Library

[8]

J. Demmel, L. Grigori, M. Hoemmen, and J. Langou. Communication-optimal parallel and sequential QR and LU factorizations. SIAM Journal on Scientific Computing, 34(1):206--239, 2012.

Digital Library

[9]

A. P. Dempster, N. M. Laird, and D. B. Rubin. Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society, pages 1--38, 1977.

[10]

T. V. Duong, H. H. Bui, D. Q. Phung, and S. Venkatesh. Activity recognition and abnormality detection with the switching hidden semi-Markov model. In Computer Vision and Pattern Recognition, volume 1, pages 838--845, 2005.

Digital Library

[11]

Federal Aviation Administration. Advisory circular. Flight operational quality assurance. Federal Aviation Administration, 12(2), 2004.

[12]

Federal Aviation Administration. FAA Terminal area forecast summary report. Available at stylesame http://www.faa.gov/about/office_org/headquarters_offices/apl/aviation_forecasts/taf_reports/media/TAF_Summary_Report_FY2013-2040.pdf, 2013.

[13]

Federal Aviation Administration. Next Generation Air Transportation System. Available at stylesame http://www.faa.gov/nextgen/, 2016.

[14]

Z. Ghahramani and G. E. Hinton. Variational learning for switching state-space models. Neural computation, 12(4):831--864, 2000.

Digital Library

[15]

G. H. Golub and C. F. Van Loan. Matrix computations. John Hopkins University Press, 3 edition, 2012.

[16]

D. Gorinevsky, B. Matthews, and R. Martin. Aircraft anomaly detection using performance models trained on fleet data. In Conference on intelligent data understanding, pages 17--23, 2012.

[17]

J. D. Hamilton. A new approach to the economic analysis of nonstationary time series and the business cycle. Econometrica: Journal of the Econometric Society, pages 357--384, 1989.

[18]

U. Hassler. Stochastic Processes and Calculus: An Elementary Introduction with Applications. Springer, 2015.

[19]

J. Janssen and N. Limnios. Semi-Markov models and applications. Springer Science & Business Media, 2013.

[20]

N. D. Le, R. D. Martin, and A. E. Raftery. Modeling flat stretches, bursts outliers in time series using mixture transition distribution models. Journal of the American Statistical Association, 91(436):1504--1515, 1996.

[21]

S. Lee, I. Hwang, and K. Leiden. Flight deck human-automation issue detection via intent inference. In International Conference on Human-Computer Interaction in Aerospace, 2014.

Digital Library

[22]

L. Li, S. Das, H. John, R. Palacios, and A. Srivastava. Analysis of flight data using clustering techniques for detecting abnormal operations. Journal of Aerospace Information Systems, 12(9):587--598, 2015.

[23]

L. Ljung. System identification: theory for the user. Springer, 1998.

Digital Library

[24]

H. Lutkepohl. New introduction to multiple time series analysis. Springer, 2007.

Digital Library

[25]

J. Ma and S. Perkins. Online novelty detection on temporal sequences. In International Conference on Knowledge Discovery and Data Mining, pages 613--618, 2003.

Digital Library

[26]

B. Matthews, S. Das, K. Bhaduri, K. Das, R. Martin, and N. Oza. Discovering anomalous aviation safety events using scalable data mining algorithms. Journal of Aerospace Information Systems, 10(10):467--475, 2013.

[27]

R. Nehl and J. Schade. Update: Concept and operation of the performance data analysis and reporting system (PDARS). In Aerospace Conference, pages 1--16, 2007.

[28]

P. Patel, E. Keogh, J. Lin, and S. Lonardi. Mining motifs in massive time series databases. In International Conference on Data Mining, pages 370--377, 2002.

Digital Library

[29]

S. M. Ross. Introduction to probability models. Academic press, 2014.

Digital Library

[30]

B. Scholkopf, J. C. Platt, J. C. Shawe-Taylor, A. J. Smola, and R. C. Williamson. Estimating the support of a high-dimensional distribution. Neural Computation, 13(7):1443--1471, 2001.

Digital Library

[31]

A. N. Srivastava. Greener aviation with virtual sensors: a case study. Data Mining and Knowledge Discovery, 24(2):443--471, 2012.

Digital Library

[32]

I. C. Statler and D. A. Maluf. NASA's aviation system monitoring and modeling project. Technical report, NASA/TP-2007-214556, 2003.

Cited By

Gao RWang JYu YWu JZhang L(2025)Enhanced graph diffusion learning with dynamic transformer for anomaly detection in multivariate time seriesNeurocomputing10.1016/j.neucom.2024.129168619(129168)Online publication date: Feb-2025
https://doi.org/10.1016/j.neucom.2024.129168
Kabashkin IPerekrestov V(2024)Ecosystem of Aviation Maintenance: Transition from Aircraft Health Monitoring to Health Management Based on IoT and AI SynergyApplied Sciences10.3390/app1411439414:11(4394)Online publication date: 22-May-2024
https://doi.org/10.3390/app14114394
Ain QAftab Ahmed Jilani AAzhar Butt NUr Rehman SAbdulrahman Alhulayyil H(2024)Anomaly Detection for Aviation Cyber-Physical System: Opportunities and ChallengesIEEE Access10.1109/ACCESS.2024.349551912(175905-175925)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3495519
Show More Cited By

Index Terms

Semi-Markov Switching Vector Autoregressive Model-Based Anomaly Detection in Aviation Systems
1. Computing methodologies
  1. Machine learning
    1. Learning paradigms
      1. Unsupervised learning
        Anomaly detection
    2. Machine learning approaches
      1. Learning in probabilistic graphical models
        Bayesian network models

Recommendations

Research on hidden Markov model for system call anomaly detection
PAISI'07: Proceedings of the 2007 Pacific Asia conference on Intelligence and security informatics

Intrusion detection, especially anomaly detection, requires sufficient security background knowledge. It is very significant to recognize system anomaly behavior under the condition of poor domain knowledge. In this paper, the general methods for system ...
An ARP-based Anomaly Detection Algorithm Using Hidden Markov Model in Enterprise Networks
ICSNC '07: Proceedings of the Second International Conference on Systems and Networks Communications

Network anomaly detection is an active research area. Behavior recognition of traffic is a process by which the ongoing observed behavior of a host is tracked and compared by a given model. Various methods for behavior recognition exist. But ...
Anomaly detection via feature-aided tracking and hidden Markov models
Special section: Best papers from the 2007 biometrics: Theory, applications, and systems (BTAS 07) conference

The problem of detecting an anomaly (or abnormal event) is such that the distribution of observations is different before and after an unknown onset time, and the objective is to detect the change by statistically matching the observed pattern with that ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

KDD '16: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

August 2016

2176 pages

ISBN:9781450342322

DOI:10.1145/2939672

General Chairs:
Balaji Krishnapuram
IBM
,
Mohak Shah
Bosch
,
Program Chairs:
Alex Smola
Amazon
,
Charu Aggarwal
IBM
,
Dou Shen
Baidu
,
Rajeev Rastogi
Amazon

Copyright © 2016 ACM.

© 2016 Association for Computing Machinery. ACM acknowledges that this contribution was authored or co-authored by an employee, contractor or affiliate of the United States government. As such, the United States Government retains a nonexclusive, royalty-free right to publish or reproduce this article, or to allow others to do so, for Government purposes only.

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 August 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

NASA
NSF

Conference

KDD '16

Sponsor:

KDD '16: The 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

August 13 - 17, 2016

California, San Francisco, USA

Acceptance Rates

KDD '16 Paper Acceptance Rate 66 of 1,115 submissions, 6%;

Overall Acceptance Rate 1,133 of 8,635 submissions, 13%

Upcoming Conference

KDD '25

Sponsor:
sigkdd
sigkdd

The 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining

August 3 - 7, 2025

Toronto , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

34
Total Citations
View Citations
748
Total Downloads

Downloads (Last 12 months)89
Downloads (Last 6 weeks)13

Reflects downloads up to 31 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Gao RWang JYu YWu JZhang L(2025)Enhanced graph diffusion learning with dynamic transformer for anomaly detection in multivariate time seriesNeurocomputing10.1016/j.neucom.2024.129168619(129168)Online publication date: Feb-2025
https://doi.org/10.1016/j.neucom.2024.129168
Kabashkin IPerekrestov V(2024)Ecosystem of Aviation Maintenance: Transition from Aircraft Health Monitoring to Health Management Based on IoT and AI SynergyApplied Sciences10.3390/app1411439414:11(4394)Online publication date: 22-May-2024
https://doi.org/10.3390/app14114394
Ain QAftab Ahmed Jilani AAzhar Butt NUr Rehman SAbdulrahman Alhulayyil H(2024)Anomaly Detection for Aviation Cyber-Physical System: Opportunities and ChallengesIEEE Access10.1109/ACCESS.2024.349551912(175905-175925)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3495519
Wadinger MKvasnica M(2024)Adaptable and Interpretable Framework for Anomaly Detection in SCADA-based industrial systemsExpert Systems with Applications: An International Journal10.1016/j.eswa.2024.123200246:COnline publication date: 15-Jul-2024
https://dl.acm.org/doi/10.1016/j.eswa.2024.123200
Li DLi YZhang TCai JZuo HZhang Y(2023)Dynamic Health Monitoring of Aero-Engine Gas-Path System Based on SFA-GMM-BIDElectronics10.3390/electronics1214319912:14(3199)Online publication date: 24-Jul-2023
https://doi.org/10.3390/electronics12143199
Xu LDing XZhao DLiu AZhang Z(2023)A Three-Dimensional ResNet and Transformer-Based Approach to Anomaly Detection in Multivariate Temporal–Spatial DataEntropy10.3390/e2502018025:2(180)Online publication date: 17-Jan-2023
https://doi.org/10.3390/e25020180
Xiang ZGao ZLiu JZhang Y(2023)Using Catalyst Mass-Based Clustering Analysis to Identify Adverse Events during ApproachAerospace10.3390/aerospace1005048310:5(483)Online publication date: 19-May-2023
https://doi.org/10.3390/aerospace10050483
Su YGong YSong S(2023)Time Series Data ValidityProceedings of the ACM on Management of Data10.1145/35889391:1(1-26)Online publication date: 30-May-2023
https://dl.acm.org/doi/10.1145/3588939
Zhu XPang JLiang X(2023)Anomaly Detection for Multivariate Telemetry Series of Satellite with Improved HVAE2023 IEEE 16th International Conference on Electronic Measurement & Instruments (ICEMI)10.1109/ICEMI59194.2023.10270355(309-314)Online publication date: 9-Aug-2023
https://doi.org/10.1109/ICEMI59194.2023.10270355
Jasra SValentino GMuscat ACamilleri R(2022)Hybrid Machine Learning–Statistical Method for Anomaly Detection in Flight DataApplied Sciences10.3390/app12201026112:20(10261)Online publication date: 12-Oct-2022
https://doi.org/10.3390/app122010261
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents