Information Discovery and Analysis Systems (IDAS) are designed to correlate multiple sources of data and use data mining techniques to identify potential significant events. Application domains for IDAS are numerous and include the emerging area of homeland security.Developing test cases for an IDAS requires background data sets into which hypothetical future scenarios can be overlaid. The IDAS can then be measured in terms of false positive and false negative error rates. Obtaining the test data sets can be an obstacle due to both privacy issues and also the time and cost associated with collecting a diverse set of data sources.In this paper, we give an overview of the design and architecture of an IDAS Data Set Generator (IDSG) that enables a fast and comprehensive test of an IDAS. The IDSG generates data using statistical and rule-based algorithms and also semantic graphs that represent interdependencies between attributes. A credit card transaction application is used to illustrate the approach.

References

[1]

Abowd, J.M. and Lane, J.I. Synthetic Data and Confidentiality Protection. U.S. Census Bureau, LEHD Program Technical Paper No. TP-2003-10, (2003).

Google Scholar

[2]

Chan, P.K., Fan, W., Prodromidis, A.L., and Stolfo, S.J. Distributed Data Mining in Credit Card Fraud Detection. IEEE Intelligent Systems 14(6), 67--74. (1999).

Digital Library

Google Scholar

[3]

Department of Defense, Office of the Inspector General. Information Technology Management: Terrorism Information Awareness Program. Report No. D-2004-033. (2004).

Google Scholar

[4]

General Accounting Office, Data Mining: Federal Efforts Cover a Wide Range of Uses. GAO-04-548. (2004).

Google Scholar

[5]

Kusiak, A., Kernstine, K.H., Kern, J.A., McLaughlin, K.A., and Tseng, T.L. Data Mining: Medical and Engineering Case Studies. Proceedings of the Industrial Engineering Research 2000 Conference, Cleveland, Ohio, May 21-23, (2000), 1--7.

Google Scholar

[6]

Leskovec, J. Grobelnik, M., and Millic-Frayling, N. Learning Sub-structures of Document Semantic Graphs for Document Summarization. LinkKDD 2004, August 2004, Seattle WA, USA. (2004).

Google Scholar

[7]

Ormerod, T., Morley, N., Ball, L., Langley, C., and Spenser, C. Using Ethnography To Design a Mass Detection Tool (MDT) For The Early Discovery of Insurance Fraud. CHI 2003, April 5--10, 2003, Ft. Lauderdale, Florida, USA. ACM 1-58113-637-4/03/0004 (2003).

Digital Library

Google Scholar

[8]

Prince, E., and Nicholson, W.L. A Test of a Robust/Resistant Refinement Procedure on Synthetic Data. Acta Cryst., A39, (1983), 407--410.

Crossref

Google Scholar

[9]

Rogers, M. Graham, J., and Tonge, R.P. Using Statistical Image Models for Objective Evaluation of Spot Detection in Two-Dimensional Gels. Proteomics, June, 3(6) (2003), 879--886.

Google Scholar

[10]

Varga, T. and Bunke, H. Generation of Synthetic Training Data for an HMM-based Handwriting Recognition System. Proceedings of the Seventh International Conference on Document Analysis and Recognition (ICDAR 2003) 0-7695-1960-1/03 IEEE Computer Society (2003).

Digital Library

Google Scholar

[11]

Widdows, D. and Dorow, B. A Graph Model for Unsupervised Lexical Acquisition. 19th International Conference on Computational Linguistics (COLING 19). Taipei, August (2002) 1093--1099.

Digital Library

Google Scholar

[12]

Yun, W.T., Stefanova, L., Mitra, A.K., and Krishnamurti, T.N. Multi-Model Synthetic Superensemble Prediction System. Acta Cryst., A39, (1983), 407--410.

Google Scholar

[13]

Zhu, X., Aref, W.G., Fan, J., Catlin, A.C., and Elmagarmid, A.K. Medical Video Mining for Efficient Database Indexing, Management, and Access. IEEE Int. Conf. On Data Engineering (ICDE '03), Bangalore, India, March 5-March 8, (2003), 1--12.

Crossref

Google Scholar

Cited By

View all

Suchetha GPushpalatha KSooraj SNaik VTanishka Saniha T(2024)Enhancing Machine Learning Model Accuracy through Novel SDNIoT Dataset Generation2024 International Conference on Intelligent Systems for Cybersecurity (ISCS)10.1109/ISCS61804.2024.10581240(01-06)Online publication date: 3-May-2024
https://doi.org/10.1109/ISCS61804.2024.10581240
Ling XMenzies THazard CShu JBeel J(2024)Trading Off Scalability, Privacy, and Performance in Data SynthesisIEEE Access10.1109/ACCESS.2024.336655612(26642-26654)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3366556
Idehen IJang WOverbye T(2020)Large-Scale Generation and Validation of Synthetic PMU DataIEEE Transactions on Smart Grid10.1109/TSG.2020.297734911:5(4290-4298)Online publication date: Sep-2020
https://doi.org/10.1109/TSG.2020.2977349
Show More Cited By

Index Terms

Generation of synthetic data sets for evaluating the accuracy of knowledge discovery systems

Recommendations

Development of a Synthetic Data Set Generator for Building and Testing Information Discovery Systems
ITNG '06: Proceedings of the Third International Conference on Information Technology: New Generations

Data mining research has yielded many significant and useful results such as discovering consumer-spending habits, detecting credit card fraud, and identifying anomalous social behavior. Information Discovery and Analysis Systems (IDAS) extract ...
Mining uncertain data for constrained frequent sets
IDEAS '09: Proceedings of the 2009 International Database Engineering & Applications Symposium

Data mining aims to search for implicit, previously unknown, and potentially useful pieces of information---such as sets of items that are frequently co-occurring together---that are embedded in data. The mined frequent sets can be used in the discovery ...
Flexible Frameworks for Actionable Knowledge Discovery

Most data mining algorithms and tools stop at the mining and delivery of patterns satisfying expected technical interestingness. There are often many patterns mined but business people either are not interested in them or do not know what follow-up ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

KDD '05: Proceedings of the eleventh ACM SIGKDD international conference on Knowledge discovery in data mining

August 2005

844 pages

ISBN:159593135X

DOI:10.1145/1081870

General Chair:
Robert Grossman
University of Illinois at Chicago & Open Data Partners, USA
,
Program Chairs:
Roberto Bayardo
IBM Almaden Research, USA
,
Kristin Bennett
RPI, USA

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 21 August 2005

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

KDD05

Sponsor:

KDD05: The Eleventh ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

August 21 - 24, 2005

Illinois, Chicago, USA

Acceptance Rates

Overall Acceptance Rate 1,133 of 8,635 submissions, 13%

Upcoming Conference

KDD '25

Sponsor:
sigkdd
sigkdd

The 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining

August 3 - 7, 2025

Toronto , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

22
Total Citations
View Citations
1,017
Total Downloads

Downloads (Last 12 months)21
Downloads (Last 6 weeks)0

Reflects downloads up to 21 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Suchetha GPushpalatha KSooraj SNaik VTanishka Saniha T(2024)Enhancing Machine Learning Model Accuracy through Novel SDNIoT Dataset Generation2024 International Conference on Intelligent Systems for Cybersecurity (ISCS)10.1109/ISCS61804.2024.10581240(01-06)Online publication date: 3-May-2024
https://doi.org/10.1109/ISCS61804.2024.10581240
Ling XMenzies THazard CShu JBeel J(2024)Trading Off Scalability, Privacy, and Performance in Data SynthesisIEEE Access10.1109/ACCESS.2024.336655612(26642-26654)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3366556
Idehen IJang WOverbye T(2020)Large-Scale Generation and Validation of Synthetic PMU DataIEEE Transactions on Smart Grid10.1109/TSG.2020.297734911:5(4290-4298)Online publication date: Sep-2020
https://doi.org/10.1109/TSG.2020.2977349
Idehen IJang WOverbye T(2019)PMU Data Feature Considerations for Realistic, Synthetic Data Generation2019 North American Power Symposium (NAPS)10.1109/NAPS46351.2019.9000335(1-6)Online publication date: Oct-2019
https://doi.org/10.1109/NAPS46351.2019.9000335
Popic SPavkovic BVelikic ITeslic N(2019)Data generators: a short survey of techniques and use cases with focus on testing2019 IEEE 9th International Conference on Consumer Electronics (ICCE-Berlin)10.1109/ICCE-Berlin47944.2019.8966202(189-194)Online publication date: Sep-2019
https://doi.org/10.1109/ICCE-Berlin47944.2019.8966202
Peng TTelle ALara Torralbo JAljawarneh S(2018)A tool for generating synthetic dataProceedings of the First International Conference on Data Science, E-learning and Information Systems10.1145/3279996.3280018(1-6)Online publication date: 1-Oct-2018
https://dl.acm.org/doi/10.1145/3279996.3280018
Babalola KJennings OUrdiales EDeBardelaben J(2018)Statistical Methods for Generating Synthetic Email Data Sets2018 IEEE International Conference on Big Data (Big Data)10.1109/BigData.2018.8622601(3986-3990)Online publication date: Dec-2018
https://doi.org/10.1109/BigData.2018.8622601
Chandrasekaran JFeng HLei YKuhn DKacker R(2017)Applying Combinatorial Testing to Data Mining Algorithms2017 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)10.1109/ICSTW.2017.46(253-261)Online publication date: Mar-2017
https://doi.org/10.1109/ICSTW.2017.46
Schulz CNocaj AEl-Assady MFrey SHlawatsch MHund MKarch GNetzel RSchätzle CButt MKeim DErtl TBrandes UWeiskopf D(2016)Generative Data Models for Validation and Evaluation of Visualization TechniquesProceedings of the Sixth Workshop on Beyond Time and Errors on Novel Evaluation Methods for Visualization10.1145/2993901.2993907(112-124)Online publication date: 24-Oct-2016
https://dl.acm.org/doi/10.1145/2993901.2993907
Syahaneim Hazwani RWahida NShafikah SZuraini Ellyza P(2016)Automatic Artificial Data Generator: Framework and implementation2016 International Conference on Information and Communication Technology (ICICTM)10.1109/ICICTM.2016.7890777(56-60)Online publication date: 2016
https://doi.org/10.1109/ICICTM.2016.7890777
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Development of a Synthetic Data Set Generator for Building and Testing Information Discovery Systems

Mining uncertain data for constrained frequent sets

Flexible Frameworks for Actionable Knowledge Discovery