More Web Proxy on the site http://driver.im/

article

The problem of disguised missing data

Author:

Ronald K. PearsonAuthors Info & Claims

ACM SIGKDD Explorations Newsletter, Volume 8, Issue 1

Pages 83 - 92

https://doi.org/10.1145/1147234.1147247

Published: 01 June 2006 Publication History

Abstract

Missing data is a well-recognized problem in large datasets, widely discussed in the statistics and data analysis literature. Many programming environments provide explicit codes for missing data, but these are not standardized and are not always used. This lack of standardization is one of the leading causes of the subtle problem of disguised missing data, in which unknown, inapplicable, or otherwise nonspecified responses are encoded as valid data values. Following a brief overview of the problem of explicitly coded missing data, this paper discusses sources, consequences, and detection of disguised missing data, including two real-world examples. As the first of these examples illustrates, the consequences of disguised missing data can be quite serious. The key to its detection lies in first, recognizing disguised missing data as a possibility and second, finding a sufficiently informative view of the data to reveal its presence.

References

[1]

P. Adriaans and D. Zantinge. Data Mining. Addison-Wesley, 1996.

Digital Library

[2]

V. Barnett and T. Lewis. Outliers in Statistical Data. Wiley, 3rd edition, 1994.

[3]

J. Breault. Data mining diabetic databases: Are rough sets a useful addition? In Proc. 33rd Symposium on the Interface, Computing Science and Statistics, Fairfax, VA, 2001.

[4]

L. Breiman. Bagging predictors. Machine Learning, 24:123--140, 1996.

[5]

L. Breiman. Heuristics of instability and stabilization in model selection. Ann. Statist., 24:2350--2383, 1996.

[6]

C. Date. An Introduction to Database Systems. Addison-Wesley, 7th edition, 2000.

Digital Library

[7]

D. DesJardins. Outliers, inliers, and just plain liars---new EDA+ (EDA Plus) techniques for understanding data. In Proc. SAS User's Group Intl. Conf., SUGI26, Long Beach, CA, 2001. Paper 169.

[8]

A. Feelders. Handling missing data in trees: surrogate splits or statistical imputation? In Principles of Data Mining and Knowledge Discovery (PKDD99), pages 329--334, 1999.

Digital Library

[9]

D. Heitjan. Ignorability and coarse data: Some biomedical examples. Biometrics, 49:1099--1109, 1993.

[10]

D. Heitjan and D. Rubin. Ignorability and coarse data. Ann. Statist., 19:2244--2253, 1991.

[11]

N. Horton and S. Lipsitz. Multiple imputation in practice: Comparison of software packages for regression models with missing variables. American Statistician, 55:244--254, 2001.

[12]

M. Huisman. Missing data in behavioral science research: Investigation of a collection of datasets. Kwantitatieve Methoden, 57:69--93, 1998. (in English).

[13]

M. Huisman. Post-stratification to correct for nonresponse: classification of zip code areas. In J. Bethlehem and P. van der Heijden, editors, Proc. 14th Symposium Computational Statistics, COMPSTAT 2000, pages 325--330, Utrecht, 2000.

[14]

M. Huisman and J. van der Zouwen. Item nonresponse in scale data from surveys: Types, determinants, and measures. Technical report, University of Groningen, 1998.

[15]

M. Jaeger. Ignorability for categorical data. Ann. Statist, 33:1964--1981, 2005.

[16]

G. John, R. Kohavi, and K. Pfleger. Irrelevant features and the subset selection problem. In W. Cohen and H. Hirsch, editors, Machine Learning: Proc. 11th International Conf., pages 121--129, 1994.

[17]

R. Little and D. Rubin. Statistical Analysis with Missing Data. Wiley, 2002.

Digital Library

[18]

G. McLachlan and T. Krishnan. The EM Algorithm and Extensions. Wiley, 1997.

[19]

J. Mistiaen and M. Ravallion. Survey compliance and the distribution of income. Policy Research Working Paper WPS2956, The World Bank, Development Research Group, Poverty Team, available at http://econ.worldbank.org, 2003.

[20]

J. Myllymaki. Effective web data extraction with standard XML technologies. In Proc. 10th International World Wide Web Conf., Hong Kong, 2001.

Digital Library

[21]

R. Pearson. Mining Imperfect Data: Dealing with Contamination and Incomplete Records. SIAM, 2005.

Digital Library

[22]

J. Schafer and J. Graham. Missing data: Our view of the state of the art. Psychological Methods, 7:147--177, 2002.

[23]

A. Trontell. How the US Food and Drug Administration defines and detects adverse drug events. Current Therapeutic Research, 62:641--649, 2001.

[24]

W. Venables and B. Ripley. Modern Applied Statistics with S. Springer, 2002.

Digital Library

[25]

Y. Wei, K. Detre, and J. Everhart. The NIDDK liver transplantation database. Liver Transplant Surgery, 3:10--22, 1997.

Cited By

Wen ZLi LKirkland CLi SSun XLei JXu BHou Z(2024)A machine learning approach to discrimination of igneous rocks and ore deposits by zircon trace elementsAmerican Mineralogist10.2138/am-2022-8899109:6(1129-1142)Online publication date: 1-Jun-2024
https://doi.org/10.2138/am-2022-8899
Sun MWang YBolton CMa YLi TZhao J(2024)Investigating User Estimation of Missing Data in Visual AnalysisProceedings of the 50th Graphics Interface Conference10.1145/3670947.3670977(1-13)Online publication date: 3-Jun-2024
https://dl.acm.org/doi/10.1145/3670947.3670977
Oh ELee H(2024)Quantum mechanics-based missing value estimation framework for industrial dataExpert Systems with Applications10.1016/j.eswa.2023.121385236(121385)Online publication date: Feb-2024
https://doi.org/10.1016/j.eswa.2023.121385
Show More Cited By

Index Terms

The problem of disguised missing data
1. Applied computing
  1. Computers in other domains
    1. Digital libraries and archives
2. Information systems
  1. Information retrieval
    1. Document representation
      1. Document collection models
  2. Information systems applications
    1. Data mining
    2. Digital libraries and archives

Recommendations

Cleaning disguised missing data: a heuristic approach
KDD '07: Proceedings of the 13th ACM SIGKDD international conference on Knowledge discovery and data mining

In some applications such as filling in a customer information form on the web, some missing values may not be explicitly represented as such, but instead appear as potentially valid data values. Such missing values are known as disguised missing data, ...
Privacy-preserving imputation of missing data

Handling missing data is a critical step to ensuring good results in data mining. Like most data mining algorithms, existing privacy-preserving data mining algorithms assume data is complete. In order to maintain privacy in the data mining process while ...
Missing Data Imputation Techniques

Intelligent data analysis techniques are useful for better exploring real-world data sets. However, the real-world data sets always are accompanied by missing data that is one major factor affecting data quality. At the same time, good intelligent data ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM SIGKDD Explorations Newsletter

ACM SIGKDD Explorations Newsletter Volume 8, Issue 1

June 2006

104 pages

ISSN:1931-0145

EISSN:1931-0153

DOI:10.1145/1147234

Issue’s Table of Contents

Copyright © 2006 Author.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 June 2006

Published in SIGKDD Volume 8, Issue 1

Check for updates

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

58
Total Citations
View Citations
598
Total Downloads

Downloads (Last 12 months)35
Downloads (Last 6 weeks)7

Reflects downloads up to 21 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Wen ZLi LKirkland CLi SSun XLei JXu BHou Z(2024)A machine learning approach to discrimination of igneous rocks and ore deposits by zircon trace elementsAmerican Mineralogist10.2138/am-2022-8899109:6(1129-1142)Online publication date: 1-Jun-2024
https://doi.org/10.2138/am-2022-8899
Sun MWang YBolton CMa YLi TZhao J(2024)Investigating User Estimation of Missing Data in Visual AnalysisProceedings of the 50th Graphics Interface Conference10.1145/3670947.3670977(1-13)Online publication date: 3-Jun-2024
https://dl.acm.org/doi/10.1145/3670947.3670977
Oh ELee H(2024)Quantum mechanics-based missing value estimation framework for industrial dataExpert Systems with Applications10.1016/j.eswa.2023.121385236(121385)Online publication date: Feb-2024
https://doi.org/10.1016/j.eswa.2023.121385
Katsuki TOsogami TKrause ABrunskill ECho KEngelhardt BSabato SScarlett J(2023)Regression with sensor data containing incomplete observationsProceedings of the 40th International Conference on Machine Learning10.5555/3618408.3619061(15911-15927)Online publication date: 23-Jul-2023
https://dl.acm.org/doi/10.5555/3618408.3619061
Hu XZhan ZMa DZhang S(2023)Spatiotemporal Generative Adversarial Imputation Networks: An Approach to Address Missing Data for Wind TurbinesIEEE Transactions on Instrumentation and Measurement10.1109/TIM.2023.331249372(1-8)Online publication date: 2023
https://doi.org/10.1109/TIM.2023.3312493
Hapfelmeier AHornung RHaller B(2023)Efficient permutation testing of variable importance measures by the example of random forestsComputational Statistics & Data Analysis10.1016/j.csda.2022.107689181(107689)Online publication date: May-2023
https://doi.org/10.1016/j.csda.2022.107689
Zaman NHau YLeong MAl-ashwal R(2023)A review on the significance of body temperature interpretation for early infectious disease diagnosisArtificial Intelligence Review10.1007/s10462-023-10528-x56:12(15449-15494)Online publication date: 19-Jun-2023
https://dl.acm.org/doi/10.1007/s10462-023-10528-x
Sun MMa YWang YLi TZhao JLiu YZhong P(2022)Toward Systematic Considerations of Missingness in Visual Analytics2022 IEEE Visualization and Visual Analytics (VIS)10.1109/VIS54862.2022.00031(110-114)Online publication date: Oct-2022
https://doi.org/10.1109/VIS54862.2022.00031
Shi JSun YDu X(2022)The detection algorithm for disguised missing value based on filter-KmeansMultimedia Tools and Applications10.1007/s11042-022-13421-x82:5(7583-7598)Online publication date: 2-Sep-2022
https://dl.acm.org/doi/10.1007/s11042-022-13421-x
Rizzo SChen YPang LLucas JKaoudi ZQuiane JChawla S(2022)Uncertainty-bounded reinforcement learning for revenue optimization in air cargo: a prescriptive learning approachKnowledge and Information Systems10.1007/s10115-022-01713-564:9(2515-2541)Online publication date: 2-Aug-2022
https://doi.org/10.1007/s10115-022-01713-5
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents