research-article

Public Access

GeoFlux: Hands-Off Data Integration Leveraging Join Key Knowledge

Authors:

Jie Song,

Danai Koutra,

Murali Mani,

H. V. JagadishAuthors Info & Claims

SIGMOD '18: Proceedings of the 2018 International Conference on Management of Data

Pages 1797 - 1800

https://doi.org/10.1145/3183713.3193546

Published: 27 May 2018 Publication History

PDF eReader

Abstract

Data integration is frequently required to obtain the full value of data from multiple sources. In spite of extensive research on tools to assist users, data integration remains hard, particularly for users with limited technical proficiency. To address this barrier, we study how much we can do with no user guidance. Our vision is that the user should merely specify two input datasets to be joined and get a meaningful integrated result. It turns out that our vision can be realized if the system can correctly determine the join key, for example based on domain knowledge.

We demonstrate this notion by considering a broad domain: socioeconomic data aggregated by geography, a widespread category that accounts for 80% of the data published by government agencies. Intuitively two such datasets can be integrated by joining on the geographic unit column. Although it sounds easy, this task has many challenges: How can we automatically identify columns corresponding to geographic units, other dimension variables and measure variables, respectively? If multiple geographic types exist, which one should be chosen for the join? How to join tables with idiosyncratic schema, different geographic units of aggregation or no aggregation at all?

We have developed GeoFlux, a data integration system that handles all these challenges and joins tabular data by automatically aggregating geographic information with a new, advanced crosswalk algorithm. In this demo paper, we overview the architecture of the system and its user-friendly interfaces, and then demonstrate via a real-world example that it is general, fully automatic and easy-to-use. In the demonstration, we invite users to interact with GeoFlux to integrate more sample socioeconomic data from data.ny.gov.

References

[1]

Michael J. Cafarella, Alon Halevy, and Nodira Khoussainova. 2009. Data Integration for the Relational Web. Proc. VLDB Endow., Vol. 2, 1 (Aug. 2009), 1090--1101. //books.google.com/books?id=kwTSupXTvwYC

Digital Library

Google Scholar

[2]

Hong-Hai Do and Erhard Rahm. 2002. COMA: A System for Flexible Combination of Schema Matching Approaches VLDB. VLDB Endowment, 610--621. http://dl.acm.org/citation.cfm?id=1287369.1287422

Digital Library

Google Scholar

[3]

Carl Franklin. 1992. An Introduction to Geographic Information Systems: Linking Maps to Databases. Database, Vol. 15, 2 (1992), 12--21.

Digital Library

Google Scholar

[4]

Eduard Hovy, José Luis Ambite, and Andrew Philpot. {n. d.}. Addressing a Bottleneck in Data Integration using Automated Learning Techniques. (.{n. d.}).

Google Scholar

[5]

Li Qian, Michael J. Cafarella, and H. V. Jagadish. 2012. Sample-driven Schema Mapping. In SIGMOD. ACM, New York, NY, USA, 73--84.

Digital Library

Google Scholar

[6]

Jie Song, Danai Koutra, Murali Mani, and H. V. Jagadish. 2018. GeoAlign: Interpolating Aggregates over Unaligned Partitions Proceedings of the 21th International Conference on Extending Database Technology, EDBT 2018, Vienna, Austria, March 26-29, 2018. 361--372.

Google Scholar

[7]

United States Census Bureau. 2010. Standard Hierarchy of Census Geographic Entities. Available from https://census.gov/. (2010).

Google Scholar

[8]

Hadley Wickham. 2014. Tidy data. The Journal of Statistical Software Vol. 59 (2014). Issue 10. http://www.jstatsoft.org/v59/i10/

Crossref

Google Scholar

Cited By

View all

Tang RAridas NTalip M(2023)Design of a data processing method for the farmland environmental monitoring based on improved Spark componentsFrontiers in Big Data10.3389/fdata.2023.12823526Online publication date: 20-Nov-2023
https://doi.org/10.3389/fdata.2023.1282352
Paramonov VShigarov AVetrova V(2021)Rule Driven Spreadsheet Data Extraction from Statistical Tables: Case StudyInformation and Software Technologies10.1007/978-3-030-88304-1_7(84-95)Online publication date: 7-Oct-2021
https://doi.org/10.1007/978-3-030-88304-1_7
Paramonov VShigarov AVetrova V(2020)Table Header Correction Algorithm Based on Heuristics for Improving Spreadsheet Data ExtractionInformation and Software Technologies10.1007/978-3-030-59506-7_13(147-158)Online publication date: 8-Oct-2020
https://doi.org/10.1007/978-3-030-59506-7_13

Index Terms

GeoFlux: Hands-Off Data Integration Leveraging Join Key Knowledge
1. Information systems
  1. Data management systems
    1. Information integration
      1. Extraction, transformation and loading
      2. Mediators and data integration
  2. Information systems applications
    1. Spatial-temporal systems
      1. Geographic information systems

Recommendations

Government transparency in the U.S?: the ethics, functional literacy, and usability of open geographic government data
SIGDOC '15: Proceedings of the 33rd Annual International Conference on the Design of Communication

This presentation reports on interviews conducted with Open Code Orlando Brigade members in regards to public information and geodata. Open Code Orlando Brigade is a volunteer group of coders who strive to make public data accessible for both business ...
Location and the web (LocWeb 2008)
WWW '08: Proceedings of the 17th international conference on World Wide Web

The World Wide Web has become the world's largest networked information resource, but references to geographical locations remain unstructured and typically implicit in nature. This lack of explicit spatial knowledge within the Web makes it difficult to ...
Interactive graphics an aid to health planning and decision making
SIGGRAPH '77: Proceedings of the 4th annual conference on Computer graphics and interactive techniques

One of the major functions of State and Federal Public Health agencies is the collection and maintenance of large amounts of data. The utilization of this data for research and planning may be impaired by such factors as: lack of organization and ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

SIGMOD '18: Proceedings of the 2018 International Conference on Management of Data

May 2018

1874 pages

ISBN:9781450347037

DOI:10.1145/3183713

General Chairs:
Gautam Das
University of Texas at Arlington, USA
,
Christopher Jermaine
Rice University, USA
,
Philip Bernstein
Microsoft Research, USA

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 May 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Science Foundation

Conference

SIGMOD/PODS '18

Sponsor:

SIGMOD

SIGMOD/PODS '18: International Conference on Management of Data

June 10 - 15, 2018

TX, Houston, USA

Acceptance Rates

SIGMOD '18 Paper Acceptance Rate 90 of 461 submissions, 20%;

Overall Acceptance Rate 785 of 4,003 submissions, 20%

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
355
Total Downloads

Downloads (Last 12 months)56
Downloads (Last 6 weeks)7

Reflects downloads up to 14 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Tang RAridas NTalip M(2023)Design of a data processing method for the farmland environmental monitoring based on improved Spark componentsFrontiers in Big Data10.3389/fdata.2023.12823526Online publication date: 20-Nov-2023
https://doi.org/10.3389/fdata.2023.1282352
Paramonov VShigarov AVetrova V(2021)Rule Driven Spreadsheet Data Extraction from Statistical Tables: Case StudyInformation and Software Technologies10.1007/978-3-030-88304-1_7(84-95)Online publication date: 7-Oct-2021
https://doi.org/10.1007/978-3-030-88304-1_7
Paramonov VShigarov AVetrova V(2020)Table Header Correction Algorithm Based on Heuristics for Improving Spreadsheet Data ExtractionInformation and Software Technologies10.1007/978-3-030-59506-7_13(147-158)Online publication date: 8-Oct-2020
https://doi.org/10.1007/978-3-030-59506-7_13

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Government transparency in the U.S?: the ethics, functional literacy, and usability of open geographic government data

Location and the web (LocWeb 2008)

Interactive graphics an aid to health planning and decision making