research-article

Juneau: data lake management for Jupyter

Authors:

Yi Zhang,

Zachary G. IvesAuthors Info & Claims

Proceedings of the VLDB Endowment, Volume 12, Issue 12

Pages 1902 - 1905

https://doi.org/10.14778/3352063.3352095

Published: 01 August 2019 Publication History

Get Access

Abstract

In collaborative settings such as multi-investigator laboratories, data scientists need improved tools to manage not their data records but rather their data sets and data products, to facilitate both provenance tracking and data (and code) reuse within their data lakes and file systems. We demonstrate the Juneau System, which extends computational notebook software (Jupyter Notebook) as an instrumentation and data management point for overseeing and facilitating improved dataset usage, through capabilities for indexing, searching, and recommending "complementary" data sources, previously extracted machine learning features, and additional training data. This demonstration focuses on how we help the user find related datasets via search.

References

[1]

L. A. Carvalho, R. Wang, Y. Gil, and D. Garijo. Niw: Converting notebooks into workflows to capture dataflow and provenance. In Conference on Knowledge Capture (K-CAP), 2017.

Google Scholar

[2]

D. Deng, R. C. Fernandez, Z. Abedjan, S. Wang, M. Stonebraker, A. K. Elmagarmid, I. F. Ilyas, S. Madden, M. Ouzzani, and N. Tang. The data civilizer system. In CIDR, 2017.

Google Scholar

[3]

R. Fagin, A. Lotem, and M. Naor. Optimal aggregation algorithms for middleware. Journal of computer and system sciences, 66(4):614--656, 2003.

Digital Library

Google Scholar

[4]

R. C. Fernandez, Z. Abedjan, F. Koko, G. Yuan, S. Madden, and M. Stonebraker. Aurum: A data discovery system. In 2018 IEEE 34th International Conference on Data Engineering (ICDE), pages 1001--1012. IEEE, 2018.

Crossref

Google Scholar

[5]

Z. G. Ives, S. Han, Y. Zhang, and N. Zheng. Data relationship management systems. 2019.

Google Scholar

[6]

D. Koop and J. Patel. Dataflow notebooks: encoding and tracking dependencies of cells. In TaPP, 2017.

Digital Library

Google Scholar

[7]

D. Mottin, M. Lissandrini, Y. Velegrakis, and T. Palpanas. Exemplar queries: Give me an example of what you need. PVLDB, 7(5):365--376, 2014.

Digital Library

Google Scholar

[8]

F. Nargesian, E. Zhu, K. Q. Pu, and R. J. Miller. Table union search on open data. PVLDB, 11(7):813--825, 2018.

Digital Library

Google Scholar

[9]

nteract team. Papermill: Parameterize, execute, and analyze notebooks. https://papermill.readthedocs.io/en/latest/, 2018.

Google Scholar

[10]

T. Petricek, J. Geddes, and C. Sutton. Wrattler: Reproducible, live and polyglot notebooks. In TaPP. USENIX Association, 2018.

Digital Library

Google Scholar

Cited By

View all

Chai CDeng YZhan YCao ZZhang YCao LWang YZhang ZYuan YWang GTang N(2024)LakeCompass: An End-to-End System for Data Maintenance, Search and Analysis in Data LakesProceedings of the VLDB Endowment10.14778/3685800.368588017:12(4381-4384)Online publication date: 8-Nov-2024
https://dl.acm.org/doi/10.14778/3685800.3685880
Dharmawan KKawediya CGong YYudhistira ZZhu ZGalhotra SKrisnadhi ACastro Fernandez RBarcelo PSanchez-Pi NMeliou ASudarshan S(2024)Demonstration of Ver: View Discovery in the WildCompanion of the 2024 International Conference on Management of Data10.1145/3626246.3654748(428-431)Online publication date: 9-Jun-2024
https://dl.acm.org/doi/10.1145/3626246.3654748
Ionescu AVasilev KBuse FHai RKatsifodimos A(2024)AutoFeat: Transitive Feature Discovery over Join Paths2024 IEEE 40th International Conference on Data Engineering (ICDE)10.1109/ICDE60146.2024.00150(1861-1873)Online publication date: 13-May-2024
https://doi.org/10.1109/ICDE60146.2024.00150
Show More Cited By

Recommendations

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Proceedings of the VLDB Endowment

Proceedings of the VLDB Endowment Volume 12, Issue 12

August 2019

547 pages

ISSN:2150-8097

Editors:
Lei Chen,
Fatma Özcan

Issue’s Table of Contents

Publisher

VLDB Endowment

Publication History

Published: 01 August 2019

Published in PVLDB Volume 12, Issue 12

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

18
Total Citations
View Citations
250
Total Downloads

Downloads (Last 12 months)27
Downloads (Last 6 weeks)4

Reflects downloads up to 20 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Chai CDeng YZhan YCao ZZhang YCao LWang YZhang ZYuan YWang GTang N(2024)LakeCompass: An End-to-End System for Data Maintenance, Search and Analysis in Data LakesProceedings of the VLDB Endowment10.14778/3685800.368588017:12(4381-4384)Online publication date: 8-Nov-2024
https://dl.acm.org/doi/10.14778/3685800.3685880
Dharmawan KKawediya CGong YYudhistira ZZhu ZGalhotra SKrisnadhi ACastro Fernandez RBarcelo PSanchez-Pi NMeliou ASudarshan S(2024)Demonstration of Ver: View Discovery in the WildCompanion of the 2024 International Conference on Management of Data10.1145/3626246.3654748(428-431)Online publication date: 9-Jun-2024
https://dl.acm.org/doi/10.1145/3626246.3654748
Ionescu AVasilev KBuse FHai RKatsifodimos A(2024)AutoFeat: Transitive Feature Discovery over Join Paths2024 IEEE 40th International Conference on Data Engineering (ICDE)10.1109/ICDE60146.2024.00150(1861-1873)Online publication date: 13-May-2024
https://doi.org/10.1109/ICDE60146.2024.00150
Paton NChen JWu Z(2023)Dataset Discovery and Exploration: A SurveyACM Computing Surveys10.1145/362652156:4(1-37)Online publication date: 9-Nov-2023
https://dl.acm.org/doi/10.1145/3626521
Hai RKoutras CQuix CJarke M(2023)Data Lakes: A Survey of Functions and SystemsIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2023.327010135:12(12571-12590)Online publication date: 25-Apr-2023
https://dl.acm.org/doi/10.1109/TKDE.2023.3270101
Gong YZhu ZGalhotra SFernandez R(2023)Ver: View Discovery in the Wild2023 IEEE 39th International Conference on Data Engineering (ICDE)10.1109/ICDE55515.2023.00045(503-516)Online publication date: Apr-2023
https://doi.org/10.1109/ICDE55515.2023.00045
Nambiar AMundra D(2022)An Overview of Data Warehouse and Data Lake in Modern Enterprise Data ManagementBig Data and Cognitive Computing10.3390/bdcc60401326:4(132)Online publication date: 7-Nov-2022
https://doi.org/10.3390/bdcc6040132
Wieder PNolte H(2022)Toward data lakes as central building blocks for data management and analysisFrontiers in Big Data10.3389/fdata.2022.9457205Online publication date: 19-Aug-2022
https://doi.org/10.3389/fdata.2022.945720
Couto JRuiz D(2022)An overview about data integration in data lakes2022 17th Iberian Conference on Information Systems and Technologies (CISTI)10.23919/CISTI54924.2022.9820576(1-7)Online publication date: 22-Jun-2022
https://doi.org/10.23919/CISTI54924.2022.9820576
Shivashankar KMartini A(2022)Maintainability Challenges in ML: A Systematic Literature Review2022 48th Euromicro Conference on Software Engineering and Advanced Applications (SEAA)10.1109/SEAA56994.2022.00018(60-67)Online publication date: Aug-2022
https://doi.org/10.1109/SEAA56994.2022.00018
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Recommendations

Internet Password Organizer: Password Reminder Book - Alphabetical Organizer Journal Notebook - 5x8 With 110 Pages Vol.5 Password Book

Password Book: Password Journal / Password Organizer / Password Keeper / Internet Usernames and Passwords / Internet Password Logbook A Passkey Log ... seamless pattern collection) (Volume 48)

Password Book: Password Organizer / Web Password Book / Password Book / Password Journal / Internet Password Book / Internet Password Book / Password ... bunch on blue table) (Volume 52)