More Web Proxy on the site http://driver.im/

research-article

Astronomical data processing in EXTASCID

Authors:

Florin RusuAuthors Info & Claims

SSDBM '13: Proceedings of the 25th International Conference on Scientific and Statistical Database Management

Article No.: 47, Pages 1 - 4

https://doi.org/10.1145/2484838.2484875

Published: 29 July 2013 Publication History

Abstract

Scientific data have dual structure. Raw data are preponderantly ordered multi-dimensional arrays or sequences while metadata and derived data are best represented as unordered relations. Scientific data processing requires complex operations over arrays and relations. These operations cannot be expressed using only standard linear and relational algebra operators, respectively. Existing scientific data processing systems are designed for a single data model and handle complex processing at the application level.

EXTASCID is a complete and extensible system for scientific data processing. It supports both array and relational data natively. Complex processing is handled by a metaoperator that can execute any user code. As a result, EXTASCID can process full scientific workflows inside the system, with minimal data movement and application code. We illustrate the overall process on a real dataset and workflow from astronomy---starting with a set of sky images, the goal is to identify and classify transient astrophysical objects.

References

[1]

Palomar Transient Factory. www.astro.caltech.edu/ptf.

[2]

Sloan Digital Sky Survey. www.sdss3.org.

[3]

S. Arumugam and al. The DataPath System: A Data-Centric Analytic Processing Engine for Large Data Warehouses. In SIGMOD 2010.

Digital Library

[4]

P. Baumann and al. The Multidimensional Database System RasDaMan. In SIGMOD 1998.

Digital Library

[5]

J. B. Buck and al. SciHadoop: Array-based Query Processing in Hadoop. In SC 2011.

Digital Library

[6]

C. Chang and al. Titan: A High-Performance Remote Sensing Database. In ICDE 1997.

Digital Library

[7]

C. Chang and al. T2: A Customizable Parallel Database for Multi-Dimensional Data. SIGMOD Rec., 27(1), 1998.

Digital Library

[8]

Y. Cheng and al. GLADE: Big Data Analytics Made Easy. In SIGMOD 2012.

Digital Library

[9]

R. Cornacchia and al. Flexible and Efficient IR using Array Databases. VLDBJ, 17, 2008.

Digital Library

[10]

J. Dean and S. Ghemawat. MapReduce: Simplified Data Processing on Large Clusters. Commun. ACM, 51(1), 2008.

Digital Library

[11]

P. Furtado and P. Baumann. Storage of Multidimensional Arrays Based on Arbitrary Tiling. In ICDE 1999.

[12]

S. Idreos and al. MonetDB: Two Decades of Research in Column-oriented Database Architectures. IEEE Data Eng. Bull., 35(1), 2012.

[13]

M. Ivanova and al. Data Vaults: A Symbiosis between Database Technology and Scientific File Repositories. In SSDBM 2012.

Digital Library

[14]

S. Sarawagi and M. Stonebraker. Efficient Organization of Large Multidimensional Arrays. In ICDE 1994.

Digital Library

[15]

E. Soroush and al. ArrayStore: A Storage Manager for Complex Parallel Array Processing. In SIGMOD 2011.

Digital Library

[16]

M. Stonebraker and al. The Architecture of SciDB. In SSDBM 2011.

Digital Library

[17]

A. R. van Ballegooij. RAM: A Multidimensional Array DBMS. In EDBT Workshops 2004.

Digital Library

[18]

Y. Zhang and al. SciQL: Bridging the Gap between Science and Relational DBMS. In IDEAS 2011.

Digital Library

Cited By

Reznichenko V(2022)60 Years of Databases (part three)PROBLEMS IN PROGRAMMING10.15407/pp2022.01.034(034-066)Online publication date: Mar-2022
https://doi.org/10.15407/pp2022.01.034
Villarroya SBaumann P(2022)A survey on machine learning in array databasesApplied Intelligence10.1007/s10489-022-03979-253:9(9799-9822)Online publication date: 12-Aug-2022
https://dl.acm.org/doi/10.1007/s10489-022-03979-2
Wang PWoodcock RTaib RPaget MHeld A(2022)A Data Cube Architecture for Cloud‐Based Earth Observation AnalyticsBig Data Analytics in Earth, Atmospheric, and Ocean Sciences10.1002/9781119467557.ch5(95-113)Online publication date: Nov-2022
https://doi.org/10.1002/9781119467557.ch5
Show More Cited By

Recommendations

Astronomical pipeline processing using fuzzy logic
Towards a provenance framework for sub-image processing for astronomical data
CIKM '10: Proceedings of the 19th ACM international conference on Information and knowledge management

While there has been advances in observational equipment that generate huge high quality images, the processing of these images remains a major bottleneck. We show that provenance data collected during the processing of data can be reused to perform ...
Services for Astronomical Data Management and Sharing
eScience on Distributed Computing Infrastructure - Volume 8500

As part of the AstroGrid-PL project, we have implemented a large scale data management system for the Polish astronomical community within the framework of the PLGrid Plus project, with built-in metadata services, replication and distributed storage ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

SSDBM '13: Proceedings of the 25th International Conference on Scientific and Statistical Database Management

July 2013

401 pages

ISBN:9781450319218

DOI:10.1145/2484838

Copyright © 2013 Copyright is held by the owner/author(s).

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 29 July 2013

Check for updates

Qualifiers

Research-article

Conference

SSDBM '13

SSDBM '13: Conference on Scientific and Statistical Database Management

July 29 - 31, 2013

Maryland, Baltimore, USA

Acceptance Rates

Overall Acceptance Rate 56 of 146 submissions, 38%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

13
Total Citations
View Citations
102
Total Downloads

Downloads (Last 12 months)3
Downloads (Last 6 weeks)0

Reflects downloads up to 17 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Reznichenko V(2022)60 Years of Databases (part three)PROBLEMS IN PROGRAMMING10.15407/pp2022.01.034(034-066)Online publication date: Mar-2022
https://doi.org/10.15407/pp2022.01.034
Villarroya SBaumann P(2022)A survey on machine learning in array databasesApplied Intelligence10.1007/s10489-022-03979-253:9(9799-9822)Online publication date: 12-Aug-2022
https://dl.acm.org/doi/10.1007/s10489-022-03979-2
Wang PWoodcock RTaib RPaget MHeld A(2022)A Data Cube Architecture for Cloud‐Based Earth Observation AnalyticsBig Data Analytics in Earth, Atmospheric, and Ocean Sciences10.1002/9781119467557.ch5(95-113)Online publication date: Nov-2022
https://doi.org/10.1002/9781119467557.ch5
Villarroya SBaumann P(2020)On the Integration of Machine Learning and Array Databases2020 IEEE 36th International Conference on Data Engineering (ICDE)10.1109/ICDE48307.2020.00170(1786-1789)Online publication date: Apr-2020
https://doi.org/10.1109/ICDE48307.2020.00170
(2018)Automatic identification and classification of Palomar Transient Factory astrophysical objects in GLADEInternational Journal of Computational Science and Engineering10.1504/IJCSE.2018.1001495516:4(337-349)Online publication date: 1-Jan-2018
https://dl.acm.org/doi/10.1504/IJCSE.2018.10014955
Baumann P(2018)Array DatabasesEncyclopedia of Database Systems10.1007/978-1-4614-8265-9_2061(165-177)Online publication date: 7-Dec-2018
https://doi.org/10.1007/978-1-4614-8265-9_2061
Liaukevich VMišev DBaumann PMerticariu VChoudhary AWu KDong B(2017)Location and Processing Aware Datacube CachingProceedings of the 29th International Conference on Scientific and Statistical Database Management10.1145/3085504.3085539(1-6)Online publication date: 27-Jun-2017
https://dl.acm.org/doi/10.1145/3085504.3085539
Dong BWu KByna SLiu JZhao WRusu FHuang HWeissman JIamnitchi AIosup A(2017)ArrayUDFProceedings of the 26th International Symposium on High-Performance Parallel and Distributed Computing10.1145/3078597.3078599(53-64)Online publication date: 26-Jun-2017
https://dl.acm.org/doi/10.1145/3078597.3078599
Merticariu GMisev DBaumann P(2016)Towards a General Array Database Benchmark: Measuring Storage AccessBig Data Benchmarking10.1007/978-3-319-49748-8_3(40-67)Online publication date: 1-Dec-2016
https://doi.org/10.1007/978-3-319-49748-8_3
Baumann P(2016)Array DatabasesEncyclopedia of Database Systems10.1007/978-1-4899-7993-3_2061-2(1-12)Online publication date: 21-Dec-2016
https://doi.org/10.1007/978-1-4899-7993-3_2061-2
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents