More Web Proxy on the site http://driver.im/

Article

The architecture of SciDB

Authors:

Michael Stonebraker,

Suchi RamanAuthors Info & Claims

SSDBM'11: Proceedings of the 23rd international conference on Scientific and statistical database management

Pages 1 - 16

Published: 20 July 2011 Publication History

Abstract

SciDB is an open-source analytical database oriented toward the data management needs of scientists. As such it mixes statistical and linear algebra operations with data management ones, using a natural nested multidimensional array data model. We have been working on the code for two years, most recently with the help of venture capital backing. Release 11.06 (June 2011) is downloadable from our website (SciDB.org).

This paper presents the main design decisions of SciDB. It focuses on our decisions concerning a high-level, SQL-like query language, the issues facing our query optimizer and executor and efficient storage management for arrays. The paper also discusses implementation of features not usually present in DBMSs, including version control, uncertainty and provenance.

References

[1]

http://arxiv.org/abs/0805.2366

[2]

Becla, J., Lim, K.-T.: Report from the First Workshop on Extremely Large Databases. Data Science Journal 7 (2008).

[3]

Szalay, A.: Private communication.

[4]

Branco, M., Cameron, D., Gaidioz, B., Garonne, V., Koblitz, B., Lassnig, M., Rocha, R., Salgado, P., Wenaus, T.: Managing ATLAS data on a petabyte-scale with DQ2. Journal of Physics: Conference Series 119 (2008).

[5]

Szalay, A.: The Sloan Digital Sky Survey and Beyond. In: SIGMOD Record (June 2008).

Digital Library

[6]

Cudre-Mauroux, P., et al.: A Demonstration of SciDB: a Science-oriented DBMS. VLDB 2(2), 1534-1537 (2009).

Digital Library

[7]

Becla, J., Lim, K.-T.: Report from the Second Workshop on Extremely Large Databases, http://www-conf.slac.stanford.edu/xldb08/, http://www.jstage.jst.go.jp/article/dsj/7/0/1/_pdf

[8]

Becla, J., Lim, K.-T.: Report from the Third Workshop on Extremely Large Databases, http://www-conf.slac.stanford.edu/xldb09/

[9]

Becla, J., Lim, K.-T.: Report from the Fourth Workshop on Extremely Large Databases, http://www-conf.slac.stanford.edu/xldb10/

[10]

Cudre-Maroux, P., et al.: SS-DB: A Standard Science DBMS Benchmark (submitted for publication).

[11]

http://www.hdfgroup.org/HDF5/

[12]

http://en.wikipedia.org/wiki/APLprogramming_language

[13]

http://en.wikipedia.org/wiki/Functional_programming

[14]

http://kx.com/

[15]

Stonebraker, M., Rowe, L.A., Hirohama, M.: The Implementation of POSTGRES. IEEE Transactions on Knowledge and Data Engineering 2(1), 125-142 (1990).

Digital Library

[16]

http://developer.postgresql.org/docs/postgres/xaggr.html

[17]

http://www.netlib.org/ScaLAPACK/

[18]

Sarawagi, S., Stonebraker, M.: Efficient organization of large multidimensional arrays. In: ICDE, pp. 328-336 (1994), http://citeseer.ist.psu.edu/article/sarawagi94efficient.html

Digital Library

[19]

Soroush, E., et al.: ArrayStore: A Storage Manager for Complex Parallel Array Processing. In: Proc. 2011 SIGMOD Conference (2011).

Digital Library

[20]

Seering, A., et al.: Efficient Versioning for Scientific Arrays (submitted for publication).

[21]

Mutsuzaki, M., Theobald, M., de Keijzer, A., Widom, J., Agrawal, P., Benjelloun, O., Das Sarma, A., Murthy, R., Sugihara, T.: Trio-One: Layering Uncertainty and Lineage on a Conventional DBMS. In: Proceedings of the 2007 CIDR Conference, Asilomar, CA (January 2007).

[22]

Wu, E., et al.: The SciDB Provenance System (in preparation).

[23]

Cohen, J., et al.: Mad Skills: New Analysis Practices for Big Data. In: Proc. 2009 VLDB Conference

Digital Library

[24]

http://www.r-project.org/

[25]

http://monetdb.cwi.nl/

[26]

van Ballegooij, A., Cornacchia, R., de Vries, A.P., Kersten, M.L.: Distribution Rules for Array Database Queries. In: Andersen, K.V., Debenham, J., Wagner, R. (eds.) DEXA 2005. LNCS, vol. 3588, pp. 55-64. Springer, Heidelberg (2005).

Digital Library

Cited By

Koo KKim SKim WChoi YHan JKim BMoon B(2024)PreVision: An Out-of-Core Matrix Computation System with Optimal Buffer ReplacementProceedings of the ACM on Management of Data10.1145/36392972:1(1-25)Online publication date: 26-Mar-2024
https://dl.acm.org/doi/10.1145/3639297
Zhou LChen JDas AMin HYu LZhao MZou J(2022)Serving deep learning models with deduplication from relational databasesProceedings of the VLDB Endowment10.14778/3547305.354732515:10(2230-2243)Online publication date: 7-Sep-2022
https://dl.acm.org/doi/10.14778/3547305.3547325
Zhang WKim JRoss KSedlar EStadler L(2021)Adaptive code generation for data-intensive analyticsProceedings of the VLDB Endowment10.14778/3447689.344769714:6(929-942)Online publication date: 12-Apr-2021
https://dl.acm.org/doi/10.14778/3447689.3447697
Show More Cited By

The architecture of SciDB
1. Information systems
  1. Data management systems

Recommendations

Overview of sciDB: large scale array storage, processing and analysis
SIGMOD '10: Proceedings of the 2010 ACM SIGMOD International Conference on Management of data

SciDB [4, 3] is a new open-source data management system intended primarily for use in application domains that involve very large (petabyte) scale array data; for example, scientific applications such as astronomy, remote sensing and climate modeling, ...
Efficient Storage Allocation of Large-Scale Extendible Multi-dimensional Scientific Datasets
SSDBM '06: Proceedings of the 18th International Conference on Scientific and Statistical Database Management

Data analyses in scientific domains involve storage, retrieval, processing and visualization of large scale multidimensional datasets. The datasets incrementally grow by appending new data to the dataset without reorganizing the already allocated data ...
Selective Scan for Filter Operator of SciDB
SSDBM '16: Proceedings of the 28th International Conference on Scientific and Statistical Database Management

Recently there has been an increasing interest in analyzing scientific data generated by observations and scientific experiments. For managing these data efficiently, SciDB, a multi-dimensional array-based DBMS, is suggested. When SciDB processes a ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Guide Proceedings

SSDBM'11: Proceedings of the 23rd international conference on Scientific and statistical database management

July 2011

601 pages

ISBN:9783642223501

Editors:
Judith Bayard Cushing
The Evergreen State College, Olympia,WA
,
James French
CNRI and University of Virginia, Charlottesville, VA
,
Shawn Bowers
Gonzaga University, Spokane, WA

Sponsors

Paradigm4 Inc.: Paradigm4 Inc.
Microsoft Research: Microsoft Research
Gordon and Betty Moore Foundation: Gordon and Betty Moore Foundation
eScience Institute: eScience Institute

Publisher

Springer-Verlag

Berlin, Heidelberg

Publication History

Published: 20 July 2011

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

54
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 17 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Koo KKim SKim WChoi YHan JKim BMoon B(2024)PreVision: An Out-of-Core Matrix Computation System with Optimal Buffer ReplacementProceedings of the ACM on Management of Data10.1145/36392972:1(1-25)Online publication date: 26-Mar-2024
https://dl.acm.org/doi/10.1145/3639297
Zhou LChen JDas AMin HYu LZhao MZou J(2022)Serving deep learning models with deduplication from relational databasesProceedings of the VLDB Endowment10.14778/3547305.354732515:10(2230-2243)Online publication date: 7-Sep-2022
https://dl.acm.org/doi/10.14778/3547305.3547325
Zhang WKim JRoss KSedlar EStadler L(2021)Adaptive code generation for data-intensive analyticsProceedings of the VLDB Endowment10.14778/3447689.344769714:6(929-942)Online publication date: 12-Apr-2021
https://dl.acm.org/doi/10.14778/3447689.3447697
Choi DYoon HChung Y(2021)Subarray Skyline Query Processing in Array DatabasesProceedings of the 33rd International Conference on Scientific and Statistical Database Management10.1145/3468791.3468799(37-48)Online publication date: 6-Jul-2021
https://dl.acm.org/doi/10.1145/3468791.3468799
Mofrad MMelhem RAhmad YHammoud M(2020)GraphiteProceedings of the VLDB Endowment10.14778/3380750.338075113:6(783-797)Online publication date: 11-Mar-2020
https://dl.acm.org/doi/10.14778/3380750.3380751
Dolmatova OAugsten NBöhlen MMaier DPottinger RDoan ATan WAlawini ANgo H(2020)A Relational Matrix Algebra and its Implementation in a Column StoreProceedings of the 2020 ACM SIGMOD International Conference on Management of Data10.1145/3318464.3389747(2573-2587)Online publication date: 11-Jun-2020
https://dl.acm.org/doi/10.1145/3318464.3389747
Choi DPark CChung Y(2019)Progressive top-k subarray query processing in array databasesProceedings of the VLDB Endowment10.14778/3329772.332977612:9(989-1001)Online publication date: 1-May-2019
https://dl.acm.org/doi/10.14778/3329772.3329776
Schüle MBungeroth MKemper AGünnemann SNeumann T(2019)MLearnProceedings of the 3rd International Workshop on Data Management for End-to-End Machine Learning10.1145/3329486.3329494(1-4)Online publication date: 30-Jun-2019
https://dl.acm.org/doi/10.1145/3329486.3329494
Li SChen LKumar ABoncz PManegold SAilamaki ADeshpande AKraska T(2019)Enabling and Optimizing Non-linear Feature Interactions in Factorized Linear AlgebraProceedings of the 2019 International Conference on Management of Data10.1145/3299869.3319878(1571-1588)Online publication date: 25-Jun-2019
https://dl.acm.org/doi/10.1145/3299869.3319878
Han DNam YLee JPark KKim HKim MBoncz PManegold SAilamaki ADeshpande AKraska T(2019)DistMEProceedings of the 2019 International Conference on Management of Data10.1145/3299869.3319865(759-774)Online publication date: 25-Jun-2019
https://dl.acm.org/doi/10.1145/3299869.3319865
Show More Cited By

View Options

View options

Media

Figures

Other

Tables

View Table of Contents