More Web Proxy on the site http://driver.im/

research-article

PNUTS: Yahoo!'s hosted data serving platform

Authors:

Brian F. Cooper,

Raghu Ramakrishnan,

Utkarsh Srivastava,

Adam Silberstein,

Philip Bohannon,

Hans-Arno Jacobsen,

Ramana YerneniAuthors Info & Claims

Proceedings of the VLDB Endowment, Volume 1, Issue 2

Pages 1277 - 1288

https://doi.org/10.14778/1454159.1454167

Published: 01 August 2008 Publication History

Abstract

We describe PNUTS, a massively parallel and geographically distributed database system for Yahoo!'s web applications. PNUTS provides data storage organized as hashed or ordered tables, low latency for large numbers of concurrent requests including updates and queries, and novel per-record consistency guarantees. It is a hosted, centrally managed, and geographically distributed service, and utilizes automated load-balancing and failover to reduce operational complexity. The first version of the system is currently serving in production. We describe the motivation for PNUTS and the design and implementation of its table storage and replication layers, and then present experimental results.

References

[1]

Eventually consistent. http://www.allthingsdistributed.com/2007/12/-eventually_consistent.html.

[2]

Trading consistency for scalability in distributed architectures. http://www.infoq.com/news/2008/03/ebaybase, 2008.

[3]

M. K. Aguilera, A. Merchant, M. Shah, A. Veitch, and C. Karamanolis. Sinfonia: A new paradigm for building scalable distributed systems. In SOSP, 2007.

Digital Library

[4]

P. Bernstein, N. Dani, B. Khessib, R. Manne, and D. Shutt. Data management issues in supporting large-scale web services. IEEE Data Engineering Bulletin, December 2006.

[5]

P. Bernstein, V. Hadzilacos, and N. Goodman. Concurrency Control and Recovery in Database Systems. Addison-Wesley, 1987.

Digital Library

[6]

P. A. Bernstein and N. Goodman. Timestamp-based algorithms for concurrency control in distributed database systems. In Proc. VLDB, 1980.

Digital Library

[7]

L. Breslau, P. Cao, L. Fan, G. Phillips, and S. Shenker. Web caching and zipf-like distributions: Evidence and implications. In Proc. INFOCOM, 1999.

[8]

F. Chang et al. Bigtable: A distributed storage system for structured data. In OSDI, 2006.

Digital Library

[9]

F. Dabek, M. F. Kaashoek, D. R. Karger, R. Morris, and I. Stoica. Wide-area cooperative storage with CFS. In Proc. SOSP, 2001.

Digital Library

[10]

K. Daudjee and K. Salem. Lazy database replication with snapshot isolation. In Proc. VLDB, 2006.

Digital Library

[11]

J. Dean and S. Ghemawat. MapReduce: Simplified data processing on large clusters. In OSDI, 2004.

Digital Library

[12]

G. DeCandia et al. Dynamo: Amazon's highly available key-value store. In SOSP, 2007.

Digital Library

[13]

D. J. DeWitt and J. Gray. Parallel database systems: The future of high performance database processing. CACM, 36(6), June 1992.

Digital Library

[14]

I. Stoica et al. Consistent hashing and random trees: distributed caching protocols for relieving hot spots on the World Wide Web. In Proc. ACM STOC, 1997.

Digital Library

[15]

S. Ghemawat, H. Gobioff, and S.-T. Leung. The Google File System. In Proc. SOSP, 2003.

Digital Library

[16]

J. Gray and A. Reuter. Transaction Processing: Concepts and Techniques. Morgan Kaufmann, 1993.

Digital Library

[17]

P. Helland. Life beyond distributed transactions: an apostate's opinion. In Proc. Conference on Innovative Data Systems Research (CIDR), 2007.

[18]

Ryan Huebsch, Joseph M. Hellerstein, Nick Lanham, Boon Thau Loo, Scott Shenker, and Ion Stoica. Querying the internet with pier. In Proc. VLDB, 2003.

Digital Library

[19]

D. Kossmann. The state of the art in distributed query processing. ACM Computing Surveys, 32(4):422--469, 2000.

Digital Library

[20]

J. MacCormick, N. Murphy, M. Najork, C. A. Thekkath, and L. Zhou. Boxwood: Abstractions as the foundation for storage infrastructure. In OSDI, 2004.

Digital Library

[21]

C. Olston, B. Reed, U. Srivastava, R. Kumar, and A. Tomkins. Pig Latin: A not-so-foreign language for data processing. In Proc. SIGMOD, 2008.

Digital Library

[22]

E. Pacitti, P. Minet, and E. Simon. Fast algorithms for maintaining replica consistency in lazy master replicated databases. In VLDB, 1999.

Digital Library

[23]

K. Petersen, M. J. Spreitzer, D. B. Terry, M. M. Theimer, and A. J. Demers. Flexible update propagation for weakly consistent replication. In Proc. SOSP, 1997.

Digital Library

[24]

A. Rowstron and P. Druschel. Pastry: Scalable, decentralized object location and routing for large-scale peer-to-peer systems. In Middleware, 2001.

Digital Library

[25]

A. Silberstein, B. F. Cooper, U. Srivastava, E. Vee, R. Yerneni, and R. Ramakrishnan. Efficient bulk insertion into a distributed ordered table. In Proc. SIGMOD, 2008.

Digital Library

[26]

I. Stoica, R. Morris, D. Karger, M. F. Kaashoek, and H. Balakrishnan. Chord: A scalable peer-to-peer lookup service for internet applications. In Proc. SIGCOMM, 2001.

Digital Library

[27]

S. A. Weil, S. A. Brandt, E. L. Miller, D. D. E. Long, and C. Maltzahn. Ceph: A scalable, high-performance distributed file system. In Proc. OSDI, 2006.

Digital Library

[28]

S. A. Weil, S. A. Brandt, E. L. Miller, and C. Maltzahn. CRUSH: Controlled, scalable, decentralized placement of replicated data. In Proc. Supercomputing (SC), 2006.

Digital Library

Cited By

Lai ZCui FFan HLo EZhou WLi F(2024)Occam's Razor for Distributed ProtocolsProceedings of the 2024 ACM Symposium on Cloud Computing10.1145/3698038.3698514(618-636)Online publication date: 20-Nov-2024
https://dl.acm.org/doi/10.1145/3698038.3698514
Wooders SMo XNarang ALin KStoica IHellerstein JCrooks NGonzalez J(2023)RALF: Accuracy-Aware Scheduling for Feature Store MaintenanceProceedings of the VLDB Endowment10.14778/3632093.363211617:3(563-576)Online publication date: 1-Nov-2023
https://dl.acm.org/doi/10.14778/3632093.3632116
Hildred JAbebe MDaudjee K(2023)Caerus: Low-Latency Distributed Transactions for Geo-Replicated SystemsProceedings of the VLDB Endowment10.14778/3632093.363210917:3(469-482)Online publication date: 1-Nov-2023
https://dl.acm.org/doi/10.14778/3632093.3632109
Show More Cited By

Index Terms

PNUTS: Yahoo!'s hosted data serving platform
1. Information systems

Recommendations

A batch of PNUTS: experiences connecting cloud batch and serving systems
SIGMOD '11: Proceedings of the 2011 ACM SIGMOD International Conference on Management of data

Cloud data management systems are growing in prominence, particularly at large Internet companies like Google, Yahoo!, and Amazon, which prize them for their scalability and elasticity. Each of these systems trades off between low-latency serving ...
PNUTS in Flight: Web-Scale Data Serving at Yahoo

Data management for stateful Web applications is extremely challenging. Applications must scale as they grow in popularity, serve their content with low latency on a global scale, and be highly available, even in the face of hardware failures. This need ...
Big Data Analytics with R and Hadoop

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Proceedings of the VLDB Endowment

Proceedings of the VLDB Endowment Volume 1, Issue 2

August 2008

461 pages

ISSN:2150-8097

Issue’s Table of Contents

Publisher

VLDB Endowment

Publication History

Published: 01 August 2008

Published in PVLDB Volume 1, Issue 2

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

309
Total Citations
View Citations
4,327
Total Downloads

Downloads (Last 12 months)577
Downloads (Last 6 weeks)63

Reflects downloads up to 10 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Lai ZCui FFan HLo EZhou WLi F(2024)Occam's Razor for Distributed ProtocolsProceedings of the 2024 ACM Symposium on Cloud Computing10.1145/3698038.3698514(618-636)Online publication date: 20-Nov-2024
https://dl.acm.org/doi/10.1145/3698038.3698514
Wooders SMo XNarang ALin KStoica IHellerstein JCrooks NGonzalez J(2023)RALF: Accuracy-Aware Scheduling for Feature Store MaintenanceProceedings of the VLDB Endowment10.14778/3632093.363211617:3(563-576)Online publication date: 1-Nov-2023
https://dl.acm.org/doi/10.14778/3632093.3632116
Hildred JAbebe MDaudjee K(2023)Caerus: Low-Latency Distributed Transactions for Geo-Replicated SystemsProceedings of the VLDB Endowment10.14778/3632093.363210917:3(469-482)Online publication date: 1-Nov-2023
https://dl.acm.org/doi/10.14778/3632093.3632109
Margara ACugola GFelicioni NCilloni S(2023)A Model and Survey of Distributed Data-Intensive SystemsACM Computing Surveys10.1145/360480156:1(1-69)Online publication date: 26-Aug-2023
https://dl.acm.org/doi/10.1145/3604801
Nguyen CMiller JAbadi D(2023)Detock: High Performance Multi-region Transactions at ScaleProceedings of the ACM on Management of Data10.1145/35892931:2(1-27)Online publication date: 20-Jun-2023
https://dl.acm.org/doi/10.1145/3589293
Han RYu JZhang RHerlihy MNarula N(2022)Analysing and Improving Shard Allocation Protocols for Sharded BlockchainsProceedings of the 4th ACM Conference on Advances in Financial Technologies10.1145/3558535.3559783(198-216)Online publication date: 19-Sep-2022
https://dl.acm.org/doi/10.1145/3558535.3559783
Houshmand FSaberlatibari JLesani MJhala RDillig I(2022)Hamband: RDMA replicated data typesProceedings of the 43rd ACM SIGPLAN International Conference on Programming Language Design and Implementation10.1145/3519939.3523426(348-363)Online publication date: 9-Jun-2022
https://dl.acm.org/doi/10.1145/3519939.3523426
VanBenschoten NAjmani AGartner MMatei AShah ASharif IShraer AStorm ATaft RTan OWoods AWalters PIves ZBonifati AEl Abbadi A(2022)Enabling the Next Generation of Multi-Region Applications with CockroachDBProceedings of the 2022 International Conference on Management of Data10.1145/3514221.3526053(2312-2325)Online publication date: 10-Jun-2022
https://dl.acm.org/doi/10.1145/3514221.3526053
Dharamjeet Chen YChen TKuan YChang Y(2022)LLSM: A Lifetime-Aware Wear-Leveling for LSM-Tree on NAND Flash MemoryIEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems10.1109/TCAD.2022.319754241:11(3946-3956)Online publication date: 1-Nov-2022
https://dl.acm.org/doi/10.1109/TCAD.2022.3197542
Li YEldawy AXue JKnorozova NMokbel MJanardan R(2022)Scalable computational geometry in MapReduceThe VLDB Journal — The International Journal on Very Large Data Bases10.1007/s00778-018-0534-528:4(523-548)Online publication date: 10-Mar-2022
https://dl.acm.org/doi/10.1007/s00778-018-0534-5
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents