More Web Proxy on the site http://driver.im/

Article

Flashshare: punching through server storage stack from kernel to firmware for ultra-low latency SSDs

Authors:

Mohammad Alian,

Myoungjun Chun,

Mahmut Taylan Kandemir,

Myoungsoo JungAuthors Info & Claims

OSDI'18: Proceedings of the 13th USENIX conference on Operating Systems Design and Implementation

Pages 477 - 492

Published: 08 October 2018 Publication History

Abstract

A modern datacenter server aims to achieve high energy efficiency by co-running multiple applications. Some of such applications (e.g., web search) are latency sensitive. Therefore, they require low-latency I/O services to fast respond to requests from clients. However, we observe that simply replacing the storage devices of servers with Ultra-Low-Latency (ULL) SSDs does not notably reduce the latency of I/O services, especially when co-running multiple applications. In this paper, we propose FLASHSHARE to assist ULL SSDs to satisfy different levels of I/O service latency requirements for different co-running applications. Specifically, FLASHSHARE is a holistic cross-stack approach, which can significantly reduce I/O interferences among co-running applications at a server without any change in applications. At the kernel-level, we extend the data structures of the storage stack to pass attributes of (co-running) applications through all the layers of the underlying storage stack spanning from the OS kernel to the SSD firmware. For given attributes, the block layer and NVMe driver of FLASHSHARE differently manage the I/O scheduler and interrupt handler of NVMe. We also enhance the NVMe controller and cache layer at the SSD firmware-level, by dynamically partitioning DRAM in the ULL SSD and adjusting its caching strategies to meet diverse user requirements. The evaluation results demonstrate that FLASHSHARE can shorten the average and 99th-percentile turnaround response times of co-running applications by 22% and 31%, respectively.

References

[1]

ALIAN, M., ABULILA, A. H., JINDAL, L., KIM, D., AND KIM, N. S. Ncap: Network-driven, packet context-aware power management for client-server architecture. In High Performance Computer Architecture (HPCA), 2017 IEEE International Symposium on (2017), IEEE, pp. 25-36.

[2]

BINKERT, N., BECKMANN, B., BLACK, G., REINHARDT, S. K., SAIDI, A., BASU, A., HESTNESS, J., HOWER, D. R., KRISHNA, T., SARDASHTI, S., SEN, R., SEWELL, K., SHOAIB, M., VAISH, N., HILL, M. D., AND WOOD, D. A. The gem5 simulator. SIGARCH Comput. Archit. News 39, 2 (Aug. 2011), 1-7.

Digital Library

[3]

CAULFIELD, A. M., DE, A., COBURN, J., MOLLOW, T. I., GUPTA, R. K., AND SWANSON, S. Moneta: A high-performance storage array architecture for next-generation, nonvolatile memories. In Proceedings of the 2010 43rd Annual IEEE/ACM International Symposium on Microarchitecture (2010), IEEE Computer Society, pp. 385-395.

Digital Library

[4]

CHANG, L.-P. On efficient wear leveling for large-scale flash-memory storage systems. In Proceedings of the 2007 ACM symposium on Applied computing (2007), ACM, pp. 1126-1130.

Digital Library

[5]

CHANG, L.-P., KUO, T.-W., AND LO, S.-W. Real-time garbage collection for flash-memory storage systems of real-time embedded systems. ACM Transactions on Embedded Computing Systems (TECS) 3, 4 (2004), 837-863.

Digital Library

[6]

CHEONG, W., YOON, C., WOO, S., HAN, K., KIM, D., LEE, C., CHOI, Y., KIM, S., KANG, D., YU, G., ET AL. A flash memory controller for 15ms ultra-low-latency ssd using high-speed 3d nand flash with 3ms read time. In Solid-State Circuits Conference-(ISSCC), 2018 IEEE International (2018), IEEE, pp. 338-340.

[7]

EISENMAN, A., GARDNER, D., ABDELRAHMAN, I., AXBOE, J., DONG, S., HAZELWOOD, K., PETERSEN, C., CIDON, A., AND KATTI, S. Reducing dram footprint with nvm in facebook. In Proceedings of the Thirteenth EuroSys Conference (2018), ACM, p. 42.

Digital Library

[8]

FARRINGTON, N., AND ANDREYEV, A. Facebook's data center network architecture. In Optical Interconnects Conference, 2013 IEEE (2013), Citeseer, pp. 49-50.

[9]

FREE SOFTWARE FOUNDATION. nice. http://www.gnu.org/software/coreutils/manual/coreutils.html#nice-invocation.

[10]

GOUK, D., KWON, M., ZHANG, J., KOH, S., CHOI, W., KIM, N. S., KANDEMIR, M., AND JUNG, M. Amber: Enabling precise full-system simulation with detailed modeling of all ssd resources. In Proceedings of the 2018 51rd Annual IEEE/ACM International Symposium on Microarchitecture (2018), IEEE Computer Society.

Digital Library

[11]

GRUPP, L. M., CAULFIELD, A. M., COBURN, J., SWANSON, S., YAAKOBI, E., SIEGEL, P. H., AND WOLF, J. K. Characterizing flash memory: anomalies, observations, and applications. In Microarchitecture, 2009. MICRO-42. 42nd Annual IEEE/ACM International Symposium on (2009), IEEE, pp. 24-33.

Digital Library

[12]

GRUPP, L. M., DAVIS, J. D., AND SWANSON, S. The harey tortoise: Managing heterogeneous write performance in ssds. In USENIX Annual Technical Conference (2013), pp. 79-90.

Digital Library

[13]

INTEL CORPORATION. Intel 535. https://ark.intel.com/products/86734/Intel-SSD-535-Series-240GB-2_5in-SATA-6Gbs-16nm-MLC, 2015.

[14]

INTEL CORPORATION. Intel 750. https://ark.intel.com/products/86740/Intel-SSD-750-Series-400GB-12-Height-PCIe-3_0-20nm-MLC, 2015.

[15]

INTEL CORPORATION. Intel optane technology. https://www.intel.com/content/www/us/en/architecture-and-technology/intel-optane-technology.html, 2016.

[16]

JEONG, S., LEE, K., LEE, S., SON, S., AND WON, Y. I/o stack optimization for smartphones. In Presented as part of the 2013 USENIX Annual Technical Conference (USENIX ATC 13) (San Jose, CA, 2013), USENIX, pp. 309-320.

Digital Library

[17]

JIN, Y., WEN, Y., AND CHEN, Q. Energy efficiency and server virtualization in data centers: An empirical investigation. In Computer Communications Workshops (INFOCOM WKSHPS), 2012 IEEE Conference on (2012), IEEE, pp. 133-138.

[18]

JUNG, M. Exploring parallel data access methods in emerging non-volatile memory systems. IEEE Transactions on Parallel & Distributed Systems, 3 (2017), 746-759.

Digital Library

[19]

JUNG, M., AND KANDEMIR, M. Middleware - firmware cooperation for high-speed solid state drives. In Middleware '12 (2012).

Digital Library

[20]

JUNG, M., AND KANDEMIR, M. Revisiting widely held ssd expectations and rethinking system-level implications. In ACM SIGMETRICS Performance Evaluation Review (2013), vol. 41, ACM, pp. 203-216.

Digital Library

[21]

JUNG, M., ZHANG, J., ABULILA, A., KWON, M., SHAHIDI, N., SHALF, J., KIM, N. S., AND KANDEMIR, M. Simplessd: Modeling solid state drives for holistic system simulation. IEEE Computer Architecture Letters PP, 99 (2017), 1-1.

Digital Library

[22]

JUNG, M.-S., IK PARK, C., AND OH, S.-J. Cooperative memory management. In US 8745309 (2009).

[23]

KIM, J., LEE, D., AND NOH, S. H. Towards slo complying ssds through ops isolation. In FAST (2015), pp. 183-189.

Digital Library

[24]

KIM, S., KIM, H., LEE, J., AND JEONG, J. Enlightening the i/o path: A holistic approach for application performance. In Proceedings of the 15th Usenix Conference on File and Storage Technologies (Berkeley, CA, USA, 2017), FAST'17, USENIX Association, pp. 345-358.

Digital Library

[25]

KIM, T., HAHN, S. S., LEE, S., HWANG, J., LEE, J., AND KIM, J. Pcstream: automatic stream allocation using program contexts. In 10th USENIX Workshop on Hot Topics in Storage and File Systems (HotStorage 18) (2018), USENIX Association.

Digital Library

[26]

LEVERICH, J., AND KOZYRAKIS, C. Reconciling high server utilization and sub-millisecond quality-of-service. In Proceedings of the Ninth European Conference on Computer Systems (New York, NY, USA, 2014), EuroSys '14, ACM, pp. 4:1-4:14.

Digital Library

[27]

LIU, J., WU, J., AND PANDA, D. K. High performance rdma-based mpi implementation over infiniband. International Journal of Parallel Programming 32, 3 (2004), 167-198.

Digital Library

[28]

LO, D., CHENG, L., GOVINDARAJU, R., BARROSO, L. A., AND KOZYRAKIS, C. Towards energy proportionality for large-scale latency-critical workloads. In ACM SIGARCH Computer Architecture News (2014), vol. 42, IEEE Press, pp. 301-312.

Digital Library

[29]

LO, D., CHENG, L., GOVINDARAJU, R., RANGANATHAN, P., AND KOZYRAKIS, C. Heracles: improving resource efficiency at scale. In ACM SIGARCH Computer Architecture News (2015), vol. 43, ACM, pp. 450-462.

Digital Library

[30]

MARS, J., TANG, L., HUNDT, R., SKADRON, K., AND SOFFA, M. L. Bubble-up: Increasing utilization in modern warehouse scale computers via sensible co-locations. In Proceedings of the 44th annual IEEE/ACM International Symposium on Microarchitecture (2011), ACM, pp. 248-259.

Digital Library

[31]

MEGIDDO, N., AND MODHA, D. S. Arc: A self-tuning, low overhead replacement cache. In FAST (2003), vol. 3, pp. 115-130.

Digital Library

[32]

MESNIER, M., CHEN, F., LUO, T., AND AKERS, J. B. Differentiated storage services. In Proceedings of the Twenty-Third ACM Symposium on Operating Systems Principles (2011), ACM, pp. 57-70.

Digital Library

[33]

MURUGAN, M., AND DU, D. H. Rejuvenator: A static wear leveling algorithm for nand flash memory with minimized overhead. In Mass Storage Systems and Technologies (MSST), 2011 IEEE 27th Symposium on (2011), IEEE, pp. 1-12.

Digital Library

[34]

PATTERSON, R. H., GIBSON, G. A., GINTING, E., STODOLSKY, D., AND ZELENKA, J. Informed prefetching and caching, vol. 29. ACM, 1995.

Digital Library

[35]

RHO, E., JOSHI, K., SHIN, S.-U., SHETTY, N. J., HWANG, J.- Y., CHO, S., LEE, D. D., AND JEONG, J. Fstream: managing flash streams in the file system. In 16th USENIX Conference on File and Storage Technologies (2018), p. 257.

Digital Library

[36]

SAMSUNG MEMORY SOLUTIONS LAB. Ultra-low latency with samsung z-nand ssd. https://www.samsung.com/us/labs/pdfs/collateral/Samsung_Z-NAND_Technology_Brief_v5.pdf, 2017.

[37]

SHIM, H., SEO, B.-K., KIM, J.-S., AND MAENG, S. An adaptive partitioning scheme for dram-based cache in solid state drives. In Mass Storage Systems and Technologies (MSST), 2010 IEEE 26th Symposium on (2010), IEEE, pp. 1-12.

Digital Library

[38]

SMITH, K. Garbage collection. SandForce, Flash Memory Summit, Santa Clara, CA (2011), 1-9.

[39]

SOURCE, I. O. org. storage performance development kit (spdk), 2016.

[40]

SUNGJOON KOH, CHANGRIM LEE, M. K. M. J. Exploring system challenges of ultra-low latency solid state drives. In The 10th USENIX Workshop on Hot Topics in Storage and File Systems (HotStorage) (2018), IEEE.

Digital Library

[41]

VUČINIú, D., WANG, Q., GUYOT, C., MATEESCU, R., BLAGOJEVIú, F., FRANCA-NETO, L., LE MOAL, D., BUNKER, T., XU, J., SWANSON, S., ET AL. Dc express: shortest latency protocol for reading phase change memory over pci express. In Proceedings of the 12th USENIX conference on File and Storage Technologies (2014), USENIX Association, pp. 309-315.

Digital Library

[42]

WANG, J., PARK, D., PAPAKONSTANTINOU, Y., AND SWANSON, S. Ssd in-storage computing for search engines. IEEE Transactions on Computers (2016).

[43]

WANG, L., ZHAN, J., LUO, C., ZHU, Y., YANG, Q., HE, Y., GAO, W., JIA, Z., SHI, Y., ZHANG, S., ZHENG, C., LU, G., ZHAN, K., LI, X., AND QIU, B. Bigdatabench: A big data benchmark suite from internet services. In 2014 IEEE 20th International Symposium on High Performance Computer Architecture (HPCA) (Feb 2014), pp. 488-499.

[44]

WEST DIGIT. I/o latency optimization with polling. https://events.static.linuxfound.org/sites/events/files/slides/lemoal-nvme-polling-vault-2017-final_0.pdf.

[45]

WIKI. Cpu cache. https://en.wikipedia.org/wiki/CPU_cache.

[46]

WORKGROUP, N. E. Nvm express revision 1.3. https://nvmexpress.org/wp-content/uploads/NVM_Express_Revision_1.3.pdf.

[47]

YADGAR, G., FACTOR, M., LI, K., AND SCHUSTER, A. Management of multilevel, multiclient cache hierarchies with application hints. ACM Transactions on Computer Systems (TOCS) 29, 2 (2011), 5.

Digital Library

[48]

YANG, J., MINTURN, D. B., AND HADY, F. When poll is better than interrupt. In FAST (2012), vol. 12, pp. 3-3.

Digital Library

[49]

YANG, J., PANDURANGAN, R., CHOI, C., AND BALAKRISHNAN, V. Autostream: automatic stream management for multistreamed ssds. In Proceedings of the 10th ACM International Systems and Storage Conference (2017), ACM, p. 3.

Digital Library

[50]

YANG, S., HARTER, T., AGRAWAL, N., KOWSALYA, S. S., KRISHNAMURTHY, A., AL-KISWANY, S., KAUSHIK, R. T., ARPACI-DUSSEAU, A. C., AND ARPACI-DUSSEAU, R. H. Split-level i/o scheduling. In Proceedings of the 25th Symposium on Operating Systems Principles (New York, NY, USA, 2015), SOSP '15, ACM, pp. 474-489.

Digital Library

[51]

YANG, T., LIU, T., BERGER, E. D., KAPLAN, S. F., AND MOSS, J. E. B. Redline: First class support for interactivity in commodity operating systems. In OSDI (2008), vol. 8, pp. 73-86.

Digital Library

[52]

YONG, H., JEONG, K., LEE, J., AND KIM, J.-S. vstream: virtual stream management for multi-streamed ssds. In 10th USENIX Workshop on Hot Topics in Storage and File Systems (HotStorage 18) (2018), USENIX Association.

Digital Library

Cited By

Zhang JKwon MGouk DKoh SKim NKandemir MJung MMartínez JDuato JJohn L(2021)Revamping storage class memory with hardware automated memory-over-storage solutionProceedings of the 48th Annual International Symposium on Computer Architecture10.1109/ISCA52012.2021.00065(762-775)Online publication date: 14-Jun-2021
https://dl.acm.org/doi/10.1109/ISCA52012.2021.00065
Liao XLu YXu EShu JLu SHowell J(2020)Write dependency disentanglement with HORAEProceedings of the 14th USENIX Conference on Operating Systems Design and Implementation10.5555/3488766.3488797(549-565)Online publication date: 4-Nov-2020
https://dl.acm.org/doi/10.5555/3488766.3488797
Lee GShin SSong WHam TLee JJeong JDan TDahlia M(2019)Asynchronous I/O stackProceedings of the 2019 USENIX Conference on Usenix Annual Technical Conference10.5555/3358807.3358858(603-616)Online publication date: 10-Jul-2019
https://dl.acm.org/doi/10.5555/3358807.3358858
Show More Cited By

Flashshare: punching through server storage stack from kernel to firmware for ultra-low latency SSDs
1. General and reference
  1. Cross-computing tools and techniques

Recommendations

TLB Improvements for Chip Multiprocessors: Inter-Core Cooperative Prefetchers and Shared Last-Level TLBs

Translation Lookaside Buffers (TLBs) are critical to overall system performance. Much past research has addressed uniprocessor TLBs, lowering access times and miss rates. However, as Chip MultiProcessors (CMPs) become ubiquitous, TLB design and ...
SELECTIVE VICTIM CACHING: A METHOD TO IMPROVE THE PERFORMANCE OF DIRECT-MAPPED CACHES
High performance cache replacement using re-reference interval prediction (RRIP)
ISCA '10

Practical cache replacement policies attempt to emulate optimal replacement by predicting the re-reference interval of a cache block. The commonly used LRU replacement policy always predicts a near-immediate re-reference interval on cache hits and ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

OSDI'18: Proceedings of the 13th USENIX conference on Operating Systems Design and Implementation

October 2018

815 pages

ISBN:9781931971478

Program Chairs:
Andrea Arpaci-Dusseau
University of Wisconsin-Madison
,
Geoff Voelker
University of California, San Diego

Sponsors

NetApp
Google Inc.
NSF
Microsoft: Microsoft
Facebook: Facebook

In-Cooperation

SIGOPS: ACM Special Interest Group on Operating Systems

Publisher

USENIX Association

United States

Publication History

Published: 08 October 2018

Check for updates

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

7
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 29 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Zhang JKwon MGouk DKoh SKim NKandemir MJung MMartínez JDuato JJohn L(2021)Revamping storage class memory with hardware automated memory-over-storage solutionProceedings of the 48th Annual International Symposium on Computer Architecture10.1109/ISCA52012.2021.00065(762-775)Online publication date: 14-Jun-2021
https://dl.acm.org/doi/10.1109/ISCA52012.2021.00065
Liao XLu YXu EShu JLu SHowell J(2020)Write dependency disentanglement with HORAEProceedings of the 14th USENIX Conference on Operating Systems Design and Implementation10.5555/3488766.3488797(549-565)Online publication date: 4-Nov-2020
https://dl.acm.org/doi/10.5555/3488766.3488797
Lee GShin SSong WHam TLee JJeong JDan TDahlia M(2019)Asynchronous I/O stackProceedings of the 2019 USENIX Conference on Usenix Annual Technical Conference10.5555/3358807.3358858(603-616)Online publication date: 10-Jul-2019
https://dl.acm.org/doi/10.5555/3358807.3358858
Liang SWang YLu YYang ZLi HLi XDan TDahlia M(2019)Cognitive SSDProceedings of the 2019 USENIX Conference on Usenix Annual Technical Conference10.5555/3358807.3358841(395-410)Online publication date: 10-Jul-2019
https://dl.acm.org/doi/10.5555/3358807.3358841
Liu JArpaci-Dusseau AArpaci-Dusseau RKannan SYadgar GNoh S(2019)File systems as processesProceedings of the 11th USENIX Conference on Hot Topics in Storage and File Systems10.5555/3357062.3357081(14-14)Online publication date: 8-Jul-2019
https://dl.acm.org/doi/10.5555/3357062.3357081
Liu JWang FFeng D(2019)CostPIProceedings of the 48th International Conference on Parallel Processing10.1145/3337821.3337879(1-10)Online publication date: 5-Aug-2019
https://dl.acm.org/doi/10.1145/3337821.3337879
Fevgas ABozanis PDesai BAnagnostopoulos DManolopoulos YNikolaidou M(2019)A spatial index for hybrid storageProceedings of the 23rd International Database Applications & Engineering Symposium10.1145/3331076.3331091(1-8)Online publication date: 10-Jun-2019
https://dl.acm.org/doi/10.1145/3331076.3331091

View Options

View options

Figures

Tables

Media

View Table of Conten