Volley: Accelerating Write-Read Orders in Disaggregated Storage

Modern data centers deploy disaggregated storage systems (e.g., NVMe over Fabrics, NVMe-oF) for fine-grained resource elasticity and high resource utilization. A client-side writeback cache is used to absorb writes and buffer frequently accessed data, thereby eliminating unnecessary remote storage accesses and improving performance. Yet, a cache miss on the full cache triggers an evict-and-fetch operation which evicts the old entries before new data blocks are fetched. Existing systems perform the evict-and-fetch operation by sequentially executing write and read I/O operations, which reduces the concurrency and makes it challenging to fully utilize the fast network and storage devices.

To overcome this problem, we propose Volley, a network storage protocol that guarantees the execution order of the write and read I/O operations, enabling the writeback cache to issue eviction and fetch operations simultaneously. We implement the Volley protocol by extending NVMe-oF atop commodity network and storage hardware and further the ACK free-rides and notification acceleration techniques to reduce the network overhead. We adopt Volley into two writeback caching systems (V-Cache and V-TriCache) for virtual machine storage and out-of-core computing, respectively. Evaluations show that V-Cache outperforms Linux page cache and SPDK OCF (a state-of-the-art caching system) by up to 6.84× and 3.01×. Experiments on a production workload of Facebook (Mixgraph) show that V-TriCache reduces the total running time by up to 16.7% compared with the state-of-the-art out-of-core computing system, TriCache.