Linux IOCost Cgroup Controller and resctl-bench

Resource isolation is a fundamental requirement in datacenter environments. However, our production experience in Meta’s largescale datacenters shows that existing IO control mechanisms for block storage are inadequate in containerized environments. IO control needs to provide proportional resources to containers while taking into account the hardware heterogeneity of storage devices and the idiosyncrasies of the workloads deployed in datacenters. The speed of modern SSDs requires IO control to execute with low-overheads. Furthermore, IO control should strive for work conservation, take into account the interactions with the memory management subsystem, and avoid priority inversions that lead to isolation failures.

IOCost, an IO control solution, is created to address these challenges. It is designed for containerized environments and provides scalable, work-conserving, and low-overhead IO control for heterogeneous storage devices and diverse workloads in datacenters. IOCost performs offline profiling to build a device model and uses it to estimate device occupancy of each IO request. To minimize runtime overhead, it separates IO control into a fast per-IO issue path and a slower periodic planning path. A novel work-conserving budget donation algorithm enables containers to dynamically share unused budget. We have deployed IOCost across the entirety of Meta’s datacenters comprised of millions of machines, upstreamed IOCost to the Linux kernel, and open-sourced our device-profiling tools. IOCost has been running in productionfor two years, providing IO control for Meta’s fleet. We describe the design of IOCost and share our experience deploying it at scale.

For more details, please refer to the following paper:

https://www.cs.cmu.edu/~dskarlat/publications/iocost_asplos22.pdf