Manish Gill

When ClickHouse Cloud was launched - we still had Persistent Volume Claims (PVCs) backed by EBS (and other equivalent) volumes to store critical metadata in the hot path. We did not have true separation of Compute and Storage. 

Over the course of the last few years, we have been slowly but surely working our way to remove this limitation. This talk is the culmination of everything that was done - important fixes in the Core database as well as the cloud database Operator - to finally get rid of those volumes. Now when we allocate database server pods in the cloud - there are no Network Block Volumes attached to them. 

This talk documents the challenges of implementing such a critical feature in the journey of a cloud database, as well as the subsequent migration of customers in this cloud to this mode of operation. We will cover important topics such as S3-based Storage Engine, Catalog and S3-Plain Re-writable Disks as well as the Kubernetes Operator changes to enable these features in the Cloud.

ClickHouse recently undertook a major refactoring of our operator to change how we orchestrate customer-provisioned databases through Kubernetes StatefulSets.

The challenge? Migrating thousands of services to the new orchestration system without any downtime. This talk explores how we accomplished this migration, diving into the complex challenges we faced and solved. We'll cover our custom migration controller, durable execution workflows, and database synchronization hurdles that emerged while keeping ClickHouse Cloud's fleet running smoothly.