We Migrated to Loki and Survived: Lessons from the Trenches

When ZipRecruiter's monthly logging bill started competing with our cloud infrastructure costs, we knew something had to change. Logz.io had served us well, but as our scale grew, so did our pain points: unpredictable costs, retention limitations, and the feeling we were paying premium prices for commodity infrastructure. Enter Grafana Loki - the "like Prometheus, but for logs" solution that promised cost savings, better Kubernetes integration, and control over our own destiny. The migration wasn't quite as smooth as the blog posts made it sound, though. This talk walks through ZipRecruiter's journey migrating our logging infrastructure and the interesting discoveries and painful lessons we learned along the way. We'll cover how we justified the move to leadership, the architecture choices we made (and some we regretted), and why we learned the hard way that Loki is definitely not Elasticsearch. You'll hear about running dual logging systems during the transition, the cardinality problems that kept us up at night, and the real incidents we faced in production. We'll share actual cost comparisons, performance tuning strategies, team feedback, and where Loki exceeded and fell short of our expectations. No marketing fluff - just honest stories from the trenches, complete with monitoring dashboards and the messy details other talks skip over. Whether you're considering a similar migration, already running Loki, or just curious about the trade-offs of managed vs. self-hosted logging infrastructure, you'll walk away with actionable insights and realistic expectations about what it really takes to run Loki at scale.