Tools shaping culture

     Mark will talk about his journey at Ticketmaster to introduce open source tools to the organization and how those tools changed the engineering culture for the better.

     With the Gitlab scm tool, Mark teamed with a few early adopters migrating from SVN to Git, and witnessed how Gitlab changed the way people worked.  With SVN the code could be opaque.   Conversely, with Gitlab the code became transparent, even to non-engineers.    Day to day tasks became easier:  Code review was easier from a workflow perspective, and people could start contributing code to other team’s projects very easily through the browser based merge request feature.    Greater transparency and collaboration helped the TM engineering teams deliver quality products much faster both internally within their own teams, and collaboratively with other teams throughout the organization.

     With Rundeck, Mark introduced the tool to facilitate both automated code deployments and allow software engineers to self-service deployments all the way to production.   Previously the organization had a deployment hand-off model where engineers wrote code and handed it to various downstream groups for testing and production deployment activities.    By using rundeck and deployment automation, code could be delivered safely to production environments by the same teams that engineered the code, resulting in more rapid deployment cycles and better mean time to repair in the event outages occurred.   Using this tool shifted the mindset from a waterfall delivery approach to a devops ownership model.   Engineers develop, ship and ultimately own their code in production.   Shifting resposibility left has dramatically changed the way Ticketmaster delivers their products by increasing velocity and reducing business disruptions. 

      Zabbix is a popular open-source alerting solution used by many companies.   In our organization monitoring and alerting was the domain of the systems teams primarily focused on server health.   Our team rolled out zabbix along with automated connectors that made onboarding through the API both simple and self-service.   With application and business level SLA alerting in place and proper demonstration, adoption ballooned with the product and engineering teams started taking notice of how their products fared in the production environments.   Where previously business interuptions were mostly perceived as an operations problem, visibly providing product health transparency with Zabbix and putting development teams in the primary escalation path resulted in the engineering teams responding much faster to service interuptions increasing both mean time to detect and mean time to repair.   

     All three of these tools are open source and freely available.  Their use has dramatically changed the culture at Ticketmaster as engineering teams are better able to take ownership of their code, product delivery, and product uptime in the wild.  Rather than mandate cultural change from the executive level, Ticketmaster achieved this organically from the bottom up by simply taking popular open source tools and making them widely available to engineers in the organization.   If the tools are built and they are excellent, adoption and subsequently cultural change will occur.