Apache Mesos and Apache Aurora have been crucial in growing a microservices culture at Twitter. The initial phase of adoption at Aurora at Twitter saw many teams abandon their own deploy tooling and frameworks in favor of concise Aurora configuration files.
But over time a whole new class of bespoke deploy tooling has emerged as service owners had to target large deployment matrices consisting of multiple pre-production environments (for things like performance testing, integration testing, regression testing, canary testing, etc.) across multiple availability zones (i.e. multiple Mesos clusters). As the matrix grows, manual orchestration quickly becomes untenable.
The custom tooling also had a strong emphasis on supporting rolling back services to a previous good state, which only becomes more complicated as your deployment matrix grows.
In this talk I’ll outline the ways in which Mesos and Aurora helped our engineers to implement more sophisticated DevOps processes by making it easy to grow their deployment matrix. I’ll outline how that process surfaced some of the holes in currently available tooling and how it led to a huge amount of duplicate effort for our service owners. Finally I’ll describe a system we’ve built at Twitter to support our CI/CD pipeline that we feel closes those gaps.
