Automatic Canary Analysis: Critical Success Factor for Release Pipeline

Netflix began streaming services with a small-scale microservice ecosystem. As our global reach grew, so did the complexity of the underlying ecosystem. Membership Lifecycle Ecosystem is a cluster of services that handles all the backend business logic related to membership, signup, billing, and payments. This ecosystem consists of 100+ internal microservices and a similar number of external partners and payment processors - spanning across multiple teams. Each team has an asynchronous release cadence and the quality of each release has a direct impact on the quality of the ecosystem and end-user experiences. Automatic Canary Analysis (ACA) is a release deployment feature that enables us to conduct automatic analysis of key metrics during a production change. The fundamental principle around ACA is to carry out a delta-analysis of the behavior of new change against the current baseline, by deploying and diverting a very small subset of production traffic to the new service build. In this session, we will talk about how to use ACA to define system metrics comparison, as well as business logic metrics comparison to ensure the quality of the new deployment’s performance and functionality. Additionally, this session will discuss how ACA helped us to achieve a fully automated release pipeline.