Agile + DevOps West 2020 Concurrent Session : Software Breaks in Mysterious Ways: Stories and Lessons from Netflix

Conference archive


Thursday, June 11, 2020 - 10:00am to 11:00am

Software Breaks in Mysterious Ways: Stories and Lessons from Netflix

Unexpected regressions, bugs with no apparent reason—they happen, especially if you work in a complex, fast-paced ecosystem with multiple moving parts. Is it possible to avoid failures if you don’t know what can fail? How do you even find out about such failures? The Netflix Growth Engineering team owns the infrastructure for user signup for over 190 different countries. The team supports many device types and payment methods, run A/B experiments, and integrate with dozens of services owned by multiple other teams as well as external partners. There are many opportunities for things to go wrong. Marek Kiszkis will present techniques employed at Netflix Growth Engineering to prevent and detect unknown errors. Learn how Netflix embraced end-to-end testing (which now comprises 70% of the team's testing pyramid), and how they made it cheaper, faster, and more reliable. He'll also cover how Netflix leveraged A/B testing for migrations from legacy systems or large-scale refactorings. Come learn how accepting the fact that software will break in unknown ways changes your day-to-day work, from development workflow to deployments to monitoring and alerting.

Marek Kiszkis

Marek Kiszkis has been developing software for the last 10+ years, most recently at Netflix. Some time ago, he used to think that it’s possible to have bug-free software. Now he realizes there are bugs everywhere, you just have to look close enough. He learned to get along with the bugs, but still tries to keep their population in check. Marek's experience has taught him to always look for simple solutions to complicated problems. To that end, he searches for and contributes to efforts related to software quality, clean code and reducing technical debt. He actually, sincerely enjoys working with legacy code.