QA for ML: Testing around the AI Black Box

There's been a lot of discussion around leveraging AI and machine learning (ML) for testing software, but seemingly less on how to test and provide QA for an AI- or ML-driven application. But Lauren Pehnke was faced with doing just that for a product that would combine natural language processing, computer vision, and image classification to drive ranked search results. She will outline the research on AI and ML she did before taking on this role, including basic vocabulary for the variants of AI involved and interesting AI "failures" to better inform what areas of the process would require a more critical eye. She will also cover testing approaches and strategies that have been useful in this space, borrowing from the scientific method and exploratory and black box testing, some ways to quantify qualitative evaluations of results when developing models, and a reminder that while the shiny new tech is cool, eventually it has to fit into an end-user experience. Come with your questions and your experiences to contribute to this conversational session so that we can all learn more about working with ML.