Testing Uncertainty for a Conversational AI

Uncertainty has always been a key challenge for testers. But testing a chatbot adds a completely new level of uncertainty. There are a lot of platforms and tools available for chatbot development, but what we lack is a standardized chatbot testing strategy. The way testing is performed on chatbots differs a lot from "traditional" testing (like for an app or web portal) due to the apparent randomness of a conversation with a chatbot. From testing numerous clients' chatbots. Rajni Singh has experienced that it is impossible to anticipate all the situations that can happen during a conversation. As they introduced learning components to conversational ai such as machine learning and intent training, the chatbot evolved and changed its behavior compared to previous test runs. This increased the need for regression tests and complicated them at the same time. Rajni will talk about the challenges faced during chatbot testing and how to mitigate them with different strategies. She'll share her experience using their own advanced automation framework as well as commercial and open source tools.