STARWEST 2017 - Exploratory Testing

Critical thinking is the kind of thinking that specifically looks for problems and mistakes. Regular people don't do a lot of it. However, if you want to be a great tester, you need to be a great critical thinker. Critically-thinking testers save projects from dangerous assumptions and ultimately from disasters. The good news is that critical thinking is not just innate intelligence or a talent—it's a learnable and improvable skill you can master. Michael Bolton shares the specific techniques and heuristics of critical thinking and presents realistic testing puzzles that help you practice...

In the tradition of James Whittaker’s book series, How to Break Software, Jon Hagar applies the testing “attack” concept to mobile and embedded software systems. First, Jon defines the environments of mobile and embedded software. He then examines the issues of software product failures caused by defects found in these types of software. Next, Jon shares a set of attacks against mobile and embedded software based on common modes of failure that teams can direct against their software. Like different kinds of software design patterns, attacks are test design patterns that must be...

As testers, we know that we can define many more test cases than we will ever have time to design, execute, and report. The key problem in testing is choosing a small, “smart” subset—from the almost infinite number of tests available—that will find a large percentage of the defects. Join Lee Copeland to discover how to design test cases using formal black-box techniques, including equivalence class testing, boundary value testing, decision tables, and state-transition diagrams. Explore examples of each of these techniques in action. Don’t just pick test cases randomly. Learn to selectively...

Many organizations find that test automation does not work as well as they thought it would. In many cases, these failures are due to generic technical reasons, which can be fixed with relative ease. Solutions that have worked well for others are patterns; these test automation patterns are common to automation efforts at any level with whatever tools you are using. Dot Graham focuses on often-neglected technical issues—i.e., non-management issues—and the patterns that help solve them. These are not development or code patterns—this is a code-free tutorial. Using a set of patterns...

Have you ever worked on a project where you felt testing was thorough and complete—all features were covered and all tests passed—yet in the first week in production the software had serious issues and problems? Join Dawn Haynes to learn how to inject robustness testing into your projects to uncover those issues before release. Robustness—an important and often overlooked area of testing—is the degree to which a system operates correctly in the presence of exceptional inputs or stressful environmental conditions. Dawn shows you how—by expanding basic tests and incorporating specific...

A startup is an organization created to deliver a new product or service under conditions of extreme uncertainty. Approximately 40 percent of all startups will cease operation with investors losing everything; 95 percent will fall short of their financial projections. And the number one cause of startup failure? No one wants to buy their product. Eric Ries, author of The Lean Startup, learned that under conditions of extreme uncertainty, classical management methods do not bring success. Based on his and others’ experiences, he formulated the Lean Startup methodology consisting of five...

In chess, the word blunder means a very bad move by someone who should know better. Even though functional test automation has been around for a long time, people still make some very bad moves and serious blunders. The most common misconception in automation is thinking that manual testing is the same as automated testing. And this thinking accounts for most of the blunders in system level test automation. Dorothy Graham takes us on a tour of these blunders, including: the Stable-Application Myth (you can’t start automating until the application is stable), Inside-the-Box Thinking (...

Many organizations today are adopting DevOps to accelerate software delivery. However, once they have invested significant time and money optimizing most parts of the software delivery process, testing often holds them back from achieving the desired results. Why? Because software testing is still dominated by yesterday’s tools and processes—which don’t meet the needs of today’s accelerated development processes. How can you ensure that you and your team help the organization achieve its objectives? Wayne Ariola says that the key is continuous testing—and Wayne doesn’t just mean test...

Think of manually executed test scripts—like pulling a wagon without wheels. Eventually the wagon will make it to the final destination, but the journey itself will be long and painful. Many people think test scripts are outdated because of the long, painful process of writing and running them. Andrea Fox says that their analytics team shared this way of thinking. To make things worse, the team also was dealing with defects being constantly introduced because the restrictive scripts were not catching the issues. Change was vital to provide more efficiency in manual testing, as well...

Modern software products tend to have a rich UI that supports many user workflows, all of which need to be covered in testing. Agile organizations quickly discover that manual end-to-end testing neither supports their velocity nor provides respectable regression coverage. A common progression is to move from fully manual testing to record/replay, then to Selenium IDE style tests, then to automation based on Selenium WebDriver, perhaps with a BDD overlay. Daria Mehra has practiced this approach and shares her experience. She compares the Selenium style of automation to an...

Today, three things are undeniable facts of business—projects are becoming more agile, teams are learning to function well remotely, and the tester’s role is evolving. Mike Hrycyk believes that testers in agile teams face daunting challenges and often struggle to keep up with the pace of new feature development while performing all the needed regression testing activities. Mike offers a strategy as an alternate path: the creation of a regression testing team to augment feature teams, one that works in parallel, handling the testing regression cycle while the feature teams—...

Sabermetrics turned the baseball world upside down by challenging decades-old measures of individual performance and their perceived linkage to team success. After cementing their legacy as the Lovable Losers for 108 years, the Chicago Cubs were able to leverage a data-driven approach to finally win a World Series. A high-school football coach, devoted to statistical analysis, has won three state championships—by never punting. Formula 1 racing teams collect staggering amounts of telemetry data from their race cars for the purpose of eking out seconds during the course of a race....

Automation is a powerful tool to help testing but too often it is used to replicate existing manual tests. This leads organizations to spend large amounts of time and money constantly updating flaky automated tests and test teams to suffer frustration from having to focus on activities that are not truly testing. This cost and frustration can be avoided by using automation as a tool to assist testing—not to replace tests. Jeffrey Martin shares some real-world examples of using automation to supplement testing by leveraging its true value—the replication and repetition of tasks...

Stories help us learn. They can be fun or scary, exciting or relaxing. People worldwide tell and listen to stories, and we access them through books, film, TV, and IT. But the direct experience of face-to-face storytelling is still a powerful experience. When Isabel Evans was young, there was a program on the radio called Listen with Mother. For fifteen minutes, mothers and children across the country would sit and listen to a story. Join Isabel and become your inner child. Bring your testing parent and listen to her stories. In fifteen-minute sections, Isabel recounts stories drawn from...

Even seemingly simple software systems can be a dense forest of intersecting logical pathways which may leave you wondering if your testing was robust enough. Traditional test cases are flawed since they only execute the pathways the tester considered at the time the test case was written, and they will execute the same way—every time and without variation. Jon Fetrow shows how, using model-based testing, you can create a map of your software forest and answer the question “Did you test enough?” Jon discusses the use of models to catch defects in the requirements and design phase...