Test Management

In spite of many great testing “how-to” books, people involved with system and software testing—testers, requirements engineers, system/software architects, system and software engineers, technical leaders, managers, and customers—continue to make many different types of testing-related mistakes. Think of these commonly-occurring human errors as a system of software testing pitfalls. And when projects fall into these pitfalls, testing is less effective at uncovering defects, people are less productive when testing, and project morale is damaged.

You want to integrate skilled testing and development work. But how do you accomplish this without developers accidentally subverting the testing process or testers becoming an obstruction? Efficient, deep testing requires “critical distance” from the development process, commitment and planning to build a testable product, dedication to uncovering the truth, responsiveness among team members, and often a skill set that developers alone—or testers alone—do not ordinarily possess.

As testers, we can feel overwhelmed by the sheer volume of things that require our attention. We are pressured to meet the demands of a fast-paced development environment while grappling with the extreme complexities inherent in today’s software. How can we remember everything while prioritizing our work in a way that allows us to test thoroughly and with confidence? Kirk Lee shares how the proper use of checklists provides a lightweight yet powerful solution.

Many projects implicitly use some kind of risk-based approach for prioritizing testing activities. However, critical testing decisions should be based on a product risk assessment process using key business drivers as its foundation. For agile projects, this assessment should be both thorough and lightweight. Erik van Veenendaal discusses PRISMA (PRoduct RISk MAnagement), a highly practical method for performing systematic product risk assessments. Learn how to employ PRISMA techniques in agile projects using Risk Poker.

Most traditional teams do testing at least twice—once during development as new features are created and again during release candidate testing right before release. As a system grows, regression testing takes more and more time, making tight releases impossible—or at least risky—and adding to the burden of maintaining automated tests. Matt Heusser suggests that adopting continuous integration (with its continuous testing) and continuous delivery (with its associated production monitoring) can eliminate the need for classic regression testing.

Have you found yourself forced to use outdated test tools because the cost to migrate was prohibitive? Have you abandoned or rewritten existing tests because it was easier (and cheaper) than migrating? With technology ever changing, most businesses struggle to keep up with producing high-quality products for the lowest price possible. And it is usually testers who suffer the most, as they are forced to use tools that are outdated, or no longer supported, because the company cannot afford the migration cost.

Today, most of us struggle with non-production environments. Either the test data is not right or consistent, the dependencies are mismanaged, or “They just aren't quite like production.”  Instead of striving for simpler environments, most organizations add test environments―pre-prod, UAT, stage, QAB, and so on. And they end up spending more and more time troubleshooting and maintaining environments rather than building and learning. It does not have to be this way.

In the world of agile, automation is king. When faced with testing multiple versions of software, either while migrating or supporting multiple versions in the field, many teams give up, convinced that automation cannot be achieved. Marquis Waller and Jeff Sikkink provide insights into how using tools—Jenkins, VMware API, Selenium, and others—can allow you to create a rich set of migration tests. They discuss the challenges they face maintaining migration testing for a large enterprise workflow product that runs on three different operating systems (AIX, Linux, Windows).

A great deal of confusion surrounds the concepts of release automation, continuous integration, continuous delivery, and continuous deployment. Even some industry experts are confused about the differences. How these concepts work progressively to achieve high quality software delivery is generating a lot of discussion and controversy. Bryan Linder defines the methodology, processes, and tools associated with release automation, as well as the differences between its maturity levels.

In our work we assess the quality of software to give well-grounded advice on the “go live” decision. We test software to prevent bad things from happening to users once the software is deployed. However, in some cases, the mere act of testing breaches safety barriers and can put companies on the spot, causing embarrassment, damage, or even death. The worst test ever to go bad—the Chernobyl meltdown which cost approximately 200,000 lives―was caused by a stress test executed in production.

Every day thousands of mobile apps are built, and many are released with poor quality. Dozens of new mobile devices become available every day. Immense pressure mounts on organizations to test mobile apps with shorter go-to-market cycles. Mobile app testing becomes overwhelming due to multiple platforms, varying OS versions, device manufacturers, screen resolutions, and more. Parimala Hariprasad presents an approach to designing test strategies for mobile apps. She addresses such questions as: What devices to test? How to select them? Can we use simulators/emulators?

As software increasingly becomes the face of the business, defects can lead to embarrassment, financial loss, and even business failure. Nevertheless, in response to today's demand for speed and “continuous everything,” the software delivery conveyer belt keeps moving faster and faster. It's foolhardy to expect that speeding up an already-troubled implementation process will achieve the desired results. Wayne Ariola shares why and how to evolve from automated to continuous testing and discusses the methods to help you do so.

Suddenly, application performance is important to your business, and you have been given the budget to improve it. You’re in a hurry because customers are complaining or because you expect jumps in transaction volume and your application needs to scale quickly. Do you know where to start? Join Ryan Riehle as he shares his experiences developing enterprise performance testing programs. Ryan covers the key techniques and heuristics that lead to an effective performance improvement effort.

Thousands of times each day, network perimeter security defenses fail to recognize new and obfuscated attacks. Rather than attempting to build security firewalls, Declan O’Riordan asserts that project teams must design, code, and test security into applications―and that requires skills that are in short supply. As testers, we need to recognize which security tests we can perform and which require delegation to experts. Let’s stop our passive acceptance of designs that are weak on security and instead conduct analysis of the security features before we plan the system testing.

Have you ever stood in line at midnight to buy the latest release of a product? Have you worked on a product that created such delight in customers that they camped out overnight to be the first to buy it? Though this level of customer devotion is rare, it is possible to create everyday products that your customers will love. In the past, the designers and developers have received the lion’s share of the credit, but the role of quality teams is just as important in creating this level of success.

Do you find yourself confused about the definition of mobile testing? Do you understand the challenges of mobile testing and where to start? Is this your first mobile testing project? Joe Larizza and Eran Kinsbruner describe the techniques of mobile testing and the steps necessary to help testing teams transform to face these new challenges. Learn about test automation, testing tools, new methodologies—DevOps, DevTest, Shift Left and Right—and how to build a strategic mobile test road map to increase your market awareness and avoid common pitfalls affecting mobile testing teams.