Consulting

In chess, the word blunder means a very bad move by someone who should know better. Even though functional test automation has been around for a long time, people still make some very bad moves and serious blunders. The most common misconception in automation is thinking that manual testing is the same as automated testing. And this thinking accounts for most of the blunders in system level test automation.

You want to integrate skilled testing and development work. But how do you accomplish this without developers accidentally subverting the testing process or testers becoming an obstruction? Efficient, deep testing requires “critical distance” from the development process, commitment and planning to build a testable product, dedication to uncovering the truth, responsiveness among team members, and often a skill set that developers alone—or testers alone—do not ordinarily possess.

As testers, we can feel overwhelmed by the sheer volume of things that require our attention. We are pressured to meet the demands of a fast-paced development environment while grappling with the extreme complexities inherent in today’s software. How can we remember everything while prioritizing our work in a way that allows us to test thoroughly and with confidence? Kirk Lee shares how the proper use of checklists provides a lightweight yet powerful solution.

In today's fast-paced IT world, companies follow “best” testing trends and practices with the assumption that, by applying these methodologies, their product quality will improve. But that does not always happen. Why? Liana Gevorgyan questions and defines, in the language of metrics, exactly what is expected to be changed or improved, and how to implement these improvements. While your project is in progress, choosing the right metrics and looking at their trends help you understand what must change to improve your methodology.

Ever feel like your manager, development manager, product manager, product owner, or ____ (you fill in the blank) is not listening to you or your team? Are you struggling to make an impact with your messages? Are you “pushing a wet rope uphill” in championing product quality? Are you talking, but no one is listening? Mike Sowers shares practical examples of how to more effectively speak like a test manager and offers concrete advice based on his experiences in the technology, financial, transportation, and professional services sectors.

Have you found yourself forced to use outdated test tools because the cost to migrate was prohibitive? Have you abandoned or rewritten existing tests because it was easier (and cheaper) than migrating? With technology ever changing, most businesses struggle to keep up with producing high-quality products for the lowest price possible. And it is usually testers who suffer the most, as they are forced to use tools that are outdated, or no longer supported, because the company cannot afford the migration cost.

When we talk about product quality, test team efficiency, and productivity, we always talk numbers. However, very few companies implement metrics programs in a way that supports solid decision making. Many have tried and failed, leaving a negative impression of metrics. Kris Kosyk explains what metrics like Defect Removal Efficiency tell us and how it is impacted by Test Coverage and Defect Backlog Change Rate. Moving up a level, Kris explains how to use operational testing metrics to understand the development lifecycle process.

Today, most of us struggle with non-production environments. Either the test data is not right or consistent, the dependencies are mismanaged, or “They just aren't quite like production.”  Instead of striving for simpler environments, most organizations add test environments―pre-prod, UAT, stage, QAB, and so on. And they end up spending more and more time troubleshooting and maintaining environments rather than building and learning. It does not have to be this way.

In the world of agile, automation is king. When faced with testing multiple versions of software, either while migrating or supporting multiple versions in the field, many teams give up, convinced that automation cannot be achieved. Marquis Waller and Jeff Sikkink provide insights into how using tools—Jenkins, VMware API, Selenium, and others—can allow you to create a rich set of migration tests. They discuss the challenges they face maintaining migration testing for a large enterprise workflow product that runs on three different operating systems (AIX, Linux, Windows).

A great deal of confusion surrounds the concepts of release automation, continuous integration, continuous delivery, and continuous deployment. Even some industry experts are confused about the differences. How these concepts work progressively to achieve high quality software delivery is generating a lot of discussion and controversy. Bryan Linder defines the methodology, processes, and tools associated with release automation, as well as the differences between its maturity levels.

In our work we assess the quality of software to give well-grounded advice on the “go live” decision. We test software to prevent bad things from happening to users once the software is deployed. However, in some cases, the mere act of testing breaches safety barriers and can put companies on the spot, causing embarrassment, damage, or even death. The worst test ever to go bad—the Chernobyl meltdown which cost approximately 200,000 lives―was caused by a stress test executed in production.

Every day thousands of mobile apps are built, and many are released with poor quality. Dozens of new mobile devices become available every day. Immense pressure mounts on organizations to test mobile apps with shorter go-to-market cycles. Mobile app testing becomes overwhelming due to multiple platforms, varying OS versions, device manufacturers, screen resolutions, and more. Parimala Hariprasad presents an approach to designing test strategies for mobile apps. She addresses such questions as: What devices to test? How to select them? Can we use simulators/emulators?

Classical test process improvement models no longer fit in organizations adopting the newest development approaches. Instead, a more flexible approach is required today. Solutions like SOA, virtualization, web technology, cloud computing, mobile, and the application of social media have dramatically changed the IT landscape. In addition, we are innovating the way we develop, test, and manage. Many organizations are moving toward a combination of agile/scrum, context-driven testing, continuous integration and delivery, DevOps, and TestOps.

As companies scramble to adjust to the demands of an increasingly data-driven world, testers are told “go test data quality” without any guidance as to what that entails or how to go about it. The fact that the data is often a living, flowing ecosystem, rather than just a single object, requires the use of different strategies to gain meaningful insights. Shauna Ayers and Catherine Cruz Agosto guide you through the challenges of data quality and apply a structured approach to analyze, measure, test, and monitor living data sets, and gauge the business impact of data quality issues.

Traditional applications are tested through the GUI and through all exposed APIs. However, typical mobile app testing is only done through the front-end GUI. In addition, performance and security details are not readily available from the mobile device. Max Saperstone demonstrates some benefits of testing a native mobile application on a rooted device—one with privileged access control. Although Max does not describe how to root a device, he shares how to access back-end processes and test at this detailed level.

Some problems in software testing seem timeless. Other challenges—including SOA and cloud computing—arise due to the introduction of new technologies. Scott Tilley has led a three-year project at the Florida Institute of Technology to identify hard problems in software testing as voiced by leading practitioners in the field. The problems were identified through a series of workshops, interviews, and surveys.

Many development and test organizations must work within the confines of compressed release cycles, various agile methodologies, and cloud and mobile environments for their business applications. So, how can test organizations keep up with the pace of development and increase the quality of their applications under test? Clint Sprauve describes how service virtualization and network virtualization can help your team improve speed and increase quality.

Do you find yourself confused about the definition of mobile testing? Do you understand the challenges of mobile testing and where to start? Is this your first mobile testing project? Joe Larizza and Eran Kinsbruner describe the techniques of mobile testing and the steps necessary to help testing teams transform to face these new challenges. Learn about test automation, testing tools, new methodologies—DevOps, DevTest, Shift Left and Right—and how to build a strategic mobile test road map to increase your market awareness and avoid common pitfalls affecting mobile testing teams.