Consulting

Beyond the publicity and hype about the Internet of Things (IoT), a new term is emerging—the Internet of Everything (IoE). What are people talking about? Should you be interested? What does it mean to testers and development? Paul Gerrard shares his perspective on the scale, variety, ubiquity, complexity, and challenge of this technological wave that many believe will dominate our industry into the next decade. Right now, the IoT/IoE is very confusing. Although standards are emerging, many commercial applications are bleeding edge, speculative, or exploratory. While security and privacy concerns dominate the discussion today, significant functional, user experience, integration, and complexity challenges await us. The IoE brings broader societal risks that must be addressed by organizations, individuals, and their governments. Paul presents a seven-layer architectural model to help you make sense of it all. Take back a set of key questions you need to ask and recommendations for formulating your test strategy for the Internet of Everything.

It’s time to change the conversation about testing and quality―from bugs and poor requirements to products and solutions. Instead, Janet Gregory says testers and managers working in agile organizations need to learn more about the business, the market, and the customers. With that knowledge, we have the information necessary to identify key problems and formulate solutions that reduce uncertainty and increase product value. Janet challenges testers and test managers to lead the way to change how organizations talk about quality. She explores ways to measure value and focus on the positive rather than a more negative cost-of-quality perspective. In an agile organization testing is not a job or a phase or a department; testing is a crosscutting concern and a skill which is vital to every contributor on a project and to a product. Join Janet to learn ways to rethink your priorities and ways to optimize your test practices.

To improve as a skilled tester, you don't necessarily have to attend a class or read a book. James and Jon Bach show how you can grow by recording and then dissecting as few as five minutes of your test process. Using a Testopsy, you build your skills of observation, narration, and test framing. And if you do it with a colleague or as a group, it stimulates discussion on test design and test strategy. James and Jon demonstrate a live Testopsy to underscore these points and give you specific guidelines for conducting your own. Testing is a performance—not an artifact. Much like a medical examiner narrates his autopsies into a tape recorder, you can look very carefully at what you actually do and identify your own heuristics. By putting that process into descriptive, evocative words, you can discover surprising depths in each act of testing you perform.

In this changing world of software development processes, where does software testing stand? How can we adapt our testing methodologies to keep pace with rapid development processes? Agile testing is a paradigm shift and is one of the challenges that teams face when transitioning from a traditional testing model to agile. Harini Gupta explores different agile methodologies that are used in large software development programs across Microsoft to promote software agility. Harini shares her own experiences with agile software testing methodologies including continuous integration, continuous delivery, and the ability to enable or disable features by configuration settings. Harini talks about validating features with unit/integration testing in production,flighting (aka A/B testing), metrics analysis, and how to validate the health of features on a live site by synthetic transactions and availability testing. Join Harini for this eye-opening session on how agile teams at Microsoft test today.

The Internet of Things (IoT) is here, and it brings new testing problems, complexity, and, in Jason Arbon’s opinion, a bit more fun. Tiny computers are embedded in everything from light bulbs and shoes to baby diapers. The tiny devices are attached to cattle, humans, cars, and trash cans. The number of IoT devices is set to explode into the tens of billions in the next few years. Consumers expect all these devices to simply work—and work well—with each other. It is easy to test a web app with a mouse and keyboard, or to test a mobile app with a few swipes, but IoT demands that testers determine how to vary the temperature, generate good and bad golf swings, and verify that this device plays well with all those other devices. The topics of security and privacy are more important than ever in this IoT world. Join Jason for a glimpse into the emerging and fun world of IoT testing.

Many development and test organizations must work within the confines of compressed release cycles, various agile methodologies, and cloud and mobile environments for their business applications. So, how can test organizations keep up with the pace of development and increase the quality of their applications under test? Clint Sprauve and Todd DeCapua describe how service virtualization and network virtualization can help your team improve speed and increase quality. Learn how to use service virtualization to simulate third-party or internal web services to remove wait times and reduce the need for high-cost testing infrastructures. Take back techniques to incorporate network virtualization into the testing environment to simulate real-world network conditions. Learn from Clint and Todd how the combination of service and network virtualization allows teams to implement a robust and consistent continuous testing strategy to reduce defects in production applications.

As innovative businesses reduce their time-to-market and shorten release cycles, the need for continuous delivery methods becomes inevitable. Testers are constantly asked to rapidly and reliably deliver comprehensive test results in ever-decreasing or continuous cycles. Meanwhile, testing is butting heads with hard budget, tight time, and limited resource constraints, causing frustration for everyone involved. How can we turn these pains into gains? First, Ingo Phillip shows how to measure test coverage in terms of risk coverage on both the requirement and test case level—and why this matters. Second, Ingo shows how novel test design techniques help us test smarter, not harder, while keeping the entire test case portfolio straightforward. Third, he demonstrates how this effort results in a robust synthetic test data approach with repeatable, stable test execution. Finally, Ingo explains how to enable continuous delivery through continuous testing without drowning in test cases.

Many functional tests of dialogs and web pages are done through the user interface, and the expected results of each test are clear. However, Hans Buwalda recognizes that some applications are “special” and pose challenges for automated testing. For example, oil exploration applications frequently produce complex 3D graphics output that is not easy to describe in a test case, let alone write the automation code. A game may implement randomized behavior, and a test must know where that behavior is in that game and where it needs to go next.

Models exist all around us. To the untrained, models appear rather esoteric. In fact, our brain is a fantastic modeling engine. How do you think we can move without collision, use language, reason, and make sense of our world? Paul Gerrard explains how models underpin the common test methods. We testers can use models at every level of our work—to simplify the testing problem, to scope the testing tasks, to communicate with stakeholders and peers, to inform test design/selection decisions, and to measure coverage or progress in our work.

We’ve all been there. We work incredibly hard to develop a feature and design tests based on written requirements. We build a detailed test plan that aligns the tests with the software and the documented business needs. And when we put the tests to the software, it all falls apart because the requirements were changed without informing everyone. But help is at hand. Enter behavior-driven development (BDD) and Cucumber—a tool for running automated acceptance tests and facilitating BDD. Mary Thorn explores the nuances of Cucumber and shows you how to implement BDD and agile acceptance testing. By fostering collaboration for implementing active requirements via a common language and format, Cucumber bridges the communication gap between business stakeholders and implementation teams. If you experience developers not coding to requirements, testers not getting requirements updates, or customers who feel out of the loop and don’t get what they ask for, Mary has answers for you.

Organizing testing activities under a Test Center of Excellence (TCoE) is a common practice, now seen as a logical step in the growth toward a more mature testing organization. Although it appears straightforward, the efforts of many companies soon plateau and excellence proves elusive. What can be done to reach a level of service delivery that customers will value? Wim Demey describes how the ITIL model can help in the journey to improve your TCoE. Based on a lifecycle model, ITIL is a widely accepted service management framework, consisting of five stages—service strategy, service design, service transition, service operation, and continual service improvement. From the ITIL perspective, you can consider test activities as services that need to be aligned with the needs of the business. Using an example, Wim describes the ITIL lifecycle and explains how its concepts are applied to the TCoE. Discover how one Test Center of Excellence fully integrated ITIL into its existing test organization.

At one time or another, every tester hears the dreaded question, “Why didn’t you guys catch these bugs?” We all have some standard responses, and they are most likely true). But what can we learn about our testing when we look beyond the easy answers? Pamela Gillaspie proposes that the key to improving your testing is determining the areas where bugs are slipping past your defenses. For her team, the practice is a lot like basketball. If you group the bugs into zones, you can devise a strategy to cover those zones more effectively. Some zones need a different testing approach than you’ve used; others might reveal a need for closer communication. Join Pamela as she shares her experience as defensive coordinator, addressing the developers’ playbook (What kinds of recurring problems do we see?), trick plays (The user is doing what?), and penalties (That wasn’t in the requirements!).

Each day thousands of mobile apps are built, and many are released with poor quality. Pressure mounts exponentially on organizations to test mobile apps with shorter time-to-market cycles. Mobile app testing becomes overwhelming due to multiple platforms, varying OS versions, numerous device manufacturers, wide range of screen resolutions, and more. Parimala Hariprasad presents an approach to designing test strategies for mobile apps. She addresses such questions as: What devices should we test? How do we select them? Can we use simulators/emulators? How do we handle fragmentation challenges? Which platforms are good enough? Parimala shares her experiences and highlights how analytics and user reviews can facilitate the creation of an effective test strategy that evolves over time and balances tradeoffs between cost, quality, and time-to-market in the constantly changing mobile market. Key takeaways include learning about fragmentation, the shotgun approach, mobile personas, and use of analytics to fine-tune the test strategy.

Classical test process improvement models no longer fit in organizations adopting the newest development approaches. Instead, a more flexible approach is required today. Solutions like SOA, virtualization, web technology, cloud computing, mobile, and the application of social media have dramatically changed the IT landscape. In addition, we are innovating the way we develop, test, and manage. Many organizations are moving toward a combination of agile/scrum, context-driven testing, continuous integration and delivery, DevOps, and TestOps. Effective test automation has become a prerequisite for success. All of this requires a different way of improving testing, an adaptable way that responds to innovations in both technology and development. Martin shares a roadmap that enables you to translate the triggers and objectives for test improvement into actions that can be implemented immediately. Learn how to achieve continuous test improvement in any situation, and take away a practical set of guidelines to enable a quick start.

Finding major design issues late in development—and the resulting unpredictability at a project’s end—cause many projects to suffer. The use of graphical test planning (GTP) helps eliminate this unpredictability. GTP is a revolutionary test analysis method that uses behavior modeling to capture the system design, anticipate product bugs before coding, and develop test suites before code is ready. GTP is lightweight, yet covers boundary conditions, class equivalence, domain analysis, and combinations. David Bradley shows how to use your skills and experiences to identify and remove design issues from the beginning. Learn how to capture detailed technical information that elicits input and reviews from many sources, making testing a valuable and integral part of the whole project lifecycle. Discover how GTP provides the flexibility, efficiency, and agility to fit in with today’s development processes. Explore how you can use GTP to successfully test your next project—no matter how challenging.

With hundreds of mobiles, tablets, and wearables running different versions of iOS and Android, automated continuous testing is the only sane way to address mobile testing. Tom Chavez presents a review of automated testing tools for mobile apps, including tools that require writing tests in a programming language  and tools that record mobile app actions for replay requiring no programming at all. To complement and complete your device test pool, Tom discusses when to use your own devices versus a remote device pool—even thousands of miles away from your development team. Learn how mobile app developers and test teams have made the transition from manual testing to automation. Leave convinced and ready with the resources and steps to integrate continuous automated testing into your mobile development process. Don’t let a fatal bug sneak into your apps, annoying users or, even worse, causing them to move on to your competitor.M

Big Data, a term to describe the exponential growth and availability of data, has become increasingly more important to businesses due to large-scale and location-aware social media and mobile applications. These applications generate massive amounts of data, much of it in real-time. This drives the need for scalable, real-time platforms, which can process humongous data volumes and can derive real time analytics. Unfortunately, Big Data may contain bad data which causes organizations to make poor decisions. Even worse for testers is the fact that Big Data testing is challenging due to a[WU3] embraces complex technology stack, numerous data sources, real time events and streams, complex transformations, and more. Karen Pruitt and Sushmitha Geddam describe the critical testing of focal areas with[WU4]  in Big Data batching and real-time data processing. They describe frameworks that support Big Data testing and help frameworks that support Big Data testing which helps to strengthen data quality. Join this session to learn how to handle big data integration challenges and the skill sets you’ll need to be successful. Take away valuable insights for testing your Big Data implementations.te

In the world of software-as-a-service, just about anyone with a laptop and an Internet connection can spin up their very own cloud-based web service. Software startups, in particular, are often big on ideas but small on staff. This makes streamlining the traditional develop-test-integrate-deploy-monitor pipeline critically important. Melissa Benua says that an effective way to accomplish this is to reduce the number of different test suites that verify many of the same things for each stage. Melissa explains how teams can avoid this by authoring the right set of tests and using the right frameworks. Drawing on lessons learned in companies both large and small, Melissa shows how teams can drastically slash time spent developing automation, verifying builds for release, and monitoring code in production—without sacrificing availability or reliability.

When developing and testing a mobile app, do you wonder how well it will perform under heavy WiFi load conditions? What happens if the WiFi signal drops? Will the app switch cleanly between WiFi and cellular with no data loss? Les Honniball explains that Walt Disney Parks and Resorts Technology provides the applications and infrastructure that guests use to plan, book, explore, and enjoy their stay at Disney parks and resorts. Join Les for insights into how they designed tests, both manual and automated, to meet the unique set of challenges that comes with millions of pageviews per day. He explains their successful test strategies, analytics, user experience design, and lessons learned while testing Disney’s mobile app. Hear how Les and his team overcame the challenges of using mobile device emulators vs. the actual mobile device. Learn how Les and his team performed in-the-Park testing and how valuable those testing results were vs. testing in the office or lab.

DevOps methodologies require stronger cohesion between application development and operations teams. In the DevOps paradigm, performance engineering must mature beyond the usual measure-and-tune approach. Join Sundar Narayanan as he explores principles and techniques for proactive performance engineering. Sundar shares techniques for building predictive performance models from system designs parameterized with workload characteristics and expected resource demands.

Testing a PC-based game poses interesting challenges. With different OS platforms and many different GPUs, the number of combinations to test grows quickly. The large number of hardware configurations used by the players of EVE Online makes it impossible to test them all. This can lead to the situation where reproducing defects in the test lab is not possible. Björgvin Reynisson shares how CCP Games got fed up with the situation and decided to make a crowd-testing app that tests the game’s graphics engine. He describes the app, the framework for deploying the app, and how they handled the test results coming in from testers. Björgvin shares how the app has become both a development tool and an automated testing tool. Discover how this approach has helped identify hard-to-catch bugs, and discover some of the challenges of crowd testing. See how this project took an unexpected turn and led the team to change how the game content is distributed.tes

Mobile users, driven by their actions as consumers, have high expectations. They also expect to be able to use their own mobile devices for work. All this change brings uncertainty. Nearly half of enterprises have concerns with IT’s ability to keep up with the rapid pace of mobile. Success depends on your enterprise’s ability to fully embrace and leverage these changes for optimal user experiences, making functional and performance testing critical to your strategy. Ron Anderson shares practical guidance on how to create testing processes that keep the user top of mind, fueled by best practices within an enterprise mobile environment. He presents several use cases to serve as the backdrop, giving you practical insight into the methods presented. Leave with significantly increased clarity on how to understand and address user expectations, plan for functional testing mapped to user characteristics, and develop strategies to ensure responsiveness.