Software Testing at Scale – Its like counting stars in the universe!

by September 15, 2017

Systems and Integration Complexities Make Software Testing Even More Challenging

Testing software and systems across different architectures and configurations such as operating systems, browsers, and hardware is particularly challenging and overwhelming for testers.  Software testing at scale presents a problem due to the infinite number of combinations of inputs in a system, and integration points between other systems. Exploring every configuration, integration, and hardware state can lead to more tests than stars in the universe which is not feasible for execution. Simply, using traditional software testing methodologies is just not possible to test every combination or path using physical hardware. The question then becomes, “How do you test software effectively on different systems?”

Employing Combinatorial Software Testing Techniques

For many, Pairwise Testing or All-Pairs Testing provides a way to test all possible discrete combinations of parameters, for each pair of input parameters to a system.  ISTQB defines All-Pairs Testing (or Pairwise Testing) as: A black-box test design technique in which test cases are designed to execute all possible discrete combinations of each pair of input parameters.

According to Wikipedia “one of the main strengths of this combinatorial technique is that it enables a significant reduction in the number of test cases without compromising functional coverage.”

Driving Testing Effectiveness Using the All-Pairs Software Testing Method

All-Pairs testing therefore generates a subset of combinations to satisfy all factors and provides an intermediate level of coverage which should give the confidence needed when testing a system, however its not as big a job as running all combinations or paths. You should pair every test value at least once. All-Pairs testing is very useful in configuration testing for example as it can be used to generate a sensible subset of test combinations.

In this market report produced by Bloor Research, Philip Howard says All-Pairs testing of a system or subsystem represents a reasonable cost-benefit compromise between often computationally infeasible higher-order combinatorial testing methods, and less exhaustive methods which fail to exercise all possible pairs of parameters.

We all know that effective software testing which meets tight schedules is needed and test design techniques should be very effective to achieve maximum test coverage and a high defect yield rate.

CA has teamed up with Paul Gerrard, champion of the New Model Testing, to produce a series of videos to help users to understand the use of modeling techniques for software testing. Watch this video below to see how he uses Pairwise software testing technique in configuration testing and also considers other examples where All-Pairs testing could be used effectively.

If you would like to find out more about optimizing test cases try CA Agile Requirements Designer and download the free 30-day trial.