Efficiently Displaying Test Results for Multi-Dimensional Test Matrices

Efficiently Displaying Test Results for Multi-Dimensional Test Matrices

3 July 2024 Stephan Petzl Leave a comment QA

When managing a complex software product, testing across multiple dimensions such as operating systems, databases, and reporting engines can become a daunting task. Each intersection of these parameters represents a unique test case that needs to be executed and documented. This article will guide you through a practical approach to display these results efficiently, ensuring comprehensive coverage while maintaining manageability.

Understanding the Challenge

Imagine you have 11 or 12 dimensions for your test matrix. Each dimension, like the OS or the DB, contains several variables. The total number of test cases is the product of all these variables, leading to potentially millions of tests. Running and reporting these tests manually is not feasible, so a strategic approach is necessary.

Solution Overview

To tackle this problem, consider using a pairwise coverage approach or combinatorial test design. These methods help in selecting a manageable set of test combinations that ensure thorough coverage without the need to execute every possible test case.

Step-by-Step Guide

1. Identify Test Inputs

Begin by identifying all the variables within each dimension of your test matrix. For example, your OS dimension might include Windows, Mac, and Linux.

2. Create Pairwise Tests

Using a tool like Hexawise, create a set of pairwise tests. This method ensures that every possible pair of test inputs is tested at least once. For instance, if you have 559,872 possible test cases, pairwise testing might reduce this to just 22 tests while still achieving significant coverage.

3. Analyze Coverage

Utilize coverage analysis tools to understand the extent of your test coverage. After executing a subset of tests, you can determine the percentage of potential pairs that have been tested. This helps in identifying coverage gaps and prioritizing further tests if needed.

4. Report Results

Summarize the results in a clear and concise manner. Using a spreadsheet can be very effective. Organize the data by dimensions and highlight the pass/fail status of each test. This makes it easy to spot patterns and identify problematic areas.

Practical Example

Consider the following simplified test matrix:

Test No.    OS        DB      RepEngine  Mode     Result     Notes
0001        Windows   MySQL   Velocity   Easy     PASS       
0002        Windows   MySQL   Velocity   Advanced FAIL       Schema failure
0003        Windows   MySQL   Crystal    Easy     PASS       
0004        Windows   MySQL   Crystal    Advanced FAIL       Schema failure

By exporting this data to a CSV and using a spreadsheet, you can sort and analyze the results efficiently, identifying patterns and potential issues.

Advanced Testing Techniques

If you require more thorough coverage, you can adjust your testing strategy to include combinations of three or more variables. While this increases the number of tests, it also ensures a higher level of coverage and can uncover more complex issues.


Managing a multi-dimensional test matrix can be challenging, but with the right approach, you can achieve comprehensive coverage without excessive testing. Tools like Hexawise and pairwise testing methods are invaluable in this process.

For those looking for a no-code solution to streamline their testing process, consider using Repeato. Repeato is a no-code test automation tool for iOS and Android that allows you to create, run, and maintain automated tests quickly and efficiently. Its computer vision and AI-based approach make it particularly fast and easy to use, ensuring high-quality assurance for your apps.

For more detailed guides and resources, visit our documentation page.

Like this article? there’s more where that came from!