~/qa-guides/pairwise-testing
>_ Pairwise Testing: Examples and Test Cases
Build and audit pairwise test sets with three worked models, valid-combination constraints, coverage calculations, and 15 test cases.
Short answer
Pairwise testing selects test configurations so that every required pair of values from two different parameters appears together in at least one configuration. It is also called all-pairs or 2-way combinatorial testing.
Define the parameters, representative values, and valid combinations, then audit the selected rows and give each an expected result. The coverage claim applies to that model; it does not establish that every full combination or workflow was tested.
This guide uses a preview, an export operation, and a constrained compatibility matrix to show selection, coverage calculations, and 15 executable test cases. Their business rules are example fixtures to adapt to your requirements.
What pairwise coverage means
A parameter, also called a factor, is an input or configuration dimension. Its values, sometimes called levels, are the alternatives included in the model. A configuration assigns one value to every parameter.
A pair identifies both the parameters and their selected values. Language English with theme dark is a different target from channel push with theme dark, even though both mention dark.
For three binary parameters, there are three parameter pairs. Each parameter pair has four value combinations, giving 3 x 4 = 12 pair targets. A complete configuration covers one target for each parameter pair, so it covers three targets in this model.
The NIST introduction to covering arrays describes this property as 2-coverage: inspect any two columns and every required value combination appears. A covering array may repeat pairs across rows; one occurrence is enough for the coverage criterion.
For a model with constraints, a required pair must be extendable to at least one valid complete configuration. Testing whether the two values violate an obvious pair-level rule is insufficient when constraints involve additional parameters.
For a fixed model, pair coverage equals distinct covered required pairs divided by all required pairs. Record the domains and constraints with the percentage.
Pairwise testing versus equivalence partitioning
Equivalence partitioning groups inputs by expected handling and selects representatives from those groups. Those representatives can become values in a pairwise model.
Covering every value separately does not cover their interactions. With binary parameters A, B, and C, configurations 000 and 111 exercise both values of every parameter. They cover only 6 of the 12 pair targets: each parameter pair sees 00 and 11, while 01 and 10 remain absent.
Pairwise testing adds an interaction criterion. It requires each pair of selected values, subject to the model's constraints. The distinction matters when one browser setting works independently and one theme works independently, but their combination fails.
Both techniques belong to black-box test design. Neither criterion measures code coverage, and neither chooses the expected business result automatically.
When a pairwise model is useful
Consider the technique when several independently selectable options create more combinations than the team can reasonably execute for the intended test scope. Examples include report options, configuration flags, supported client environments, and import modes.
For environment selection, the mobile device testing checklist explains the broader device matrix. Pair coverage can organize selected dimensions, while the matrix still needs representative supported versions and important real devices.
Keep mandatory regression cases and business-critical combinations even if they repeat pairs already covered. Their inclusion can be justified by a known defect, unusual cost, or another risk rather than by adding a new pair.
How to build and audit a pairwise test set
1. Scope the operation and its parameters
Choose one operation to exercise in every row and define its starting data, permissions, and observable checkpoint. Configuration coverage collected during login is not evidence for an export on the same device.
List independently selectable factors and keep derived values separate. For example, a named time zone determines its offset for a given instant; treating the two as freely selectable creates impossible rows. Record constants such as account role so the claim's scope remains clear.
2. Choose concrete representatives
Use requirements and equivalence classes to select values, including exact supported versions where relevant. Record excluded domains: two ordinary source records do not represent every large dataset or unusual character.
Add boundary value analysis for ordered limits. Those tests exercise the calculation or validation of a selected value rather than replacing the interaction inventory.
3. Declare compatibility constraints
Record the requirement behind each excluded configuration before generation. Include multi-parameter dependencies; checking pairs for obvious conflicts alone can admit an invalid complete row.
NIST's Practical Combinatorial Testing discusses constraints, generation, and expected-result problems. The third example below shows the lost targets that can result from generating without constraints and then deleting invalid rows.
4. Audit the selected rows
Enumerate the required targets for each parameter pair, retaining parameter names in every target. For constrained models, derive them from valid complete configurations.
Check every selected row against the domains and constraints, project it onto every parameter pair, and compare the distinct projections with the required inventory. Preserve the model, rows, and audit output together. Repeating a pair in several rows does not create a new coverage item.
5. Execute the specified operation
Turn each row into setup, an action, and assertions for visible output, saved data, and side effects. Reset relevant state between independent runs.
Capture the applied settings: a UI that silently substitutes a default has not exercised the requested row. Record which rows reached the operation checkpoint; the blocked-row example below shows how to separate that evidence from planned coverage and verdicts.
6. Re-audit changes to the set
Recalculate after adding a value, changing a constraint, deleting a row, or substituting an environment. Check expected results when a change affects the output contract.
Before removing a seemingly redundant row, audit the remaining union of pairs and review any separate regression reason for keeping that row.
7. Add coverage for other interaction risks
Keep targeted three-or-more-factor combinations when a requirement or known defect calls for them. Use decision table testing to define business outcomes for complete condition combinations, and state transition testing for behavior depending on event history.
With only two parameters, pairwise coverage requires their full cross-product for the selected values; there is no reduction relative to full combinations in that model.
Worked example 1: Three binary preview settings
Assume a notification preview with three independent parameters:
- Channel: email or push.
- Language: en or de.
- Theme: light or dark.
All eight combinations are valid. The preview reports the selected channel, shows label Ready for en or Bereit for de, and uses the specified palette. Light has background #ffffff and text #111111; dark has background #111111 and text #ffffff. Rendering a preview does not queue or send a notification.
Keep message data and all other settings fixed. These are illustrative requirements for a fictional feature, not a claim about a particular notification provider.
Construct four rows
Start with all four channel-and-language combinations, then assign themes so that each channel and each language appears with both themes:
- P1: channel email, language en, theme light.
- P2: channel email, language de, theme dark.
- P3: channel push, language en, theme dark.
- P4: channel push, language de, theme light.
The rows cover all four channel-language pairs directly. Inspect the other projections:
- Channel-theme: email-light in P1, email-dark in P2, push-dark in P3, and push-light in P4.
- Language-theme: en-light in P1, de-dark in P2, en-dark in P3, and de-light in P4.
All 12 targets appear. Each row contributes three distinct pair targets, and this particular set has no repeated pair targets: 4 x 3 = 12. Planned pair coverage is 12 / 12 = 100%.
The full configurations number 2 x 2 x 2 = 8; only four are selected. Full-combination coverage is 4 / 8 = 50%. Both percentages are correct because they measure different sets.
Show what complete pair coverage can miss
Suppose a defect exists only for channel push, language de, and theme dark together. None of P1-P4 uses that triple, so all four can pass while the defect remains.
Each pair from the missing triple is already present elsewhere: push-de in P4, push-dark in P3, and de-dark in P2. Covering them in separate configurations does not combine them into one execution.
Add that exact triple as a targeted regression when the requirement or risk calls for it. With the extra row, pair coverage remains 100% while full-combination coverage becomes 5 / 8 = 62.5%. Record the reason for the additional row rather than claiming it adds a missing pair.
Even for two-factor failures, detection depends on the chosen representatives, reaching the affected behavior, and checking its output. Pair coverage alone does not establish those conditions.
Worked example 2: Mixed values in an export operation
Assume an export feature with these independent parameters:
- Format: csv, json, or pdf.
- Delivery: download or email.
- Dataset: populated or empty.
- Locale: en or de.
All 3 x 2 x 2 x 2 = 24 configurations are valid. This is a separate fixture from any format or delivery example in another guide.
For populated data, the source contains exactly records R17 with amount 42 and R18 with amount 7, in that order. Empty means zero source records. Permissions, destination, service availability, and other fields are valid.
The example's output contract is:
- A completed operation creates exactly one export artifact containing only the selected dataset in source order.
- CSV uses header fields id and amount; an empty export contains the header without data rows. Amounts retain their canonical numeric values.
- JSON uses an array of records with id and numeric amount; an empty export contains
[]. - PDF contains the selected records, or an explicit empty-result message when there are none.
- The confirmation heading is Report for en and Bericht for de. PDF uses the same localized heading. CSV fields and JSON keys remain id and amount in both locales.
- Download returns the artifact through the download path and queues no email.
- Email queues exactly one message to the fixture recipient with the artifact attached, uses the localized heading as its subject, and initiates no browser download.
Use a message test double and inspect its recorded output. Reset artifacts and messages before each independent row so an earlier export cannot satisfy the next case.
Build six complete configurations
Use these selected rows:
- E1: format csv, delivery download, dataset populated, locale en.
- E2: format csv, delivery email, dataset empty, locale de.
- E3: format json, delivery download, dataset empty, locale de.
- E4: format json, delivery email, dataset populated, locale en.
- E5: format pdf, delivery download, dataset empty, locale en.
- E6: format pdf, delivery email, dataset populated, locale de.
Start by enumerating all six format-delivery combinations. For each format, assign both dataset values and both locales across its two rows. Then inspect the remaining parameter pairs; that last check is necessary because covering every format with both options does not by itself guarantee all other pairs.
Count and audit the 30 pair targets
There are six parameter pairs:
- Format-delivery:
3 x 2 = 6targets. - Format-dataset:
3 x 2 = 6targets. - Format-locale:
3 x 2 = 6targets. - Delivery-dataset:
2 x 2 = 4targets. - Delivery-locale:
2 x 2 = 4targets. - Dataset-locale:
2 x 2 = 4targets.
The total is 6 + 6 + 6 + 4 + 4 + 4 = 30. The six selected rows cover all 30 distinct targets. They produce 36 pair occurrences because each row projects onto six parameter pairs; repeated occurrences do not enlarge the denominator.
Inspect the smaller projections explicitly. Download occurs with populated in E1 and empty in E3/E5; email occurs with empty in E2 and populated in E4/E6. Both delivery methods occur with en and de. Dataset-locale includes populated-en in E1/E4, empty-de in E2/E3, empty-en in E5, and populated-de in E6.
Planned pair coverage is 30 / 30 = 100%. Full-combination coverage is 6 / 24 = 25%.
For this model, six is also a lower bound: the six distinct format-delivery pairs each require a row, and each row contains only one such pair. The six-row set reaches that bound. This proof applies to these domains and constraints; it is not a general promise about generator output.
Keep expected results separate from selection
Pair coverage does not tell you whether an empty CSV should contain a header or whether an emailed PDF should start a download. Those expectations come from the output contract.
For E2, inspect an empty CSV attachment, the German subject, and the absence of a browser download. For E6, inspect a PDF containing both source records, its German heading, and exactly one queued message. A successful job status without those checks provides weaker evidence than the case requires.
Worked example 3: A compatibility model with constraints
Assume a fictional reading application's release matrix has four parameters:
- OS: Windows or macOS.
- Browser: Chrome, Firefox, or Safari.
- Theme: light or dark.
- Network: online or offline.
Fix the application build and exact supported environment versions outside this model. Its support contract excludes Windows with Safari. It also excludes Windows with Firefox in offline mode. These are fixture-specific support decisions, not general statements about browser capabilities or known browser defects.
Prepare document D17, revision 5, with text Reference text in both the server fixture and the local document cache. For valid online rows, the reader displays D17 from the server and identifies it as Server copy. For valid offline rows, the already prepared reader displays the cached D17 and identifies it as Saved copy. Light and dark use the same palette rules as the preview example.
Keep the prepared cache and source revision constant. This model covers opening that document under the selected configuration; it does not cover installation, cache eviction, conflicting revisions, or an entire offline application lifecycle.
Derive feasible configurations and pairs
Without constraints, the full space contains 24 configurations and 30 pair targets, using the same parameter sizes as the export model.
Windows-Safari removes four complete configurations: two themes times two network modes. Windows-Firefox-offline removes another two: one for each theme. The exclusions do not overlap, leaving 24 - 4 - 2 = 18 valid complete configurations.
Only the OS-browser pair Windows-Safari becomes infeasible. Each other pair has a valid complete witness, giving these counts:
- OS-browser: 5 feasible targets.
- OS-theme: 4 feasible targets.
- OS-network: 4 feasible targets.
- Browser-theme: 6 feasible targets.
- Browser-network: 6 feasible targets.
- Theme-network: 4 feasible targets.
The denominator is 5 + 4 + 4 + 6 + 6 + 4 = 29 feasible pairs.
The three-value exclusion does not make Windows-Firefox, Windows-offline, or Firefox-offline individually infeasible. Windows-Firefox can run online; Windows-offline can use Chrome; Firefox-offline can run on macOS. Excluding those pairs would remove legitimate coverage obligations.
See why deleting invalid rows loses coverage
An unconstrained pairwise candidate could contain:
- U1: OS Windows, browser Chrome, theme light, network online.
- U2: OS macOS, browser Chrome, theme dark, network offline.
- U3: OS Windows, browser Firefox, theme dark, network offline.
- U4: OS macOS, browser Firefox, theme light, network online.
- U5: OS Windows, browser Safari, theme dark, network online.
- U6: OS macOS, browser Safari, theme light, network offline.
This candidate covers the 30 unconstrained pair targets, but U3 and U5 violate the support contract. Delete them and only U1, U2, U4, and U6 remain. They cover 21 of the 29 feasible pairs, or 72.4% rounded to one decimal place.
The eight missing targets are:
- OS Windows with browser Firefox.
- OS Windows with theme dark.
- OS Windows with network offline.
- Browser Firefox with theme dark.
- Browser Firefox with network offline.
- Browser Safari with theme dark.
- Browser Safari with network online.
- Theme dark with network online.
Every target above still has a valid complete witness. They cannot be removed from the denominator simply because the original rows that covered them were invalid.
Use a valid six-row set
Regenerate or repair the selection and audit the complete result:
- K1: OS Windows, browser Chrome, theme light, network offline.
- K2: OS macOS, browser Chrome, theme dark, network online.
- K3: OS Windows, browser Firefox, theme dark, network online.
- K4: OS macOS, browser Firefox, theme light, network offline.
- K5: OS macOS, browser Safari, theme light, network online.
- K6: OS macOS, browser Safari, theme dark, network offline.
All six configurations satisfy the constraints. Together they cover all 29 feasible pairs: planned coverage is 29 / 29 = 100%. Selected full-combination coverage within the valid space is 6 / 18 = 33.3%, rounded to one decimal place.
The rows themselves provide witnesses for every feasible pair. They also show why a constraint must apply to a complete row: K3 covers Windows-Firefox safely online, while K4 covers Firefox-offline on macOS.
Report a blocked row accurately
Suppose K3 cannot run because its required test environment is unavailable. The other five completed rows cover only 24 of the 29 feasible pairs, or 82.8%. Planned pair coverage remains 100%, but exercised pair coverage is 82.8% until the missing targets are exercised.
If K3 reaches the reading operation with the correct configuration and then fails its output assertion, all 29 pairs may have been exercised, with one failing result. Record that failure separately. A coverage percentage cannot turn an incorrect rendering into a pass.
15 pairwise test cases with expected results
Use the fixtures above and reset relevant state between independent rows. Cases 1-14 each use one selected configuration. Case 15 has two separately recorded Safari variants so that the complete collection exercises all six constrained rows.
Test case 1: Preview email in English with the light theme
- Preconditions: The preview fixture is ready and no notifications are queued.
- Input: P1: channel email, language en, theme light.
- Steps: Apply all three values, render the preview, and inspect its selected channel, label, palette, and notification output.
- Expected result: Channel is email and the label is Ready. Background is white and text is near-black as specified. No notification is queued or sent.
Test case 2: Preview email in German with the dark theme
- Preconditions: Reset the preview and notification output.
- Input: P2: channel email, language de, theme dark.
- Steps: Render the preview and verify that the requested language and theme were applied together.
- Expected result: Channel remains email, label is Bereit, background is the specified near-black, and text is white. No notification side effect occurs.
Test case 3: Preview push in English with the dark theme
- Preconditions: Reset the preview fixture.
- Input: P3: channel push, language en, theme dark.
- Steps: Render and inspect the channel, label, and palette after confirming all selections.
- Expected result: Channel is push, label is Ready, and the dark palette is applied. No push or email is queued.
Test case 4: Preview push in German with the light theme
- Preconditions: Reset the preview fixture.
- Input: P4: channel push, language de, theme light.
- Steps: Render the preview and inspect its output and notification records.
- Expected result: Channel is push, label is Bereit, and the light palette is applied. No notification is queued. Together with cases 1-3, the applied rows exercise all 12 preview pair targets.
Test case 5: Download populated CSV in English
- Preconditions: Prepare the populated R17/R18 dataset; clear export artifacts and message output.
- Input: E1: format csv, delivery download, dataset populated, locale en.
- Steps: Generate the export and inspect the downloaded file, confirmation heading, and message output.
- Expected result: Exactly one CSV artifact has header id and amount, followed by R17 with 42 and R18 with 7 in source order. Heading is Report. No email is queued.
Test case 6: Email empty CSV in German
- Preconditions: Prepare the empty dataset and clear artifacts and messages.
- Input: E2: format csv, delivery email, dataset empty, locale de.
- Steps: Generate the export, inspect the recorded message and attached file, and check browser download activity.
- Expected result: One message with subject Bericht has one CSV attachment containing the header and zero data rows. The confirmation heading is Bericht, and no browser download starts.
Test case 7: Download empty JSON in German
- Preconditions: Prepare the empty dataset and reset output.
- Input: E3: format json, delivery download, dataset empty, locale de.
- Steps: Generate and parse the downloaded artifact, then inspect confirmation and messages.
- Expected result: Exactly one JSON artifact contains an empty array. Heading is Bericht. No prior dataset appears and no email is queued.
Test case 8: Email populated JSON in English
- Preconditions: Prepare R17/R18 and clear prior artifacts and messages.
- Input: E4: format json, delivery email, dataset populated, locale en.
- Steps: Generate the export and parse the attachment from the recorded message.
- Expected result: One message with subject Report has one JSON attachment containing exactly R17 with numeric amount 42 and R18 with numeric amount 7, in source order. Keys remain id and amount. No browser download starts.
Test case 9: Download empty PDF in English
- Preconditions: Prepare the empty dataset and clear output.
- Input: E5: format pdf, delivery download, dataset empty, locale en.
- Steps: Generate and open the PDF, then inspect confirmation and message output.
- Expected result: Exactly one PDF has heading Report and an explicit empty-result message, with no stale data records. Confirmation is in English and no email is queued.
Test case 10: Email populated PDF in German
- Preconditions: Prepare R17/R18 and clear output.
- Input: E6: format pdf, delivery email, dataset populated, locale de.
- Steps: Generate the export, open the recorded attachment, and inspect message count and download activity.
- Expected result: One message with subject Bericht has one PDF attachment headed Bericht and containing R17 with 42 and R18 with 7 in source order. No browser download starts. Cases 5-10 collectively exercise all 30 export pair targets when their actual configurations match the selected rows.
Test case 11: Read the cached document on Windows Chrome in light mode
- Preconditions: Prepare D17 revision 5 in the reader cache and select the pinned Windows/Chrome environment.
- Input: K1: OS Windows, browser Chrome, theme light, network offline.
- Steps: Apply the theme and offline mode, open the prepared document, and inspect its identifier, revision, text, source indicator, and palette.
- Expected result: D17 revision 5 displays Reference text with indicator Saved copy and the specified light palette. No successful server response is required to supply this document.
Test case 12: Read the server document on macOS Chrome in dark mode
- Preconditions: Prepare matching server and cache fixtures in the pinned macOS/Chrome environment.
- Input: K2: OS macOS, browser Chrome, theme dark, network online.
- Steps: Open D17 and inspect the source used, content, and palette.
- Expected result: D17 revision 5 displays Reference text with indicator Server copy and the specified dark palette. The saved-copy indicator does not conceal an unintended offline path.
Test case 13: Read online on Windows Firefox in dark mode
- Preconditions: Prepare the pinned Windows/Firefox environment and server fixture.
- Input: K3: OS Windows, browser Firefox, theme dark, network online.
- Steps: Verify online mode, open D17, and inspect the rendered content and palette.
- Expected result: D17 revision 5 displays Reference text, Server copy, and the dark palette. The row is valid because the fixture excludes Windows/Firefox only with offline mode.
Test case 14: Read offline on macOS Firefox in light mode
- Preconditions: Prepare the pinned macOS/Firefox environment with D17 revision 5 cached.
- Input: K4: OS macOS, browser Firefox, theme light, network offline.
- Steps: Apply offline mode and open the prepared document in the reader.
- Expected result: D17 revision 5 displays Reference text, Saved copy, and the light palette. The offline Firefox pair remains a required feasible target because this complete row satisfies the constraints.
Test case 15: Exercise both Safari variants on macOS
- Preconditions: Prepare the pinned macOS/Safari environment with matching server and cache fixtures. Reset reader state between variants while retaining the declared cache fixture.
- Inputs: K5 uses light theme with online mode; K6 uses dark theme with offline mode. Both use macOS and Safari.
- Steps: Execute and record each variant independently, applying its theme and network mode before opening D17.
- Expected result: K5 displays D17 revision 5, Reference text, Server copy, and the light palette. K6 displays the same document and revision with Saved copy and the dark palette. Cases 11-15 cover all 29 feasible compatibility pairs when both variants and all preceding rows are exercised.
FAQ
Yes, for a small model. Enumerate the required pairs, select valid complete configurations, and compare their projections with that inventory. Use an independent audit because an apparently balanced list can omit pairs. Larger models generally benefit from generation and automated coverage verification.
Separate unsupported configurations from inputs whose rejection is part of the required behavior. Keep the positive compatibility model feasible, and design negative cases for supported rejection paths with clear expected results. Combining several invalid values can hide later validation behind the first error.
No. An orthogonal array balances occurrences of the relevant value combinations equally. A covering array requires their presence and can repeat some more often than others. Pairwise coverage is the 2-way coverage goal; the particular array structure must be checked against the model, especially when values have different counts or constraints.