Questions the comparison answers
- Which tests fail now but passed in the baseline?
- Which failures were fixed?
- Which tests disappeared from the current report?
- Which tests changed status or became materially slower?
- Do duplicate identities or inconsistent suite totals make the report unreliable?
How tests are matched
By default, TestSignal matches tests by classname + test name. This is more stable than comparing XML lines because report generators can reorder suites, split files, or rewrite summary attributes.
| Change | Meaning |
|---|---|
| New failure or error | The test was absent or passing in the baseline and now fails. |
| Fixed | A previously failing or errored test now passes. |
| Missing | A baseline test is not present in the current run. |
| Duration regression | The test exceeds both the percentage and absolute-duration thresholds. |
| Persistent failure | The problem remains in both runs and is not misreported as new. |
A repeatable comparison workflow
- Use the last trusted run as baseline.
A main-branch or release report makes the result easier to interpret. - Add the current run.
You can select multiple XML shards on either side. - Review regression priorities.
New failures and worse statuses are placed ahead of routine changes. - Adjust slow-test thresholds only when needed.
The default requires at least 25% and 100ms of additional duration. - Export evidence.
Share filtered HTML, CSV, or Markdown in a pull request or issue.
What the result does not claim
Two runs cannot prove that a test is flaky, and aggregate duration is not a substitute for a benchmark. The tool reports observed status and duration changes without making unsupported diagnoses.