fix: prevent SubtestReport from overwriting parent test result

[jasonwbarnett]

Problem

When using pytest subtests (pytest>=9.0 built-in or the pytest-subtests plugin), SubtestReport objects share the same nodeid as the parent test. The collector's pytest_runtest_logreport hook processes these reports via update_test_result(), which stores results in self.in_flight[nodeid]. Each subtest report overwrites the previous one (last-write-wins), and the parent test's own call-phase report arrives with outcome="passed" (because the subtest context manager swallows exceptions), overwriting any subtest failures.

Result: Tests with failing subtests are reported as "passed" in the JSON output and Buildkite Test Engine — a silent false positive.

Reproduction

https://github.com/altana-ai/bktec-subtest-repro — 14 test permutations, reproducible in 3 commands:

git clone https://github.com/altana-ai/bktec-subtest-repro.git && cd bktec-subtest-repro
uv sync
uv run pytest test_permutations.py --json=results.json -v
python3 -c "import json; [print(f'{e[\"name\"]:45s} result={e[\"result\"]}') for e in json.load(open('results.json'))]"

10 out of 10 tests with subtest failures are silently reported as "passed".

Why "ignore subtest reports" (option 1 from #93) doesn't work

In pytest 9.0+, the parent test's own call-phase TestReport arrives with outcome="passed" even when subtests fail. The __exit__ method on the subtest context manager returns True, swallowing exceptions. While pytest 9 does mutate the outcome to "failed" via pytest_report_teststatus, that mutation happens inside the terminal reporter's pytest_runtest_logreport — the collector's hook runs first and sees "passed".

If you just ignore SubtestReport objects, the collector only sees the parent's "passed" report → same broken result.

Solution

Three targeted changes to pytest_runtest_logreport (~30 lines of new logic):

1. Detect SubtestReport objects via duck-typing (hasattr(report.context, 'msg')) — works for both pytest>=9.0 built-in and the pytest-subtests plugin without importing either.

2. Propagate subtest failures to the parent's in-flight entry. Ignore passing/skipped subtests so they don't overwrite a previous failure.

3. Guard against the parent test's "passed" call-phase report overwriting a subtest-induced failure.

The fix produces one JSON entry per parent test with the correct "result": "failed" status and the subtest's failure reason. The scope and name fields contain the real pytest nodeid, so bktec retries, muting, and test splitting continue to work without any changes.

Compatibility

pytest version	subtests source	Works?
>=9.0	Built-in subtests fixture	Yes
<9.0	pytest-subtests plugin	Yes
Any	No subtests used	Zero behavior change — _is_subtest_report() returns False immediately

Validated: 10/10 broken tests fixed, 0 regressions

After applying the fix and re-running the same 14 permutations from the reproduction repo:

Test	Before (broken)	After (patched)	Failure Reason
test_all_subtests_pass	passed	passed	n/a
test_first_fails_rest_pass	passed	failed	first subtest fails
test_last_fails_rest_pass	passed	failed	last subtest fails
test_middle_fails	passed	failed	middle subtest fails
test_all_subtests_fail	passed	failed	C fails
test_single_subtest_fails	passed	failed	single subtest fails
test_code_outside_subtests_passes	passed	failed	B fails
test_code_outside_subtests_fails	failed	failed	n/a (already correct)
test_no_subtests_passes/fails	correct	correct	n/a (no regression)
test_subtests_with_kwargs	passed	failed	assert 0 > 0
test_subtests_msg_and_kwargs	passed	failed	assert -1 > 0
test_many_subtests_one_failure	passed	failed	iteration 7 fails
test_subtest_no_msg	passed	failed	unnamed subtest fails

Test plan

• 18 new unit tests covering: detection, failure propagation, parent guard, teardown override, finalize cleanup, JSON output, and no-subtest regression
• Integration test (subprocess pytest invocation, auto-skipped on pytest<9.0)
• uv run pytest — all 85 existing tests pass, 5 integration tests skipped (need pytest>=9.0)
• uv run pylint src/ — 10.00/10
• Validated against 14-permutation reproduction harness
• Validated on Python 3.9 + pytest 8.4.2 + pytest-subtests 0.15.0 — same 10/10 fixes, 0 regressions. The duck-typing detection works identically for the pytest-subtests plugin's SubTestReport (capital T) and pytest 9.0+'s built-in SubtestReport
• CI passes across Python 3.9, 3.10, 3.11, 3.12, 3.13 (pytest + pylint)

Fixes #93

[jasonwbarnett]

I forgot to mention we are running this fix in production already.

[niceking]

Hey @jasonwbarnett thanks for this! I am not a python expert but I had a look and the changes look sound (they only add new behaviour related to subtests) and there are tests and you've verified that it works on production. I'll just get another review from the team as a sanity check due to my aforementioned lack of python knowledge 😅