Skip to content

Fix the hooks ordering issue that caused failed test to be reported as passed#45

Merged
nprizal merged 1 commit intomainfrom
pie-3532-failed-test-reported-as-passed
Mar 27, 2025
Merged

Fix the hooks ordering issue that caused failed test to be reported as passed#45
nprizal merged 1 commit intomainfrom
pie-3532-failed-test-reported-as-passed

Conversation

@nprizal
Copy link
Contributor

@nprizal nprizal commented Mar 27, 2025

#44 introduced a bug where failed tests are reported as passed. This happened because I moved the execution collection (i.e. adding it to payload) from after call to after teardown lifecycle. Apparently, the pytest_runtest_logreport hook that called after teardown return a report object that contains the outcome of teardown process, not the outcome of the test call. Therefore, when the test call result is failed, it is reported as passed because the teardown process was successful.

This PR address the issue by fixing the hook ordering to be:

pytest_runtest_logstart:
  create test data and store it to `in_flight` object
pytest_runtest_setup:
  -
pytest_runtest_call:
  -
pytest_runtest_logreport (only after call):
  collect the test result and update the test data
pytest_runtest_teardown:
  collect the execution tags
  add test to payload

@nprizal nprizal requested a review from zhming0 March 27, 2025 06:55
@nprizal nprizal self-assigned this Mar 27, 2025
@nprizal nprizal force-pushed the pie-3532-failed-test-reported-as-passed branch from 8f654ae to 3d5529f Compare March 27, 2025 21:57
@nprizal nprizal changed the title Fix the execution handling order that caused failed test to be reported as passed Fix the hook handling order that caused failed test to be reported as passed Mar 27, 2025
@nprizal nprizal force-pushed the pie-3532-failed-test-reported-as-passed branch from 3d5529f to dad36ab Compare March 27, 2025 22:05
Copy link
Contributor

@zhming0 zhming0 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting! Looks good 👍🏿 . But since we cannot easily write a test for this, it's probably better to add more code comments for future travellers.

@nprizal nprizal force-pushed the pie-3532-failed-test-reported-as-passed branch from dad36ab to e758659 Compare March 27, 2025 22:50
@nprizal nprizal force-pushed the pie-3532-failed-test-reported-as-passed branch from e758659 to 30dde3a Compare March 27, 2025 22:53
@nprizal nprizal merged commit 3f06c46 into main Mar 27, 2025
11 checks passed
@nprizal nprizal deleted the pie-3532-failed-test-reported-as-passed branch March 27, 2025 23:03
@nprizal nprizal mentioned this pull request Mar 28, 2025
@nprizal nprizal changed the title Fix the hook handling order that caused failed test to be reported as passed Fix the hooks ordering issue that caused failed test to be reported as passed Mar 28, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants