test_runner: support expecting a test-case to fail · nodejs/node@81a9668

Triggered via pull request December 21, 2025 19:18

JakobJingleheimer

synchronize #60669

JakobJingleheimer:test_runner/feat/should-fail

Status Cancelled

Total duration 1h 0m 51s

Artifacts 1

test-shared.yml

on: pull_request

Build slim tarball

1m 29s

Matrix: build

Annotations

9 errors

aarch64-darwin: with shared libraries

Process completed with exit code 2.

aarch64-darwin: with shared libraries: test/test-runner/test-output-output.mjs#L0

--- stderr --- node:internal/modules/run_main:107 triggerUncaughtException( ^ AssertionError [ERR_ASSERTION]: Expected values to be strictly equal: + actual - expected + 'TAP version 13\n' + + '# Subtest: sync expect fail (method)\n' + + 'ok 1 - sync expect fail (method) # EXPECTED FAILURE\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: sync expect fail (options)\n' + + 'ok 2 - sync expect fail (options) # EXPECTED FAILURE\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: async expect fail (method)\n' + + 'ok 3 - async expect fail (method) # EXPECTED FAILURE\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: async expect fail (options)\n' + + 'ok 4 - async expect fail (options) # EXPECTED FAILURE\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: sync todo with expect fail\n' + + 'ok 5 - sync todo with expect fail # TODO\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: sync skip expect fail\n' + + 'ok 6 - sync skip expect fail # SKIP\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: sync pass todo\n' + + 'ok 7 - sync pass todo # TODO\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: sync pass todo with message\n' + + 'ok 8 - sync pass todo with message # TODO this is a passing todo\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: sync fail todo\n' + + 'not ok 9 - sync fail todo # TODO\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + " location: '/test/fixtures/test-runner/output/output.js:(LINE):1'\n" + + " failureType: 'testCodeFailure'\n" + + " error: 'thrown from sync fail todo'\n" + + " code: 'ERR_TEST_FAILURE'\n" + + ' stack: |-\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' ...\n' + + '# Subtest: sync fail todo with message\n' + + 'not ok 10 - sync fail todo with message # TODO this is a failing todo\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + " location: '/test/fixtures/test-runner/output/output.js:(LINE):1'\n" + + " failureType: 'testCodeFailure'\n" + + " error: 'thrown from sync fail todo with message'\n" + + " code: 'ERR_TEST_FAILURE'\n" + + ' stack: |-\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' ...\n' + + '# Subtest: sync skip pass\n' + + 'ok 11 - sync skip pass # SKIP\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: sync skip pass with message\n' + + 'ok 12 - sync skip pass with message # SKIP this is skipped\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: sync pass\n' + + 'ok 13 - sync pass\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# this test should pass\n' + + '# Subtest: sync throw fail\n' + + 'not ok 14 - sync throw fail\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + " location: '/test/fixtures/test-runner/output/output.js:(LINE):1'\n" + + " failureType: 'testCodeFailure'\n" + + " error: 'thrown from sync throw fail'\n" + + " code: 'ERR_TEST_FAILURE'\n" + + ' stack: |-\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' ...\n' + + '# Subtest: async skip pass\n' + + 'ok 15 - async skip pass # SKIP\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: async pass\n' + + 'ok 16 - async pass\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n

aarch64-linux: with shared libraries

Process completed with exit code 2.

aarch64-linux: with shared libraries: test/test-runner/test-output-output.mjs#L0

--- stderr --- node:internal/modules/run_main:107 triggerUncaughtException( ^ AssertionError [ERR_ASSERTION]: Expected values to be strictly equal: + actual - expected + 'TAP version 13\n' + + '# Subtest: sync expect fail (method)\n' + + 'ok 1 - sync expect fail (method) # EXPECTED FAILURE\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: sync expect fail (options)\n' + + 'ok 2 - sync expect fail (options) # EXPECTED FAILURE\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: async expect fail (method)\n' + + 'ok 3 - async expect fail (method) # EXPECTED FAILURE\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: async expect fail (options)\n' + + 'ok 4 - async expect fail (options) # EXPECTED FAILURE\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: sync todo with expect fail\n' + + 'ok 5 - sync todo with expect fail # TODO\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: sync skip expect fail\n' + + 'ok 6 - sync skip expect fail # SKIP\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: sync pass todo\n' + + 'ok 7 - sync pass todo # TODO\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: sync pass todo with message\n' + + 'ok 8 - sync pass todo with message # TODO this is a passing todo\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: sync fail todo\n' + + 'not ok 9 - sync fail todo # TODO\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + " location: '/test/fixtures/test-runner/output/output.js:(LINE):1'\n" + + " failureType: 'testCodeFailure'\n" + + " error: 'thrown from sync fail todo'\n" + + " code: 'ERR_TEST_FAILURE'\n" + + ' stack: |-\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' ...\n' + + '# Subtest: sync fail todo with message\n' + + 'not ok 10 - sync fail todo with message # TODO this is a failing todo\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + " location: '/test/fixtures/test-runner/output/output.js:(LINE):1'\n" + + " failureType: 'testCodeFailure'\n" + + " error: 'thrown from sync fail todo with message'\n" + + " code: 'ERR_TEST_FAILURE'\n" + + ' stack: |-\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' ...\n' + + '# Subtest: sync skip pass\n' + + 'ok 11 - sync skip pass # SKIP\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: sync skip pass with message\n' + + 'ok 12 - sync skip pass with message # SKIP this is skipped\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: sync pass\n' + + 'ok 13 - sync pass\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# this test should pass\n' + + '# Subtest: sync throw fail\n' + + 'not ok 14 - sync throw fail\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + " location: '/test/fixtures/test-runner/output/output.js:(LINE):1'\n" + + " failureType: 'testCodeFailure'\n" + + " error: 'thrown from sync throw fail'\n" + + " code: 'ERR_TEST_FAILURE'\n" + + ' stack: |-\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' ...\n' + + '# Subtest: async skip pass\n' + + 'ok 15 - async skip pass # SKIP\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: async pass\n' + + 'ok 16 - async pass\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n

x86_64-linux: with shared libraries

Process completed with exit code 2.

x86_64-linux: with shared libraries: test/test-runner/test-output-output.mjs#L0

--- stderr --- node:internal/modules/run_main:107 triggerUncaughtException( ^ AssertionError [ERR_ASSERTION]: Expected values to be strictly equal: + actual - expected + 'TAP version 13\n' + + '# Subtest: sync expect fail (method)\n' + + 'ok 1 - sync expect fail (method) # EXPECTED FAILURE\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: sync expect fail (options)\n' + + 'ok 2 - sync expect fail (options) # EXPECTED FAILURE\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: async expect fail (method)\n' + + 'ok 3 - async expect fail (method) # EXPECTED FAILURE\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: async expect fail (options)\n' + + 'ok 4 - async expect fail (options) # EXPECTED FAILURE\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: sync todo with expect fail\n' + + 'ok 5 - sync todo with expect fail # TODO\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: sync skip expect fail\n' + + 'ok 6 - sync skip expect fail # SKIP\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: sync pass todo\n' + + 'ok 7 - sync pass todo # TODO\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: sync pass todo with message\n' + + 'ok 8 - sync pass todo with message # TODO this is a passing todo\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: sync fail todo\n' + + 'not ok 9 - sync fail todo # TODO\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + " location: '/test/fixtures/test-runner/output/output.js:(LINE):1'\n" + + " failureType: 'testCodeFailure'\n" + + " error: 'thrown from sync fail todo'\n" + + " code: 'ERR_TEST_FAILURE'\n" + + ' stack: |-\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' ...\n' + + '# Subtest: sync fail todo with message\n' + + 'not ok 10 - sync fail todo with message # TODO this is a failing todo\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + " location: '/test/fixtures/test-runner/output/output.js:(LINE):1'\n" + + " failureType: 'testCodeFailure'\n" + + " error: 'thrown from sync fail todo with message'\n" + + " code: 'ERR_TEST_FAILURE'\n" + + ' stack: |-\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' ...\n' + + '# Subtest: sync skip pass\n' + + 'ok 11 - sync skip pass # SKIP\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: sync skip pass with message\n' + + 'ok 12 - sync skip pass with message # SKIP this is skipped\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: sync pass\n' + + 'ok 13 - sync pass\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# this test should pass\n' + + '# Subtest: sync throw fail\n' + + 'not ok 14 - sync throw fail\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + " location: '/test/fixtures/test-runner/output/output.js:(LINE):1'\n" + + " failureType: 'testCodeFailure'\n" + + " error: 'thrown from sync throw fail'\n" + + " code: 'ERR_TEST_FAILURE'\n" + + ' stack: |-\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' *\n' + + ' ...\n' + + '# Subtest: async skip pass\n' + + 'ok 15 - async skip pass # SKIP\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: async pass\n' + + 'ok 16 - async pass\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n

Test Shared libraries

Canceling since a higher priority waiting request for Test Shared libraries-test_runner/feat/should-fail exists

x86_64-darwin: with shared libraries

Canceling since a higher priority waiting request for Test Shared libraries-test_runner/feat/should-fail exists

x86_64-darwin: with shared libraries

The operation was canceled.