Skip to content

tests: fuzzing improvements #469

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
qdeslandes wants to merge 1 commit into
base: main
Choose a base branch
from
+30 −6
Open

tests: fuzzing improvements #469

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/ci.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -122,7 +122,7 @@ jobs:
run: ctest --test-dir $GITHUB_WORKSPACE/build -L e2e --verbose

- name: Run fuzzing tests
run: ctest --test-dir $GITHUB_WORKSPACE/build -L fuzzing --verbose
run: ctest --test-dir $GITHUB_WORKSPACE/build -R fuzzing.parser.60 --verbose
Copy link

@github-actions github-actions bot Mar 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude: Minor consistency note — the other CI test steps use label-based selection (-L unit, -L e2e), while this now uses regex (-R). This works in practice, but the unescaped . characters are regex wildcards. For robustness you could escape them or keep using -L:

Suggested change
run: ctest --test-dir $GITHUB_WORKSPACE/build -R fuzzing.parser.60 --verbose
run: ctest --test-dir $GITHUB_WORKSPACE/build -R 'fuzzing\.parser\.60$' --verbose

- name: Upload fuzzer findings
uses: actions/upload-artifact@v4
if: always()
Expand Down
2 changes: 1 addition & 1 deletion CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,7 @@ if (NOT ${NO_DOCS})
endif ()

if (NOT ${NO_TESTS})
set(CMAKE_CTEST_ARGUMENTS "--output-on-failure")
set(CMAKE_CTEST_ARGUMENTS "--output-on-failure;--label-exclude;fuzzing")
enable_testing()
add_subdirectory(tests)
endif ()
Expand Down
3 changes: 2 additions & 1 deletion doc/developers/build.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -103,7 +103,8 @@ A full configuration (without any part disabled) will provide the following targ

- ``core``, ``bpfilter``, ``libbpfilter``, ``bfcli``: the ``bpfilter`` binaries.
- ``test_bin``: build the binaries needed to run the tests (below).
- ``test``: run all the tests. This command will run ``unit``, ``check``, ``e2e``, ``fuzzing``, and ``integration`` targets. See :doc:`tests` for more information.
- ``test``: run all the tests. This command will run ``unit``, ``check``, ``e2e``, and ``integration`` targets. See :doc:`tests` for more information.
- ``fuzzing``: fuzz the CLI parser indefinitely. Use ``fuzzing_quick`` for a quick 60 seconds fuzzing.
- ``check``: run ``clang-tidy`` and ``clang-format`` against the source files.
- ``benchmarks``: run the benchmarks on ``bpfilter``.

Expand Down
29 changes: 26 additions & 3 deletions tests/fuzz/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ file(GLOB_RECURSE libbpfilter_fuzz_srcs
${CMAKE_SOURCE_DIR}/src/libbpfilter/*.h ${CMAKE_SOURCE_DIR}/src/libbpfilter/*.c
)

add_executable(fuzz_parser
add_executable(fuzz_parser EXCLUDE_FROM_ALL
${CMAKE_CURRENT_SOURCE_DIR}/fuzz_parser.c
${CMAKE_SOURCE_DIR}/src/bfcli/helper.c
${CMAKE_SOURCE_DIR}/src/bfcli/ruleset.c
Expand Down Expand Up @@ -57,7 +57,7 @@ add_dependencies(fuzz_parser bfcli_parser bfcli_lexer)
file(MAKE_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}/corpus)
file(MAKE_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}/findings)

add_test(NAME "fuzzing.parser"
add_test(NAME "fuzzing.parser.60"
COMMAND $<TARGET_FILE:fuzz_parser>
-artifact_prefix=${CMAKE_CURRENT_BINARY_DIR}/findings/
-max_total_time=60
Expand All @@ -70,13 +70,36 @@ add_test(NAME "fuzzing.parser"
${CMAKE_CURRENT_SOURCE_DIR}/corpus
)

set_tests_properties("fuzzing.parser.60" PROPERTIES
LABELS "fuzzing"
ENVIRONMENT "ROOT_DIR=${CMAKE_SOURCE_DIR};BUILD_DIR=${CMAKE_BINARY_DIR};GEN_INC_DIR=${lib_gen_inc_dir}/include"
)

add_custom_target(fuzzing_quick
COMMAND ${CMAKE_CTEST_COMMAND} --verbose -R "fuzzing.parser.60"
DEPENDS fuzz_parser
COMMENT "Running fuzzing tests"
)

add_test(NAME "fuzzing.parser"
COMMAND $<TARGET_FILE:fuzz_parser>
-artifact_prefix=${CMAKE_CURRENT_BINARY_DIR}/findings/
-print_final_stats=1
-only_ascii=1
-close_fd_mask=3
-dict=${CMAKE_CURRENT_SOURCE_DIR}/keywords.dict
-max_len=16384
${CMAKE_CURRENT_BINARY_DIR}/corpus
${CMAKE_CURRENT_SOURCE_DIR}/corpus
)

set_tests_properties("fuzzing.parser" PROPERTIES
LABELS "fuzzing"
ENVIRONMENT "ROOT_DIR=${CMAKE_SOURCE_DIR};BUILD_DIR=${CMAKE_BINARY_DIR};GEN_INC_DIR=${lib_gen_inc_dir}/include"
)

add_custom_target(fuzzing
COMMAND ${CMAKE_CTEST_COMMAND} --output-on-failure -L fuzzing
COMMAND ${CMAKE_CTEST_COMMAND} --verbose -R "fuzzing.parser"
Copy link

@github-actions github-actions bot Mar 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude: ctest -R performs regex substring matching. The pattern fuzzing.parser matches any test name containing that substring, which includes both fuzzing.parser and fuzzing.parser.60 (. is also a regex wildcard matching any character). This means make fuzzing will run the 60-second test and the indefinite test, rather than just the indefinite fuzzer.

Consider anchoring the regex:

Suggested change
COMMAND ${CMAKE_CTEST_COMMAND} --verbose -R "fuzzing.parser"
COMMAND ${CMAKE_CTEST_COMMAND} --verbose -R "^fuzzing\\.parser$$"

($$ is needed because CMake interprets a single $; the shell receives $ which anchors end-of-string.)

Alternatively, rename the indefinite test to something that isn't a prefix of the other, e.g. fuzzing.parser.infinite.

DEPENDS fuzz_parser
COMMENT "Running fuzzing tests"
)
Loading