-
Notifications
You must be signed in to change notification settings - Fork 0
Add streaming encode support and generators #2
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: infra-fixes
Are you sure you want to change the base?
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull request overview
This PR adds comprehensive streaming encode support to zcbor, enabling CBOR encoding via callback-based writing instead of requiring a pre-allocated output buffer. This is particularly valuable for resource-constrained embedded systems with limited RAM.
Changes:
- Core streaming encode infrastructure with callback-based write operations and indefinite-length container support
- New generator option
--stream-encode-functionsto emit streaming-specific entrypoints and provider structs - Iterator-based APIs for repeated fields and chunk-based APIs for indefinite-length text/byte strings
- State management enhancements to support streaming write callbacks and provider contexts
- Comprehensive test coverage across test1_suit, test2_simple, test3_corner_cases, and test4_senml with streaming variants
- Documentation updates in README.md and ARCHITECTURE.md
Reviewed changes
Copilot reviewed 22 out of 22 changed files in this pull request and generated 16 comments.
Show a summary per file
| File | Description |
|---|---|
| zcbor/zcbor.py | Core code generation logic: added --stream-encode-functions option, streaming provider struct generation, indefinite-length container logic, and streaming-aware function name generation |
| src/zcbor_encode.c | Streaming write macros (ZCBOR_WRITE_BYTE/BYTES), indefinite-length encode helpers, iterator-based multi-encode, chunk-based string encoding APIs |
| src/zcbor_common.c | State array layout changes to accommodate larger constant state structure with streaming fields |
| include/zcbor_encode.h | New streaming API declarations: state initialization, iterator/chunk callbacks, provider getters/setters |
| include/zcbor_decode.h | State macro updates for new constant state size calculations |
| include/zcbor_common.h | Extended constant state structure with streaming fields, alignment macros, new error codes, ZCBOR_CONST_STATE_SLOTS calculation |
| src/zcbor_print.c | Added error strings for new ZCBOR_ERR_STREAM_WRITE_FAILED and ZCBOR_ERR_STREAM_READ_FAILED codes |
| tests/encode/test{1,2,3,4}_*/src/main.c | Streaming test implementations with stream_write callbacks and provider structs |
| tests/encode/test{1,2,3,4}_*/CMakeLists.txt | Build system support for STREAMING flag and --stream-encode-functions option |
| tests/encode/test{1,2,3,4}_*/testcase.yaml | New test variants for streaming and streaming+canonical modes |
| tests/unit/test1_unit_tests/src/main.c | Error string tests updated for new error codes |
| README.md | Documentation of streaming encode entrypoints and usage |
| ARCHITECTURE.md | Documentation of streaming code generation in CodeRenderer |
Comments suppressed due to low confidence (1)
zcbor/zcbor.py:1422
- This method requires 1 positional argument, whereas overridden CddlXcoder.var_name may be called with 3. This call correctly calls the base method, but does not match the signature of the overriding method.
def var_name(self):
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
2a3a06b to
3260e80
Compare
88ac9e1 to
110661a
Compare
43b39b3 to
fabf9ef
Compare
4d511a3 to
e94cdd6
Compare
Add an encode mode that writes CBOR bytes directly to a caller-provided callback instead of an intermediate output buffer. This enables low-RAM producers (e.g. UART streaming) to emit CBOR without needing to precompute container sizes. The encode implementation routes all byte writes through helper macros that either append to the payload buffer (existing behavior) or call the streaming callback (streaming mode). When streaming is enabled, list/map containers are terminated using indefinite-length encoding (break 0xFF) to avoid backtracking and header resizing. Also add helper APIs for iterator-driven repeated-field encoding and chunked indefinite-length text string encoding, plus error reporting for streaming write/provider failures. Backward compatible: NULL callback retains existing buffered encoding. Signed-off-by: Jon Ringle <jringle@gridpoint.com>
Add optional generation of streaming encode entrypoints (cbor_stream_encode_<type>) and provider structs for repeated fields (iterator) and tstr chunking. Keep default generated output unchanged unless --stream-encode-functions is enabled, and keep repeated fields as fixed arrays unless --repeated-as-pointers is enabled. Document the optional streaming encode entrypoints and the --stream-encode-functions codegen option. Update the architecture overview to mention the generated provider structs and streaming entrypoints. Signed-off-by: Jon Ringle <jringle@gridpoint.com>
- Add STREAMING=1 variants for test1_suit, test2_simple, test3_corner_cases, and test4_senml. - Exercise streaming encode paths via stream_write helpers and cbor_stream_encode_* entrypoints. Signed-off-by: Jon Ringle <jringle@gridpoint.com>
Signed-off-by: Jon Ringle <jringle@gridpoint.com>
Introduce a small sample that demonstrates chunked streaming encode/decode for tstr and bstr. The program streams a long name and payload via chunk callbacks, then decodes using chunk-in callbacks. Signed-off-by: Jon Ringle <jringle@gridpoint.com>
Treat 'foo: bar' as a label in list context even if 'foo' is a known type, while keeping map behavior unchanged. Adds regression coverage in both the Python test suite and the corner case CDDL fixtures. Signed-off-by: Jon Ringle <jringle@gridpoint.com>
e94cdd6 to
fd5cd58
Compare
This adds a streaming encode path and the related code generation and tests.
--stream-encode-functionsReplaces #1.