Skip to content

Conversation

@ringlej
Copy link
Collaborator

@ringlej ringlej commented Jan 19, 2026

This adds a streaming encode path and the related code generation and tests.

  • core encode supports streaming writes via callback, with streaming-safe list/map termination and error reporting
  • new helper APIs for iterator-based repeated fields and chunked indefinite-length text string encoding
  • optional generator support for streaming encode entrypoints and provider structs via --stream-encode-functions
  • documentation updates for streaming encode API and generated provider structs
  • encode test coverage for streaming across test1_suit, test2_simple, test3_corner_cases, and test4_senml

Replaces #1.

Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds comprehensive streaming encode support to zcbor, enabling CBOR encoding via callback-based writing instead of requiring a pre-allocated output buffer. This is particularly valuable for resource-constrained embedded systems with limited RAM.

Changes:

  • Core streaming encode infrastructure with callback-based write operations and indefinite-length container support
  • New generator option --stream-encode-functions to emit streaming-specific entrypoints and provider structs
  • Iterator-based APIs for repeated fields and chunk-based APIs for indefinite-length text/byte strings
  • State management enhancements to support streaming write callbacks and provider contexts
  • Comprehensive test coverage across test1_suit, test2_simple, test3_corner_cases, and test4_senml with streaming variants
  • Documentation updates in README.md and ARCHITECTURE.md

Reviewed changes

Copilot reviewed 22 out of 22 changed files in this pull request and generated 16 comments.

Show a summary per file
File Description
zcbor/zcbor.py Core code generation logic: added --stream-encode-functions option, streaming provider struct generation, indefinite-length container logic, and streaming-aware function name generation
src/zcbor_encode.c Streaming write macros (ZCBOR_WRITE_BYTE/BYTES), indefinite-length encode helpers, iterator-based multi-encode, chunk-based string encoding APIs
src/zcbor_common.c State array layout changes to accommodate larger constant state structure with streaming fields
include/zcbor_encode.h New streaming API declarations: state initialization, iterator/chunk callbacks, provider getters/setters
include/zcbor_decode.h State macro updates for new constant state size calculations
include/zcbor_common.h Extended constant state structure with streaming fields, alignment macros, new error codes, ZCBOR_CONST_STATE_SLOTS calculation
src/zcbor_print.c Added error strings for new ZCBOR_ERR_STREAM_WRITE_FAILED and ZCBOR_ERR_STREAM_READ_FAILED codes
tests/encode/test{1,2,3,4}_*/src/main.c Streaming test implementations with stream_write callbacks and provider structs
tests/encode/test{1,2,3,4}_*/CMakeLists.txt Build system support for STREAMING flag and --stream-encode-functions option
tests/encode/test{1,2,3,4}_*/testcase.yaml New test variants for streaming and streaming+canonical modes
tests/unit/test1_unit_tests/src/main.c Error string tests updated for new error codes
README.md Documentation of streaming encode entrypoints and usage
ARCHITECTURE.md Documentation of streaming code generation in CodeRenderer
Comments suppressed due to low confidence (1)

zcbor/zcbor.py:1422

  • This method requires 1 positional argument, whereas overridden CddlXcoder.var_name may be called with 3. This call correctly calls the base method, but does not match the signature of the overriding method.
    def var_name(self):

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@ringlej ringlej force-pushed the push-streaming branch 3 times, most recently from 88ac9e1 to 110661a Compare January 26, 2026 13:40
@ringlej ringlej closed this Jan 27, 2026
@ringlej ringlej reopened this Jan 27, 2026
@ringlej ringlej force-pushed the push-streaming branch 2 times, most recently from 43b39b3 to fabf9ef Compare January 27, 2026 15:22
Add an encode mode that writes CBOR bytes directly to a caller-provided callback
instead of an intermediate output buffer. This enables low-RAM producers (e.g.
UART streaming) to emit CBOR without needing to precompute container sizes.

The encode implementation routes all byte writes through helper macros that
either append to the payload buffer (existing behavior) or call the streaming
callback (streaming mode). When streaming is enabled, list/map containers are
terminated using indefinite-length encoding (break 0xFF) to avoid backtracking
and header resizing.

Also add helper APIs for iterator-driven repeated-field encoding and chunked
indefinite-length text string encoding, plus error reporting for streaming
write/provider failures.

Backward compatible: NULL callback retains existing buffered encoding.

Signed-off-by: Jon Ringle <jringle@gridpoint.com>
Add optional generation of streaming encode entrypoints (cbor_stream_encode_<type>)
and provider structs for repeated fields (iterator) and tstr chunking.

Keep default generated output unchanged unless --stream-encode-functions is enabled,
and keep repeated fields as fixed arrays unless --repeated-as-pointers is enabled.

Document the optional streaming encode entrypoints and the --stream-encode-functions
codegen option. Update the architecture overview to mention the generated provider
structs and streaming entrypoints.

Signed-off-by: Jon Ringle <jringle@gridpoint.com>
- Add STREAMING=1 variants for test1_suit, test2_simple, test3_corner_cases, and test4_senml.
- Exercise streaming encode paths via stream_write helpers and cbor_stream_encode_* entrypoints.

Signed-off-by: Jon Ringle <jringle@gridpoint.com>
Signed-off-by: Jon Ringle <jringle@gridpoint.com>
Introduce a small sample that demonstrates chunked streaming encode/decode
for tstr and bstr.

The program streams a long name and payload via chunk callbacks, then
decodes using chunk-in callbacks.

Signed-off-by: Jon Ringle <jringle@gridpoint.com>
Treat 'foo: bar' as a label in list context even if 'foo' is a known type, while keeping map behavior unchanged.

Adds regression coverage in both the Python test suite and the corner case CDDL fixtures.

Signed-off-by: Jon Ringle <jringle@gridpoint.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant