Add streaming encode support and generators

ARCHITECTURE.md

@@ -175,6 +175,8 @@ There are up to 4 files constructed:
 - The H file with all the struct definitions (the type file). If both decoding and encoding files are generated for the same CDDL, they can share the same type file.
 - An optional cmake file for building the generated code together with the zcbor C libraries.
 
+When `zcbor code` is invoked with `--stream-encode`, CodeRenderer also emits streaming encode entrypoints (`cbor_stream_encode_<type>`) and the associated provider structs used to supply repeated elements and chunked `tstr`/`bstr` values.
+
 CodeRenderer conducts some pruning and deduplication of the list of types and functions received from CodeGenerator.
 
 

README.md

@@ -39,6 +39,7 @@ There are samples in the [samples](samples) directory that demonstrate different
 
 1. The [hello_world sample](samples/hello_world/README.md) is a minimum examples of encoding and decoding using the C library.
 2. The [pet sample](samples/pet/README.md) shows a how to use the C library together with generated code, and how to use the script tool to do code generation and data conversion.
+3. The [streaming_chunks sample](samples/streaming_chunks/README.md) shows how to use streaming encode and decode entrypoints with chunk callbacks.
 
 The [tests](tests) also demonstrate how to use zcbor in different ways. The [encoding](tests/encode), [decoding](tests/decode), and [unit](tests/unit) tests run using [Zephyr](https://github.com/zephyrproject-rtos/zephyr) (the samples do not use Zephyr).
 
@@ -196,6 +197,34 @@ The tests require [Zephyr](https://github.com/zephyrproject-rtos/zephyr) (if you
 
 The generated C code is C++ compatible.
 
+Streaming encode/decode entrypoints
+-----------------------------------
+
+For low-RAM paths, `zcbor code` can optionally generate streaming encode and decode entrypoints:
+
+- Encode: `cbor_stream_encode_<type>(zcbor_stream_write_fn stream_write, void *stream_user_data, const <type> *input, const struct cbor_stream_io_<type> *prov, size_t *bytes_written_out)`
+- Decode: `cbor_stream_decode_<type>(const uint8_t *payload, size_t payload_len, <type> *result, const struct cbor_stream_io_<type> *prov, size_t *payload_len_out)`
+
+Enable them with `--stream-encode` and `--stream-decode`.
+
+Streaming encode entrypoints write CBOR via the supplied callback (no output buffer), and can stream:
+
+- the `zcbor_stream_write_fn` callback signature is:
+  `size_t (*)(void *user_data, const uint8_t *data, size_t len)` and should return
+  the number of bytes written (must equal `len` for success)
+- repeated fields via iterator callbacks in `struct cbor_stream_io_<type>`
+- `bstr`/`tstr` values via chunk callbacks (use `chunks_out_*` fields) that emit chunks into the encoder
+
+Streaming decode entrypoints let you consume `bstr`/`tstr` chunks without reassembly:
+
+- the `zcbor_stream_chunk_in` callback signature is:
+  `bool (*)(void *ctx, const uint8_t *data, size_t len)`
+- chunk callbacks live in `struct cbor_stream_io_<type>` (use `chunks_in_*` fields)
+- when chunk io is present, the generated decode logic calls
+  `zcbor_tstr_chunk_in()` / `zcbor_bstr_chunk_in()` instead of
+  decoding into a single contiguous buffer, so the corresponding `zcbor_string`
+  field is not populated and the callback is responsible for capturing data
+
 Build system
 ------------
 
@@ -436,7 +465,8 @@ zcbor code --help
 
 ```
 usage: zcbor code [-h] -c CDDL [--no-prelude] [-v]
-                  [--default-max-qty DEFAULT_MAX_QTY] [--output-c OUTPUT_C]
+                  [--default-max-qty DEFAULT_MAX_QTY] [--repeated-as-pointers]
+                  [--stream-encode] [--stream-decode] [--output-c OUTPUT_C]
                   [--output-h OUTPUT_H] [--output-h-types OUTPUT_H_TYPES]
                   [--copy-sources] [--output-cmake OUTPUT_CMAKE]
                   -t ENTRY_TYPES [ENTRY_TYPES ...] [-d] [-e] [--time-header]
@@ -477,6 +507,25 @@ options:
                         as sometimes the value is needed for internal
                         computations. If so, the script will raise an
                         exception.
+  --repeated-as-pointers
+                        Represent repeated fields (max_qty > 1) as pointer +
+                        count instead of embedding a fixed-size array in the
+                        generated types. This can significantly reduce the
+                        size of top-level unions/structs at the cost of
+                        requiring the caller to provide storage for decode,
+                        and a readable array for encode.
+  --stream-encode       Also generate streaming encode entrypoints
+                        (cbor_stream_encode_<type>) for each entry type. These
+                        use zcbor_new_encode_state_streaming() and are
+                        intended for low-RAM UART write paths. Streaming
+                        encode entrypoints support: - repeated fields via
+                        zcbor_multi_encode_iter_minmax() (iterator callback) -
+                        tstr/bstr fields via chunk callbacks
+  --stream-decode       Also generate streaming decode entrypoints
+                        (cbor_stream_decode_<type>) for each entry type. These
+                        use zcbor_tstr_chunk_in()/zcbor_bstr_chunk_in()
+                        callbacks and allow indefinite-length tstr/bstr values
+                        to be processed without reassembly.
   --output-c, --oc OUTPUT_C
                         Path to output C file. If both --decode and --encode
                         are specified, _decode and _encode will be appended to

include/zcbor_common.h

@@ -152,8 +152,61 @@ struct zcbor_state_constant {
 #ifdef ZCBOR_MAP_SMART_SEARCH
 	uint8_t *map_search_elem_state_end; /**< The end of the @ref map_search_elem_state buffer. */
 #endif
+	size_t (*stream_write)(void *user_data, const uint8_t *data, size_t len);
+	void *stream_user_data;
+	size_t stream_bytes_written;  /**< Total bytes written in streaming mode. */
+
+	const void *stream_io;
+
 };
 
+#ifndef ZCBOR_STREAM_IO_HELPERS
+#define ZCBOR_STREAM_IO_HELPERS
+static inline void zcbor_set_stream_io(zcbor_state_t *state, const void *io)
+{
+	if (state && state->constant_state) {
+		state->constant_state->stream_io = io;
+	}
+}
+
+static inline const void *zcbor_get_stream_io(const zcbor_state_t *state)
+{
+	return (state && state->constant_state) ? state->constant_state->stream_io : NULL;
+}
+#endif
+
+/* Alignment helper for state storage. */
+#if defined(__cplusplus) && (__cplusplus >= 201103L)
+#define ZCBOR_ALIGNAS(type) alignas(type)
+#define ZCBOR_ALIGNOF(type) alignof(type)
+#define ZCBOR_STATIC_ASSERT(cond, msg) static_assert((cond), msg)
+#elif defined(__STDC_VERSION__) && (__STDC_VERSION__ >= 201112L)
+#define ZCBOR_ALIGNAS(type) _Alignas(type)
+#define ZCBOR_ALIGNOF(type) _Alignof(type)
+#define ZCBOR_STATIC_ASSERT(cond, msg) _Static_assert((cond), msg)
+#else
+/* Require GCC extensions when C11/C++11 alignment is unavailable. */
+#if defined(__GNUC__)
+#define ZCBOR_ALIGNAS(type) __attribute__((aligned(__alignof__(type))))
+#define ZCBOR_ALIGNOF(type) __alignof__(type)
+#define ZCBOR_STATIC_ASSERT(cond, msg) _Static_assert((cond), msg)
+#else
+#error "Unsupported compiler: zcbor requires alignment and static assertion support."
+#endif
+#endif
+
+/* We store struct zcbor_state_constant in state storage that is aligned as
+ * zcbor_state_t. Ensure that is sufficient for the constant state object.
+ */
+ZCBOR_STATIC_ASSERT((ZCBOR_ALIGNOF(zcbor_state_t) >= ZCBOR_ALIGNOF(struct zcbor_state_constant)),
+	"zcbor_state_t alignment must be >= zcbor_state_constant alignment.");
+
+/** Number of zcbor_state_t slots required to store a struct zcbor_state_constant
+ *  object at the end of the state array.
+ */
+#define ZCBOR_CONST_STATE_SLOTS \
+	((sizeof(struct zcbor_state_constant) + sizeof(zcbor_state_t) - 1) / sizeof(zcbor_state_t))
+
 #ifdef ZCBOR_CANONICAL
 #define ZCBOR_ENFORCE_CANONICAL_DEFAULT true
 #else
@@ -285,6 +338,8 @@ do { \
 #define ZCBOR_ERR_MAP_FLAGS_NOT_AVAILABLE 20
 #define ZCBOR_ERR_INVALID_VALUE_ENCODING 21 ///! When ZCBOR_CANONICAL is defined, and the incoming data is not encoded with minimal length, or uses indefinite length array.
 #define ZCBOR_ERR_CONSTANT_STATE_MISSING 22
+#define ZCBOR_ERR_STREAM_WRITE_FAILED 23 ///! Streaming callback returned error or wrote fewer bytes than requested
+#define ZCBOR_ERR_STREAM_READ_FAILED 24 ///! Streaming provider callback returned error or invalid data
 #define ZCBOR_ERR_UNKNOWN 31
 
 /** The largest possible elem_count. */

include/zcbor_decode.h

@@ -44,10 +44,15 @@ void zcbor_new_decode_state(zcbor_state_t *state_array, size_t n_states,
  *                            including elements in nested unordered maps.
  */
 #define ZCBOR_STATE_D(name, num_backups, payload, payload_size, elem_count, n_flags) \
-zcbor_state_t name[((num_backups) + 2 + ZCBOR_FLAG_STATES(n_flags))]; \
+ZCBOR_ALIGNAS(zcbor_state_t) uint8_t name##_storage[ \
+	(((num_backups) + 1 + ZCBOR_FLAG_STATES(n_flags) + ZCBOR_CONST_STATE_SLOTS) * sizeof(zcbor_state_t)) \
+]; \
+zcbor_state_t *name = (zcbor_state_t *)name##_storage; \
 do { \
-	zcbor_new_decode_state(name, ZCBOR_ARRAY_SIZE(name), payload, payload_size, elem_count, \
-			(uint8_t *)&name[(num_backups) + 1], ZCBOR_FLAG_STATES(n_flags) * sizeof(zcbor_state_t)); \
+	zcbor_new_decode_state(name, ((num_backups) + 1 + ZCBOR_FLAG_STATES(n_flags) + ZCBOR_CONST_STATE_SLOTS), \
+			payload, payload_size, elem_count, \
+			(uint8_t *)&name[(num_backups) + 1], \
+			ZCBOR_FLAG_STATES(n_flags) * sizeof(zcbor_state_t)); \
 } while(0)
 
 
@@ -73,6 +78,10 @@ bool zcbor_uint_decode(zcbor_state_t *state, void *result, size_t result_size);
 bool zcbor_bstr_decode(zcbor_state_t *state, struct zcbor_string *result); /* bstr */
 bool zcbor_tstr_decode(zcbor_state_t *state, struct zcbor_string *result); /* tstr */
 bool zcbor_tag_decode(zcbor_state_t *state, uint32_t *result);  /* CBOR tag */
+
+typedef bool (*zcbor_stream_chunk_in)(void *ctx, const uint8_t *data, size_t len);
+bool zcbor_tstr_chunk_in(zcbor_state_t *state, zcbor_stream_chunk_in call, void *ctx);
+bool zcbor_bstr_chunk_in(zcbor_state_t *state, zcbor_stream_chunk_in call, void *ctx);
 bool zcbor_bool_decode(zcbor_state_t *state, bool *result); /* boolean CBOR simple value */
 bool zcbor_float16_decode(zcbor_state_t *state, float *result); /* IEEE754 float16 */
 bool zcbor_float16_bytes_decode(zcbor_state_t *state, uint16_t *result); /* IEEE754 float16 raw bytes */

include/zcbor_encode.h

@@ -40,9 +40,13 @@ void zcbor_new_encode_state(zcbor_state_t *state_array, size_t n_states,
  *  @param[in]  elem_count    The starting elem_count (typically 1).
  */
 #define ZCBOR_STATE_E(name, num_backups, payload, payload_size, elem_count) \
-zcbor_state_t name[((num_backups) + 2)]; \
+ZCBOR_ALIGNAS(zcbor_state_t) uint8_t name##_storage[ \
+	(((num_backups) + 1 + ZCBOR_CONST_STATE_SLOTS) * sizeof(zcbor_state_t)) \
+]; \
+zcbor_state_t *name = (zcbor_state_t *)name##_storage; \
 do { \
-	zcbor_new_encode_state(name, ZCBOR_ARRAY_SIZE(name), payload, payload_size, elem_count); \
+	zcbor_new_encode_state(name, ((num_backups) + 1 + ZCBOR_CONST_STATE_SLOTS), \
+			payload, payload_size, elem_count); \
 } while(0)
 
 
@@ -188,6 +192,29 @@ bool zcbor_multi_encode_minmax(size_t min_encode, size_t max_encode,
 		const size_t *num_encode, zcbor_encoder_t encoder,
 		zcbor_state_t *state, const void *input, size_t input_len);
 
+/**
+ * @brief Iterator callback for streaming encode of repeated fields.
+ *
+ * Return values:
+ *  -  1: produced one element, stored at *elem_out
+ *  -  0: done (normal end)
+ *  - <0: error (negative errno-style)
+ *
+ * The pointer stored in *elem_out must remain valid until the next call to next()
+ * (or until encoding is finished), whichever comes first.
+ */
+typedef int (*zcbor_stream_iter)(void *ctx, const void **elem_out);
+
+/**
+ * @brief Encode a repeated field by iterating elements from a callback.
+ *
+ * This is intended for streaming encode where the caller does not want (or
+ * cannot afford) to materialize an array + count upfront.
+ */
+bool zcbor_multi_encode_iter_minmax(size_t min_encode, size_t max_encode,
+		zcbor_encoder_t encoder, zcbor_state_t *state,
+		zcbor_stream_iter next, void *ctx);
+
 
 /* Supplementary string (bstr/tstr) encoding functions: */
 
@@ -231,6 +258,74 @@ bool zcbor_bstr_start_encode(zcbor_state_t *state);
  */
 bool zcbor_bstr_end_encode(zcbor_state_t *state, struct zcbor_string *result);
 
+/**
+ * @brief Chunk iterator for streaming encode of indefinite-length text/byte strings.
+ *
+ * Return values:
+ *  -  1: produced one chunk (*ptr, *len)
+ *  -  0: done (normal end)
+ *  - <0: error (negative errno-style)
+ */
+typedef int (*zcbor_next_chunk_fn)(void *ctx, const uint8_t **ptr, size_t *len);
+
+/* Encode an indefinite-length tstr using a chunk iterator. */
+bool zcbor_tstr_encode_indefinite_chunks(zcbor_state_t *state,
+		zcbor_next_chunk_fn next_chunk, void *ctx);
+bool zcbor_bstr_encode_indefinite_chunks(zcbor_state_t *state,
+		zcbor_next_chunk_fn next_chunk, void *ctx);
+
+/**
+ * @brief Push callback for streaming encode of indefinite-length text/byte strings.
+ *
+ * The callback should emit 0 or more chunks using zcbor_*str_encode_chunk(),
+ * and return true on success.
+ */
+typedef bool (*zcbor_stream_chunk_out)(void *ctx, zcbor_state_t *state);
+
+/* Indefinite-length tstr/bstr using a chunk callback. */
+bool zcbor_tstr_chunk_out(zcbor_state_t *state,
+		zcbor_stream_chunk_out call, void *ctx);
+bool zcbor_bstr_chunk_out(zcbor_state_t *state,
+		zcbor_stream_chunk_out call, void *ctx);
+
+/* Push-oriented helpers for indefinite-length tstr/bstr. */
+bool zcbor_tstr_start_encode_indefinite(zcbor_state_t *state);
+bool zcbor_bstr_start_encode_indefinite(zcbor_state_t *state);
+bool zcbor_tstr_encode_chunk(zcbor_state_t *state, const uint8_t *ptr, size_t len);
+bool zcbor_bstr_encode_chunk(zcbor_state_t *state, const uint8_t *ptr, size_t len);
+bool zcbor_tstr_end_encode_indefinite(zcbor_state_t *state);
+bool zcbor_bstr_end_encode_indefinite(zcbor_state_t *state);
+
+/** Stream write callback type.
+ *
+ * @param user_data User-provided context (e.g., UART device pointer)
+ * @param data Pointer to data to write
+ * @param len Number of bytes to write
+ * @return Number of bytes written (must equal len for success)
+ */
+typedef size_t (*zcbor_stream_write_fn)(void *user_data, const uint8_t *data, size_t len);
+
+/** Initialize encoding state for streaming mode.
+ *
+ * Bytes are written directly via the callback instead of buffering.
+ * In streaming mode, arrays and maps use indefinite-length encoding
+ * (0x9F/0xBF + items + 0xFF) to avoid backtracking.
+ *
+ * @param state_array Array of states (must have at least 2 elements for constant_state)
+ * @param n_states Number of states in array
+ * @param stream_write Callback function to write bytes
+ * @param stream_user_data User data passed to callback (e.g., UART device pointer)
+ * @param elem_count Starting element count (typically 1)
+ */
+void zcbor_new_encode_state_streaming(zcbor_state_t *state_array, size_t n_states,
+		zcbor_stream_write_fn stream_write, void *stream_user_data, size_t elem_count);
+
+size_t zcbor_stream_bytes_written(const zcbor_state_t *state);
+
+int zcbor_stream_entry_function(void *input, zcbor_state_t *states, size_t n_states,
+		zcbor_encoder_t func, zcbor_stream_write_fn stream_write, void *stream_user_data,
+		size_t elem_count, size_t *bytes_written_out);
+
 #ifdef __cplusplus
 }
 #endif

samples/pet/CMakeLists.txt

@@ -14,6 +14,7 @@ if (REGENERATE_ZCBOR)
     zcbor code # Invoke code generation
     --decode --encode # Generate both encoding and decoding code
     --short-names # Attempt to make generated symbol names shorter (at the risk of collision)
+    --stream-encode # Generate streaming encode entrypoints
     -c ${CMAKE_CURRENT_LIST_DIR}/../../tests/cases/pet.cddl # Generate code for the data structures in pet.cddl
     -t Pet # Create a public API for decoding/encoding the "Pet" type from pet.cddl
     --output-cmake ${CMAKE_CURRENT_LIST_DIR}/pet.cmake # The generated cmake file will be placed here

samples/pet/README.md

@@ -7,6 +7,7 @@ structure. The 3 data structures are created in 3 different ways:
 1. Converted from YAML using the zcbor script (See [CMakeLists.txt](CMakeLists.txt), [pet1.yml](pet1.yml), and pet1.h).
 2. Encoded using the zcbor C API.
 3. Encoded using zcbor-generated C code.
+4. Encoded using zcbor-generated streaming entrypoints.
 
 The generated code is found in [src](src) and [include](include).
 To regenerate the files, invoke the [CMakeLists.txt](CMakeLists.txt) file with `-DREGENERATE_ZCBOR=Y`.
@@ -41,3 +42,7 @@ build/app
 > Name: Gary Giraffe
 > Birthday: 0x010203040a0b0c0d
 > Species: Other
+> 
+> Name: Sammy Streaming
+> Birthday: 0xaabbccddeeff1122
+> Species: Cat

samples/pet/include/pet_decode.h

@@ -15,23 +15,32 @@
 #include <stdbool.h>
 #include <stddef.h>
 #include <string.h>
+#include "zcbor_encode.h"
+#include "zcbor_decode.h"
 #include "pet_types.h"
 
 #ifdef __cplusplus
 extern "C" {
 #endif
 
-#if DEFAULT_MAX_QTY != 3
+#if ZCBOR_GENERATED_DEFAULT_MAX_QTY != 3
 #error "The type file was generated with a different default_max_qty than this file"
 #endif
 
-
 int cbor_decode_Pet(
 		const uint8_t *payload, size_t payload_len,
 		struct Pet *result,
 		size_t *payload_len_out);
 
 
+
+
+
+
+
+
+
+
 #ifdef __cplusplus
 }
 #endif