pg-debyte

Core building blocks for PostgreSQL extensions that decode bytea into JSON. This repository provides reusable Rust crates plus a small example extension.

Workspace

• pg_debyte_core: envelope parser, registry, codecs/actions, limits, errors.
• pg_debyte_macros: helper macros for registering typed decoders.
• pg_debyte_pgrx: PG15/PG17 helper glue (GUC limits and decoding helpers).
• pg_debyte_ext: example PG17 extension crate with a demo registry and decoder.
• pg_debyte_tools: helper binaries (demo payload generator).

MVP status

• Envelope format parsing (magic + version + type_id + schema_version + codec + actions).
• Action pipeline (decode in reverse) with bounded zstd decode.
• Bincode codec with size limits.
• Static registry for decoders/codecs/actions.
• Known schema SQL functions (per-type decoding without envelope).
• PG15/PG17 helper for GUC limits and decoding (to be called from extension).
• Panic protection around decoding (catch_unwind in pgrx).

Notes

• pg_debyte_ext is only an example implementation; you will create your own extension crate.
• PG15/PG17 are supported via pgrx feature flags.

Examples

The code blocks below are backed by real crates in examples/:

• examples/readme_known_schema
• examples/readme_by_id
• examples/readme_envelope

Quick start: build your own extension

Common setup:

cargo new my_pg_debyte_ext --lib
cd my_pg_debyte_ext

[dependencies]
pgrx = { version = "0.16.1", default-features = false, features = ["pg17"] }
# or: features = ["pg15"]
pg_debyte_core = "0.2.1"
pg_debyte_macros = "0.2.1"
pg_debyte_pgrx = { version = "0.2.1", default-features = false }
serde = { version = "1.0", features = ["derive"] }
uuid = "1.8"

Enable the matching pg_debyte_pgrx feature:

[features]
pg15 = ["pg_debyte_pgrx/pg15"]
pg17 = ["pg_debyte_pgrx/pg17"]

Build and install once (choose your PG major, matching features):

cargo pgrx init --pg15 /path/to/pg15
cargo pgrx install --features pg15 --sudo

cargo pgrx init --pg17 /path/to/pg17
cargo pgrx install --features pg17 --sudo

Enable in Postgres:

CREATE EXTENSION my_pg_debyte_ext;

Known schema (per-type function)

Use this when the payload schema is fixed and you want a dedicated SQL function per type. The function is generated by declare_know_schema! and decodes raw payload without an envelope.

src/lib.rs:

use pgrx::prelude::*;
use pg_debyte_core::{BincodeCodec, StaticRegistry};
use pg_debyte_macros::declare_know_schema;
use serde::{Deserialize, Serialize};
use uuid::Uuid as CoreUuid;

pg_module_magic!();

#[derive(Debug, Deserialize, Serialize)]
struct MyRecord {
    id: u32,
    label: String,
}

const MY_TYPE_ID: CoreUuid = CoreUuid::from_bytes([0x11; 16]);
const MY_SCHEMA_VERSION: u16 = 1;
const MY_CODEC_ID: u16 = 1;
const MY_CODEC: BincodeCodec = BincodeCodec::new(MY_CODEC_ID, 32 * 1024 * 1024);

declare_know_schema!(
    MY_DECODER,
    ty = MyRecord,
    type_id = MY_TYPE_ID,
    schema_version = MY_SCHEMA_VERSION,
    codec = MY_CODEC,
    codec_ty = BincodeCodec,
    actions = [],
    fn_name = bytea_to_json_my_record
);

static REGISTRY: StaticRegistry = StaticRegistry::new(&[&MY_DECODER], &[]);

#[pg_guard]
pub unsafe extern "C-unwind" fn _PG_init() {
    pg_debyte_pgrx::init_gucs();
    pg_debyte_pgrx::set_registry(&REGISTRY);
}

Usage:

Generate a raw payload with the helper binary:

cargo run -p pg_debyte_tools --bin demo_payload

SELECT bytea_to_json_my_record(decode('<hex-encoded-payload>', 'hex'));

Registry + by_id

Use this when you have raw payloads but need flexible routing by type_id + schema_version. You register decoders in a registry and call a single SQL function with explicit type info.

src/lib.rs:

use pgrx::prelude::*;
use pgrx::JsonB;
use pg_debyte_core::{BincodeCodec, DecodeError, StaticRegistry};
use pg_debyte_macros::declare_decoder;
use serde::{Deserialize, Serialize};
use uuid::Uuid as CoreUuid;

pg_module_magic!();

#[derive(Debug, Deserialize, Serialize)]
struct MyRecord {
    id: u32,
    label: String,
}

const MY_TYPE_ID: CoreUuid = CoreUuid::from_bytes([0x11; 16]);
const MY_SCHEMA_VERSION: u16 = 1;
const MY_CODEC_ID: u16 = 1;
const MY_CODEC: BincodeCodec = BincodeCodec::new(MY_CODEC_ID, 32 * 1024 * 1024);

declare_decoder!(
    MY_DECODER,
    ty = MyRecord,
    type_id = MY_TYPE_ID,
    schema_version = MY_SCHEMA_VERSION,
    codec = MY_CODEC,
    codec_ty = BincodeCodec,
    actions = []
);

static REGISTRY: StaticRegistry = StaticRegistry::new(&[&MY_DECODER], &[]);

#[pg_guard]
pub unsafe extern "C-unwind" fn _PG_init() {
    pg_debyte_pgrx::init_gucs();
    pg_debyte_pgrx::set_registry(&REGISTRY);
}

#[pg_extern]
fn bytea_to_json_by_id(
    data: Vec<u8>,
    type_id: pgrx::Uuid,
    schema_version: i16,
) -> Result<JsonB, DecodeError> {
    let limits = pg_debyte_pgrx::limits();
    let core_uuid = CoreUuid::from_bytes(*type_id.as_bytes());
    let value = pg_debyte_pgrx::decode_by_id(&data, core_uuid, schema_version, &limits)?;
    Ok(JsonB(value))
}

Usage (generate payload with the helper binary):

cargo run -p pg_debyte_tools --bin demo_payload

SELECT bytea_to_json_by_id(
  decode('<hex-encoded-payload>', 'hex'),
  '22222222-2222-2222-2222-222222222222'::uuid,
  1::smallint
);

Envelope (auto)

Use this when payloads include an envelope with type, version, codec, and actions. bytea_to_json_auto parses the envelope and dispatches to the matching decoder automatically.

src/lib.rs:

use pgrx::prelude::*;
use pgrx::JsonB;
use pg_debyte_core::{BincodeCodec, DecodeError, StaticRegistry};
use pg_debyte_macros::declare_decoder;
use serde::{Deserialize, Serialize};
use uuid::Uuid as CoreUuid;

pg_module_magic!();

#[derive(Debug, Deserialize, Serialize)]
struct MyRecord {
    id: u32,
    label: String,
}

const MY_TYPE_ID: CoreUuid = CoreUuid::from_bytes([0x22; 16]);
const MY_SCHEMA_VERSION: u16 = 1;
const MY_CODEC_ID: u16 = 1;
const MY_CODEC: BincodeCodec = BincodeCodec::new(MY_CODEC_ID, 32 * 1024 * 1024);

declare_decoder!(
    MY_DECODER,
    ty = MyRecord,
    type_id = MY_TYPE_ID,
    schema_version = MY_SCHEMA_VERSION,
    codec = MY_CODEC,
    codec_ty = BincodeCodec,
    actions = []
);

static REGISTRY: StaticRegistry = StaticRegistry::new(&[&MY_DECODER], &[]);

#[pg_guard]
pub unsafe extern "C-unwind" fn _PG_init() {
    pg_debyte_pgrx::init_gucs();
    pg_debyte_pgrx::set_registry(&REGISTRY);
}

#[pg_extern]
fn bytea_to_json_auto(data: Vec<u8>) -> Result<JsonB, DecodeError> {
    let limits = pg_debyte_pgrx::limits();
    let value = pg_debyte_pgrx::decode_auto(&data, &limits)?;
    Ok(JsonB(value))
}

Usage:

Generate an envelope with the helper binary (type_id 0x11, schema_version 1):

cargo run -p pg_debyte_tools --bin demo_envelope

SELECT bytea_to_json_auto(decode('<hex-encoded-envelope>', 'hex'));

Example usage (PG15/PG17)

Build and install the example extension (PG15/PG17):

cargo pgrx init --pg15 /path/to/pg15
cargo pgrx install -p pg_debyte_ext --features pg15 --sudo

cargo pgrx init --pg17 /path/to/pg17
cargo pgrx install -p pg_debyte_ext --features pg17 --sudo

Enable in Postgres:

CREATE EXTENSION pg_debyte_ext;

Generate a demo payload hex:

# run the helper binary
cargo run -p pg_debyte_tools --bin demo_payload

Decode in SQL (raw payload, no envelope):

SELECT bytea_to_json_by_id(
  decode('<hex-encoded-payload>', 'hex'),
  '11111111-1111-1111-1111-111111111111'::uuid,
  1::smallint
);

Generate a demo payload hex for a known schema:

cargo run -p pg_debyte_tools --bin demo_second_payload

Decode in SQL (known schema, per-type function):

SELECT bytea_to_json_demo_record_second(decode('<hex-encoded-payload>', 'hex'));

Generate a demo envelope hex:

#run the helper binary
cargo run -p pg_debyte_tools --bin demo_envelope

Decode in SQL (auto envelope):

SELECT bytea_to_json_auto(decode('<hex-encoded-envelope>', 'hex'));

Generate a full SQL example for auto envelope:

cargo run -p pg_debyte_tools --bin demo_auto_sql

Running pg tests in Docker

If host permissions make cargo pgrx test difficult, use the Docker runner:

source scripts/dev.sh
docker_tests_pg_extensions pg17