Synthdocs 🧪📄

Generate realistic synthetic case documents for testing RAG and agentic systems.

Why Synthdocs?

Synthdocs generates realistic-looking case files (multiple documents + evolving facts) to baseline-test RAG and agentic workflows.

It's not a statistically faithful mirror of your corpus—it's a stable, shareable test set so you can iterate fast when real docs are sensitive or change often. Useful for regression testing, grounding/citation checks, and tightening your pipelines without waiting on production data. ✅

Installation

uv sync

For the interactive viewer (optional):

uv sync --extra viewer

Quick Start

Generate a single document:

uv run examples/generate_sample.py --output my_document

This creates:

• my_document.md — the generated document with YAML frontmatter
• my_document.meta.yaml — metadata with exact locations of each fact

Core Workflows

1. Generate Cases

Generate multiple cases, each with multiple documents:

uv run examples/generate_multi_case.py --count 3 --output ./output

Or use the Python API:

from pathlib import Path

from synthdocs import (
    CaseTemplate, DocumentTypeSpec, EntitySchema, EntityField,
    FactType, FactField, generate_case_batch, MistralBackend,
)

template = CaseTemplate(
    name="lease-dispute",
    description="Tenant {{ tenant.name }} renting from {{ landlord.name }}.",
    entity_schemas=[
        EntitySchema(name="Tenant", description="The tenant", fields=[
            EntityField(name="name", description="Full name", field_type="str"),
        ]),
        EntitySchema(name="Landlord", description="The landlord", fields=[
            EntityField(name="name", description="Full name", field_type="str"),
        ]),
    ],
    fact_types=[
        FactType(name="LeaseTerms", description="Lease terms", fields=[
            FactField(name="monthly_rent", description="Rent", field_type="int"),
        ], template="Rent: {{ monthly_rent }} per month"),
    ],
    document_types=[
        DocumentTypeSpec(
            name="Lease Agreement",
            description="Signed lease contract",
            introduces_fact_types=["LeaseTerms"],
        ),
    ],
)

results = generate_case_batch(
    template=template,
    count=5,
    backend=MistralBackend(),
    output_dir=Path("./output"),
)

See docs/case-templates.md for the full template reference.

Output structure:

output/
└── cases/
    └── <case_id>/
        ├── case.meta.yaml
        └── documents/
            ├── 01-lease-agreement.md
            ├── 01-lease-agreement.meta.yaml
            └── ...

2. Evaluate Generated Cases

Evaluate fact-location accuracy on a generated output folder:

# Evaluate tool-produced fact locations
uv run synthdocs eval fact-locations --target output/

# Compare user predictions against ground truth
uv run synthdocs eval fact-locations --target output/ --predictions-dir predictions/

Reports are written under the target folder (output/reports/fact_location/...).

For more details on evaluation modes, judge configuration, and metrics, see docs/evaluation.md.

3. Viewer

Browse generated documents with highlighted facts:

uv run synthdocs viewer output/

Features:

• Document browser sidebar
• Color-coded fact highlighting with all occurrences marked
• Click facts to cycle through their locations in the document
• Shows found/not-found status with fact type names and occurrence counts

More Docs

• docs/case-templates.md — define case templates (entities, facts, document types)
• docs/custom-llm-backend.md — implement your own LLMBackend + evaluate the judge with your backend
• docs/evaluation.md — how synthdocs evaluates fact-location quality (spec + case-folder modes)

Environment Variables

• MISTRAL_API_KEY — required for Mistral backend
• OPENAI_API_KEY — required for OpenAI backend

Development

# Run tests
uv run pytest

# Run linter
uv run ruff check src/ tests/

# Run type checker
uv run mypy src/ tests/

License

MIT

TODO / Ideas

• Write documents as we go along instead of all at once
• Add multi-backend eval mode (e.g., --backends mistral,openai) to compare models
• When evaluating predictions from a prediction folder, also report how many required facts were missing entirely from the predictions