diff --git a/README.md b/README.md index b0eac7a..abe47fe 100644 --- a/README.md +++ b/README.md @@ -21,6 +21,8 @@ end ## Guides and Examples +- User guide track (numbered by complexity): [`docs/user/README.md`](./docs/user/README.md) +- Developer guide track (numbered by architecture depth): [`docs/developer/README.md`](./docs/developer/README.md) - Quickstart: [`docs/quickstart.md`](./docs/quickstart.md) - Integration boundaries: [`docs/integration-boundaries.md`](./docs/integration-boundaries.md) - Troubleshooting: [`docs/troubleshooting.md`](./docs/troubleshooting.md) @@ -33,6 +35,8 @@ end - Minimal sample: `mix run examples/minimal_api_sample.exs` - Session concurrency sample: `mix run examples/session_concurrency_sample.exs` - Crash recovery sample: `mix run examples/crash_recovery_sample.exs` +- `jido_skill` format sample: `examples/skill_format_example.md` +- `jido_command` format sample: `examples/command_format_example.md` ## API Contract Snapshot diff --git a/docs/developer/01-architecture-overview.md b/docs/developer/01-architecture-overview.md new file mode 100644 index 0000000..ab31714 --- /dev/null +++ b/docs/developer/01-architecture-overview.md @@ -0,0 +1,71 @@ +# 01 - Architecture Overview + +`jido_document` is a session-oriented document runtime focused on: + +- in-memory document state management +- file-backed persistence with divergence detection +- render preview generation +- operational safety, history, and recovery + +This library is intentionally non-UI. Transport and presentation are external +integration concerns. + +## Core architectural boundaries + +- Stable API boundary: + - `Jido.Document` + - `Jido.Document.Agent` + - `Jido.Document.Document` + - `Jido.Document.Frontmatter` + - `Jido.Document.Renderer` + - `Jido.Document.SchemaMigration` + - `Jido.Document.SessionRegistry` + - `Jido.Document.Signal` + - `Jido.Document.SignalBus` +- Internal implementation boundary: + - `Jido.Document.Actions.*` + - `Jido.Document.Persistence` + - `Jido.Document.History` + - `Jido.Document.Checkpoint` + - `Jido.Document.Revision` + - `Jido.Document.Authorization` + - `Jido.Document.Audit` + - `Jido.Document.Reliability` + - `Jido.Document.Safety` + - `Jido.Document.PathPolicy` + +## High-level component map + +```mermaid +flowchart LR + Caller["Caller"] --> Entry["Jido.Document / Jido.Document.Agent"] + Entry --> Registry["Jido.Document.SessionRegistry"] + Entry --> Agent["Session Process (Jido.Document.Agent)"] + Registry --> SessionSup["Jido.Document.SessionSupervisor"] + SessionSup --> Agent + Agent --> Actions["Jido.Document.Actions.*"] + Actions --> DocModel["Jido.Document.Document + Frontmatter"] + Actions --> Render["Jido.Document.Renderer"] + Actions --> Persist["Jido.Document.Persistence"] + Agent --> History["History + Revision + Audit"] + Agent --> Recovery["Checkpoint + Recovery"] + Agent --> Bus["Jido.Document.SignalBus"] +``` + +## Canonical flow + +1. Caller resolves/starts a session. +2. Caller sends action commands through `Agent.command/4`. +3. Agent executes one action module with a normalized context. +4. Agent updates session state, history, revision, and audit metadata. +5. Agent emits session-scoped signals for subscribers. +6. Save operations persist using atomic-write and divergence checks. + +## Architectural invariants + +- Document state is session-owned and process-local (`Agent`). +- Frontmatter and markdown body are modeled explicitly and normalized. +- Save is guarded by baseline snapshots to avoid silent overwrite. +- Signal payloads are versioned and bounded. +- Recovery is explicit (checkpoint discovery, recover, discard). + diff --git a/docs/developer/02-supervision-and-runtime-topology.md b/docs/developer/02-supervision-and-runtime-topology.md new file mode 100644 index 0000000..db5218b --- /dev/null +++ b/docs/developer/02-supervision-and-runtime-topology.md @@ -0,0 +1,51 @@ +# 02 - Supervision and Runtime Topology + +`Jido.Document.Application` defines a small supervision tree with clear +separation between shared runtime services and dynamic session processes. + +## Topology + +```mermaid +flowchart TD + App["Jido.Document.Application"] --> RootSup["Jido.Document.Supervisor (one_for_one)"] + RootSup --> Metrics["Jido.Document.Render.Metrics"] + RootSup --> Queue["Jido.Document.Render.JobQueue"] + RootSup --> Bus["Jido.Document.SignalBus"] + RootSup --> SessionSup["Jido.Document.SessionSupervisor"] + RootSup --> Registry["Jido.Document.SessionRegistry"] + Registry --> SessionSup + SessionSup --> AgentA["Jido.Document.Agent (session-a)"] + SessionSup --> AgentN["Jido.Document.Agent (session-n)"] +``` + +## Responsibilities by process + +- `Render.Metrics`: + - tracks render strategy and quality metrics. +- `Render.JobQueue`: + - handles async render jobs. +- `SignalBus`: + - session-scoped pub/sub fanout with queue-aware dropping. +- `SessionSupervisor`: + - dynamic supervisor for session agents. +- `SessionRegistry`: + - session discovery, path-indexing, lock ownership, idle cleanup. + +## Startup behavior + +Before supervisor start, `Application.start/2` runs render plugin startup +checks (`PluginManager.startup_check/2`) and logs compatibility diagnostics. + +## Session lifecycle + +1. `SessionRegistry.ensure_session/3` or `ensure_session_by_path/3` is called. +2. Registry starts a new `Agent` under `SessionSupervisor` if missing. +3. Registry monitors agent PID and removes stale entries on `:DOWN`. +4. Idle reclaim can terminate old sessions (`reclaim_idle/2`). + +## Failure model + +- `one_for_one` supervision isolates process crashes. +- Session failures do not inherently crash sibling sessions. +- Registry and signal bus are shared services and should remain lightweight. + diff --git a/docs/developer/03-agent-command-pipeline.md b/docs/developer/03-agent-command-pipeline.md new file mode 100644 index 0000000..ecba8e8 --- /dev/null +++ b/docs/developer/03-agent-command-pipeline.md @@ -0,0 +1,71 @@ +# 03 - Agent Command Pipeline + +`Jido.Document.Agent` is the stateful orchestrator. It owns: + +- active `Document` +- history model +- checkpoint and recovery state +- revision sequence +- preview/fallback state +- audit trail window + +## Command execution flow + +```mermaid +sequenceDiagram + participant Caller + participant Agent as Jido.Document.Agent + participant Action as Jido.Document.Action + participant Impl as Jido.Document.Actions.* + + Caller->>Agent: command(session, action, params, opts) + Agent->>Agent: normalize params + inject defaults + Agent->>Agent: guard action / lock if needed + Agent->>Action: execute(action_module, params, context) + Action->>Action: authorize + telemetry wrapper + Action->>Impl: run(params, context) + Impl-->>Action: {:ok, value} | {:error, reason} + Action-->>Agent: Result.t() + Agent->>Agent: apply_success/apply_failure + Agent->>Agent: history/revision/audit/signal updates + Agent-->>Caller: Result.t() +``` + +## Key contracts + +- Action behavior (`Jido.Document.Action`): + - `name/0` + - `idempotency/0` + - `run/2` +- Result envelope (`Jido.Document.Action.Result`): + - `status` + - `value` or `error` + - metadata (`action`, `idempotency`, `correlation_id`, `duration_us`) +- Context envelope (`Jido.Document.Action.Context`): + - session identity, actor, document snapshot, options + +## Cross-cutting behaviors in pipeline + +- Authorization: + - applied in `Action.execute/3` + - role matrix + optional custom hook policy +- Retry: + - controlled by `Jido.Document.Reliability.with_retry/2` + - retries only retryable errors +- Optimistic rollback: + - failed mutation actions can revert to previous state +- Render fallback: + - on render failure, agent emits a fallback preview payload + +## Sync vs async + +- Sync mode (`mode: :sync`): + - `GenServer.call`, immediate `Result.t()`. +- Async mode (`mode: :async`): + - `GenServer.cast`, returns `:ok`. + +## History commands + +`:undo` and `:redo` are handled via history transitions and require a loaded +document. They update revision/audit metadata similarly to regular actions. + diff --git a/docs/developer/04-document-model-and-frontmatter-engine.md b/docs/developer/04-document-model-and-frontmatter-engine.md new file mode 100644 index 0000000..a37ee58 --- /dev/null +++ b/docs/developer/04-document-model-and-frontmatter-engine.md @@ -0,0 +1,57 @@ +# 04 - Document Model and Frontmatter Engine + +`Jido.Document.Document` is the canonical in-memory representation: + +- `path` +- `frontmatter` map +- `body` string +- `raw` source content +- schema reference +- dirty/revision tracking + +## Parse and serialize pipeline + +```mermaid +flowchart LR + Raw["Raw file content"] --> Split["Frontmatter.split/1"] + Split --> ParseFM["Frontmatter.parse/2"] + ParseFM --> NewDoc["Document.new/1"] + NewDoc --> Canon["Document.canonicalize/2"] + Canon --> Serialize["Document.serialize/2"] + Serialize --> Output["Persistable content"] +``` + +## Frontmatter semantics + +- Supported delimiters: + - YAML: `---` + - TOML: `+++` +- Frontmatter is optional: + - no delimiter => `frontmatter: %{}` +- Serialization defaults: + - empty frontmatter omitted unless `emit_empty_frontmatter: true` + +## Mutation semantics + +- `update_frontmatter/3`: + - `:merge` (default) or `:replace` +- `update_body/3`: + - body normalization policies (line endings, trailing whitespace) +- `apply_body_patch/3`: + - full replace string + - function patch + - `%{search, replace, global}` + +All effective mutations increment revision and set dirty state. + +## Schema validation + +`Jido.Document.Schema` validates frontmatter against `Field` declarations: + +- required fields +- defaulting +- type coercion (`string`, `integer`, `float`, `boolean`, arrays, enums) +- unknown key policy (`:warn`, `:ignore`, `:reject`) + +This layer validates metadata contracts without coupling to UI or transport. + diff --git a/docs/developer/05-rendering-and-preview-subsystem.md b/docs/developer/05-rendering-and-preview-subsystem.md new file mode 100644 index 0000000..5bcb9f1 --- /dev/null +++ b/docs/developer/05-rendering-and-preview-subsystem.md @@ -0,0 +1,58 @@ +# 05 - Rendering and Preview Subsystem + +Rendering is split between orchestration and transformation components: + +- orchestration: + - `Jido.Document.Actions.Render` + - `Jido.Document.Render.ChangeTracker` + - `Jido.Document.Render.JobQueue` + - `Jido.Document.Render.Metrics` +- transformation: + - `Jido.Document.Renderer` + - `Jido.Document.Render.PluginManager` + - `Jido.Document.Render.ThemeRegistry` + +## Render flow + +```mermaid +flowchart TD + Cmd["Agent command(:render)"] --> Decision["ChangeTracker.plan/3"] + Decision --> Safety["Safety.scan + Safety.redact"] + Safety --> Mode{"async?"} + Mode -- "yes" --> Queue["JobQueue.enqueue/6"] + Mode -- "no" --> Render["Renderer.render/2"] + Render --> Plugins["PluginManager.apply_plugins/3"] + Render --> Adapter["Adapter selection (:mdex/:simple/:auto)"] + Adapter --> Preview["Preview payload"] + Preview --> Metrics["Render.Metrics"] +``` + +## Renderer design points + +- Frontmatter stripping is default for preview. +- Adapter auto-fallback: + - prefers `:mdex` when available + - falls back to `:simple` with diagnostics +- Fallback preview exists for degraded render conditions. + +## Plugin model + +- Contract: + - `Jido.Document.Render.Plugin.transform/2` + - `Jido.Document.Render.Plugin.compatible?/1` +- Manager behavior: + - ordered execution by priority + - startup compatibility checks + - failure isolation with diagnostics, not hard crash + +## Output contract + +Render output includes: + +- `html` +- `toc` +- `diagnostics` +- `cache_key` +- `adapter` +- `metadata` (extensions, syntax highlight settings, plugin list, etc.) + diff --git a/docs/developer/06-persistence-divergence-and-recovery.md b/docs/developer/06-persistence-divergence-and-recovery.md new file mode 100644 index 0000000..dd02e0d --- /dev/null +++ b/docs/developer/06-persistence-divergence-and-recovery.md @@ -0,0 +1,65 @@ +# 06 - Persistence, Divergence, and Recovery + +This subsystem combines path policy, atomic writes, baseline snapshots, and +checkpoint-based crash recovery. + +## Save pipeline + +```mermaid +sequenceDiagram + participant Caller + participant Agent + participant Save as Actions.Save + participant Path as PathPolicy + participant Persist as Persistence + + Caller->>Agent: command(:save, params) + Agent->>Save: run(params, context) + Save->>Path: resolve_path(path, workspace_root) + Save->>Persist: detect_divergence(path, baseline) + Save->>Persist: atomic_write(path, serialized) + Save->>Persist: snapshot(path) + Save-->>Agent: {:ok, saved_payload} + Agent-->>Caller: Result.ok +``` + +## Divergence protection + +- Save compares current file state against baseline snapshot. +- On mismatch, conflict error is returned unless explicit strategy allows: + - `:overwrite` + - `:merge_hook` + +## Atomic write strategy + +`Persistence.atomic_write/3` performs: + +1. write fsynced temp file +2. optional metadata preservation +3. atomic rename to target +4. best-effort directory sync + +This minimizes partial-write corruption risk. + +## Checkpoint and recovery model + +`Jido.Document.Checkpoint` stores session payload snapshots to disk: + +- schema version +- session id +- document struct +- disk snapshot baseline +- capture timestamp + +Recovery APIs on `Agent`: + +- `recovery_status/1` +- `recover/2` +- `discard_recovery/1` +- `list_recovery_candidates/1` + +## Path security boundary + +`Jido.Document.PathPolicy.resolve_path/2` enforces workspace-root containment to +prevent path escapes from session load/save operations. + diff --git a/docs/developer/07-session-registry-locking-and-signals.md b/docs/developer/07-session-registry-locking-and-signals.md new file mode 100644 index 0000000..28983c6 --- /dev/null +++ b/docs/developer/07-session-registry-locking-and-signals.md @@ -0,0 +1,54 @@ +# 07 - Session Registry, Locking, and Signals + +Multi-client coordination is handled by `SessionRegistry` and `SignalBus`. + +## Session registry responsibilities + +- deterministic file-based session id generation +- lazy session startup under `SessionSupervisor` +- path-to-session indexing +- idle-session reclaim +- optimistic lock ownership with revisioned tokens + +## Lock lifecycle + +```mermaid +stateDiagram-v2 + [*] --> Unlocked + Unlocked --> Locked: acquire_lock(owner) + Locked --> Locked: validate_lock(token) + Locked --> Unlocked: release_lock(valid token) + Locked --> Locked: force_takeover(new owner) +``` + +## Lock semantics + +- `acquire_lock/4`: + - returns `lock_token`, `lock_revision`, owner +- `validate_lock/3`: + - optimistic token check for write-authorized caller +- `release_lock/3`: + - token-required release +- `force_takeover/4`: + - operator override path with `previous_owner` + +## Signal bus model + +`Jido.Document.SignalBus` provides session-scoped fanout with queue-aware +dropping. + +```mermaid +flowchart LR + Agent["Session Agent"] --> Build["Signal.build/4"] + Build --> Bus["SignalBus.broadcast/5"] + Bus --> SubA["Subscriber A"] + Bus --> SubB["Subscriber B"] + Bus --> Drop["Drop when queue > max_queue_len"] +``` + +Design notes: + +- Signals are versioned (`schema_version`) and type-checked. +- Payloads are size-bounded with truncation metadata. +- Monitoring cleans up dead subscriber PIDs automatically. + diff --git a/docs/developer/08-extension-points-and-api-evolution.md b/docs/developer/08-extension-points-and-api-evolution.md new file mode 100644 index 0000000..732199a --- /dev/null +++ b/docs/developer/08-extension-points-and-api-evolution.md @@ -0,0 +1,63 @@ +# 08 - Extension Points and API Evolution + +This guide covers where the system is intentionally extensible and how API +stability is enforced across releases. + +## Primary extension points + +### Action modules + +- Contract: `Jido.Document.Action` behavior +- Requirements: + - `name/0` + - `idempotency/0` + - `run/2` +- Execution harness provides: + - context normalization + - authorization integration + - telemetry + - canonical result wrapping + +### Render plugins + +- Contract: `Jido.Document.Render.Plugin` + - `transform/2` + - `compatible?/1` +- Runtime: + - configured plugin list + - startup compatibility checks + - isolated plugin failure diagnostics + +### Migration templates + +- Built-in template directory: + - `priv/migration/templates/*.exs` +- Runtime tooling: + - `Jido.Document.Migrator` + - `mix jido.migrate_docs` + +## Stability and release controls + +- Stable API manifest: + - `Jido.Document.PublicApi` + - `priv/api/public_api_manifest.exs` +- Semver governance: + - [`../semver-policy.md`](../semver-policy.md) +- Release gating: + - [`../release-blocking-criteria.md`](../release-blocking-criteria.md) + +## Quality controls in repo + +- CI alias: + - `mix ci` -> format, compile warnings-as-errors, API manifest check, tests +- Integration-heavy regression coverage: + - `test/jido/document/*integration_test.exs` +- Property coverage: + - `property_document_roundtrip_test.exs` + +## Change strategy for maintainers + +1. Update docs and contracts first. +2. Keep stable public API backward compatible for minor releases. +3. Add migration notes before deprecation. +4. Use manifest checks to detect accidental API drift. diff --git a/docs/developer/README.md b/docs/developer/README.md new file mode 100644 index 0000000..51b5cb2 --- /dev/null +++ b/docs/developer/README.md @@ -0,0 +1,14 @@ +# Developer Guides + +This guide set is ordered from architecture fundamentals to advanced +operational and evolution concerns. + +1. [01 - Architecture Overview](./01-architecture-overview.md) +2. [02 - Supervision and Runtime Topology](./02-supervision-and-runtime-topology.md) +3. [03 - Agent Command Pipeline](./03-agent-command-pipeline.md) +4. [04 - Document Model and Frontmatter Engine](./04-document-model-and-frontmatter-engine.md) +5. [05 - Rendering and Preview Subsystem](./05-rendering-and-preview-subsystem.md) +6. [06 - Persistence, Divergence, and Recovery](./06-persistence-divergence-and-recovery.md) +7. [07 - Session Registry, Locking, and Signals](./07-session-registry-locking-and-signals.md) +8. [08 - Extension Points and API Evolution](./08-extension-points-and-api-evolution.md) + diff --git a/docs/user/01-getting-started.md b/docs/user/01-getting-started.md new file mode 100644 index 0000000..0556813 --- /dev/null +++ b/docs/user/01-getting-started.md @@ -0,0 +1,77 @@ +# 01 - Getting Started + +This guide covers the minimum setup to start a document session and run a full +load-update-render-save flow. + +## Components in this guide + +- `Jido.Document` +- `Jido.Document.Agent` +- `Jido.Document.Action.Result` + +## 1. Add the dependency + +```elixir +def deps do + [ + {:jido_document, "~> 0.1"} + ] +end +``` + +Then install: + +```bash +mix deps.get +``` + +## 2. Run a minimal lifecycle + +```elixir +Application.ensure_all_started(:jido_document) + +alias Jido.Document.Agent +alias Jido.Document.Action.Result + +workspace_root = Path.expand(".") +tmp_dir = Path.join(workspace_root, "tmp/jido_document_user_guide") +File.mkdir_p!(tmp_dir) + +path = Path.join(tmp_dir, "getting_started.md") +File.write!(path, "---\ntitle: \"Getting Started\"\n---\nHello\n") + +{:ok, agent} = Jido.Document.start_session(session_id: "user-guide-01") + +context_opts = [context_options: %{workspace_root: workspace_root}] + +{:ok, _loaded, _meta} = + Agent.command(agent, :load, %{path: path}, context_opts) + |> Result.unwrap() + +{:ok, _updated, _meta} = + Agent.command(agent, :update_body, %{body: "Updated from guide 01\n"}) + |> Result.unwrap() + +{:ok, preview_payload, _meta} = + Agent.command(agent, :render, %{}) + |> Result.unwrap() + +{:ok, saved, _meta} = + Agent.command(agent, :save, %{path: path}, context_opts) + |> Result.unwrap() + +IO.puts("Rendered HTML bytes: #{byte_size(preview_payload.preview.html)}") +IO.puts("Saved bytes: #{saved.bytes}") +``` + +## 3. What happens in this flow + +- `:load` parses frontmatter and markdown into a canonical document struct. +- `:update_body` edits in-memory markdown and bumps revision state. +- `:render` produces preview output (`html`, `toc`, `diagnostics`). +- `:save` serializes and writes to disk with divergence protection. + +## Next + +Continue with [02 - Document Structure: Frontmatter and Markdown](./02-document-structure-frontmatter-and-markdown.md). + diff --git a/docs/user/02-document-structure-frontmatter-and-markdown.md b/docs/user/02-document-structure-frontmatter-and-markdown.md new file mode 100644 index 0000000..899561b --- /dev/null +++ b/docs/user/02-document-structure-frontmatter-and-markdown.md @@ -0,0 +1,90 @@ +# 02 - Document Structure: Frontmatter and Markdown + +This guide explains how `Jido.Document` models a document as: + +- frontmatter map +- markdown body string + +Frontmatter is optional. Markdown-only documents are valid. + +## Components in this guide + +- `Jido.Document.Document` +- `Jido.Document.Frontmatter` + +## 1. Supported frontmatter delimiters + +- YAML frontmatter: `---` +- TOML frontmatter: `+++` + +Examples: + +```markdown +--- +title: "YAML Example" +count: 3 +--- +# Body +``` + +```markdown ++++ +title = "TOML Example" +count = 3 ++++ +# Body +``` + +```markdown +# Markdown Only + +No frontmatter is present. +``` + +## 2. Parse into the canonical struct + +```elixir +alias Jido.Document.Document + +{:ok, yaml_doc} = + Document.parse("---\ntitle: \"Hello\"\ncount: 3\n---\n# Body\n") + +{:ok, plain_doc} = + Document.parse("# Markdown only\n") + +yaml_doc.frontmatter +# => %{"count" => 3, "title" => "Hello"} + +plain_doc.frontmatter +# => %{} +``` + +## 3. Serialize back to file content + +```elixir +{:ok, content} = Document.serialize(yaml_doc, syntax: :yaml) +{:ok, body_only} = Document.serialize(plain_doc) +{:ok, forced_frontmatter} = Document.serialize(plain_doc, emit_empty_frontmatter: true) +``` + +Behavior: + +- Empty frontmatter is omitted by default. +- `emit_empty_frontmatter: true` forces delimiters even for empty metadata. + +## 4. Update frontmatter and body + +```elixir +{:ok, doc1} = Document.update_frontmatter(yaml_doc, %{author: "Pascal"}, mode: :merge) +{:ok, doc2} = Document.update_body(doc1, "# Updated Body\r\n", line_endings: :lf) +``` + +`mode` options for frontmatter updates: + +- `:merge` (default): merge keys +- `:replace`: replace the entire frontmatter map + +## Next + +Continue with [03 - Working with the Document API](./03-working-with-the-document-api.md). + diff --git a/docs/user/03-working-with-the-document-api.md b/docs/user/03-working-with-the-document-api.md new file mode 100644 index 0000000..628e46e --- /dev/null +++ b/docs/user/03-working-with-the-document-api.md @@ -0,0 +1,62 @@ +# 03 - Working with the Document API + +This guide focuses on pure document operations without running a session +process. + +## Components in this guide + +- `Jido.Document.Document` + +## 1. Build and validate documents in memory + +```elixir +alias Jido.Document.Document + +{:ok, doc} = + Document.new( + path: "/tmp/demo.md", + frontmatter: %{title: "Demo", priority: 1}, + body: "Initial body\n" + ) +``` + +Validation rules include: + +- `frontmatter` must be a map +- `body` must be a string +- `path`, when set, must be a string + +## 2. Apply edits + +```elixir +{:ok, doc1} = Document.update_frontmatter(doc, %{owner: "ops"}, mode: :merge) +{:ok, doc2} = Document.apply_body_patch(doc1, %{search: "Initial", replace: "Updated"}) +{:ok, doc3} = Document.update_body(doc2, "Updated body with LF\r\n", line_endings: :lf) +``` + +`apply_body_patch/3` supports: + +- full replacement string +- patch map (`search`/`replace`/`global`) +- unary function (`fn body -> ... end`) + +## 3. Normalize and serialize deterministically + +```elixir +canonical = Document.canonicalize(doc3, line_endings: :lf, trailing_whitespace: :trim) +{:ok, output} = Document.serialize(canonical, syntax: :yaml) +``` + +## 4. Parse-serialize roundtrip pattern + +```elixir +{:ok, parsed} = Document.parse(output) +{:ok, output_again} = Document.serialize(parsed, syntax: :yaml) +``` + +For stable pipelines, compare `output == output_again`. + +## Next + +Continue with [04 - Session Workflows with Agent](./04-session-workflows-with-agent.md). + diff --git a/docs/user/04-session-workflows-with-agent.md b/docs/user/04-session-workflows-with-agent.md new file mode 100644 index 0000000..be096ae --- /dev/null +++ b/docs/user/04-session-workflows-with-agent.md @@ -0,0 +1,95 @@ +# 04 - Session Workflows with Agent + +This guide moves from pure document functions to stateful session orchestration. + +## Components in this guide + +- `Jido.Document.Agent` +- `Jido.Document.Action.Result` +- `Jido.Document.Signal` + +## 1. Start a session + +```elixir +Application.ensure_all_started(:jido_document) +{:ok, agent} = Jido.Document.Agent.start_link(session_id: "user-guide-04") +``` + +## 2. Run action commands + +```elixir +alias Jido.Document.Agent +alias Jido.Document.Action.Result + +workspace_root = Path.expand(".") +path = Path.join(workspace_root, "tmp/agent_flow.md") + +File.mkdir_p!(Path.dirname(path)) +File.write!(path, "---\ntitle: \"Agent Flow\"\n---\nBody\n") + +opts = [context_options: %{workspace_root: workspace_root}] + +{:ok, loaded, _} = + Agent.command(agent, :load, %{path: path}, opts) + |> Result.unwrap() + +{:ok, _updated, _} = + Agent.command(agent, :update_frontmatter, %{changes: %{owner: "docs"}}) + |> Result.unwrap() + +{:ok, rendered, _} = + Agent.command(agent, :render, %{}) + |> Result.unwrap() + +{:ok, saved, _} = + Agent.command(agent, :save, %{path: loaded.path}, opts) + |> Result.unwrap() + +IO.puts("toc entries: #{length(rendered.preview.toc)}") +IO.puts("saved revision: #{saved.revision}") +``` + +Supported action names: + +- `:load` +- `:save` +- `:update_frontmatter` +- `:update_body` +- `:render` +- `:undo` +- `:redo` + +## 3. Sync and async command modes + +```elixir +# default: synchronous call +result = Agent.command(agent, :update_body, %{body: "Sync edit\n"}) + +# async fire-and-forget +:ok = Agent.command(agent, :update_body, %{body: "Async edit\n"}, mode: :async) +``` + +## 4. Subscribe to session signals + +```elixir +:ok = Agent.subscribe(agent) + +receive do + {:jido_document_signal, signal} -> + IO.inspect(signal.type, label: "signal") +after + 1_000 -> :ok +end +``` + +## 5. Inspect session state + +```elixir +state = Agent.state(agent) +IO.inspect(state.document.revision, label: "current revision") +``` + +## Next + +Continue with [05 - Rendering and Preview Pipeline](./05-rendering-and-preview-pipeline.md). + diff --git a/docs/user/05-rendering-and-preview-pipeline.md b/docs/user/05-rendering-and-preview-pipeline.md new file mode 100644 index 0000000..90d1d3b --- /dev/null +++ b/docs/user/05-rendering-and-preview-pipeline.md @@ -0,0 +1,71 @@ +# 05 - Rendering and Preview Pipeline + +This guide focuses on preview generation and render diagnostics. + +## Components in this guide + +- `Jido.Document.Renderer` +- `Jido.Document.Actions.Render` (via `Agent.command/4`) + +## 1. Render directly from markdown + +~~~elixir +alias Jido.Document.Renderer + +markdown = """ +--- +title: "Render Demo" +--- +# Heading + +```elixir +IO.puts("hello") +``` +""" + +{:ok, preview} = Renderer.render(markdown, adapter: :auto, strip_frontmatter: true) + +IO.puts("adapter=#{preview.adapter}") +IO.puts("toc_count=#{length(preview.toc)}") +IO.puts("diagnostics=#{length(preview.diagnostics)}") +~~~ + +Preview payload keys: + +- `html` +- `toc` +- `diagnostics` +- `cache_key` +- `adapter` +- `metadata` + +## 2. Render through an active session + +```elixir +alias Jido.Document.Agent +alias Jido.Document.Action.Result + +{:ok, payload, _meta} = + Agent.command(agent, :render, %{render_opts: [adapter: :simple]}) + |> Result.unwrap() + +preview = payload.preview +IO.puts(preview.html) +``` + +## 3. Render options you will likely use + +- `adapter`: `:mdex`, `:simple`, or `:auto` +- `strip_frontmatter`: remove frontmatter from preview source +- `syntax_highlight`: theme and language support controls +- `plugins`: transform pipeline hooks +- `max_code_block_lines`: truncate very large fenced blocks + +## 4. Fallback behavior + +When primary rendering fails, the session-level render flow provides a fallback +preview so clients still have readable output while reporting diagnostics. + +## Next + +Continue with [06 - Concurrency with Session Registry](./06-concurrency-with-session-registry.md). diff --git a/docs/user/06-concurrency-with-session-registry.md b/docs/user/06-concurrency-with-session-registry.md new file mode 100644 index 0000000..a7632d4 --- /dev/null +++ b/docs/user/06-concurrency-with-session-registry.md @@ -0,0 +1,76 @@ +# 06 - Concurrency with Session Registry + +This guide covers multi-client coordination and lock ownership. + +## Components in this guide + +- `Jido.Document.SessionRegistry` +- `Jido.Document.SessionSupervisor` + +## 1. Ensure a session by path + +```elixir +alias Jido.Document.SessionRegistry + +path = Path.expand("tmp/concurrent_doc.md") +File.mkdir_p!(Path.dirname(path)) +File.write!(path, "# Concurrency demo\n") + +{:ok, session} = SessionRegistry.ensure_session_by_path(SessionRegistry, path) + +IO.puts("session_id=#{session.session_id}") +IO.puts("pid=#{inspect(session.pid)}") +``` + +## 2. Acquire and validate locks + +```elixir +{:ok, lock} = SessionRegistry.acquire_lock(SessionRegistry, session.session_id, "client-a") + +:ok = + SessionRegistry.validate_lock( + SessionRegistry, + session.session_id, + lock.lock_token + ) +``` + +If another client tries to lock the same session, it receives a conflict error. + +## 3. Force takeover for operational recovery + +```elixir +{:ok, takeover} = + SessionRegistry.force_takeover( + SessionRegistry, + session.session_id, + "client-b", + reason: "operator override" + ) +``` + +Takeover payload includes: + +- new `lock_token` +- incremented `lock_revision` +- `previous_owner` + +## 4. Release and reclaim + +```elixir +:ok = SessionRegistry.release_lock(SessionRegistry, session.session_id, takeover.lock_token) +{:ok, reclaimed_ids} = SessionRegistry.reclaim_idle(SessionRegistry, 60_000) +``` + +## 5. Deterministic IDs for file sessions + +```elixir +session_id = SessionRegistry.session_id_for_path(path) +``` + +Use this when you need stable session identity across restarts. + +## Next + +Continue with [07 - History, Checkpoints, and Safe Persistence](./07-history-checkpoints-and-safe-persistence.md). + diff --git a/docs/user/07-history-checkpoints-and-safe-persistence.md b/docs/user/07-history-checkpoints-and-safe-persistence.md new file mode 100644 index 0000000..7e21879 --- /dev/null +++ b/docs/user/07-history-checkpoints-and-safe-persistence.md @@ -0,0 +1,85 @@ +# 07 - History, Checkpoints, and Safe Persistence + +This guide covers reliability features used in production editing workflows. + +## Components in this guide + +- `Jido.Document.Agent` (`:undo`, `:redo`, recovery APIs) +- `Jido.Document.Persistence` (through `:save`) +- `Jido.Document.Safety` (through `:render` and `:save`) + +## 1. History-based undo and redo + +```elixir +alias Jido.Document.Agent + +%{status: :ok} = Agent.command(agent, :update_body, %{body: "Body v1\n"}) +%{status: :ok} = Agent.command(agent, :update_body, %{body: "Body v2\n"}) + +%{status: :ok, value: undo_value} = Agent.command(agent, :undo, %{}) +%{status: :ok, value: redo_value} = Agent.command(agent, :redo, %{}) + +IO.puts(undo_value.document.body) +IO.puts(redo_value.document.body) +``` + +## 2. Checkpoint and recovery lifecycle + +```elixir +{:ok, recovering_agent} = + Agent.start_link( + session_id: "recoverable-session", + checkpoint_dir: Path.expand("tmp/checkpoints"), + checkpoint_on_edit: true, + autosave_interval_ms: 15_000 + ) + +pending = Agent.recovery_status(recovering_agent) + +case pending do + nil -> :ok + _checkpoint -> Agent.recover(recovering_agent) +end +``` + +Other recovery utilities: + +- `Agent.discard_recovery/1` +- `Agent.list_recovery_candidates/1` + +## 3. Save divergence protection + +Session saves include a baseline disk snapshot by default. If file content on +disk diverges externally, save returns a conflict error unless you choose an +explicit conflict strategy. + +```elixir +Agent.command(agent, :save, %{ + path: path, + on_conflict: :overwrite +}) +``` + +Conflict policies: + +- `:reject` (default) +- `:overwrite` +- `:merge_hook` + +## 4. Sensitive content checks + +Safety options can be applied to render and save operations: + +```elixir +safety = %{block_severities: [:high], approved_codes: ["email"]} + +Agent.command(agent, :render, %{safety: safety}) +Agent.command(agent, :save, %{path: path, safety: safety}) +``` + +If high-severity findings are unapproved, save is blocked. + +## Next + +Continue with [08 - Schema Validation and Migration](./08-schema-validation-and-migration.md). + diff --git a/docs/user/08-schema-validation-and-migration.md b/docs/user/08-schema-validation-and-migration.md new file mode 100644 index 0000000..9ca9772 --- /dev/null +++ b/docs/user/08-schema-validation-and-migration.md @@ -0,0 +1,95 @@ +# 08 - Schema Validation and Migration + +This guide covers strict frontmatter contracts and migration workflows. + +## Components in this guide + +- `Jido.Document.Field` +- `Jido.Document.Schema` +- `Jido.Document.SchemaMigration` +- `Jido.Document.Migrator` +- `mix jido.migrate_docs` + +## 1. Define a schema for frontmatter + +```elixir +defmodule BlogSchema do + @behaviour Jido.Document.Schema + alias Jido.Document.Field + + @impl true + def fields do + [ + %Field{name: :title, type: :string, required: true}, + %Field{name: :slug, type: :string, required: true}, + %Field{name: :published, type: :boolean, default: false}, + %Field{name: :tags, type: {:array, :string}} + ] + end +end +``` + +## 2. Validate frontmatter maps + +```elixir +alias Jido.Document.Schema + +frontmatter = %{"title" => "Hello", "slug" => "hello", "published" => "true"} + +{:ok, normalized, warnings} = + Schema.validate_frontmatter(frontmatter, BlogSchema, unknown_keys: :warn) +``` + +`unknown_keys` policies: + +- `:warn` +- `:ignore` +- `:reject` + +## 3. Run schema migration operations + +```elixir +alias Jido.Document.SchemaMigration + +ops = [ + {:rename, "name", "title"}, + {:coerce, "priority", :integer} +] + +{:ok, dry_run} = SchemaMigration.dry_run(%{"name" => "Doc", "priority" => "3"}, ops) +{:ok, applied} = SchemaMigration.apply(%{"name" => "Doc", "priority" => "3"}, ops) +``` + +Destructive operations such as `{:drop, key}` require `allow_destructive: true`. + +## 4. Migrate whole directories + +Dry run: + +```bash +mix jido.migrate_docs --source ./content --template blog_frontmatter +``` + +Apply changes with backups: + +```bash +mix jido.migrate_docs \ + --source ./content \ + --template blog_frontmatter \ + --apply \ + --backup-dir ./migration_backups +``` + +List bundled templates: + +```bash +mix jido.migrate_docs --list-templates +``` + +## 5. Production recommendation + +- Use `dry_run` first. +- Save reports with `--report`. +- Apply in controlled batches. +- Keep backups until post-migration validation passes. + diff --git a/docs/user/README.md b/docs/user/README.md new file mode 100644 index 0000000..ce6f4e7 --- /dev/null +++ b/docs/user/README.md @@ -0,0 +1,13 @@ +# User Guides + +This guide set is ordered from basic usage to advanced operational topics. + +1. [01 - Getting Started](./01-getting-started.md) +2. [02 - Document Structure: Frontmatter and Markdown](./02-document-structure-frontmatter-and-markdown.md) +3. [03 - Working with the Document API](./03-working-with-the-document-api.md) +4. [04 - Session Workflows with Agent](./04-session-workflows-with-agent.md) +5. [05 - Rendering and Preview Pipeline](./05-rendering-and-preview-pipeline.md) +6. [06 - Concurrency with Session Registry](./06-concurrency-with-session-registry.md) +7. [07 - History, Checkpoints, and Safe Persistence](./07-history-checkpoints-and-safe-persistence.md) +8. [08 - Schema Validation and Migration](./08-schema-validation-and-migration.md) + diff --git a/examples/command_format_example.md b/examples/command_format_example.md new file mode 100644 index 0000000..848b013 --- /dev/null +++ b/examples/command_format_example.md @@ -0,0 +1,24 @@ +--- +name: code-review +description: Review changed files +model: sonnet +allowed-tools: + - Read + - Grep + - Bash(git diff:*) +jido: + command_module: Jido.Code.Command.Commands.CodeReview + hooks: + pre: true + after: true + schema: + target_file: + type: string + required: true + doc: File path to review + mode: + type: atom + default: standard +--- +Review {{target_file}} using mode {{mode}} and summarize findings. + diff --git a/examples/skill_format_example.md b/examples/skill_format_example.md new file mode 100644 index 0000000..a603b26 --- /dev/null +++ b/examples/skill_format_example.md @@ -0,0 +1,34 @@ +--- +name: pdf-processor +description: Extract text and tables from PDFs +version: 1.0.0 +allowed-tools: Read, Write, Bash(python:*) +jido: + actions: + - MyApp.Actions.ExtractPdfText + - MyApp.Actions.ExtractPdfTables + router: + - "pdf/extract/text": ExtractPdfText + - "pdf/extract/tables": ExtractPdfTables + hooks: + pre: + enabled: true + signal_type: "skill/pdf_processor/pre" + bus: ":jido_code_bus" + data: + source: "frontmatter" + post: + enabled: true + signal_type: "skill/pdf_processor/post" + bus: ":jido_code_bus" + data: + source: "frontmatter" +--- +# PDF Processor + +Use this skill to parse uploaded PDF files and return structured output: + +- Extract plain text from each page. +- Extract table-like regions for downstream processing. +- Emit pre and post hook signals for observability. + diff --git a/mix.exs b/mix.exs index 395df59..e8d3f64 100644 --- a/mix.exs +++ b/mix.exs @@ -85,6 +85,24 @@ defmodule JidoDocument.MixProject do "docs/semver-policy.md", "docs/release-blocking-criteria.md", "docs/quickstart.md", + "docs/user/README.md", + "docs/user/01-getting-started.md", + "docs/user/02-document-structure-frontmatter-and-markdown.md", + "docs/user/03-working-with-the-document-api.md", + "docs/user/04-session-workflows-with-agent.md", + "docs/user/05-rendering-and-preview-pipeline.md", + "docs/user/06-concurrency-with-session-registry.md", + "docs/user/07-history-checkpoints-and-safe-persistence.md", + "docs/user/08-schema-validation-and-migration.md", + "docs/developer/README.md", + "docs/developer/01-architecture-overview.md", + "docs/developer/02-supervision-and-runtime-topology.md", + "docs/developer/03-agent-command-pipeline.md", + "docs/developer/04-document-model-and-frontmatter-engine.md", + "docs/developer/05-rendering-and-preview-subsystem.md", + "docs/developer/06-persistence-divergence-and-recovery.md", + "docs/developer/07-session-registry-locking-and-signals.md", + "docs/developer/08-extension-points-and-api-evolution.md", "docs/integration-boundaries.md", "docs/troubleshooting.md", "docs/migration-guide.md",