Add Jaeger tracing and OTEL verification to E2E

README.md

@@ -40,14 +40,15 @@ WALLET_PATH=~/repos/browser-wallet \
 ./up.sh
 ```
 
-This runs through 7 stages:
+This runs through 8 stages:
 1. Checks prerequisites (Docker, Stellar CLI, Deno, Cargo)
-2. Starts a local Stellar network via Docker
-3. Generates accounts (admin, provider, treasury) and funds them via Friendbot
-4. Builds and deploys contracts (channel-auth, privacy-channel)
-5. Registers the provider on the channel-auth contract
-6. Starts the provider platform (PostgreSQL, migrations, server)
-7. Builds wallet extensions for Chrome and Brave with dev seeds
+2. Starts Jaeger for distributed tracing (UI at http://localhost:16686)
+3. Starts a local Stellar network via Docker
+4. Generates accounts (admin, provider, treasury) and funds them via Friendbot
+5. Builds and deploys contracts (channel-auth, privacy-channel)
+6. Registers the provider on the channel-auth contract
+7. Starts the provider platform (PostgreSQL, migrations, server)
+8. Builds wallet extensions for Chrome and Brave with dev seeds
 
 After it finishes, load `browser-wallet/dist/chrome/` or `dist/brave/` as unpacked extensions.
 
@@ -71,7 +72,15 @@ Rebuilds the Chrome and Brave wallet extensions without restarting the network o
 cd e2e && deno task e2e
 ```
 
-Runs the full 8-step E2E test (fund, auth, deposit, receive, send, withdraw) against the local stack.
+Runs the full E2E test (fund, auth, deposit, receive, send, withdraw) against the local stack. Traces are exported to Jaeger automatically.
+
+```bash
+# Verify traces were captured
+deno task verify-otel
+
+# Open Jaeger UI to inspect traces
+open http://localhost:16686
+```
 
 ## E2E in CI
 
@@ -90,3 +99,4 @@ See [RELEASES.md](RELEASES.md) for the versioning strategy and release workflows
 - **Friendbot timeout**: The local Stellar node can take a few minutes on first start. Re-run `./up.sh`, it will pick up where it left off.
 - **Provider connection fails**: Check `provider.log` in this directory for errors.
 - **Contract deployment fails**: Make sure the local Stellar container is running (`docker ps`).
+- **No traces in Jaeger**: Check `jaeger.log` and ensure `OTEL_DENO=true` is set (automatic with `deno task e2e`).

down.sh

@@ -44,6 +44,12 @@ if [ "$killed" = false ]; then
   done
 fi
 
+# --- Stop Jaeger ---
+if docker ps -a --format '{{.Names}}' | grep -q "^jaeger-local$"; then
+  docker rm -f jaeger-local >/dev/null 2>&1
+  info "Stopped Jaeger"
+fi
+
 # --- Stop PostgreSQL ---
 if [ -f "$PROVIDER_PLATFORM_PATH/docker-compose.yml" ]; then
   info "Stopping PostgreSQL..."
@@ -70,6 +76,7 @@ for f in \
   "$WALLET_PATH/.env.seed.local" \
   "$WALLET_PATH/.env.seed.local.brave" \
   "$SCRIPT_DIR/provider.log" \
+  "$SCRIPT_DIR/jaeger.log" \
 ; do
   if [ -f "$f" ]; then
     rm "$f"

e2e/.gitignore

@@ -1,2 +1,3 @@
 node_modules/
 wasms/
+e2e-trace-ids.json

e2e/README.md

@@ -9,11 +9,11 @@ Docker compose setup for running the full Moonlight stack in CI. No host depende
 │ stellar  │     │ db  │     │ provider │     │ test-runner  │
 │ (node)   │◄────│(pg) │◄────│ (deno)   │◄────│ (deno)      │
 └──────────┘     └─────┘     └──────────┘     └─────────────┘
-      ▲                            ▲
-      │         ┌───────┐          │
-      └─────────│ setup │──────────┘
-                │(1-shot)│
-                └───────┘
+      ▲                            │ traces          │ traces
+      │         ┌───────┐         ▼                  ▼
+      └─────────│ setup │    ┌──────────┐
+                │(1-shot)│   │  jaeger  │
+                └───────┘    └──────────┘
 ```
 
 Services:
@@ -22,6 +22,7 @@ Services:
 - **setup** — one-shot container that generates accounts, deploys contracts, writes config
 - **provider** — provider platform (reads config from setup, runs migrations, serves API)
 - **test-runner** — runs the E2E test suite against provider and stellar node
+- **jaeger** — all-in-one tracing backend (OTLP collector + query UI on port 16686)
 
 Each `docker compose up` creates an isolated network. Parallel runs don't interfere with each other.
 
@@ -114,3 +115,161 @@ Config is loaded from env vars first, then `/config/contracts.env` (Docker), the
 | `receive.ts` | Prepare-to-receive flow |
 | `send.ts` | Send flow |
 | `withdraw.ts` | Withdraw flow |
+| `tracer.ts` | OpenTelemetry adapter for SDK's MoonlightTracer interface |
+| `verify-otel.ts` | Jaeger trace verification (16 checks) |
+| `e2e-trace-ids.json` | Generated artifact — trace IDs from the last E2E run |
+
+## OpenTelemetry / Jaeger
+
+The E2E suite is instrumented with distributed tracing. When run with `OTEL_DENO=true` (set automatically by `deno task e2e`), traces are exported to Jaeger and can be inspected visually.
+
+### How tracing works
+
+There are **two services** producing traces:
+
+- **`moonlight-e2e`** — the E2E test process. Creates spans for each test step (`e2e.deposit`, `e2e.send`, etc.), SDK operations (`PrivacyChannel.read`, `UtxoBasedAccount.deriveBatch`), auth flows, and bundle submission/polling. Outgoing `fetch()` calls automatically carry W3C `traceparent` headers.
+
+- **`provider-platform`** — the privacy provider. Creates spans for request handling (auto-instrumented HTTP spans via `OTEL_DENO=true`) and application logic (`P_CreateChallenge`, `P_AddOperationsBundle`, `Executor.*`, `Verifier.*`, `Mempool.*`, `Bundle.*`).
+
+**Distributed traces** connect the two: when the SDK makes an HTTP request to the provider, the provider's HTTP span appears as a child of the SDK's span within the same trace. The provider's application-level spans (background services like executor, verifier, mempool) run on polling loops and appear as **separate 1-span root traces** — they are not nested inside the HTTP request traces.
+
+### Running
+
+```bash
+# Start the stack (includes Jaeger)
+./up.sh
+
+# Run E2E with tracing enabled
+cd e2e
+deno task e2e
+
+# Verify traces were captured (16 checks)
+deno task verify-otel
+
+# Open Jaeger UI
+open http://localhost:16686
+```
+
+### Navigating the Jaeger UI
+
+#### 1. View all E2E traces
+
+1. In the left sidebar, set **Service** to `moonlight-e2e`
+2. Click **Find Traces**
+3. You'll see 7 traces (one per E2E step), each showing:
+   - Root span name (`e2e.fund_accounts`, `e2e.authenticate_alice`, etc.)
+   - Total duration
+   - Span count
+   - Which services are involved (1 service = SDK only, 2 services = distributed)
+
+The traces map 1:1 to the E2E steps:
+
+| Trace | What it does | Distributed? |
+|-------|-------------|--------------|
+| `e2e.fund_accounts` | Funds test accounts via Friendbot | No (Friendbot only) |
+| `e2e.authenticate_alice` | SEP-10 auth with provider | Yes |
+| `e2e.authenticate_bob` | SEP-10 auth with provider | Yes |
+| `e2e.deposit` | Deposit XLM into privacy channel | Yes |
+| `e2e.prepare_receive` | Derive UTXOs for receiving | No (Soroban reads only) |
+| `e2e.send` | Send XLM through privacy channel | Yes |
+| `e2e.withdraw` | Withdraw XLM from privacy channel | Yes |
+
+#### 2. Read an authentication trace
+
+Click an **`e2e.authenticate_*`** trace. The waterfall view shows:
+
+```
+e2e.authenticate_alice                          ██████████████████████  (root span)
+  auth.get_challenge                            ████████                 GET challenge from provider
+    GET                                         ███████                  outgoing fetch()
+      GET [provider-platform]                   ██████                   provider handles request
+  auth.sign_challenge                               ██                   local crypto (no network)
+  auth.verify_challenge                               ████████████████   POST signed challenge
+    POST                                              ████████████████   outgoing fetch()
+      POST [provider-platform]                        ███████████████    provider verifies
+```
+
+- **Indentation** = parent-child relationship
+- **Two colors** = two services. The provider's spans are nested inside the SDK's HTTP spans — that's the `traceparent` link
+- **`auth.sign_challenge`** has no children — it's pure local cryptographic signing
+- Click any span to see its **Tags** (HTTP method, status code, URL) and **Logs/Events** (`enter`, `exit`)
+
+#### 3. Read a deposit/send/withdraw trace
+
+Click an **`e2e.deposit`** trace. This is the richest trace:
+
+```
+e2e.deposit                                     ████████████████████████████████████████
+  UtxoBasedAccount.deriveBatch                  ██                                        derive UTXO keys
+  UtxoBasedAccount.batchLoad                    ████                                      load UTXO state
+    PrivacyChannel.read                         ████                                      read on-chain data
+      POST                                      ███                                       Soroban RPC call
+  bundle.submit                                     ██                                    submit to provider
+    POST                                            ██                                    outgoing fetch()
+      POST [provider-platform]                      █                                     provider accepts
+  bundle.wait                                         ██████████████████████████████████   poll until done
+    GET → GET [provider-platform]                     █  █  █                              polling requests
+```
+
+**Three phases:**
+1. **Account setup** (first ~200ms) — derive keys, load on-chain UTXO state via Soroban RPC
+2. **Bundle submission** (small sliver) — POST the privacy operations to the provider
+3. **Waiting** (the long bar) — poll the provider every 5s until the bundle is processed. The gaps between GET spans are the sleep intervals
+
+The `e2e.send` trace is similar but also includes `UtxoBasedAccount.selectUTXOsForTransfer` (a fast synchronous span for UTXO selection).
+
+**Zoom in**: click and drag on the timeline header to zoom into the account setup phase. Click "Reset Zoom" (top-right) to restore.
+
+#### 4. View provider application spans
+
+The provider's background services (executor, verifier, mempool) run on polling loops independent of HTTP requests. Their spans appear as **separate 1-span root traces**, not nested inside the E2E traces.
+
+1. Set **Service** to `provider-platform`
+2. Set **Operation** to a specific span (e.g., `P_AddOperationsBundle`, `Executor.executeNext`)
+3. Click **Find Traces**
+4. Each result is a single-span trace. Click one to see its **Logs/Events** — these show the internal state machine:
+   - `P_AddOperationsBundle`: `enter` → `session_valid` → `operations_classified` → `fee_calculated` → `exit`
+   - `Executor.executeNext`: `enter` → `slot_found` → `transaction_built` → `submitted` → `exit`
+
+To filter to spans from your E2E run (vs background noise), use **Min Duration** or narrow the **Lookback** time window.
+
+#### 5. Inspect the distributed link
+
+1. Open any distributed trace (e.g., `e2e.deposit`)
+2. Find a provider-platform span (different color) — e.g., the POST inside `bundle.submit`
+3. Click it to expand the detail panel
+4. Look at **References** — it shows `CHILD_OF` with a parent span ID pointing to the SDK's outgoing HTTP span
+
+This proves W3C traceparent propagation is working: the SDK's `fetch()` injected the trace context, and the provider's `Deno.serve()` extracted it.
+
+#### 6. Compare operation durations
+
+1. Service: `provider-platform`, Operation: `Executor.submitTransactionToNetwork`
+2. Find Traces
+3. Compare durations across the 3 bundles (deposit, send, withdraw) to see which transaction type is slowest on-chain
+
+#### 7. Service dependency graph
+
+Click **System Architecture** (or **Dependencies**) in the top navigation to see a graph of `moonlight-e2e` → `provider-platform` — the service dependency map derived from trace data.
+
+### Sidebar filter reference
+
+| Filter | Use case |
+|--------|----------|
+| Service | `moonlight-e2e` for SDK traces, `provider-platform` for server traces |
+| Operation | Filter by span name (e.g., `e2e.deposit`, `P_AddOperationsBundle`) |
+| Tags | Filter by attributes (e.g., `http.status_code=500` to find errors) |
+| Min/Max Duration | Find slow operations (e.g., Min: `10s` to find long bundle waits) |
+| Lookback | Narrow time window to filter out old/background traces |
+
+### Trace verification
+
+`deno task verify-otel` runs 16 automated checks against Jaeger:
+
+- **Provider-platform** (8 checks): function-level spans, auth create/verify spans, bundle processing, Bundle.* helpers, spans with events, HTTP spans, background service spans
+- **SDK** (7 checks): all 7 E2E step spans present, auth/bundle E2E spans, PrivacyChannel spans, UtxoBasedAccount spans, SDK spans with events
+- **Distributed tracing** (2 checks): shared trace IDs between services, CHILD_OF parent-child references
+
+It uses two query strategies to avoid background span noise:
+1. **Trace-by-ID** — fetches exact E2E traces (for SDK + distributed checks)
+2. **Time-windowed query** — fetches provider spans within the E2E time window (for application-level checks)

e2e/account.ts

@@ -1,5 +1,6 @@
 import {
   ChannelReadMethods,
+  type MoonlightTracer,
   PrivacyChannel,
   StellarDerivator,
   UtxoBasedStellarAccount,
@@ -13,6 +14,7 @@ export async function setupAccount(
   secretKey: string,
   config: Config,
   minFreeUtxos: number,
+  tracer?: MoonlightTracer,
 ): Promise<{
   accountHandler: UtxoBasedStellarAccount;
   channelClient: PrivacyChannel;
@@ -27,13 +29,15 @@ export async function setupAccount(
     config.channelContractId,
     config.channelAuthId,
     config.channelAssetContractId,
+    tracer ? { tracer } : undefined,
   );
 
   const accountHandler = new UtxoBasedStellarAccount({
     root: secretKey as Ed25519SecretKey,
     derivator: stellarDerivator,
     options: {
       batchSize: 50,
+      tracer,
       fetchBalances(publicKeys: Uint8Array[]) {
         return channelClient.read({
           method: ChannelReadMethods.utxo_balances,

e2e/auth.ts

@@ -1,5 +1,6 @@
 import { Keypair, TransactionBuilder } from "stellar-sdk";
 import type { Config } from "./config.ts";
+import { withE2ESpan } from "./tracer.ts";
 
 export async function authenticate(
   keypair: Keypair,
@@ -8,39 +9,45 @@ export async function authenticate(
   const publicKey = keypair.publicKey();
 
   // 1. Get auth challenge
-  const challengeRes = await fetch(
-    `${config.providerUrl}/api/v1/stellar/auth?account=${publicKey}`,
-  );
-  if (!challengeRes.ok) {
-    throw new Error(
-      `Auth challenge failed: ${challengeRes.status} ${await challengeRes.text()}`,
+  const challengeXdr = await withE2ESpan("auth.get_challenge", async () => {
+    const challengeRes = await fetch(
+      `${config.providerUrl}/api/v1/stellar/auth?account=${publicKey}`,
     );
-  }
-  const challengeData = await challengeRes.json();
-  const challengeXdr: string = challengeData.data.challenge;
+    if (!challengeRes.ok) {
+      throw new Error(
+        `Auth challenge failed: ${challengeRes.status} ${await challengeRes.text()}`,
+      );
+    }
+    const challengeData = await challengeRes.json();
+    return challengeData.data.challenge as string;
+  });
 
   // 2. Co-sign the challenge transaction
-  const tx = TransactionBuilder.fromXDR(challengeXdr, config.networkPassphrase);
-  tx.sign(keypair);
-  const signedXdr = tx.toXDR();
+  const signedXdr = withE2ESpan("auth.sign_challenge", async () => {
+    const tx = TransactionBuilder.fromXDR(challengeXdr, config.networkPassphrase);
+    tx.sign(keypair);
+    return tx.toXDR();
+  });
 
   // 3. Submit signed challenge
-  const authRes = await fetch(`${config.providerUrl}/api/v1/stellar/auth`, {
-    method: "POST",
-    headers: { "Content-Type": "application/json" },
-    body: JSON.stringify({ signedChallenge: signedXdr }),
-  });
-  if (!authRes.ok) {
-    throw new Error(
-      `Auth verify failed: ${authRes.status} ${await authRes.text()}`,
-    );
-  }
-  const authData = await authRes.json();
-  const jwt: string = authData.data.jwt;
+  return withE2ESpan("auth.verify_challenge", async () => {
+    const authRes = await fetch(`${config.providerUrl}/api/v1/stellar/auth`, {
+      method: "POST",
+      headers: { "Content-Type": "application/json" },
+      body: JSON.stringify({ signedChallenge: await signedXdr }),
+    });
+    if (!authRes.ok) {
+      throw new Error(
+        `Auth verify failed: ${authRes.status} ${await authRes.text()}`,
+      );
+    }
+    const authData = await authRes.json();
+    const jwt: string = authData.data.jwt;
 
-  if (!jwt) {
-    throw new Error(`No JWT in auth response: ${JSON.stringify(authData)}`);
-  }
+    if (!jwt) {
+      throw new Error(`No JWT in auth response: ${JSON.stringify(authData)}`);
+    }
 
-  return jwt;
+    return jwt;
+  });
 }