Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

52 changes: 52 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -434,6 +434,58 @@ curl -X POST http://127.0.0.1:3000/v1/export/dataset \

Creates an async export job for a Silver dataset. Supported formats: `jsonl` (default), `csv`. Optional filters: `target_id`, `network`, `time_start`, `time_end`. Exportable datasets: `token_transfers`, `native_balance_deltas`, `decoded_events`, `hl_fills`, `hl_funding`, `positions`. Maximum 100,000 records per export.

#### Sink delivery

Export jobs support an optional `sink` field to deliver completed export data to an external destination. When a sink is configured, data is delivered to the sink **and** stored in-memory for download via the existing `/v1/export/jobs/:job_id/download` endpoint (backward compatible).

Supported sink types:

| Sink type | Description | Required fields |
|-----------|-------------|-----------------|
| `local_file` | Write export data to a local file path | `file_path` |
| `webhook` | POST export data to an HTTP(S) URL | `url`, optional `headers` |
| `database` | Deliver to an external database (not yet implemented) | `connection_string`, `table` |

Example with local file sink:

```bash
curl -X POST http://127.0.0.1:3000/v1/export/dataset \
-H "Authorization: Bearer <API_KEY>" \
-H "Content-Type: application/json" \
-d '{
"dataset": "token_transfers",
"format": "jsonl",
"sink": {
"sink_type": "local_file",
"file_path": "/tmp/token_transfers_export.jsonl"
}
}'
```

Example with webhook sink:

```bash
curl -X POST http://127.0.0.1:3000/v1/export/dataset \
-H "Authorization: Bearer <API_KEY>" \
-H "Content-Type: application/json" \
-d '{
"dataset": "hl_fills",
"format": "csv",
"sink": {
"sink_type": "webhook",
"url": "https://example.com/receive-export",
"headers": {"X-API-Key": "your-key"}
}
}'
```

When a sink is configured, the export job status response includes additional fields:

- `delivered_to`: destination description (file path, URL, etc.) — set on successful delivery
- `delivery_status`: one of `"pending"`, `"delivered"`, or `"failed"`

Webhook URLs are validated against the same rules as `callback_url` (HTTPS/HTTP only, no private/loopback addresses). Local file paths must not contain `..` path traversal. The `database` sink type is reserved but not yet implemented at runtime.

### GET /v1/export/jobs/:job_id

```bash
Expand Down
1 change: 1 addition & 0 deletions api/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ tracing-subscriber = { version = "0.3", features = ["env-filter"] }
sqlx = { version = "0.8.6", features = ["postgres", "runtime-tokio-rustls", "uuid", "chrono", "bigdecimal"] }
dotenvy = "0.15"
anyhow = "1.0"
async-trait = "0.1"
bigdecimal = { version = "0.4", features = ["serde"] }
uuid = { version = "1.19", features = ["v4", "serde"] }
chrono = { version = "0.4", features = ["serde"] }
Expand Down
Loading
Loading