Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 37 additions & 8 deletions agent-simulations/introduction.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -83,17 +83,40 @@ script=[
- **Simple integration** - Just implement one `call()` method
- **Multi-language support** - Python, TypeScript, and Go

## Two Ways to Create Simulations

LangWatch offers two approaches to agent testing:

### On-Platform Scenarios (No Code)

Create and run simulations directly in the LangWatch UI:
- Define situations and evaluation criteria visually
- Run against HTTP agents or managed prompts
- Ideal for quick iteration and non-technical team members

[Get started with On-Platform Scenarios →](/scenarios/overview)

### Scenario Library (Code-Based)

Write simulations in code for maximum control:
- Full programmatic control over conversation flow
- Complex assertions and tool call verification
- CI/CD integration for automated testing
- **Trace-based evaluation** via OpenTelemetry integration

[Get started with the Scenario library →](/agent-simulations/getting-started)

Both approaches produce simulations that appear in the same visualizer, so you can mix and match based on your needs.

## Visualizing Simulations in LangWatch

Once you've set up your agent tests with Scenario, LangWatch provides powerful visualization tools to:
The Simulations visualizer helps you analyze results from both On-Platform Scenarios and code-based tests:

- **Organize simulations** into sets and batches
- **Debug agent behavior** by stepping through conversations
- **Track performance** over time with run history
- **Collaborate** with your team on agent improvements

The rest of this documentation will show you how to use LangWatch's simulation visualizer to get the most out of your agent testing.

<img
src="/images/simulations/simulation-set-overview.png"
alt="Simulations Sets"
Expand All @@ -102,8 +125,14 @@ The rest of this documentation will show you how to use LangWatch's simulation v

## Next Steps

- [Overview](/agent-simulations/overview) - Learn about LangWatch's simulation visualizer
- [Getting Started](/agent-simulations/getting-started) - Set up your first simulation
- [Individual Run Analysis](/agent-simulations/individual-run) - Learn how to debug specific scenarios
- [Batch Runs](/agent-simulations/batch-runs) - Understand how to organize multiple tests
- [Scenario Documentation](https://langwatch.ai/scenario/) - Deep dive into the testing framework
### On-Platform Scenarios
- [Overview](/scenarios/overview) - Create scenarios in the UI
- [Creating Scenarios](/scenarios/creating-scenarios) - Write effective test cases
- [Running Scenarios](/scenarios/running-scenarios) - Execute and analyze results

### Scenario Library
- [Visualizer Overview](/agent-simulations/overview) - Learn about the simulation visualizer
- [Library Getting Started](/agent-simulations/getting-started) - Set up your first code-based simulation
- [Individual Run Analysis](/agent-simulations/individual-run) - Debug specific scenarios
- [Batch Runs](/agent-simulations/batch-runs) - Organize multiple tests
- [Scenario Documentation](https://langwatch.ai/scenario/) - Deep dive into the testing library
201 changes: 201 additions & 0 deletions agents/http-agents.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,201 @@
---
title: HTTP Agents
description: Configure HTTP endpoints as testable agents for LangWatch scenarios
sidebarTitle: HTTP Agents
---

HTTP Agents let you test any AI agent deployed as an API endpoint. Configure your endpoint once, then use it across multiple scenarios.

## Creating an HTTP Agent

1. Navigate to **Scenarios** in the sidebar
2. When running a scenario, click **Add New Agent** in the target selector
3. Configure the HTTP agent settings

<Frame>
<img src="/images/agents/http-agent-form.png" alt="HTTP Agent configuration form" />
</Frame>

## Configuration

### Basic Settings

| Field | Description | Example |
|-------|-------------|---------|
| **Name** | Descriptive name | "Production Chat API" |
| **URL** | Endpoint to call | `https://api.example.com/chat` |
| **Method** | HTTP method | `POST` |

### Authentication

Choose how to authenticate requests:

| Type | Description | Header |
|------|-------------|--------|
| **None** | No authentication | - |
| **Bearer Token** | OAuth/JWT token | `Authorization: Bearer <token>` |
| **API Key** | Custom API key header | `X-API-Key: <key>` (configurable) |
| **Basic Auth** | Username/password | `Authorization: Basic <base64>` |

<Frame>
<img src="/images/agents/auth-config.png" alt="Authentication configuration" />
</Frame>

### Body Template

Define the JSON body sent to your endpoint. Use placeholders for dynamic values:

```json
{
"messages": {{messages}},
"stream": false,
"max_tokens": 1000
}
```

**Available placeholders:**

| Placeholder | Type | Description |
|-------------|------|-------------|
| `{{messages}}` | Array | Full conversation history (OpenAI format) |
| `{{input}}` | String | Latest user message only |
| `{{threadId}}` | String | Unique conversation identifier |

**Messages format:**

The `{{messages}}` placeholder expands to an OpenAI-compatible message array:

```json
[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Hello!"},
{"role": "assistant", "content": "Hi! How can I help?"},
{"role": "user", "content": "I need help with my order"}
]
```

### Response Extraction

Use JSONPath to extract the assistant's response from your API's response format.

**Common patterns:**

| API Response Format | Response Path |
|--------------------|---------------|
| `{"choices": [{"message": {"content": "..."}}]}` | `$.choices[0].message.content` |
| `{"response": "..."}` | `$.response` |
| `{"data": {"reply": "..."}}` | `$.data.reply` |
| `{"message": "..."}` | `$.message` |

<Tip>
If your endpoint returns the message directly as a string (not JSON), leave
the response path empty.
</Tip>

## Example Configurations

### OpenAI-Compatible Endpoint

```
Name: OpenAI Compatible API
URL: https://api.yourcompany.com/v1/chat/completions
Method: POST
Auth: Bearer Token

Body Template:
{
"model": "gpt-4",
"messages": {{messages}},
"temperature": 0.7
}

Response Path: $.choices[0].message.content
```

### Simple Chat API

```
Name: Simple Chat Service
URL: https://chat.yourcompany.com/api/message
Method: POST
Auth: API Key (X-API-Key)

Body Template:
{
"message": {{input}},
"conversation_id": {{threadId}}
}

Response Path: $.reply
```

### Custom Agent with Context

```
Name: Customer Support Agent
URL: https://support.yourcompany.com/agent
Method: POST
Auth: Bearer Token

Body Template:
{
"messages": {{messages}},
"context": {
"source": "scenario_test",
"timestamp": "{{threadId}}"
}
}

Response Path: $.response.content
```

## Managing Agents

### Editing Agents

HTTP Agents are project-level resources. To edit an existing agent:

1. Open any scenario
2. Click the target selector
3. Find the agent in the HTTP Agents section
4. Click the edit icon

### Deleting Agents

Deleting an agent won't affect past scenario runs, but will prevent future runs against that agent.

## Troubleshooting

### Common Issues

| Problem | Possible Cause | Solution |
|---------|---------------|----------|
| 401 Unauthorized | Invalid or expired token | Check authentication credentials |
| 404 Not Found | Wrong URL | Verify endpoint URL |
| Timeout | Slow response | Check endpoint performance |
| Invalid JSON | Malformed body template | Validate JSON syntax |
| Empty response | Wrong response path | Test JSONPath against actual response |

### Testing Your Configuration

Before running scenarios:

1. Test your endpoint manually (curl, Postman)
2. Verify the response format matches your JSONPath
3. Check that authentication works

<Warning>
HTTP Agent credentials are stored in your project. Use environment-specific
agents (dev, staging, prod) rather than sharing credentials.
</Warning>

## Next Steps

<CardGroup cols={2}>
<Card title="Running Scenarios" icon="play" href="/scenarios/running-scenarios">
Test your agent with scenarios
</Card>
<Card title="Creating Scenarios" icon="plus" href="/scenarios/creating-scenarios">
Write test cases for your agent
</Card>
</CardGroup>
57 changes: 57 additions & 0 deletions agents/overview.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
---
title: Agents Overview
description: Configure HTTP agents to test your deployed AI agents with LangWatch scenarios
sidebarTitle: Overview
---

**Agents** in LangWatch represent external AI systems you want to test. When you run a scenario, you test it against an agent to evaluate its behavior.

## Agent Types

Currently, LangWatch supports **HTTP Agents** - external API endpoints that receive conversation messages and return responses.

<Frame>
<img src="/images/agents/agent-list.png" alt="Agent list showing configured HTTP agents" />
</Frame>

## When to Use Agents

Use HTTP Agents when you want to test:

- **Deployed agents** - Your production or staging AI endpoints
- **External services** - Third-party AI APIs
- **Custom implementations** - Any HTTP endpoint that handles conversations

For testing prompts directly (without a deployed endpoint), use [Prompt targets](/prompt-management/overview) instead.

## Key Concepts

### HTTP Agent

An HTTP Agent configuration includes:

| Field | Description |
|-------|-------------|
| **Name** | Descriptive name for the agent |
| **URL** | The endpoint to call |
| **Authentication** | How to authenticate requests |
| **Body Template** | JSON body format with message placeholders |
| **Response Path** | JSONPath to extract the response |

### Agent vs. Prompt

| Testing... | Use |
|------------|-----|
| A deployed endpoint (API) | HTTP Agent |
| A prompt before deployment | Prompt (from Prompt Management) |

## Next Steps

<CardGroup cols={2}>
<Card title="HTTP Agents" icon="globe" href="/agents/http-agents">
Configure HTTP agents for scenario testing
</Card>
<Card title="Running Scenarios" icon="play" href="/scenarios/running-scenarios">
Test your agents with scenarios
</Card>
</CardGroup>
30 changes: 25 additions & 5 deletions docs.json
Original file line number Diff line number Diff line change
Expand Up @@ -66,11 +66,31 @@
"group": "Agent Simulations",
"pages": [
"agent-simulations/introduction",
"agent-simulations/overview",
"agent-simulations/getting-started",
"agent-simulations/set-overview",
"agent-simulations/batch-runs",
"agent-simulations/individual-run"
{
"group": "On-Platform Scenarios",
"pages": [
"scenarios/overview",
"scenarios/creating-scenarios",
"scenarios/running-scenarios"
]
},
{
"group": "Scenario Library",
"pages": [
"agent-simulations/overview",
"agent-simulations/getting-started",
"agent-simulations/set-overview",
"agent-simulations/batch-runs",
"agent-simulations/individual-run"
]
},
{
"group": "Agents",
"pages": [
"agents/overview",
"agents/http-agents"
]
}
]
},
{
Expand Down
Loading
Loading