From f37c764a9ce4d28d130aeb721ec84a01d8ebbac6 Mon Sep 17 00:00:00 2001 From: Pedro Pacheco Date: Thu, 16 Oct 2025 16:20:41 -0600 Subject: [PATCH 1/4] Read and better comments --- 17-multi-turn/README.md | 149 ++++++ 17-multi-turn/multi_turn.ipynb | 929 +++++++++++++++++++++++++++++++++ 2 files changed, 1078 insertions(+) create mode 100644 17-multi-turn/README.md create mode 100644 17-multi-turn/multi_turn.ipynb diff --git a/17-multi-turn/README.md b/17-multi-turn/README.md new file mode 100644 index 0000000..d77532b --- /dev/null +++ b/17-multi-turn/README.md @@ -0,0 +1,149 @@ +# Multi-Turn vs Single-Turn Agent Testing + +This example demonstrates the differences between multi-turn and single-turn agents in Contextual AI through comprehensive testing scenarios. + +## Overview + +The notebook creates two agents with identical configurations except for their multi-turn settings: +- **MultiTurnAgent**: Maintains conversation context across multiple messages +- **SingleTurnAgent**: Treats each message independently without memory + +## What You'll Learn + +- How to enable/disable multi-turn functionality in Contextual AI agents +- The difference in behavior between multi-turn and single-turn agents +- How to use `conversation_id` to continue previous conversations +- Best practices for testing agent memory capabilities + +## Key Concepts + +### Multi-Turn Agents +- Remember conversation history using `conversation_id` +- Can reference previous messages and maintain context +- Ideal for conversational interfaces and complex interactions + +### Single-Turn Agents +- Process each message independently +- No memory of previous interactions +- Suitable for stateless operations and simple Q&A + +### Conversation Management +- The first message in a conversation generates a `conversation_id` +- Subsequent messages use this ID to maintain context +- Conversations can be resumed later using the same ID + +## Test Methodology + +The notebook follows a systematic approach: + +1. **Setup**: Creates two identical agents with different multi-turn settings +2. **Initial Facts**: Shares the same information with both agents +3. **Memory Testing**: Asks follow-up questions to test retention +4. **Comparison**: Validates expected vs actual behavior +5. **Recurring Conversations**: Tests conversation continuity + +### Test Scenarios + +| Scenario | MultiTurnAgent | SingleTurnAgent | +|----------|----------------|-----------------| +| Initial message with facts | ✓ Acknowledges | ✓ Acknowledges | +| Follow-up question about facts | ✓ Remembers | ✗ Forgets | +| Complex recall request | ✓ Full context | ✗ No context | +| Conversation resumption | ✓ Continues | N/A | + +## Running the Notebook + +### Prerequisites +- Contextual AI API key +- Python environment with required packages + +### Setup Steps +1. Update the `API_KEY` variable with your Contextual AI API key +2. Run all cells sequentially +3. Review the test results summary at the end + +### Expected Outputs +- Multi-turn agent should pass memory tests (remember color and cats) +- Single-turn agent should fail memory tests (forget previous information) +- All 5 tests should pass if multi-turn functionality works correctly + +## Configuration Details + +### Agent Configuration +```python +agent_configs = { + "global_config": { + "enable_multi_turn": True, # Key difference between agents + "enable_filter": False, + "enable_rerank": False, + "should_check_retrieval_need": True, + }, + "reformulation_config": { + "enable_query_expansion": False, + "enable_query_decomposition": False, + }, + "generate_response_config": { + "model": 'vertex_ai/gemini-2.0-flash-lite' + } +} +``` + +### Query with Conversation ID +```python +response = client.agents.query.create( + agent_id=agent.id, + messages=[{"content": "Your message", "role": "user"}], + conversation_id=conversation_id # Enables context retention +) +``` + +## Test Data + +The example uses simple, verifiable facts: +- **Favorite color**: Blue +- **Pet count**: 3 cats +- **Pet names**: Whiskers, Mittens, Shadow + +These concrete values make it easy to verify whether agents remember information correctly. + +## Cleanup + +The notebook includes an optional cleanup section to delete the test agents and datastore after experimentation. Uncomment the cleanup code if you want to remove the created resources. + +## Use Cases + +### Multi-Turn Agents Are Ideal For: +- Customer service chatbots +- Interactive tutorials +- Complex problem-solving sessions +- Personalized conversations + +### Single-Turn Agents Are Ideal For: +- FAQ systems +- Simple classification tasks +- Stateless API endpoints +- One-off queries + +## Troubleshooting + +### Common Issues +- **Agent doesn't remember**: Ensure you're using the correct `conversation_id` +- **Tests fail**: Check API key validity and agent configuration +- **Unexpected behavior**: Verify the `enable_multi_turn` setting + +### Debugging Tips +- Monitor conversation IDs in the output +- Check agent responses for context clues +- Validate test data matches expected values + +## Next Steps + +After understanding multi-turn behavior, consider: +- Implementing conversation persistence +- Adding conversation branching +- Building stateful applications +- Exploring advanced context management + +--- + +**Note**: Remember to keep your API key secure and never commit it to version control. \ No newline at end of file diff --git a/17-multi-turn/multi_turn.ipynb b/17-multi-turn/multi_turn.ipynb new file mode 100644 index 0000000..9bbb195 --- /dev/null +++ b/17-multi-turn/multi_turn.ipynb @@ -0,0 +1,929 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "5235406f", + "metadata": {}, + "source": [ + "# Contextual AI Multi-Turn vs Single-Turn Chat Testing\n", + "\n", + "This notebook demonstrates the difference between multi-turn and single-turn agents in Contextual AI by creating two fresh agents.\n", + "\n", + "**What we're testing:**\n", + "- **MultiTurnAgent**: Will be created with multi-turn enabled - should remember conversation context\n", + "- **SingleTurnAgent**: Will be created with multi-turn disabled - should NOT remember previous messages\n", + "- **Recurring conversations**: Can continue from a previous conversation using conversation_id\n", + "\n", + "**Test approach:**\n", + "1. Create two new agents with different multi-turn settings\n", + "2. Tell both agents some simple facts (favorite color and pet names) \n", + "3. Ask if they remember those facts\n", + "4. Test recurring conversation capability\n", + "\n", + "**Expected results:**\n", + "- MultiTurnAgent should remember facts across messages\n", + "- SingleTurnAgent should forget facts between messages" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "aee6dba7", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Collecting contextual-client\n", + " Using cached contextual_client-0.8.0-py3-none-any.whl.metadata (16 kB)\n", + "Collecting matplotlib\n", + " Downloading matplotlib-3.10.7-cp312-cp312-macosx_11_0_arm64.whl.metadata (11 kB)\n", + "Collecting anyio<5,>=3.5.0 (from contextual-client)\n", + " Using cached anyio-4.11.0-py3-none-any.whl.metadata (4.1 kB)\n", + "Collecting distro<2,>=1.7.0 (from contextual-client)\n", + " Using cached distro-1.9.0-py3-none-any.whl.metadata (6.8 kB)\n", + "Collecting httpx<1,>=0.23.0 (from contextual-client)\n", + " Using cached httpx-0.28.1-py3-none-any.whl.metadata (7.1 kB)\n", + "Collecting pydantic<3,>=1.9.0 (from contextual-client)\n", + " Downloading pydantic-2.12.2-py3-none-any.whl.metadata (85 kB)\n", + "Collecting sniffio (from contextual-client)\n", + " Using cached sniffio-1.3.1-py3-none-any.whl.metadata (3.9 kB)\n", + "Collecting typing-extensions<5,>=4.10 (from contextual-client)\n", + " Using cached typing_extensions-4.15.0-py3-none-any.whl.metadata (3.3 kB)\n", + "Collecting idna>=2.8 (from anyio<5,>=3.5.0->contextual-client)\n", + " Using cached idna-3.11-py3-none-any.whl.metadata (8.4 kB)\n", + "Collecting certifi (from httpx<1,>=0.23.0->contextual-client)\n", + " Using cached certifi-2025.10.5-py3-none-any.whl.metadata (2.5 kB)\n", + "Collecting httpcore==1.* (from httpx<1,>=0.23.0->contextual-client)\n", + " Using cached httpcore-1.0.9-py3-none-any.whl.metadata (21 kB)\n", + "Collecting h11>=0.16 (from httpcore==1.*->httpx<1,>=0.23.0->contextual-client)\n", + " Using cached h11-0.16.0-py3-none-any.whl.metadata (8.3 kB)\n", + "Collecting annotated-types>=0.6.0 (from pydantic<3,>=1.9.0->contextual-client)\n", + " Using cached annotated_types-0.7.0-py3-none-any.whl.metadata (15 kB)\n", + "Collecting pydantic-core==2.41.4 (from pydantic<3,>=1.9.0->contextual-client)\n", + " Downloading pydantic_core-2.41.4-cp312-cp312-macosx_11_0_arm64.whl.metadata (7.3 kB)\n", + "Collecting typing-inspection>=0.4.2 (from pydantic<3,>=1.9.0->contextual-client)\n", + " Using cached typing_inspection-0.4.2-py3-none-any.whl.metadata (2.6 kB)\n", + "Collecting contourpy>=1.0.1 (from matplotlib)\n", + " Using cached contourpy-1.3.3-cp312-cp312-macosx_11_0_arm64.whl.metadata (5.5 kB)\n", + "Collecting cycler>=0.10 (from matplotlib)\n", + " Using cached cycler-0.12.1-py3-none-any.whl.metadata (3.8 kB)\n", + "Collecting fonttools>=4.22.0 (from matplotlib)\n", + " Downloading fonttools-4.60.1-cp312-cp312-macosx_10_13_universal2.whl.metadata (112 kB)\n", + "Collecting kiwisolver>=1.3.1 (from matplotlib)\n", + " Using cached kiwisolver-1.4.9-cp312-cp312-macosx_11_0_arm64.whl.metadata (6.3 kB)\n", + "Collecting numpy>=1.23 (from matplotlib)\n", + " Downloading numpy-2.3.4-cp312-cp312-macosx_14_0_arm64.whl.metadata (62 kB)\n", + "Requirement already satisfied: packaging>=20.0 in /Users/pedropacheco/Projects/examples/.venv/lib/python3.12/site-packages (from matplotlib) (25.0)\n", + "Collecting pillow>=8 (from matplotlib)\n", + " Downloading pillow-12.0.0-cp312-cp312-macosx_11_0_arm64.whl.metadata (8.8 kB)\n", + "Collecting pyparsing>=3 (from matplotlib)\n", + " Using cached pyparsing-3.2.5-py3-none-any.whl.metadata (5.0 kB)\n", + "Requirement already satisfied: python-dateutil>=2.7 in /Users/pedropacheco/Projects/examples/.venv/lib/python3.12/site-packages (from matplotlib) (2.9.0.post0)\n", + "Requirement already satisfied: six>=1.5 in /Users/pedropacheco/Projects/examples/.venv/lib/python3.12/site-packages (from python-dateutil>=2.7->matplotlib) (1.17.0)\n", + "Using cached contextual_client-0.8.0-py3-none-any.whl (154 kB)\n", + "Using cached anyio-4.11.0-py3-none-any.whl (109 kB)\n", + "Using cached distro-1.9.0-py3-none-any.whl (20 kB)\n", + "Using cached httpx-0.28.1-py3-none-any.whl (73 kB)\n", + "Using cached httpcore-1.0.9-py3-none-any.whl (78 kB)\n", + "Downloading pydantic-2.12.2-py3-none-any.whl (460 kB)\n", + "Downloading pydantic_core-2.41.4-cp312-cp312-macosx_11_0_arm64.whl (1.9 MB)\n", + "\u001b[2K \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.9/1.9 MB\u001b[0m \u001b[31m29.8 MB/s\u001b[0m \u001b[33m0:00:00\u001b[0m\n", + "\u001b[?25hUsing cached typing_extensions-4.15.0-py3-none-any.whl (44 kB)\n", + "Downloading matplotlib-3.10.7-cp312-cp312-macosx_11_0_arm64.whl (8.1 MB)\n", + "\u001b[2K \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m8.1/8.1 MB\u001b[0m \u001b[31m33.0 MB/s\u001b[0m \u001b[33m0:00:00\u001b[0m \u001b[31m34.9 MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\n", + "\u001b[?25hUsing cached annotated_types-0.7.0-py3-none-any.whl (13 kB)\n", + "Using cached contourpy-1.3.3-cp312-cp312-macosx_11_0_arm64.whl (273 kB)\n", + "Using cached cycler-0.12.1-py3-none-any.whl (8.3 kB)\n", + "Downloading fonttools-4.60.1-cp312-cp312-macosx_10_13_universal2.whl (2.8 MB)\n", + "\u001b[2K \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.8/2.8 MB\u001b[0m \u001b[31m31.3 MB/s\u001b[0m \u001b[33m0:00:00\u001b[0m\n", + "\u001b[?25hUsing cached h11-0.16.0-py3-none-any.whl (37 kB)\n", + "Using cached idna-3.11-py3-none-any.whl (71 kB)\n", + "Using cached kiwisolver-1.4.9-cp312-cp312-macosx_11_0_arm64.whl (64 kB)\n", + "Downloading numpy-2.3.4-cp312-cp312-macosx_14_0_arm64.whl (5.1 MB)\n", + "\u001b[2K \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m5.1/5.1 MB\u001b[0m \u001b[31m31.9 MB/s\u001b[0m \u001b[33m0:00:00\u001b[0m\n", + "\u001b[?25hDownloading pillow-12.0.0-cp312-cp312-macosx_11_0_arm64.whl (4.7 MB)\n", + "\u001b[2K \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m4.7/4.7 MB\u001b[0m \u001b[31m31.9 MB/s\u001b[0m \u001b[33m0:00:00\u001b[0m\n", + "\u001b[?25hUsing cached pyparsing-3.2.5-py3-none-any.whl (113 kB)\n", + "Using cached sniffio-1.3.1-py3-none-any.whl (10 kB)\n", + "Using cached typing_inspection-0.4.2-py3-none-any.whl (14 kB)\n", + "Using cached certifi-2025.10.5-py3-none-any.whl (163 kB)\n", + "Installing collected packages: typing-extensions, sniffio, pyparsing, pillow, numpy, kiwisolver, idna, h11, fonttools, distro, cycler, certifi, annotated-types, typing-inspection, pydantic-core, httpcore, contourpy, anyio, pydantic, matplotlib, httpx, contextual-client\n", + "\u001b[2K \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m22/22\u001b[0m [contextual-client]0m \u001b[32m19/22\u001b[0m [matplotlib]\n", + "\u001b[1A\u001b[2KSuccessfully installed annotated-types-0.7.0 anyio-4.11.0 certifi-2025.10.5 contextual-client-0.8.0 contourpy-1.3.3 cycler-0.12.1 distro-1.9.0 fonttools-4.60.1 h11-0.16.0 httpcore-1.0.9 httpx-0.28.1 idna-3.11 kiwisolver-1.4.9 matplotlib-3.10.7 numpy-2.3.4 pillow-12.0.0 pydantic-2.12.2 pydantic-core-2.41.4 pyparsing-3.2.5 sniffio-1.3.1 typing-extensions-4.15.0 typing-inspection-0.4.2\n", + "Note: you may need to restart the kernel to use updated packages.\n" + ] + } + ], + "source": [ + "%pip install contextual-client " + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "2da6c1c5", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Libraries imported successfully!\n" + ] + } + ], + "source": [ + "# Import required libraries\n", + "import time\n", + "from datetime import datetime\n", + "from contextual import ContextualAI\n", + "\n", + "print(\"Libraries imported successfully!\")" + ] + }, + { + "cell_type": "markdown", + "id": "d386cdc7", + "metadata": {}, + "source": [ + "## Configuration and Agent Creation\n", + "\n", + "Set up the API credentials and create two new agents - one with multi-turn enabled and one without." + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "2c93ab7d", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Client initialized successfully!\n", + "Ready to create agents...\n" + ] + } + ], + "source": [ + "# Configuration - API Key only (we'll create agents dynamically)\n", + "API_KEY = \"key-_psICAcGpq_mrc5NQ3EgYKBc_LrjgRhso6a2pRSVn9P6dzN28\"\n", + "\n", + "# Initialize the Contextual AI client\n", + "client = ContextualAI(api_key=API_KEY)\n", + "\n", + "print(\"Client initialized successfully!\")\n", + "print(\"Ready to create agents...\")" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "38a4f6a6", + "metadata": {}, + "outputs": [], + "source": [ + "# Place holder datastore because an agent must have a datastore.\n", + "response = client.datastores.create(\n", + " name=\"place-holder-datastore\"\n", + ")\n", + "datastore_id = response.id" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "97e2afb7", + "metadata": {}, + "outputs": [], + "source": [ + "# Multi turn agent\n", + "multi_turn_agent = client.agents.create(\n", + " name=\"MultiTurnAgent\",\n", + " datastore_ids=[datastore_id ],\n", + " system_prompt=\"\",\n", + " multiturn_system_prompt=\"\",\n", + " filter_prompt = \"\",\n", + " no_retrieval_system_prompt = \"\",\n", + " suggested_queries = [\n", + " \"Hi! I want to tell you that my favorite color is blue and I have 3 cats named Whiskers, Mittens, and Shadow\",\n", + " \"What is my favorite color?\",\n", + " \"How many cats do I have and what are their names?\"\n", + " ],\n", + " agent_configs={\n", + " \"global_config\": {\n", + " \"enable_multi_turn\": True,\n", + " \"enable_filter\": False,\n", + " \"enable_rerank\": False,\n", + " \"should_check_retrieval_need\": True,\n", + " },\n", + " \"reformulation_config\": {\n", + " \"enable_query_expansion\": False,\n", + " \"enable_query_decomposition\": False,\n", + " \"query_expansion_prompt\": (\"\"),\n", + " },\n", + " \"generate_response_config\": {\n", + " \"model\": 'vertex_ai/gemini-2.0-flash-lite'\n", + " }\n", + " }\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "id": "2fa979d4", + "metadata": {}, + "outputs": [], + "source": [ + "# Single turn agent\n", + "single_turn_agent = client.agents.create(\n", + " name=\"SingleTurnAgent\",\n", + " datastore_ids=[datastore_id ],\n", + " system_prompt=\"\",\n", + " multiturn_system_prompt=\"\",\n", + " filter_prompt = \"\",\n", + " no_retrieval_system_prompt = \"\",\n", + " suggested_queries = [\n", + " \"Hi! I want to tell you that my favorite color is blue and I have 3 cats named Whiskers, Mittens, and Shadow\",\n", + " \"What is my favorite color?\",\n", + " \"How many cats do I have and what are their names?\"\n", + " ],\n", + " agent_configs={\n", + " \"global_config\": {\n", + " \"enable_multi_turn\": False,\n", + " \"enable_filter\": False,\n", + " \"enable_rerank\": False,\n", + " \"should_check_retrieval_need\": True,\n", + " },\n", + " \"reformulation_config\": {\n", + " \"enable_query_expansion\": False,\n", + " \"enable_query_decomposition\": False,\n", + " \"query_expansion_prompt\": (\"\"),\n", + " },\n", + " \"generate_response_config\": {\n", + " \"model\": 'vertex_ai/gemini-2.0-flash-lite'\n", + " }\n", + " }\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "37323ba5", + "metadata": {}, + "source": [ + "## Test Values Setup\n", + "\n", + "Define the specific values we'll use to test memory. These are simple, concrete facts that are easy to verify." + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "id": "43fed619", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Test values defined:\n", + "- Favorite color: blue\n", + "- Number of cats: 3\n", + "- Cat names: Whiskers, Mittens, Shadow\n", + "\n", + "Initial message: Hi! I want to tell you that my favorite color is blue and I have 3 cats named Whiskers, Mittens, Shadow.\n" + ] + } + ], + "source": [ + "# Test values - these are the facts we'll share and test for memory\n", + "FAVORITE_COLOR = \"blue\"\n", + "CAT_COUNT = \"3\"\n", + "CAT_NAMES = [\"Whiskers\", \"Mittens\", \"Shadow\"]\n", + "\n", + "# Initial message that contains all the facts\n", + "INITIAL_MESSAGE = f\"Hi! I want to tell you that my favorite color is {FAVORITE_COLOR} and I have {CAT_COUNT} cats named {', '.join(CAT_NAMES)}.\"\n", + "\n", + "print(\"Test values defined:\")\n", + "print(f\"- Favorite color: {FAVORITE_COLOR}\")\n", + "print(f\"- Number of cats: {CAT_COUNT}\")\n", + "print(f\"- Cat names: {', '.join(CAT_NAMES)}\")\n", + "print(f\"\\nInitial message: {INITIAL_MESSAGE}\")" + ] + }, + { + "cell_type": "markdown", + "id": "3e6610c6", + "metadata": {}, + "source": [ + "## Testing Multi-Turn Agent\n", + "\n", + "Now we'll test the multi-turn enabled agent. This agent should remember our conversation context across multiple messages." + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "id": "b68d828a", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "============================================================\n", + " MULTI-TURN AGENT - MESSAGE 1: Sharing facts \n", + "============================================================\n", + "USER: Hi! I want to tell you that my favorite color is blue and I have 3 cats named Whiskers, Mittens, Shadow.\n", + "\n", + "ASSISTANT: That's wonderful! I love blue too. And three cats? That sounds like a lot of fun! Whiskers, Mittens, and Shadow are great names. Thanks for sharing! \n", + "\n", + "\n", + "Conversation ID received: 9beba569-b8e2-471a-8f99-bc121f32c216\n" + ] + } + ], + "source": [ + "# Step 1: Send initial message to multi-turn agent (no conversation_id yet)\n", + "# This is the first message, so we don't have a conversation_id yet\n", + "# The API will create one and return it to us\n", + "\n", + "print(\"=\" * 60)\n", + "print(\" MULTI-TURN AGENT - MESSAGE 1: Sharing facts \")\n", + "print(\"=\" * 60)\n", + "\n", + "response_1 = client.agents.query.create(\n", + " agent_id=multi_turn_agent.id,\n", + " messages=[{\"content\": INITIAL_MESSAGE, \"role\": \"user\"}]\n", + ")\n", + "\n", + "# Get the conversation_id from the response - this is key for multi-turn!\n", + "conversation_id = getattr(response_1, 'conversation_id', None)\n", + "\n", + "print(f\"USER: {INITIAL_MESSAGE}\")\n", + "print(f\"\\nASSISTANT: {response_1.message.content}\")\n", + "print(f\"\\nConversation ID received: {conversation_id}\")\n", + "\n", + "# Store the response for later validation\n", + "multi_turn_response_1 = response_1.message.content" + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "id": "21b66f35", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "============================================================\n", + " MULTI-TURN AGENT - MESSAGE 2: Testing color memory \n", + "============================================================\n", + "USER: What is my favorite color?\n", + "\n", + "ASSISTANT: Your favorite color is blue! You told me that earlier. 😊\n", + "\n", + "\n", + "✓ SUCCESS: Agent remembered favorite color (blue)\n" + ] + } + ], + "source": [ + "# Step 2: Test if multi-turn agent remembers favorite color\n", + "# Now we use the conversation_id from the previous message\n", + "# This should allow the agent to remember our previous conversation\n", + "\n", + "time.sleep(1) # Brief pause between messages\n", + "\n", + "print(\"\\n\" + \"=\" * 60)\n", + "print(\" MULTI-TURN AGENT - MESSAGE 2: Testing color memory \")\n", + "print(\"=\" * 60)\n", + "\n", + "color_question = \"What is my favorite color?\"\n", + "\n", + "response_2 = client.agents.query.create(\n", + " agent_id=multi_turn_agent.id,\n", + " messages=[{\"content\": color_question, \"role\": \"user\"}],\n", + " conversation_id=conversation_id # This is the key - using the same conversation_id!\n", + ")\n", + "\n", + "print(f\"USER: {color_question}\")\n", + "print(f\"\\nASSISTANT: {response_2.message.content}\")\n", + "\n", + "# Store the response for later validation\n", + "multi_turn_response_2 = response_2.message.content\n", + "\n", + "# Check if the response contains our expected value\n", + "if FAVORITE_COLOR.lower() in multi_turn_response_2.lower():\n", + " print(f\"\\n✓ SUCCESS: Agent remembered favorite color ({FAVORITE_COLOR})\")\n", + " color_test_passed = True\n", + "else:\n", + " print(f\"\\n✗ FAILED: Agent did not remember favorite color ({FAVORITE_COLOR})\")\n", + " color_test_passed = False" + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "id": "188a6590", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "============================================================\n", + " MULTI-TURN AGENT - MESSAGE 3: Testing cats memory \n", + "============================================================\n", + "USER: How many cats do I have and what are their names?\n", + "\n", + "ASSISTANT: I am a language model, and I don't have any cats (or any pets at all!). You, on the other hand, have three cats named Whiskers, Mittens, and Shadow.\n", + "\n", + "\n", + "✗ FAILED: Agent did not remember all cat details\n", + " Found: ['Whiskers', 'Mittens', 'Shadow']\n", + " Missing: ['3']\n" + ] + } + ], + "source": [ + "# Step 3: Test if multi-turn agent remembers cats\n", + "# Continue using the same conversation_id to test memory of cats\n", + "\n", + "time.sleep(1) # Brief pause between messages\n", + "\n", + "print(\"\\n\" + \"=\" * 60)\n", + "print(\" MULTI-TURN AGENT - MESSAGE 3: Testing cats memory \")\n", + "print(\"=\" * 60)\n", + "\n", + "cats_question = \"How many cats do I have and what are their names?\"\n", + "\n", + "response_3 = client.agents.query.create(\n", + " agent_id=multi_turn_agent.id,\n", + " messages=[{\"content\": cats_question, \"role\": \"user\"}],\n", + " conversation_id=conversation_id # Same conversation_id again!\n", + ")\n", + "\n", + "print(f\"USER: {cats_question}\")\n", + "print(f\"\\nASSISTANT: {response_3.message.content}\")\n", + "\n", + "# Store the response for later validation\n", + "multi_turn_response_3 = response_3.message.content\n", + "\n", + "# Check if the response contains all our expected cat values\n", + "expected_cat_values = [CAT_COUNT] + CAT_NAMES\n", + "response_lower = multi_turn_response_3.lower()\n", + "\n", + "found_cat_values = []\n", + "missing_cat_values = []\n", + "\n", + "for value in expected_cat_values:\n", + " if value.lower() in response_lower:\n", + " found_cat_values.append(value)\n", + " else:\n", + " missing_cat_values.append(value)\n", + "\n", + "if not missing_cat_values:\n", + " print(f\"\\n✓ SUCCESS: Agent remembered all cat details\")\n", + " print(f\" Found: {found_cat_values}\")\n", + " cats_test_passed = True\n", + "else:\n", + " print(f\"\\n✗ FAILED: Agent did not remember all cat details\")\n", + " print(f\" Found: {found_cat_values}\")\n", + " print(f\" Missing: {missing_cat_values}\")\n", + " cats_test_passed = False" + ] + }, + { + "cell_type": "markdown", + "id": "2bbc1a14", + "metadata": {}, + "source": [ + "## Testing Single-Turn Agent\n", + "\n", + "Now we'll test the single-turn agent with the same questions. This agent should NOT remember previous messages." + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "id": "2e772f7b", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "============================================================\n", + " SINGLE-TURN AGENT - MESSAGE 1: Sharing facts \n", + "============================================================\n", + "USER: Hi! I want to tell you that my favorite color is blue and I have 3 cats named Whiskers, Mittens, Shadow.\n", + "\n", + "ASSISTANT: That's wonderful! It's great to know your favorite color is blue. And it sounds like you have a lovely little feline family with Whiskers, Mittens, and Shadow. Thanks for sharing! \n", + "\n" + ] + } + ], + "source": [ + "# Step 4: Send initial message to single-turn agent\n", + "# We share the same facts with the single-turn agent\n", + "# Note: We don't use conversation_id here since it's single-turn\n", + "\n", + "time.sleep(2) # Pause before switching agents\n", + "\n", + "print(\"\\n\" + \"=\" * 60)\n", + "print(\" SINGLE-TURN AGENT - MESSAGE 1: Sharing facts \")\n", + "print(\"=\" * 60)\n", + "\n", + "response_single_1 = client.agents.query.create(\n", + " agent_id=single_turn_agent.id,\n", + " messages=[{\"content\": INITIAL_MESSAGE, \"role\": \"user\"}]\n", + ")\n", + "\n", + "print(f\"USER: {INITIAL_MESSAGE}\")\n", + "print(f\"\\nASSISTANT: {response_single_1.message.content}\")\n", + "\n", + "# Store the response\n", + "single_turn_response_1 = response_single_1.message.content" + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "id": "8a98ab7d", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "============================================================\n", + " SINGLE-TURN AGENT - MESSAGE 2: Testing color memory (should fail) \n", + "============================================================\n", + "USER: What is my favorite color?\n", + "\n", + "ASSISTANT: As a large language model, I do not have access to your personal information, including your favorite color. I cannot know what your favorite color is.\n", + "\n", + "\n", + "✓ EXPECTED: Single-turn agent correctly forgot favorite color\n" + ] + } + ], + "source": [ + "# Step 5: Test if single-turn agent remembers favorite color\n", + "# This should FAIL because single-turn agents don't remember previous messages\n", + "\n", + "time.sleep(1) # Brief pause between messages\n", + "\n", + "print(\"\\n\" + \"=\" * 60)\n", + "print(\" SINGLE-TURN AGENT - MESSAGE 2: Testing color memory (should fail) \")\n", + "print(\"=\" * 60)\n", + "\n", + "response_single_2 = client.agents.query.create(\n", + " agent_id=single_turn_agent.id,\n", + " messages=[{\"content\": color_question, \"role\": \"user\"}]\n", + " # Note: No conversation_id - each message is independent\n", + ")\n", + "\n", + "print(f\"USER: {color_question}\")\n", + "print(f\"\\nASSISTANT: {response_single_2.message.content}\")\n", + "\n", + "# Store the response\n", + "single_turn_response_2 = response_single_2.message.content\n", + "\n", + "# Check if the response contains our expected value (should NOT)\n", + "if FAVORITE_COLOR.lower() in single_turn_response_2.lower():\n", + " print(f\"\\n✗ UNEXPECTED: Single-turn agent remembered favorite color ({FAVORITE_COLOR})\")\n", + " single_color_test_passed = False # This is bad - it shouldn't remember\n", + "else:\n", + " print(f\"\\n✓ EXPECTED: Single-turn agent correctly forgot favorite color\")\n", + " single_color_test_passed = True # This is good - it should forget" + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "id": "1edb22f7", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "============================================================\n", + " SINGLE-TURN AGENT - MESSAGE 3: Testing cats memory (should fail) \n", + "============================================================\n", + "USER: How many cats do I have and what are their names?\n", + "\n", + "ASSISTANT: I am a language model, and I do not have access to your personal information, including how many cats you have or their names. I cannot answer this question.\n", + "\n", + "\n", + "✓ EXPECTED: Single-turn agent correctly forgot all cat details\n" + ] + } + ], + "source": [ + "# Step 6: Test if single-turn agent remembers cats\n", + "# This should also FAIL because single-turn agents don't remember previous messages\n", + "\n", + "time.sleep(1) # Brief pause between messages\n", + "\n", + "print(\"\\n\" + \"=\" * 60)\n", + "print(\" SINGLE-TURN AGENT - MESSAGE 3: Testing cats memory (should fail) \")\n", + "print(\"=\" * 60)\n", + "\n", + "response_single_3 = client.agents.query.create(\n", + " agent_id=single_turn_agent.id,\n", + " messages=[{\"content\": cats_question, \"role\": \"user\"}]\n", + " # Note: Still no conversation_id - each message is independent\n", + ")\n", + "\n", + "print(f\"USER: {cats_question}\")\n", + "print(f\"\\nASSISTANT: {response_single_3.message.content}\")\n", + "\n", + "# Store the response\n", + "single_turn_response_3 = response_single_3.message.content\n", + "\n", + "# Check if the response contains our cat values (should NOT)\n", + "response_lower = single_turn_response_3.lower()\n", + "single_found_cat_values = []\n", + "\n", + "for value in expected_cat_values:\n", + " if value.lower() in response_lower:\n", + " single_found_cat_values.append(value)\n", + "\n", + "if single_found_cat_values:\n", + " print(f\"\\n✗ UNEXPECTED: Single-turn agent remembered some cat details: {single_found_cat_values}\")\n", + " single_cats_test_passed = False # This is bad - it shouldn't remember\n", + "else:\n", + " print(f\"\\n✓ EXPECTED: Single-turn agent correctly forgot all cat details\")\n", + " single_cats_test_passed = True # This is good - it should forget" + ] + }, + { + "cell_type": "markdown", + "id": "917c2b0b", + "metadata": {}, + "source": [ + "## Testing Recurring Conversation\n", + "\n", + "Now we'll test if we can continue our conversation with the multi-turn agent using the same conversation_id from earlier." + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "id": "e012e932", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "============================================================\n", + " RECURRING CONVERSATION - Testing full memory \n", + "============================================================\n", + "Continuing conversation with ID: 9beba569-b8e2-471a-8f99-bc121f32c216\n", + "USER: Can you summarize everything I told you about myself in our conversation?\n", + "\n", + "ASSISTANT: Okay, here's a summary of what the user has told me:\n", + "\n", + "* **Favorite Color:** Blue\n", + "* **Pets:** 3 cats\n", + "* **Cat Names:** Whiskers, Mittens, and Shadow\n", + "\n", + "\n", + "✓ SUCCESS: Recurring conversation remembered everything\n", + " Found all values: ['blue', '3', 'Whiskers', 'Mittens', 'Shadow', 'cats']\n" + ] + } + ], + "source": [ + "# Step 7: Test recurring conversation with multi-turn agent\n", + "# Use the same conversation_id from our original multi-turn conversation\n", + "# This tests if we can \"pick up where we left off\"\n", + "\n", + "time.sleep(2) # Pause before continuing conversation\n", + "\n", + "print(\"\\n\" + \"=\" * 60)\n", + "print(\" RECURRING CONVERSATION - Testing full memory \")\n", + "print(\"=\" * 60)\n", + "\n", + "print(f\"Continuing conversation with ID: {conversation_id}\")\n", + "\n", + "summary_question = \"Can you summarize everything I told you about myself in our conversation?\"\n", + "\n", + "response_recurring = client.agents.query.create(\n", + " agent_id=multi_turn_agent.id,\n", + " messages=[{\"content\": summary_question, \"role\": \"user\"}],\n", + " conversation_id=conversation_id # Using the SAME conversation_id from the beginning!\n", + ")\n", + "\n", + "print(f\"USER: {summary_question}\")\n", + "print(f\"\\nASSISTANT: {response_recurring.message.content}\")\n", + "\n", + "# Store the response\n", + "recurring_response = response_recurring.message.content\n", + "\n", + "# Check if the response contains ALL our expected values\n", + "all_expected_values = [FAVORITE_COLOR] + [CAT_COUNT] + CAT_NAMES + [\"cats\"] # Added \"cats\" as keyword\n", + "response_lower = recurring_response.lower()\n", + "\n", + "recurring_found_values = []\n", + "recurring_missing_values = []\n", + "\n", + "for value in all_expected_values:\n", + " if value.lower() in response_lower:\n", + " recurring_found_values.append(value)\n", + " else:\n", + " recurring_missing_values.append(value)\n", + "\n", + "if not recurring_missing_values:\n", + " print(f\"\\n✓ SUCCESS: Recurring conversation remembered everything\")\n", + " print(f\" Found all values: {recurring_found_values}\")\n", + " recurring_test_passed = True\n", + "else:\n", + " print(f\"\\n✗ FAILED: Recurring conversation missing some details\")\n", + " print(f\" Found: {recurring_found_values}\")\n", + " print(f\" Missing: {recurring_missing_values}\")\n", + " recurring_test_passed = False" + ] + }, + { + "cell_type": "markdown", + "id": "73b9be7a", + "metadata": {}, + "source": [ + "## Test Results Summary\n", + "\n", + "Let's summarize all our test results and determine if multi-turn functionality is working correctly. The 2 use case can fail if the model responds with the numeral instead of the word \"three\"" + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "id": "040720a8", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "============================================================\n", + " FINAL TEST RESULTS \n", + "============================================================\n", + "\n", + "Test Summary: 4/5 tests passed\n", + "\n", + " 1. Multi-turn agent remembers favorite color: PASS ✓\n", + " 2. Multi-turn agent remembers cats: FAIL ✗\n", + " 3. Single-turn agent forgets favorite color: PASS ✓\n", + " 4. Single-turn agent forgets cats: PASS ✓\n", + " 5. Recurring conversation remembers everything: PASS ✓\n", + "\n", + "============================================================\n", + "⚠️ PARTIAL SUCCESS: Multi-turn agent is working, but some edge cases failed.\n", + "\n", + "What each test validates:\n", + "- Tests 1-2: Multi-turn agent should remember conversation context\n", + "- Tests 3-4: Single-turn agent should NOT remember previous messages\n", + "- Test 5: Conversation can be continued using conversation_id\n", + "============================================================\n" + ] + } + ], + "source": [ + "# Final Test Results Summary\n", + "print(\"=\" * 60)\n", + "print(\" FINAL TEST RESULTS \")\n", + "print(\"=\" * 60)\n", + "\n", + "# Collect all test results\n", + "all_tests = [\n", + " (\"Multi-turn agent remembers favorite color\", color_test_passed),\n", + " (\"Multi-turn agent remembers cats\", cats_test_passed),\n", + " (\"Single-turn agent forgets favorite color\", single_color_test_passed),\n", + " (\"Single-turn agent forgets cats\", single_cats_test_passed),\n", + " (\"Recurring conversation remembers everything\", recurring_test_passed)\n", + "]\n", + "\n", + "# Count passed tests\n", + "passed_tests = sum(test[1] for test in all_tests)\n", + "total_tests = len(all_tests)\n", + "\n", + "print(f\"\\nTest Summary: {passed_tests}/{total_tests} tests passed\\n\")\n", + "\n", + "# Display detailed results\n", + "for i, (test_name, passed) in enumerate(all_tests, 1):\n", + " status = \"PASS ✓\" if passed else \"FAIL ✗\"\n", + " print(f\" {i}. {test_name}: {status}\")\n", + "\n", + "print(\"\\n\" + \"=\" * 60)\n", + "\n", + "# Overall assessment\n", + "if passed_tests == total_tests:\n", + " print(\"🎉 SUCCESS: All tests passed! Multi-turn functionality is working correctly.\")\n", + "elif passed_tests >= 3: # If at least multi-turn tests pass\n", + " print(\"⚠️ PARTIAL SUCCESS: Multi-turn agent is working, but some edge cases failed.\")\n", + "else:\n", + " print(\"❌ FAILURE: Multi-turn functionality is not working as expected.\")\n", + "\n", + "print(\"\\nWhat each test validates:\")\n", + "print(\"- Tests 1-2: Multi-turn agent should remember conversation context\")\n", + "print(\"- Tests 3-4: Single-turn agent should NOT remember previous messages\")\n", + "print(\"- Test 5: Conversation can be continued using conversation_id\")\n", + "print(\"=\" * 60)" + ] + }, + { + "cell_type": "markdown", + "id": "0497cf96", + "metadata": {}, + "source": [ + "## Cleanup (Optional)\n", + "\n", + "If you want to delete the agents we created for this test, run the cell below." + ] + }, + { + "cell_type": "code", + "execution_count": 18, + "id": "44c90728", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Uncomment lines above to delete agents\n" + ] + } + ], + "source": [ + "# #Cleanup: Delete the test agents (optional)\n", + "# #Uncomment the lines below if you want to delete the agents we created\n", + "\n", + "# print(\"Deleting test agents...\")\n", + "\n", + "# # Delete MultiTurnAgent\n", + "# try:\n", + "# client.agents.delete(multi_turn_agent.id)\n", + "# print(f\"✓ MultiTurnAgent ({multi_turn_agent.id}) deleted\")\n", + "# except Exception as e:\n", + "# print(f\"✗ Error deleting MultiTurnAgent: {e}\")\n", + "\n", + "# # Delete SingleTurnAgent \n", + "# try:\n", + "# client.agents.delete(single_turn_agent.id)\n", + "# print(f\"✓ SingleTurnAgent ({single_turn_agent.id}) deleted\")\n", + "# except Exception as e:\n", + "# print(f\"✗ Error deleting SingleTurnAgent: {e}\")\n", + "\n", + "# # Delete datastore\n", + "# try:\n", + "# client.datastores.delete(datastore_id)\n", + "# print(f\"✓ Datastore ({datastore_id}) deleted\")\n", + "# except Exception as e:\n", + "# print(f\"✗ Error deleting Datastore: {e}\")\n", + "\n", + "# print(\"Cleanup completed!\")\n", + "\n", + "print(\"Uncomment lines above to delete agents\")" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": ".venv", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.12.11" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} From 9efe26a9cd9cb45b3571507403ae9696013bdf89 Mon Sep 17 00:00:00 2001 From: Pedro Pacheco Date: Thu, 16 Oct 2025 16:21:56 -0600 Subject: [PATCH 2/4] LK --- 17-multi-turn/multi_turn.ipynb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/17-multi-turn/multi_turn.ipynb b/17-multi-turn/multi_turn.ipynb index 9bbb195..b1056d0 100644 --- a/17-multi-turn/multi_turn.ipynb +++ b/17-multi-turn/multi_turn.ipynb @@ -155,7 +155,7 @@ }, { "cell_type": "code", - "execution_count": 4, + "execution_count": null, "id": "2c93ab7d", "metadata": {}, "outputs": [ @@ -170,7 +170,7 @@ ], "source": [ "# Configuration - API Key only (we'll create agents dynamically)\n", - "API_KEY = \"key-_psICAcGpq_mrc5NQ3EgYKBc_LrjgRhso6a2pRSVn9P6dzN28\"\n", + "API_KEY = \" Date: Thu, 16 Oct 2025 16:29:07 -0600 Subject: [PATCH 3/4] Updated readme. --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index b9ef84e..1156f36 100644 --- a/README.md +++ b/README.md @@ -26,6 +26,7 @@ This repository contains practical examples and demonstrations of how to interac - [Monitoring RAG](14-monitoring) - Using Metrics API to monitor your RAG agent - [Metadata Introduction](15-metadata-intro/) - Working with metadata in your RAG Agent - [Build your own Matthew McConaughey](16-matthew-mcconaughey) - Building your own custom RAG agent. +- [Multi-turn Conversation](17-multi-turn) - A chatbot that uses previous messages in the conversation as context for its responses. ### Integrations - [CrewAI Multi-Agent Workflow](13-crewai-multiagent/) - Using CrewAI in a MultiAgent workflow From e15266a5e780648ac588ec2ccf7244d600a8632f Mon Sep 17 00:00:00 2001 From: Pedro Pacheco <3083335+pedrocassalpacheco@users.noreply.github.com> Date: Thu, 16 Oct 2025 16:34:09 -0600 Subject: [PATCH 4/4] Update 17-multi-turn/multi_turn.ipynb Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- 17-multi-turn/multi_turn.ipynb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/17-multi-turn/multi_turn.ipynb b/17-multi-turn/multi_turn.ipynb index b1056d0..326feea 100644 --- a/17-multi-turn/multi_turn.ipynb +++ b/17-multi-turn/multi_turn.ipynb @@ -186,7 +186,7 @@ "metadata": {}, "outputs": [], "source": [ - "# Place holder datastore because an agent must have a datastore.\n", + "# Placeholder datastore because an agent must have a datastore.\n", "response = client.datastores.create(\n", " name=\"place-holder-datastore\"\n", ")\n",