opensearch_knowledge_base

Author: michaelcizmar Version: 0.0.1 Type: tool

Description

If you are trying to build AI Agent with context retrieving capability and you are not primarily using Dify's Knowledge Base, you can use Dify's External Knowledge Base to connect with the RAG solution you prefer. This plugin will help you deploy your OpenSearch Knowledge Base client as an endpoint so Dify External Knowledge Base can seamlessly connect with it.

Before we start, make sure you have set up your OpenSearch cluster and have your data indexed. You can learn more about OpenSearch at https://opensearch.org/.

Prerequisites

• An OpenSearch cluster (self-hosted or managed service like Amazon OpenSearch Service)
• An index or alias in OpenSearch containing your knowledge base documents
• Appropriate authentication credentials for your OpenSearch cluster

Authentication Methods

This plugin supports multiple authentication methods:

1. Basic Auth (Username/Password): Standard HTTP basic authentication
2. API Key: OpenSearch API key authentication
3. AWS SigV4: For Amazon OpenSearch Service using AWS credentials
4. No Authentication: For local development or unsecured clusters

Configuration

Here's what you need to configure:

1. OpenSearch Connection Settings:

   • Host: Your OpenSearch cluster endpoint (e.g., search.example.com)
   • Port: Port number (default: 443 for SSL, 9200 for non-SSL)
   • SSL: Whether to use SSL/TLS (recommended: Yes)
   • Verify Certificates: Whether to verify SSL certificates

2. Authentication Method: Choose one of the supported methods and provide the required credentials

3. Field Names (optional, with defaults):

   • Content Field: Field name containing document content (default: content)
   • Title Field: Field name containing document title (default: title)
   • Embedding Field: Field name containing vector embeddings (default: embedding)

Knowledge ID Mapping

The knowledge_id parameter from Dify maps directly to your OpenSearch index or alias name. For example:

• If your index is named my-knowledge-base, use my-knowledge-base as the knowledge_id
• If you're using an alias, use the alias name as the knowledge_id

Setup Steps

Follow these steps to connect your Dify Knowledge Base with OpenSearch:

1. Set up the OpenSearch Endpoint:

   • Go to the plugin endpoint configuration
   • Enter your OpenSearch cluster host and port
   • Configure SSL settings
   • Select your authentication method and provide credentials
   • Optionally configure field names if your schema differs from defaults

2. Copy the Endpoint URL:

   • After creating the endpoint, copy its URL
   • The endpoint will be available at /retrieval

3. Configure Dify External Knowledge Base:

   • Go to Dify Knowledge Base settings
   • Click on "External Knowledge API" in the top right corner
   • Paste the endpoint URL (without /retrieval) into the API Endpoint field
   • Give the endpoint a name
   • IMPORTANT: Remove /retrieval from the URL when configuring in Dify!
   • For API Key, you can enter any value if no authorization is configured on the endpoint

4. Connect to External Knowledge Base:

   • Click "Connect to an external knowledge base"
   • Choose the external knowledge API you just created
   • Enter your OpenSearch index/alias name as the Knowledge ID
   • Configure top_k and score threshold before connecting

5. Test Retrieval:

   • Try a retrieval test to verify the connection
   • You should see chunks retrieved from your OpenSearch index

Security Notes

• Endpoint Security: If you're not using endpoint-level authorization, make sure your endpoint URL is kept secret
• Credentials: All credentials are stored securely by Dify's credential management system
• Network Security: Ensure your OpenSearch cluster is properly secured with network policies and authentication

Troubleshooting

• Connection Errors: Verify your OpenSearch host and port are correct, and that the cluster is accessible
• Authentication Failures: Double-check your credentials match the selected authentication method
• Index Not Found: Ensure the knowledge_id matches an existing index or alias name in your OpenSearch cluster
• No Results: Check that your index contains documents and that the field names match your configuration