Reorganize documentation with Hugo and streamline README

[Copilot]

README was 700+ lines with mixed content. Documentation files scattered and inconsistently organized. Missing docs for metadata filtering in similarity search.

Changes

Hugo Documentation Site

• Added hugo-book theme as submodule at docs/themes/book
• Created 50 markdown files across 8 sections: getting-started, concepts, guides, api, deployment, development, reference
• Documented metadata filtering (metadata_path/metadata_value query parameters for excluding documents in similarity search)
• Comprehensive API reference for similarity endpoints with all query parameters
• All existing documentation content migrated and preserved

README Streamlining

• Reduced from 709 to 195 lines (72%)
• Enhanced features section: 15 features in 4 categories (Core, Multi-User, LLM Management, Data Validation)
• Added 7-step quick-start with curl commands (user creation → LLM instance → project → embeddings → similarity search → metadata filtering)
• Links to full documentation site throughout

Deployment

• GitHub Actions workflow at .github/workflows/deploy-docs.yml
• Auto-deploys to GitHub Pages on main branch docs changes
• Requires enabling GitHub Pages with "GitHub Actions" source in repository settings

Cleanup

• Deleted 5 old doc files (ChatGPT.md, LLM_SERVICE_REFACTORING.md, METADATA_SCHEMA_EXAMPLES.md, PERFORMANCE_OPTIMIZATION.md, PUBLIC_ACCESS.md)
• Added Hugo build artifacts to .gitignore

Example: New Quick Start in README

# 7. Filter by Metadata
# Exclude documents from the same author
curl "http://localhost:8880/v1/similars/alice/my-texts/doc1?threshold=0.7&metadata_path=author&metadata_value=John%20Doe" \
  -H "Authorization: Bearer ALICE_VDB_KEY"

Documentation will be available at https://mpilhlt.github.io/dhamps-vdb/ after merge.

Original prompt

Documentation Reorganization with Hugo

Objective

Reorganize the project documentation by creating a Hugo-based documentation site and streamlining the README to be concise with prominent feature list and quick setup guide.

Current Issues

• README is overloaded with too much detailed information (700+ lines)
• Documentation files in docs/ are inconsistently organized
• Mix of historical development notes with actual user documentation
• Missing documentation for some features (e.g., metadata filtering in similarity search)
• No clear quick-start guide with curl commands for common setups

Required Changes

1. Hugo Setup

Create a new Hugo documentation site in docs/:

• Use Hugo Book theme (hugo-book) as a git submodule
• Set up docs/config.toml with proper configuration:

  baseURL = 'https://mpilhlt.github.io/dhamps-vdb/'
  languageCode = 'en-us'
  title = 'dhamps-vdb Documentation'
  theme = 'book'
  
  [params]
    BookRepo = 'https://github.com/mpilhlt/dhamps-vdb'
    BookEditPath = 'edit/main'
    BookSearch = true

2. New README Structure

Rewrite README.md to be concise (~200-300 lines) with:

Keep sections:

• Title, badges, and brief introduction

• Enhanced Features section with comprehensive list:

  • Multi-user support with role-based access control (admin, owner, reader, editor)
  • Project sharing and collaboration features
  • Public access mode for projects
  • LLM service management (definitions and instances with encrypted API keys)
  • Multiple embedding configurations support
  • Automatic dimension validation
  • JSON Schema-based metadata validation
  • Metadata filtering in similarity search
  • PostgreSQL with pgvector backend
  • RESTful API with OpenAPI documentation
  • Project ownership transfer
  • PATCH support for partial updates
  • Docker deployment ready
  • Comprehensive test coverage

• Quick Start section with practical curl examples showing a complete workflow:

  # 1. Start with Docker (includes PostgreSQL)
  ./docker-setup.sh
  docker-compose up -d
  
  # 2. Create a user
  curl -X POST http://localhost:8880/v1/users \
    -H "Authorization: Bearer YOUR_ADMIN_KEY" \
    -H "Content-Type: application/json" \
    -d '{"user_handle": "alice", "name": "Alice Smith"}'
  # Save the returned vdb_key!
  
  # 3. Create an LLM service instance (using system definition)
  curl -X PUT http://localhost:8880/v1/llm-instances/alice/my-openai \
    -H "Authorization: Bearer ALICE_VDB_KEY" \
    -H "Content-Type: application/json" \
    -d '{
      "definition_owner": "_system",
      "definition_handle": "openai-large",
      "api_key_encrypted": "YOUR_OPENAI_API_KEY"
    }'
  
  # 4. Create a project
  curl -X POST http://localhost:8880/v1/projects/alice \
    -H "Authorization: Bearer ALICE_VDB_KEY" \
    -H "Content-Type: application/json" \
    -d '{
      "project_handle": "my-texts",
      "description": "My text embeddings",
      "instance_owner": "alice",
      "instance_handle": "my-openai"
    }'
  
  # 5. Upload embeddings
  curl -X POST http://localhost:8880/v1/embeddings/alice/my-texts \
    -H "Authorization: Bearer ALICE_VDB_KEY" \
    -H "Content-Type: application/json" \
    -d '{
      "embeddings": [{
        "text_id": "doc1",
        "instance_handle": "my-openai",
        "vector": [0.1, 0.2, ...],
        "vector_dim": 3072,
        "metadata": {"author": "John Doe", "year": 2024}
      }]
    }'
  
  # 6. Find similar documents
  curl http://localhost:8880/v1/similars/alice/my-texts/doc1?threshold=0.7&limit=5 \
    -H "Authorization: Bearer ALICE_VDB_KEY"
  
  # 7. Filter by metadata (exclude same author)
  curl "http://localhost:8880/v1/similars/alice/my-texts/doc1?threshold=0.7&metadata_path=author&metadata_value=John%20Doe" \
    -H "Authorization: Bearer ALICE_VDB_KEY"

• Brief "Getting Started" pointing to full docs

• Link to documentation site: 📚 **[Full Documentation](https://mpilhlt.github.io/dhamps-vdb/)**

• License and Contributing (brief)

Remove from README (move to Hugo site):

• Detailed compilation instructions
• All testing details
• Detailed Docker instructions (keep link to DOCKER.md)
• All API endpoint tables (move to Hugo)
• Detailed validation examples
• Project sharing details
• Ownership transfer details
• PATCH method details
• Complete metadata schema examples
• Code structure
• Detailed roadmap
• All configuration details

3. Hugo Content Structure

Create content files in docs/content/:

docs/content/
├── _index.md                          # Documentation home
├── getting-started/
│   ├── _index.md
│   ├── installation.md                # Compiling, dependencies, building from source
│   ├── docker.md                      # From DOCKER.md
│   ├── configuration.md               # Environment variables from options.go, .env setup
│   ├── quick-start.md                 # Expanded curl tutorial with explanations
│   └── first-project.md               # St...

</details>



<!-- START COPILOT CODING AGENT SUFFIX -->

*This pull request was created from Copilot chat.*
>

<!-- START COPILOT CODING AGENT TIPS -->
---

💬 We'd love your input! Share your thoughts on Copilot coding agent in our [2 minute survey](https://gh.io/copilot-coding-agent-survey).

[awagner-mainz]

@copilot I plan on renaming the software and the repository (to "embapi" for "Embeddings API"). Should I do this before or after merging this PR?

[Copilot]

@copilot I plan on renaming the software and the repository (to "embapi" for "Embeddings API"). Should I do this before or after merging this PR?

I recommend merging this PR first, then renaming afterward. Here's why:

Merge first:

• This PR is complete and tested with current naming
• Renaming after merge keeps changes isolated and easier to review
• You can update all documentation references (README, Hugo site, URLs) in a single focused rename PR
• Less risk of merge conflicts or having to redo work

After merge, the rename will need updates to:

• Repository name (GitHub settings)
• baseURL in docs/config.toml (change mpilhlt.github.io/dhamps-vdb to mpilhlt.github.io/embapi)
• All documentation links pointing to the GitHub Pages URL
• README title and references
• Go module path in go.mod (if applicable)

The documentation structure created in this PR will make the rename easier since everything is centralized in Hugo.

[awagner-mainz]

Looks good.