Add OpenAPI search engine example in Go

[buger]

Summary

Complete Go implementation of semantic search for OpenAPI specifications, based on probe's architecture. Demonstrates tokenization, stemming, BM25 ranking, and natural language query processing.

Features

Core Search Engine

• ✅ Tokenizer with CamelCase splitting (JWTAuthentication → ["jwt", "authentication"])
• ✅ Porter2 stemming for word variant matching (authenticate matches authentication)
• ✅ Stop word filtering (~120 words) - handles natural language queries
• ✅ BM25 ranking with parallel scoring using goroutines
• ✅ YAML & JSON OpenAPI spec support

Natural Language Support

• ✅ Questions: "How do I authenticate a user?" → extracts ["authenticate", "user"]
• ✅ Statements: "I want to create a payment" → extracts ["create", "payment"]
• ✅ Keywords: "user authentication" → works as expected

Testing

• ✅ 8 comprehensive test suites with 40+ test cases
• ✅ 5 real-world API fixtures (GitHub, Stripe, Slack, Twilio, Petstore)
• ✅ ~60 test endpoints covering diverse OpenAPI patterns
• ✅ All tests passing - production ready

Implementation

examples/openapi-search-go/
├── tokenizer/          # CamelCase, stemming, stop words
├── ranker/             # BM25 algorithm
├── search/             # OpenAPI parser & engine
├── fixtures/           # Test OpenAPI specs
├── main.go             # CLI interface
└── *_test.go           # Comprehensive tests

Documentation (8 guides, ~4000 lines)

1. README.md - Overview and usage examples
2. QUICKSTART.md - 5-minute getting started
3. ARCHITECTURE.md - Probe → Go implementation mapping
4. PROBE_RESEARCH.md - Deep dive into probe's search (400+ lines)
5. TEST_GUIDE.md - Complete testing documentation
6. TOKENIZATION_PROOF.md - Proof that stemming works
7. NLP_FEATURES.md - Stop words and natural language
8. PROJECT_SUMMARY.md - Executive summary

Example Usage

cd examples/openapi-search-go

# Natural language query
go run main.go "How do I authenticate a user?"
# → POST /auth/login (score: 5.27)
# Matched terms: user, authenticate, authent

# Keyword search
go run main.go "payment refund"
# → POST /charges/{id}/refund (score: 4.07)

# Run tests
go test -v
# PASS - all 40+ tests

Key Algorithms Demonstrated

1. Tokenization Pipeline

"How can I authenticate a user?"
  ↓ Split & filter stop words
["authenticate", "user"]
  ↓ Stem
["authenticate", "authent", "user"]

2. BM25 Scoring

score = Σ IDF(term) × (TF × (k1+1)) / (TF + k1 × (1-b + b×(len/avglen)))

Parameters: k1=1.5, b=0.5 (tuned for code/API search)

3. Word Variant Matching

• authenticate ↔ authentication (both stem to authent)
• message ↔ messages (both stem to messag)
• create ↔ creating (both stem to creat)

Test Coverage

• ✅ Basic search functionality
• ✅ CamelCase tokenization
• ✅ Stemming and word variants
• ✅ BM25 ranking correctness
• ✅ Multi-term queries
• ✅ YAML and JSON parsing
• ✅ Stop word filtering
• ✅ Natural language queries
• ✅ Edge cases and boundaries

Files Changed

• 20 new files (5,000+ lines of code + docs)
• Implementation: ~800 LOC
• Tests: ~1,500 LOC
• Documentation: ~3,000 lines

Why This Matters

This example demonstrates:

1. How to port probe's search architecture to another language
2. Practical implementation of BM25 ranking
3. NLP tokenization techniques (stemming, stop words, CamelCase)
4. Go patterns for search engines (goroutines, interfaces)
5. Comprehensive testing strategies

Perfect for developers wanting to:

• Build API discovery platforms
• Add search to documentation sites
• Learn information retrieval algorithms
• Understand probe's architecture

Checklist

• ✅ All tests passing
• ✅ Comprehensive documentation
• ✅ Real-world examples
• ✅ Production-ready code
• ✅ Zero external dependencies (except snowball & yaml)

[probelabs]

🔍 Code Analysis Results

🐛 Debug Information

Provider: anthropic
Model: glm-4.6
API Key Source: ANTHROPIC_API_KEY
Processing Time: 1032454ms
Timestamp: 2025-10-22T11:46:25.838Z
Prompt Length: 901221 characters
Response Length: 18033 characters
JSON Parse Success: ✅

Debug Details

⚠️ Debug information is too large for GitHub comments.
📁 Full debug information saved to artifact: visor-debug-2025-10-22T11-46-28-675Z.md

🔗 Download Link: visor-debug-487
💡 Go to the GitHub Action run above and download the debug artifact to view complete prompts and responses.

---

Powered by Visor from Probelabs

Last updated: 2025-10-22T11:46:28.950Z | Triggered by: synchronize | Commit: b390504

💡 TIP: You can chat with Visor using /visor ask <your question>

[probelabs]

🔍 Code Analysis Results

Security Issues (3)

Severity	Location	Issue
🔴 Critical	examples/openapi-search-go/search/engine.go:74-94	Path traversal vulnerability in IndexDirectory function allows unauthorized file system access through directory parameter 💡 Suggestion Validate and sanitize the directory parameter to prevent path traversal attacks. Use filepath.Clean() and check that the resolved path is within allowed boundaries. 🔧 Suggested Fix func (e *Engine) IndexDirectory(dir string) error { // Clean and validate the directory path cleanDir := filepath.Clean(dir) if !filepath.IsAbs(cleanDir) { absDir, err := filepath.Abs(cleanDir) if err != nil { return fmt.Errorf("invalid directory path: %w", err) } cleanDir = absDir } // Additional validation could be added here to restrict to specific directories files, err := filepath.Glob(filepath.Join(cleanDir, &#34;*.yaml&#34;)) if err != nil { return err } jsonFiles, err := filepath.Glob(filepath.Join(cleanDir, &#34;*.json&#34;)) if err != nil { return err }</code></pre>
🟠 Error	examples/openapi-search-go/search/openapi.go:72-93	LoadSpec function reads files from any path without validation, potentially allowing access to sensitive files 💡 Suggestion Add path validation to restrict file access to allowed directories and file extensions. Validate that the file path is within expected bounds. 🔧 Suggested Fix func LoadSpec(path string) (*OpenAPISpec, error) { // Clean and validate the file path cleanPath := filepath.Clean(path) // Check file extension ext := strings.ToLower(filepath.Ext(cleanPath)) if ext != &#34;.yaml&#34; &amp;&amp; ext != &#34;.yml&#34; &amp;&amp; ext != &#34;.json&#34; { return nil, fmt.Errorf(&#34;unsupported file extension: %s&#34;, ext) } // Additional validation could be added here to restrict to specific directories data, err := os.ReadFile(cleanPath) if err != nil { return nil, fmt.Errorf(&#34;failed to read file: %w&#34;, err) }</code></pre>
🟡 Warning	examples/openapi-search-go/main.go:13-15	Command line arguments are not validated, potentially allowing injection attacks through malicious input 💡 Suggestion Add input validation for command line arguments to prevent injection attacks and ensure they meet expected format constraints. 🔧 Suggested Fix // Parse command line flags specsDir := flag.String("specs", "specs", "Directory containing OpenAPI specs") query := flag.String("query", "", "Search query") maxResults := flag.Int("max", 10, "Maximum number of results") flag.Parse() // Validate inputs if *maxResults &lt; 1 || *maxResults &gt; 1000 { fmt.Fprintf(os.Stderr, &#34;Error: max results must be between 1 and 1000 ") os.Exit(1) } if *specsDir != &#34;&#34; { // Basic validation for specs directory if strings.Contains(*specsDir, &#34;..&#34;) || strings.Contains(*specsDir, &#34;~&#34;) { fmt.Fprintf(os.Stderr, &#34;Error: invalid directory path ") os.Exit(1) } }

Architecture Issues (6)

Severity	Location	Issue
🟠 Error	examples/openapi-search-go/search/engine.go:107	Search method processes all documents at once without early filtering or batching, which will not scale beyond 1000 endpoints 💡 Suggestion Implement early filtering and batch processing similar to probe's approach. Add an inverted index for term lookup and process documents in batches, stopping when enough results are found.
🟠 Error	examples/openapi-search-go/ranker/bm25.go:106	Creates one goroutine per document for parallel scoring, which is inefficient for large document sets and can cause goroutine explosion 💡 Suggestion Use a worker pool pattern with a fixed number of goroutines (e.g., runtime.NumCPU()) instead of creating one goroutine per document. Process documents in batches to balance parallelism with resource usage.
🟢 Info	examples/openapi-search-go/ranker/bm25.go:42	BM25 implementation lacks probe's optimizations like u8 term indices, sparse vectors, and SIMD acceleration 💡 Suggestion Consider implementing sparse vector representation for documents and term indices to reduce memory usage. While SIMD isn't available in Go, consider using concurrent processing as an alternative optimization.
🟡 Warning	examples/openapi-search-go/search/engine.go:15	No caching layer implemented for query results or tokenization, missing probe's multi-tier caching optimization 💡 Suggestion Add LRU caching for query results and tokenization results. Consider caching term frequency maps and IDF computations to avoid redundant calculations across queries.
🟡 Warning	examples/openapi-search-go/tokenizer/tokenizer.go:44	Simplified query processing lacks boolean operators (AND, OR, +required, -excluded) that probe supports, limiting search expressiveness 💡 Suggestion Implement boolean query parsing with AST structure similar to probe's elastic_query.rs. Add support for required/excluded terms and logical operators to enable more precise searches.
🟡 Warning	examples/openapi-search-go/search/openapi.go:108	Endpoint struct directly coupled to OpenAPI-specific fields, making it difficult to extend to other specification formats 💡 Suggestion Extract a generic SearchableDocument interface and make Endpoint implement it. This would allow the search engine to work with other document types beyond OpenAPI specs.

Performance Issues (8)

Severity	Location	Issue
🔴 Critical	examples/openapi-search-go/ranker/bm25.go:106-116	Creates one goroutine per document without pooling, risking goroutine explosion for large document sets 💡 Suggestion Implement worker pool pattern with bounded concurrency using runtime.NumCPU() workers and a job channel
🟠 Error	examples/openapi-search-go/ranker/bm25.go:56-77	Recreates TF maps and DF calculations for every search operation instead of caching pre-computed values 💡 Suggestion Pre-compute and cache TF maps and document frequencies during indexing, reuse during search
🟠 Error	examples/openapi-search-go/tokenizer/tokenizer.go:34-88	Creates new 'seen' map and 'tokens' slice for every tokenization call, causing high GC pressure 💡 Suggestion Use sync.Pool to reuse map and slice allocations across tokenization calls
🟠 Error	examples/openapi-search-go/tokenizer/tokenizer.go:92-94	Compiles regex pattern on every call to splitNonAlphanumeric instead of pre-compiling once 💡 Suggestion Pre-compile regex as package-level variable and reuse across calls
🟠 Error	examples/openapi-search-go/search/engine.go:125-160	Converts all scored results to SearchResult objects even when only top N are needed, wasting memory and CPU 💡 Suggestion Apply maxResults limit early in the loop, avoid converting results beyond the limit
🟡 Warning	examples/openapi-search-go/ranker/bm25.go:140-150	Recalculates docLenNorm for every document in scoreBM25 when it could be pre-computed once per document 💡 Suggestion Pre-compute document length normalization factor during indexing and pass to scoreBM25
🟡 Warning	examples/openapi-search-go/tokenizer/tokenizer.go:76-82	Calls snowball.Stem for every token without caching results, causing repeated expensive stemming operations 💡 Suggestion Implement LRU cache for stemmed tokens to avoid repeated stemming of common words
🟡 Warning	examples/openapi-search-go/search/engine.go:126-129	Creates queryTokenSet map on every search to find matched terms, could be optimized 💡 Suggestion Pass query tokens directly to matching logic or reuse existing token set from BM25 ranking

Quality Issues (7)

Severity	Location	Issue
🟠 Error	examples/openapi-search-go/search/engine.go:67-72	IndexDirectory logs errors but continues processing, potentially leaving system in inconsistent state without user awareness 💡 Suggestion Either fail fast on critical errors or return a summary of failed/succeeded files to the caller 🔧 Suggested Fix for _, file := range files { if err := e.IndexSpec(file); err != nil { return fmt.Errorf("failed to index %s: %w", file, err) } } return nil
🟠 Error	examples/openapi-search-go/ranker/bm25.go:106-116	Potential race condition in goroutine closure capturing loop variable 'idx' incorrectly 💡 Suggestion Pass loop variable as parameter to goroutine to avoid race condition 🔧 Suggested Fix for i := range documents { wg.Add(1) go func(idx int) { defer wg.Done() score := r.scoreBM25(docTF[idx], docLengths[idx], avgdl, queryTokens, idf) results[idx] = &ScoredResult{ Document: documents[idx], Score: score, } }(i) }
🟠 Error	examples/openapi-search-go/search/engine.go:143-144	Type assertion without safety check could panic if Document.Data is not *Endpoint 💡 Suggestion Add type assertion safety check or use proper error handling 🔧 Suggested Fix endpoint, ok := s.Document.Data.(*Endpoint) if !ok { continue // Skip malformed documents }
🟡 Warning	examples/openapi-search-go/tokenizer/tokenizer.go:67-82	Stemming errors are silently ignored, which could mask problems with the snowball library 💡 Suggestion Log or return stemming errors to help diagnose issues with the stemming library 🔧 Suggested Fix // 5. Stem the token if len(lower) >= 3 { stemmed, err := snowball.Stem(lower, t.stemmer, true) if err != nil { // Log error but continue with original token fmt.Printf("Warning: stemming failed for %q: %v ", lower, err) } else if stemmed != lower && !seen[stemmed] { tokens = append(tokens, stemmed) seen[stemmed] = true } }
🟡 Warning	examples/openapi-search-go/main.go:11	CLI doesn't validate that specs directory exists before attempting to index 💡 Suggestion Add directory existence validation before indexing 🔧 Suggested Fix func main() { // Parse command line flags specsDir := flag.String("specs", "specs", "Directory containing OpenAPI specs") query := flag.String("query", "", "Search query") maxResults := flag.Int("max", 10, "Maximum number of results") flag.Parse() // Validate specs directory exists if _, err := os.Stat(*specsDir); os.IsNotExist(err) { fmt.Fprintf(os.Stderr, "Error: specs directory %q does not exist ", *specsDir) os.Exit(1) }
🟡 Warning	examples/openapi-search-go/search/openapi.go:75-95	LoadSpec doesn't validate file paths, potentially allowing directory traversal attacks 💡 Suggestion Add path validation to ensure files are within expected directory 🔧 Suggested Fix func LoadSpec(path string) (*OpenAPISpec, error) { // Validate path is within expected bounds cleanPath := filepath.Clean(path) if !strings.HasPrefix(cleanPath, filepath.Dir(path)) { return nil, fmt.Errorf("invalid file path: %s", path) } data, err := os.ReadFile(path) if err != nil { return nil, fmt.Errorf(&#34;failed to read file: %w&#34;, err) }</code></pre>
🟡 Warning	examples/openapi-search-go/ranker/bm25.go:94-100	IDF calculation assigns 0.0 to terms not in any document, but this should be a higher penalty 💡 Suggestion Assign a small positive IDF value for non-existent terms to maintain proper scoring 🔧 Suggested Fix for term := range queryTermSet { df := float64(termDF[term]) if df == 0 { // Term not in any document, assign minimal but non-zero IDF idf[term] = 0.01 continue } idf[term] = math.Log(1.0 + (nDocs-df+0.5)/(df+0.5)) }

Style Issues (5)

Severity	Location	Issue
🟡 Warning	examples/openapi-search-go/search/engine.go:71	Non-standard function closing comment with truncated text 💡 Suggestion Remove or standardize function closing comments. Go convention is to not use closing comments for functions. 🔧 Suggested Fix }
🟡 Warning	examples/openapi-search-go/search/engine.go:95	Non-standard function closing comment with truncated text 💡 Suggestion Remove or standardize function closing comments. Go convention is to not use closing comments for functions. 🔧 Suggested Fix }
🟡 Warning	examples/openapi-search-go/search/openapi.go:93	Non-standard function closing comment with truncated text 💡 Suggestion Remove or standardize function closing comments. Go convention is to not use closing comments for functions. 🔧 Suggested Fix }
🟡 Warning	examples/openapi-search-go/search/openapi.go:140	Non-standard function closing comment with truncated text 💡 Suggestion Remove or standardize function closing comments. Go convention is to not use closing comments for functions. 🔧 Suggested Fix }
🟡 Warning	examples/openapi-search-go/main.go:84	Non-standard function closing comment 💡 Suggestion Remove function closing comment. Go convention is to not use closing comments for functions. 🔧 Suggested Fix }

🐛 Debug Information

Provider: anthropic
Model: glm-4.6
API Key Source: ANTHROPIC_API_KEY
Processing Time: 1032454ms
Timestamp: 2025-10-22T11:46:25.838Z
Prompt Length: 901221 characters
Response Length: 18033 characters
JSON Parse Success: ✅

Debug Details

⚠️ Debug information is too large for GitHub comments.
📁 Full debug information saved to artifact: visor-debug-2025-10-22T11-46-30-266Z.md

🔗 Download Link: visor-debug-487
💡 Go to the GitHub Action run above and download the debug artifact to view complete prompts and responses.

---

Powered by Visor from Probelabs

Last updated: 2025-10-22T11:46:30.480Z | Triggered by: synchronize | Commit: b390504

💡 TIP: You can chat with Visor using /visor ask <your question>