HNSW index scan segfault (signal 11) on tables >100K rows

[stuinfla]

Bug Report: HNSW Index Scan Segfault on Large Tables

Environment

• ruvector-postgres Docker image: ruvnet/ruvector-postgres:latest
• ruvector extension version: 0.1.0
• Host: macOS (Apple M3 Max, 128GB RAM)
• Container memory: 7.6GB allocated, ~216MB used at idle
• PostgreSQL shared_buffers: 128MB
• PostgreSQL work_mem: default (4MB), also tested at 512MB

Problem

HNSW index scan causes a segmentation fault (signal 11) when querying tables with ~235K+ rows. The PostgreSQL server process crashes and requires automatic recovery.

Steps to Reproduce

-- Table: ask_ruvnet.architecture_docs (235,027 rows)
-- Index: idx_ask_ruvnet_embedding_hnsw USING hnsw (embedding ruvector_cosine_ops) WITH (m='16', ef_construction='64')
-- Index size: 1,836 MB

-- This query crashes the server:
SELECT doc_id, title,
  1 - (embedding <=> ('[' || array_to_string(ruvector_embed('vector architecture'), ',') || ']')::ruvector) AS score
FROM ask_ruvnet.architecture_docs
ORDER BY embedding <=> ('[' || array_to_string(ruvector_embed('vector architecture'), ',') || ']')::ruvector
LIMIT 5;

-- Server log shows:
-- server process (PID XXXXX) was terminated by signal 11: Segmentation fault

What Works

• Sequential scan on the same table works correctly (with SET enable_indexscan = off)
• HNSW index scan on smaller tables works (tested on tables with 10-50K rows across multiple schemas: clawdbot, scaling_up, viral_social, skillnet_ge, etc.)
• Increasing work_mem to 512MB does not prevent the crash
• The index itself builds without errors

Expected Behavior

HNSW index scan should return results without crashing, or gracefully fall back to sequential scan if index traversal fails.

Workaround

SET enable_indexscan = off;
-- query runs via sequential scan (slower but doesn't crash)
SET enable_indexscan = on;

Additional Context

• 25 HNSW indexes across 10 schemas in the same database all work on smaller tables
• The crash appears to be in the C extension's HNSW graph traversal code, not an OOM condition (container has 7.4GB free memory)
• Reproducible on every attempt (tested 3 times, crashed each time)

[ruvnet]

Fix Applied in PR #172 (merged)

Two root causes identified and fixed:

1. Hardcoded dimensions in hnsw_build (primary cause)

hnsw_am.rs:742 hardcoded let dimensions = 128; instead of reading from the column type modifier. With 384-dim embeddings stored as 128-dim, the page layout was corrupted — the index build stored truncated vectors and neighbor lists at wrong offsets. On large tables (235K+ rows), this corruption caused reads past page boundaries → segfault (signal 11).

Fix: Now reads dimensions from atttypmod:

let attr = (*tupdesc).attrs.as_ptr().offset((attnum - 1) as isize);
let typmod = (*attr).atttypmod; // 384 for ruvector(384)

2. Missing page boundary checks

read_vector() and read_neighbors() had no bounds validation. When corrupted dimension values caused reads past the 8KB page boundary, PostgreSQL's buffer system delivered a segfault.

Fix: Added bounds checking before all unsafe reads:

let total_read_end = header_size + node_header_size + vector_size + neighbor_size;
if total_read_end > pg_sys::BLCKSZ as usize {
    // Skip this block instead of segfaulting
    pg_sys::UnlockReleaseBuffer(buffer);
    return Vec::new();
}

Action Required

After updating to ruvector-postgres 2.0.2, rebuild all HNSW indexes:

REINDEX INDEX idx_ask_ruvnet_embedding_hnsw;

This regenerates the graph with correct dimensions and proper page layout.

Version: ruvector-postgres 2.0.2 | PR: #172

[ruvnet]

Fixed in ruvector-postgres v2.0.2 (PR #172). Dimension extraction + page boundary checks prevent segfaults. Please REINDEX after updating.