HNSW index: SIGSEGV on repeated queries, empty graph, wrong metric and ordering

[grparry]

Hi! Following up on #167 / #180 — while testing the HNSW index for a pgvector comparison benchmark, I found several issues that cause crashes and incorrect results. Fix is in #181.

Bugs

1. SIGSEGV on repeated k-NN queries (crash)

Running two or more ORDER BY embedding <=> query LIMIT k queries in the same psql session crashes the PostgreSQL backend with signal 11 (segfault). The fault address is typically 0x1 — stale palloc memory.

Root cause: hnsw_beginscan calls RelationGetIndexScan but never allocates the xs_orderbyvals / xs_orderbynulls arrays. The executor assumes these are valid and writes distance values into whatever memory they happen to point at. First query often works (palloc0'd to null); second query crashes.

Repro:

CREATE TABLE t (id serial, embedding ruvector(5));
INSERT INTO t (embedding) VALUES ('[1,0,0,0,0]'), ('[0,1,0,0,0]'), ('[0,0,1,0,0]');
CREATE INDEX ON t USING hnsw (embedding ruvector_cosine_ops);
SET enable_seqscan = off;

-- First query: usually works
SELECT id FROM t ORDER BY embedding <=> '[1,0,0,0,0]'::ruvector LIMIT 3;

-- Second query: SIGSEGV
SELECT id FROM t ORDER BY embedding <=> '[0,1,0,0,0]'::ruvector LIMIT 3;

2. Empty HNSW graph (no results)

connect_node_to_neighbors is a no-op TODO stub, so hnsw_build inserts nodes but never creates edges. The search traversal finds only the entry point.

3. Wrong distance metric (wrong results)

hnsw_build uses HnswConfig::default() which hardcodes DistanceMetric::Euclidean, even when the index is created with ruvector_cosine_ops. The search computes Euclidean distances on data that should use cosine similarity.

4. Wrong result ordering (wrong results)

BinaryHeap::into_iter().take(k) iterates the heap's backing array in arbitrary order, not sorted order. The results returned are k random candidates from the ef_search pool, not the k closest.

5. "index returned tuples in wrong order" (error on PG17)

If xs_recheckorderby is set to true, PG17's IndexNextWithReorder compares index-reported distances against recalculated distances from heap tuples. Floating-point precision differences cause spurious errors.

6. Use-after-free in endscan

hnsw_endscan unconditionally calls Box::from_raw on scan->opaque without checking for null, risking a double-free if called after a rescan.

Environment

• PostgreSQL 17.7
• pgrx 0.12.9
• ruvector-postgres 2.0.1
• Linux x86_64

Fix

PR #181 addresses all six issues. The same xs_orderbyvals allocation fix is also applied to ivfflat_ambeginscan.

[stuinfla]

Real-World Confirmation (235K row table)

Can confirm bugs #2 and #3 independently on a production-size dataset. Our environment:

• PostgreSQL 17.2 (Docker)
• ruvector-postgres 2.0.2 (binary) / 2.0.0 (SQL extension)
• Table: 235,301 rows × ruvector(384) (Xenova/all-MiniLM-L6-v2 ONNX embeddings)
• Index: USING hnsw (embedding ruvector_cosine_ops) WITH (m=16, ef_construction=200) — 2.7GB

Symptoms

Operator	Index Used?	Rows Returned (LIMIT 5)	Correct?
<-> (L2)	Yes (HNSW)	5	Yes — relevant results
<=> (cosine)	Yes (HNSW)	0-1	No — empty or 1 garbage row
<=> (cosine, seq scan)	No (forced off)	5	Yes — correct results

Key Finding

The HNSW index was successfully built (2.7GB, covers 252K rows per reltuples), but cosine search through the index returns empty/near-empty results. L2 search through the same index works correctly.

This aligns perfectly with bug #3 (hardcoded Euclidean metric) — the index stores L2 distances internally regardless of the operator class, so <-> works by accident while <=> fails because it expects cosine distances from the graph traversal.

Workaround

For normalized embeddings (L2-normalized, which most transformer models produce), using <-> (L2) instead of <=> (cosine) preserves correct ranking since L2 distance on unit vectors is a monotonic function of cosine distance:

||a - b||² = 2(1 - cos(a,b))

Search time with L2 through HNSW: ~0.3ms on 235K rows.

Previously Misdiagnosed

We previously attributed HNSW build crashes on this table to insufficient maintenance_work_mem (issue #164). While bumping maintenance_work_mem from 4MB to 512MB did fix the build crash, the underlying search bugs (#2-#4 above) were still present. The root cause for build crashes on large tables may also be related to the xs_orderbyvals allocation issue (bug #1).

Looking forward to PR #181 being merged. Happy to test on our 235K dataset if helpful.

[ruvnet]

Fixed in PR #181, merged to main and published as ruvector-postgres v2.0.3.

All 6 bugs reported here are resolved:

1. SIGSEGV — xs_orderbyvals/xs_orderbynulls allocation
2. Empty graph — bidirectional connections with pruning
3. Wrong metric — metric_from_index() reads opclass
4. Wrong ordering — into_sorted_vec() for deterministic results
5. xs_recheckorderby — set to false for PG17
6. Use-after-free — null-safe endscan

Available via:

• cargo install ruvector-postgres@2.0.3
• docker pull ruvnet/ruvector-postgres:2.0.3

Thanks @grparry for the detailed report and fix!