[BUG] @ruvector/ruvllm SIMD Inference produces garbled output - LLM not functional

[stuinfla]

Summary

The @ruvector/ruvllm package advertises built-in SIMD inference capabilities, but the LLM generation produces garbled, unusable output. Users expecting an integrated LLM solution are finding they still need external providers (Ollama/APIs).

Environment

• Package: @ruvector/ruvllm v0.2.4
• Platform: darwin-arm64 (Apple Silicon)
• Node.js: v22.13.1
• SIMD Support: Yes (NEON detected)

Steps to Reproduce

# Check info shows SIMD is loaded
npx @ruvector/ruvllm info
# Output: Native Module: Loaded, SIMD Support: Yes, Capabilities: NEON

# Attempt to generate text
npx @ruvector/ruvllm generate "Hello, how are you?"

Expected Behavior

Coherent text generation using the built-in SIMD inference, or a clear error message indicating an external LLM provider is required.

Actual Behavior

Garbled, unusable output:

=xmanybyon=``=on)&G&}=#)=wasmany!N=)do<5anduse`55towass>binorxOG6!manywithbytoNhowto5O<=6howOandtoashowNasNO=i<useouthadtop,byxwasG5kinhowmwasinvmandtoto<out5`withO@andfromin`~sx,totoandO<<@beinGk!use,`had<hadp!6]@i{from@use,`as]{`M~haveexG6p>wask\

Additionally, npx @ruvector/ruvllm stats throws an error:

Error: Cannot read properties of undefined (reading 'toFixed')

Impact

This creates confusion for users who expect ruvllm to include a working LLM. The package description states:

"Self-learning LLM orchestration with SONA adaptive learning, HNSW memory, FastGRNN routing, and SIMD inference"

Users interpret "SIMD inference" as meaning the package can perform LLM inference without external dependencies. Currently:

• They discover this only after attempting generation
• There's no clear documentation stating Ollama or an API is required
• The garbled output suggests a broken model rather than a missing dependency

Requested Fix

1. If SIMD inference is meant to work standalone:

   • Fix the model loading/inference pipeline
   • Ensure coherent output from built-in model

2. If external LLM is required:

   • Update package description to clarify this
   • Add clear error message: "No LLM configured. Please install Ollama or set ANTHROPIC_API_KEY"
   • Document the requirement in README

3. Fix the stats command - currently throws undefined error

Additional Context

The HNSW memory, FastGRNN routing, and embedding features work correctly. Only the LLM generation/inference is broken.

Related: ruvector-postgres ONNX embeddings work perfectly (all-MiniLM-L6-v2, BGE models, etc.). The issue is specifically with text generation in ruvllm.

[stuinfla]

Hey @ruvnet - following up on this issue.

I'm currently stuck: I can't use RuvLLM's built-in inference (produces gibberish), but I was also told Ollama isn't the right architecture for this use case. So I'm betwixt and between with no working path forward.

Current situation:

• RuvLLM SIMD inference → garbled output (this issue)
• Ollama + Qwen → works, but told it's not the right approach
• No clear alternative documented

Questions:

1. Is there an ETA on fixing the SIMD inference?
2. If Ollama isn't recommended, what IS the intended architecture for local LLM inference with RuvLLM?
3. Is there a workaround in the meantime?

Thanks for any guidance. The rest of the RuvNet stack is working great - just blocked on this piece.

[ruvnet]

Clarification: Rust Crate vs npm Package

This issue affects the npm package @ruvector/ruvllm (WASM bindings), not the Rust crate.

What's Fixed in RuvLLM v2.0 (Rust)

The optimize-mac-neon branch significantly improves the native Rust crate:

• ✅ Core ML/ANE backend for Apple Silicon
• ✅ Hybrid GPU+ANE pipeline with auto-dispatch
• ✅ NEON SIMD optimizations (1.84 Gelem/s)
• ✅ 1,114 tests passing
• ✅ Proper model loading via Candle

npm Package Status

The npm WASM package is a separate codebase that wraps the Rust library:

• Location: crates/ruvllm-wasm/
• The garbled output suggests the WASM bindings may not be correctly wiring up the inference pipeline
• The stats undefined error is a JS binding issue

Potential Fixes Needed (npm)

1. Verify WASM bindings export correct inference methods
2. Ensure model loading works in WASM environment
3. Fix stats serialization (JSON vs Debug format - related to Native SONA engine returns Rust debug format instead of JSON #78)

The Rust crate itself is working correctly with proper tests. The npm package needs separate investigation.

See also: #72 (WASM Dashboard Integration) and #78 (Debug format issue)

[ruvnet]

Fixed in PR #239.

Changes (npm/packages/ruvllm/bin/cli.js):

1. Stats crash fix: Wrapped llm.stats() in try/catch with null guard, added nullish coalescing (?? 0) on all property accesses — no more crash when stats are unavailable
2. Generate warning: Added experimental inference warning: "Built-in SIMD inference is experimental. For production use, consider llama.cpp or Ollama."

Additionally, @ruvector/ruvllm-wasm@0.1.0 has been deprecated on npm.

[ruvnet]

Resolved in PR #239 (merged). Published @ruvector/ruvllm@2.5.2.