[please test] BYOK with ollama

[olegklimov]

With the ollama project it's easy to host our own AI models.

You can set up bring-your-own-key (BYOK) to connect to ollama server, and see if you can use StarCoder2 for code completion, llama models for chat.

Does it work at all? What we need to fix to make it better?

[pardeep-singh]

@olegklimov I would like to take this up. Can you please some docs/example of how this can done? Do we need to test the integration here or make changes as well to make it work?

[olegklimov]

Oh, here https://docs.refact.ai/byok/ you can test if we have documentation that is any good :D

[avie66]

Hi @pardeep-singh
Did you had a look over Oleg's approach?

[ukrolelo]

chat_endpoint: "http://localhost:11434/v1/chat/completions"
chat_model: "starcoder2:latest"

mainThreadExtensionService.ts:79 Error: write after end
	at _write (node:internal/streams/writable:489:11)
	at Writable.write (node:internal/streams/writable:510:10)
	at c:\Users\ukro\.vscode\extensions\smallcloud.codify-6.0.4-win32-x64\node_modules\vscode-jsonrpc\lib\node\ril.js:90:29
	at new Promise (<anonymous>)
	at WritableStreamWrapper.write (c:\Users\ukro\.vscode\extensions\smallcloud.codify-6.0.4-win32-x64\node_modules\vscode-jsonrpc\lib\node\ril.js:80:16)
	at StreamMessageWriter.doWrite (c:\Users\ukro\.vscode\extensions\smallcloud.codify-6.0.4-win32-x64\node_modules\vscode-jsonrpc\lib\common\messageWriter.js:100:33)
	at c:\Users\ukro\.vscode\extensions\smallcloud.codify-6.0.4-win32-x64\node_modules\vscode-jsonrpc\lib\common\messageWriter.js:91:29
$onExtensionRuntimeError @ mainThreadExtensionService.ts:79
127.0.0.1:9084/v1/at-command-preview:1 
        
        
       Failed to load resource: the server responded with a status of 417 (Expectation Failed)
127.0.0.1:9084/v1/chat:1 
        
        
       Failed to load resource: the server responded with a status of 400 (Bad Request)
127.0.0.1:9084/v1/at-command-preview:1 
        
        
       Failed to load resource: the server responded with a status of 417 (Expectation Failed)
127.0.0.1:9084/v1/at-command-preview:1 
        
        
       Failed to load resource: the server responded with a status of 417 (Expectation Failed)
127.0.0.1:9084/v1/at-command-preview:1 
        
        
       Failed to load resource: the server responded with a status of 417 (Expectation Failed)
127.0.0.1:9084/v1/chat:1 
        
        
       Failed to load resource: the server responded with a status of 400 (Bad Request)

[ukrolelo]

chat_endpoint: "http://localhost:11434/v1/chat/completions"
chat_model: "llama3.2:1b-instruct-q8_0"

Error: Bad Request
Click to retry

 POST http://127.0.0.1:9099/v1/at-command-preview 417 (Expectation Failed)
index.umd.cjs:1 
 POST http://127.0.0.1:9099/v1/at-command-preview 417 (Expectation Failed)
index.umd.cjs:1 
 POST http://127.0.0.1:9099/v1/at-command-preview 417 (Expectation Failed)
index.umd.cjs:1 
 POST http://127.0.0.1:9099/v1/at-command-preview 417 (Expectation Failed)
index.umd.cjs:1 
 POST http://127.0.0.1:9099/v1/at-command-preview 417 (Expectation Failed)
index.umd.cjs:1 
 POST http://127.0.0.1:9099/v1/at-command-preview 417 (Expectation Failed)
Show 25 more frames
index.umd.cjs:1 
 POST http://127.0.0.1:9099/v1/at-command-preview 417 (Expectation Failed)
Show 25 more frames
index.umd.cjs:17 
 POST http://127.0.0.1:9099/v1/chat 400 (Bad Request)


Why it is sending to different port?

[olegklimov]

Why it is sending to different port?

VSCode extension talks to refact-lsp, that in turn talks to the inference server.

I think the fastest way we can fix this -- is reproduce your setup. So you have, windows, ollama with llama3.2:1b-instruct-q8_0 right, we'll try it.