Capture compiled artifacts

[deitch]

The typical onnxruntime flow is the following via llm-kvc.py (very simplified):

1. Find the onnx model in the huggingface cache as presented via the CLI flags
2. Run a compilation process, although it is more like transpilation+compilation: input is onnx model, output is a .bundle with an ELF in it. Uses et-glow, gcc, llvm, neuralizer, etc.
3. Run inference. Uses InferenceServer.

If you run the same one a second time, the process recognizes that the model already has been converted from onnx to .bundle, and tuns just the inference step. This takes about 1/10 the time.

The current nekko CLI does not recognize that there are intermediate artifacts to capture. We need to do this via the following:

• decide on a compiled artifacts cache directory default location
• have an option to the CLI to override that location
• mount the location into the container
• configure the llm-kvc.py script to support controlling that location

cc @jerenkrantz

[jerenkrantz]

LGTM!

1 & 2 is conceptually correct!

BTW, on 3, there is no separate inference server running. The Python ETGlow backend used by llm-kvc.py handles the transpilation+compilation, loading to the card, and then executing the model. (This doesn't meaningfully change anything.)

[rvs]

Sounds reasonable + what @jerenkrantz said. Also, as he pointed out InferenceServer is a separate binary. Confusingly, InferenceServer family of CLI/daemons are more specialized than ONNX runtime, however, the same GenAI model (let's say llama3) can be run via both.

[deitch]

There are two parts of this that confused me.

First, @jerenkrantz are you saying that the actual inference is handled by the same python functions? Or do you mean that the same library handles both compilation to bundle (1, 2) and inference, but does so using different functions in the library, called by llm-kvc.py? Do you mind linking to the specific lines for each stage in llm-kvc.py?

First and a half, how does llm-kvc.py do the compilation stage? Don't we need gcc+llvm for that? Or does it just call out to separate binaries?

Second, @rvs if InferenceServer is not executing the compiled bundle, what is it for?

[jerenkrantz]

The onnxruntime session is created here:

https://github.com/nekkoai/hacks/blob/04c6ddd5668c8638f1a4bb1775f7fddcfdf327ea/onnx/llm-kvc.py#L532

The Python code that drives the inference is invoked here:

https://github.com/nekkoai/hacks/blob/04c6ddd5668c8638f1a4bb1775f7fddcfdf327ea/onnx/llm-kvc.py#L543

There are two different code paths as to how the inference is driven: llm_kvc_inference and llm_kvc_inference_iobindings.

Two public blog posts from ET team:

https://www.esperanto.ai/blog/compiling-and-executing-small-language-models-with-onnxruntime-2/
https://www.esperanto.ai/blog/optimizing-small-language-models-with-onnxruntime-iobindings/

Lastly, InferenceServer is not used with onnxruntime, but different inference workflows. I think for the purposes of cli, I think we will focus on onnxruntime and llama.cpp workflows as those are two leading approaches used by the external community.

[deitch]

Huh, those are barely 50-60 lines into main(). If that is inference, where is compilation called?

[deitch]

InferenceServer is not used with onnxruntime, but different inference workflows

That has become clear 😆

So what is InferenceServer used for?

[deitch]

Solved via #14