VLLM nix packaging second attempt! #519
Conversation
arilotter
force-pushed
the
vlllm
branch
6 times, most recently
from
1d0f34e to
3fcc4c5
Compare
nix/python.nix
Outdated
| # packages that we provide to the venv via nix derivations | ||
| topLevelNixPkgs = [ | ||
| "torch" | ||
| "vllm" |
There was a problem hiding this comment.
I don't know if we want to support vllm for aarch64-darwin but I think this is not compiling on that architecture:
error: Package ‘python3.12-vllm-0.14.0’ in /nix/store/m6ji2a8jp7w4ck7vvrjh4grs4p3n7r26-source/pkgs/development/python-modules/vllm/default.nix:553 is not available on the requested hostPlatform:
hostPlatform.system = "aarch64-darwin"
Maybe we can move inside the isLinux check
There was a problem hiding this comment.
done - also made the inference node packages not build at all on macos since they won't work without vllm
|
I tried running |
This is intended - the |
Oh okay. No worries, if we see that the decentralized tests don’t crash because of that, I think we’re good |
arilotter
force-pushed
the
vlllm
branch
2 times, most recently
from
a9dc40f to
a403d98
Compare
- add vllm package and update the world fixes some cuda lib path problems in vllm updates mdbook fixes some clippy lints later, we should consider splitting the vllm code into a separate set of deps, so the regular psyche binaries don't get vllm included Original PR was #464
should drop many minutes from each test since we don't have to re-build them every time :3
- only build vllm / inference on linux - fix test build binary copies - use optionalAttrs et al. everywhere
|
oh my god the tests passed im gonna cry :) |
passes both train test and compiling cudnn stuff on h100
add vllm package and update the world
fixes some cuda lib path problems in vllm
updates mdbook
fixes some clippy lints
later, we should consider splitting the vllm code into a separate set of deps, so the regular psyche binaries don't get vllm included
Original PR was #464