rustc -C incremental: nix --option extra-sandbox-paths dynamic permissions #7
Description
Motivation
This ticket wants proper support for rustc's incremental build feature (https://blog.rust-lang.org/2016/09/08/incremental/) inside nix sandboxes .
In the nix world we call this dynamic derivations https://fzakaria.com/2025/03/11/nix-dynamic-derivations-a-practical-application.
In essence rustc also implements something similar to dynamic derivations on the compiler level. So to support rustc's incremental build feature, within a nix sandbox build, we need to map a read/write folder as a side-chain to enable fast build iterations making nix development more attractive for developers.
WARNING: What I propose here brakes pure builds and https://nix.dev/manual/nix/2.18/command-ref/new-cli/nix3-build#opt-extra-sandbox-paths enables this without a warning. I can't find any documentation what opt-extra-sandbox-paths was originally written for. A better way to support rustc's incremental builds would be to use dynamic-derivations but that would require rustc patches. The current https://lastlog.de/blog/libnix_cargo-nix-backend.html effort is quite delectate and doing this properly needs to be done at a later time (with more funding).
The working prototype hack looks like this:
The performance gain is great as the incremental cache speeds up the build from 1m40s to 39s.
Problem
The current nix (Nix) 2.31.2 implementation of extra-sandbox-paths in nix requires a static setup to work.
I want a simple way to call nix like:
time nix build --file nix/cargo_build_caller.nix target -L --option extra-sandbox-paths '/incremental-target=/home/myuser/myrustproject/target/debug/incremental'
This is a typical cargo incremental folder structure:
[nixos@nixos:/cargo-incremental-target]$ ls ~/cargo/target/debug/incremental/ -la
total 48
drwxr-xr-x 12 nixos users 4096 Jan 19 10:35 .
drwxr-xr-x 8 nixos users 4096 Jan 22 10:15 ..
drwxr-xr-x 3 nixos users 4096 Jan 19 10:31 build_script_build-2tgp5bb1llbpb
drwxr-xr-x 3 nixos users 4096 Jan 19 16:19 cargo-04kpozefqywri
drwxr-xr-x 3 nixos users 4096 Jan 19 16:19 cargo-0i29k4tub0pvo
drwxr-xr-x 3 nixos users 4096 Jan 19 10:31 cargo_credential-35r6tyg7h09g4
drwxr-xr-x 3 nixos users 4096 Jan 19 10:31 cargo_credential_libsecret-2b44gfkai8s3w
drwxr-xr-x 3 nixos users 4096 Jan 19 10:31 cargo_platform-0utfckq8jsk5l
drwxr-xr-x 3 nixos users 4096 Jan 19 10:31 cargo_util-30g8ztgxu8voo
drwxr-xr-x 3 nixos users 4096 Jan 19 10:31 cargo_util_schemas-044mp4pfxeghb
drwxr-xr-x 3 nixos users 4096 Jan 19 10:31 crates_io-05cxmv2jxlrgo
drwxr-xr-x 3 nixos users 4096 Jan 19 10:31 rustfix-0994y1k492a87
Proposal
I want something like: https://docs.docker.com/reference/dockerfile/#run---mounttypecache
And it should map target/debug/incremental to the sandbox.
And cargo clean can easily take care of the cleanup.
Static setup
First let's document how the extra-sandbox-paths is being used, since I can't find a reference in the nix documentation.
There are other confused users though: NixOS/nix#6217
1. fs preparations
as root:
mkdir -p /cargo-incremental-target
chown root:nixbld /cargo-incremental-target
Afterwards we can access it from a builder.
2. nix command line
Let's access the /cargo-incremental-target folder from a nix build run:
time nix build --file nix/cargo_build_caller.nix target -L --option extra-sandbox-paths '/incremental-target=/cargo-incremental-target'
And this is the adaption inside the nix builder:
${RUSTC} -v \
-Z incremental-info \
--crate-name cargo \
--edition=2021 src/cargo/lib.rs \
--crate-type lib \
--emit=metadata,link \
-C embed-bitcode=no \
-C debuginfo=2 \
+ -C incremental=/incremental-target \
-C split-debuginfo=unpacked \
--allow=clippy::all \
--warn=clippy::correctness \
--warn=clippy::self_named_module_files \
--out-dir /tmp/out \
...Note: --out-dir needs to be set to a static name or else rustc will discard this cache. Maybe this rustc behavior can be changed in the future as this nix specific requirement is currently an edge-case and we can hack around it on the cost of copying the result(s) to $out after the build.
3. nix directory layout adaption
While the main dir is in root:nixbld, created directories are created from nixbld1:nixbld.
[nixos@nixos:/cargo-incremental-target]$ ls -la
total 12
drwxrwxr-x 3 root nixbld 4096 Jan 22 11:30 .
drwx---r-x 20 root root 4096 Jan 19 02:18 ..
drwxr-xr-x 3 nixbld1 nixbld 4096 Jan 22 11:30 cargo-15wt6oczg67y6
[nixos@nixos:/cargo-incremental-target]$ ls -la cargo-15wt6oczg67y6/
total 28
drwxr-xr-x 3 nixbld1 nixbld 4096 Jan 22 11:30 .
drwxrwxr-x 3 root nixbld 4096 Jan 22 11:30 ..
drwxr-xr-x 2 nixbld1 nixbld 20480 Jan 22 11:30 s-hf34s7jdkx-1fq8c0z-4o05t4j34dy1btgajgn7pf2uw
-rw------- 1 nixbld1 nixbld 0 Jan 22 11:30 s-hf34s7jdkx-1fq8c0z.lock
cleanup
You need to be root to cleanup these directories and cargo clean will not be able to do so.
performance stats
numbers on cargo legacy vs. cargo nix backend performance. this section documents the nix-backend performance with using builds which already have a minimal source set using pkgs.lib.fileset.toSource.
this section shows the potential of the nix-backend but i currently don't have a clue how to get these lists cost-effective.
setup
- all root crates had hard-coded minimal file sets so it is 1:1 between cargo legacy and cargo nix-backend.
- i called
nix buildand normally the nix-backend would require ~ 2.4 seconds to create the nix build-system
test
I changed a line in /home/nixos/cargo/src/cargo/core/dependency.rs which belongs to the cargo lib target (which is a dependency of the cargo bin target)
as a result both systems would build two units:
- cargo lib
- cargo bin
legacy backend
cargo build (legacy backend)
real 0m20.553s
user 0m14.200s
sys 0m6.371s
cargo build (legacy backend) after echo 3 > /proc/sys/vm/drop_caches
real 0m36.143s
user 0m14.102s
sys 0m7.421s
cargo build (nix backend)
real 0m58.464s
user 0m3.737s
sys 0m0.761s
# cargo build (nix backend) after echo 3 > /proc/sys/vm/drop_caches
real 1m19.010s
user 0m4.168s
sys 0m1.010s
i can't explain this, the copying of the files into the sandbox requires < 1 second but the times are 1:2 with legacy-backend:nix-backend.
sandbox tests
build environment creation with sandboxes costs 5-6 seconds on my system.
let's do experiment with sandbox off by adding this to the configuration.nix i evaluated again:
nix.extraOptions = ''trusted-users = root nixos'';and a nixos-rebuild switch
all results were run several times to consider this a 'warm' build.
cargo build (nix backend) sandbox true
time nix build --file nix/cargo_build_caller.nix target -L --option sandbox true
real 1m0.414s
user 0m4.066s
sys 0m0.566s
cargo build (nix backend) - sandbox false
time nix build --file nix/cargo_build_caller.nix target -L --option sandbox false
real 1m2.853s
user 0m3.828s
sys 0m0.759s
seems to make no difference.
cargo build (nix backend) - without a source code change
time nix build --file nix/cargo_build_caller.nix target -L --option sandbox true
real 0m5.388s
user 0m3.996s
sys 0m0.592s
lib+bin: cargo legacy (holy grail reference)
real 0m19.607s
user 0m13.753s
sys 0m5.863s
- it uses incremental/cargo-0i29k4tub0pvo/s-hezavpzb8y-1r8by2t-a1ki49rk4yh64bbyy1yy0wyea/
libnix backend (what to expect)
in this scenario, with libnix supporting incremental builds, using the rustc --emit=dep-info scanner on two targets with changed files, this is the calculation:
- nix document evaluation (4-7 seconds)
- "bin" rustc with --emit=dep-info (+1 seconds)
- "lib" rustc with --emit=dep-info (+6.3 seconds)
expect "real: 0m30.000s" to "real: 0m40.000s" when everything is optimal in the libnix backend.
lib+bin: nix-backend with many -L ...
real 1m15.614s
user 0m7.804s
sys 0m0.751s
lib+bin: nix-backend with the -L dependency=.../deps hack (similar to the deps/ folder in legacy)
real 1m17.391s
user 0m7.529s
sys 0m0.748s
lib+bin: cargo legacy (with removed target/debug/incremental/cargo-*)
real 1m14.432s
user 1m56.883s
sys 0m12.990s
lib+bin: nix-backend with mapped --extra-sandbox-paths /tmp/sandbox-file
real 0m39.225s
user 0m9.744s
sys 0m1.985s
--extra-sandbox-paths /tmp/sandbox-file
time nix build --file nix/cargo_build_caller.nix target -L --option extra-sandbox-paths '/incremental-target=/cargo-incremental-target'
so nix evaluation + creating check-sums of all targets takes ~ 5.388s
i guess most of this time i spent in check-sums. the root-crates + build.rs currently consist of ~ 400 files. so i think nix does this:
- it checks all root-crates (since those are not git based read only sources like the deps
- for each root crate it generates an archive with input files and serializes them with NAR
- checks the overall NAR hash with the nix-store path (since this feeds into the store path)
- nix does this for all the 9 root crates (cargo lib + cargo bin)
- nix does this also for the script_build (which is only a build.rs file)
[nix-shell:~/cargo]$ ls -la nix/derivations/
total 232
drwxr-xr-x 3 nixos users 4096 Dec 17 15:27 .
drwxr-xr-x 4 nixos users 4096 Dec 17 15:28 ..
-rw-r--r-- 1 nixos users 27081 Jan 12 14:52 cargo-0.88.0-27e7993d9cf32df7.nix
-rw-r--r-- 1 nixos users 18165 Jan 12 14:54 cargo-0.88.0-bin-85e09d7d8299b1ad.nix
-rw-r--r-- 1 nixos users 4662 Jan 2 19:28 cargo-0.88.0-script_build-cfc654fccb259515.nix
-rw-r--r-- 1 nixos users 23299 Jan 2 23:05 cargo-0.88.0-script_build_run-f5d51778f22880c0.nix
-rw-r--r-- 1 nixos users 5483 Jan 2 18:57 cargo-credential-0.4.8-a5adc6ab9fe103b0.nix
-rw-r--r-- 1 nixos users 5044 Jan 2 18:36 cargo-credential-libsecret-0.4.13-4e698a0b35f72d06.nix
-rw-r--r-- 1 nixos users 4595 Jan 2 18:36 cargo-platform-0.2.0-c5f768769f22a333.nix
-rw-r--r-- 1 nixos users 6311 Jan 2 18:37 cargo-util-0.2.20-7087e4a73afc7b23.nix
-rw-r--r-- 1 nixos users 6104 Jan 2 18:37 cargo-util-schemas-0.8.1-bce7b79eff35b46a.nix
-rw-r--r-- 1 nixos users 5102 Jan 2 18:28 crates-io-0.40.10-cb0425982b906266.nix
-rw-r--r-- 1 nixos users 47411 Jan 2 10:54 default.nix
drwxr-xr-x 2 nixos users 36864 Dec 17 15:27 deps
-rw-r--r-- 1 nixos users 5066 Jan 2 18:37 rustfix-0.9.0-9f1c66820d29e14a.nix
-rw-r--r-- 1 nixos users 542 Dec 31 00:09 target.nix