Support installing potentially overlapping `cg_gcc` distributions

[rami3l]

Problem you are trying to solve

One specificity of cg_gcc distribution is the component structure:

let's say that we are on x64 [host]. if you want to compile x64 -> x64, rustc (and then transitively cg_gcc) will load:

/lib/rustlib/x64/codegen-backends/cg_gcc.so
/lib/rustlib/x64/codegen-backends/lib/x64/libgccjit.so
if you want to compile x64 -> aarch64, it will load

/lib/rustlib/x64/codegen-backends/cg_gcc.so
/lib/rustlib/x64/codegen-backends/lib/aarch64/libgccjit.so
(note the path to cg_gcc.so stays the same in both situations)
@Kobzol in #t-rustup > Adding support for more target-specific components @ 💬

The thing is that if we want to install rustup component add cg-gcc-x64 cg-gcc-aarch64 as-is today, the two will clash with each other WRT <root>/lib/rustlib/x64/codegen-backends/cg_gcc.so.

Solution you'd like

Assume that logically there are 3 artifacts to be installed. Let's call them a) cg-gcc-so, b) cg-gcc-libgccjit-x64 and c) cg-gcc-libgccjit-aarch64 (these are not real component names, just to show the idea; the real name of those artifacts will always be suffixed by the host tuple in addition).

I see several possible solutions here:

1. Introduce the notion of refcounted artifacts. In this interpretation, cg-gcc-x64 is a component that includes a) and b), and cg-gcc-aarch64 a) and c), i.e. both components share the artifact a).

   • When installing a component, we skip the artifacts that are already installed, but increment the refcounts of the latter (on disk?) at the same time.
   • When uninstalling a component, we decrement the refcounts of any artifact it has with a refcount >1, or remove the artifact directly otherwise.

2. Introduce the notion of component groups. In this interpretation, each of the 3 artifacts is a component in itself, so cg-gcc-x64 is a group that includes a) and b), and cg-gcc-aarch64 a) and c), i.e. both groups share the component a).

   • When installing a group, we skip the components that are already installed.
   • What it means to uninstall a group remains unclear since we must decide what to do with a). We can ignore implementing this altogether at the first step.

3. Introduce the notion of dependencies like traditional package managers (this could be abused to become a nightmare later on though). In this interpretation, each of the 3 artifacts is a component in itself, and b) and c) depend on a).

   • When installing a component that has dependencies, we skip the dependencies that are already installed.
   • When uninstalling a component that has dependencies, only that specific component is removed. Recursive uninstallation can be implemented later.

Notes

No response

[rami3l]

@rust-lang/rustup As the distribution of cg_gcc is a short-term goal for the Rust project, I'd love to receive any feedback/suggestion regarding the feature design to make this happen.

[djc]

I like the refcounts proposal least. I think we should implement what most closely matches the realities of how upstream builds these components. I think you should flesh out the design proposal to not ignore how uninstallation works, since that seems like an integral part of the implementation long term, and ignoring it just leads to technical debt.

[ChrisDenton]

I would worry about doing ref counting until we have safe concurrency for installs/uninstalls. Though maybe it can be done without risking getting into a confused state if something goes wrong (or at least, no worse than the status quo).

The second option seems like the one that "fits" as an extension to rustup's current model. But if we do become more like a traditional package manager then it could become redundant.

Heading towards being a traditional package manager may be inevitable as rust starts distributing more and more. But I don't think we are quite at that point yet.

---

Tbh, this feels like a long term design decision but we currently only have a single example cg_gcc to work from. Doing some short-term workaround might be fine so long as it doesn't constrain us in the future.

[djc]

Also,

(this could be abused to become a nightmare later on though)

This doesn't seem like a big deal. We can make sure it doesn't spiral out of control.

[ChrisDenton]

Looking at existing package mangers and following what works for them is appealing to me. But I do have some concern that the design space is fairly large and I don't want to risk over-engineering for something that's (possibly) only ever going to be used by cg_gcc.

[rami3l]

I think you should flesh out the design proposal to not ignore how uninstallation works

@djc Sure. The option 2 was my original proposal in the thread but to handle uninstallation it seemed that we'll need some kind of Garbage Collection mechanism anyway no matter what level we're doing this in (components or just artifacts), since it's a fact that the behavior of rustup component uninstall cg-gcc-x64 will and should differ according to whether cg-gcc-aarch64 is also installed. The information of this "heap" to be GC'd must be stored somewhere, say on the disk or calculated in-place.

Looking at existing package mangers and following what works for them is appealing to me. But I do have some concern that the design space is fairly large and I don't want to risk over-engineering for something that's (possibly) only ever going to be used by cg_gcc.

@ChrisDenton Exactly. My stance is that the introduction of dependencies is the most flexible of all but has risks of being too heavyweight for this one thing, and the refcounting thing is more of a hack that only responds to immediate needs.

I would worry about doing ref counting until we have safe concurrency for installs/uninstalls.

I only consider toolchain-wide locks so far. Otherwise, GC in the presence of concurrency deserves a book itself :)

[rami3l]

Aside: I really want to convince myself that (adapting a quote from B. Stroustrup) within rustup, there is a full-fledged package manager struggling to get out.

We have discussed the necessity of GC in another context (#3919, #4548) and I think we might be able to reuse the dependency tracking logic in more places than this one if we have full dependency resolution.