Update to include compatibility with RHEL9

[huston-rogers]

RHEL 9 was released after the last published update of arbiter2. A number of cgroups changes were implemented in RHEL 9. Are there plans to update arbiter to match?

The first error I'm running into is "ERROR: processes.memsw = TRUE" isn't available, but digging down, theres another error about "cgroup hierarchy doesn't exist." I can see at least part of the cgroups hierarchy available under /sys/fs/cgroup, including memory inside cgroup.controllers. I believe this indicates that cgroups has changed and that is the compatibility issue with Arbiter.

[jay-mckay]

RHEL 9 does in fact mount cgroups v2 by default. Arbiter2 relies on the cgroups v1 hierarchy, and currently does not work with v2. Work is being done to support both versions, but may take some time. In the mean time, and if your situation allows, you can mount v1 by default and arbiter should (although my testing has been very limited) work like normal. Can do this with the kernel boot parameter: "systemd.unified_cgroup_hierarchy=0".

[singhsaluja]

Thanks for your excellent work! Arbiter2 on our HPC login nodes has been a real lifesaver. We're in the process of upgrading to RHEL/9.2. Do you know when Arbiter will support RHEL9? Thanks!

[huston-rogers]

@jay-mckay is there an update for supporting cgroups v2?

[huston-rogers]

@jay-mckay we're having issues with arbiter2 not working even with the grub/kernel parameters set. Is there an update for the conversion to RHEL9?

[singhsaluja]

@snowbird294 We switched our login nodes to cgroups v1 to make arbiter work but our compute nodes are on default cgroups v2. In case you're still keen on getting Arbiter set up on RHEL9. It really comes in handy!

[huston-rogers]

Found my bug.

If you start as root and su - $USER and make a bad process happen, arbiter doesn't see it because you started as a safe user (root). user-$USER.slice wasn't created so arbiter didn't even see it.

Would still like an update on the state of v2 integration.

[jay-mckay]

@snowbird294 Hello, sorry to keep you waiting.

An update regarding v2:

Because the update changed the filesystem hierarchy, a lot of the cgroup manipulation in arbiter would have to be completely rewritten. For this reason and a few others (ease of install/deployment, maintainability of the code base) we have decided to create a new version of arbiter, which includes a major architectural redesign. This version will support both versions of cgroups out of the box without any additional configuration.

In conclusion, this means we are not going to support v2 in Arbiter2. We are in the final stages of development now, and will be at PEARC this year to showcase the work we have done. We will announce the release of this in the coming months.

We will continue to support Arbiter2, but we will probably not be adding support for v2.

[ellestad]

I see PEARC has come and gone, any update on the time frame for the next version of of arbiter2 with cgroups v2 support?

[jay-mckay]

The new version (which we are calling Arbiter 3) is in the final testing stages, and should be released before the end of the year, probably sooner rather than later.

[dch1fc]

Hi Jay,

any update on the time frame of the release of Arbiter3?

[jay-mckay]

@dch1fc Arbiter3 is still in testing, but most of the features are complete. Any interested parties can play around with the code here, but we are still finding bugs and fixing them as we test. We are looking to release the first stable version at the beginning of December of this year.

[boesr]

Hi @jay-mckay any update on the release plan?

[jay-mckay]

@boesr

Here at CHPC, we have been running Arbiter3 in our test clusters for a few months now. We are looking to switch over into production shortly, barring any unforeseen issues. We have held off on 'releasing' an official version until we have had Arbiter3 running in production here for a little while to ensure that other sites can install and use without any issue.

The current version of Arbiter3 is feature complete, and may be installed; There are some sites that currently have Arbiter3 running.

When we finally do release the 'official' first version, we will post an update here and update the documentation here as well. However, if you wish to install and use Arbiter3 before then, we are more than happy to help with the install and field any additional questions.

Thanks

[ellestad]

We are using Rocky 8 and cgroups v2. Will Arbiter3 support RHEL 8 based distributions?

We do have a plan to move to Rocky 9 before Rocky 8's EOL, but I'm not sure about the exact time frame.

[jay-mckay]

@ellestad

Arbiter3 will work with RHEL 8 and 9, and will work with either cgroups v1 or v2. Our test clusters here at CHPC are Rocky8 with cgroups v1, but we have done extensive testing with RHEL 9 and cgroups v2.

In fact, Arbiter3 will work on any modern distro that uses systemd as its init system, provided the cgroup-warden component is built for that platform.