Fix circular dependencies

[haampie]

• libxcrypt has a build dependency on perl
• perl links to libcrypt

This makes it annoying to distribute binaries (when not cross compiling).

To distribute perl binaries, you have to ensure it does not link to glibc's libcrypt, since that may not exist on other systems, so it's best to build libxcrypt and distribute together...

But to configure libxcrypt, you need perl 5.14 with open.pm, which is not available on RHEL and derivatives by default, and you may not have access to yum install perl-open.

So without root access, you have to bootstrap a minimal perl yourself to configure libxcrypt, so you can build perl with crypt support.

The latter is not satisfactory cause it's overly complex, and the bootstrap perl won't be a deterministic build (it may or may not link against glibcs libcrypt), and cannot be distributed itself.

Would it be possible to drop the build dependency on perl in favor of something that does not depend on libcrypt?

[Leont]

But to configure libxcrypt, you need perl 5.14 with open.pm, which is not available on RHEL and derivatives by default, and you may not have access to yum install perl-open.

This is an unfortunate choice in RHEL/Fedora: if you want a usable perl you need to install the perl package (which will install all core packages), not just the perl-base one (which only contains half of the core). That said, I think it would be easy to write open.pm out of the scripts.

This issue feels of a secondary nature to the main one though.

To distribute perl binaries, you have to ensure it does not link to glibc's libcrypt, since that may not exist on other systems, so it's best to build libxcrypt and distribute together...

I can see how that can turn up as a bootstrapping issue, especially on distributions that build from scratch. It's certainly suboptimal if you'd have to compile perl twice because of this to get it to work with libxcrypt.

The latter is not satisfactory cause it's overly complex, and the bootstrap perl won't be a deterministic build (it may or may not link against glibcs libcrypt), and cannot be distributed itself.

Yeah, I think that is the real issue here.

Would it be possible to drop the build dependency on perl in favor of something that does not depend on libcrypt?

How would you suggest solving this?

Perl itself has this sort of bootstrapping issue for obvious reasons, and the result is that we have a build system that depends on absolutely nothing other than a bourne shell and basic portable unix tools like sed, find, grep, cut and awk (POSIX awk no fancy gnu awk). To eliminate perl from libxcrypt's build system we'd have to do something similar here, rewrite the scripts to work the traditional unix way.

It's certainly possible to do but not trivial at all.

[haampie]

To eliminate perl from libxcrypt's build system we'd have to do something similar here, rewrite the scripts to work the traditional unix way. It's certainly possible to do but not trivial at all.

I think it's worth it.

Notice that libxcrypt went from awk to perl: #115 ;) a bit of an unfortunate change.

[zackw]

I appreciate that circular bootstrap dependencies are a serious problem.

However, the scripts you wish we hadn't replaced with Perl were a chronic maintenance burden. There is a fairly low upper limit on the complexity of an awk script before it becomes intractable to modify. Shell is not as bad but it's still bad, and all the time that I'm willing to spend on wrangling shell scripts is soaked up by my responsibilities to Autoconf.

In particular note what I said in #115 (comment):

extending hashes.conf with more fields is just too difficult as long as each consumer of hashes.conf is its own self-contained awk script with its own quirks

Now, it's possible that we no longer need the complexity of the scripts we have, because GNU libc no longer ships its own libcrypt so the sets of backward compatibility symbols we need to provide are not going to change again. If the scripts could be simplified to the point where they were maintainable in awk and shell, I wouldn't be opposed to switching back. However, I am not going to do the work myself unless someone pays me to do it; my rate for this job would be US$120/hour and off the top of my head I think it would be at least eighty hours of work (i.e. two fulltime weeks).

If someone else, such as yourself, did the work, I would review the pull request, but I would need to be thoroughly convinced that the replacement scripts were not going to become a chronic maintenance burden again, and you will not convince me of this with a straight translation, only by bringing major simplifications to the entire build process to the table.

[Leont]

Would it help to offer canned values for people who want some default set of ciphers and such? (and predefined not to do compatibility symbols?). That way one could choose build without depending on perl, at cost of configurability.

Just trying to think out of the box here, I'm not sure if such an approach is viable or useful.

[zackw]

@Leont Unfortunately, that would be almost as much work as replacing the perl scripts entirely.