OpenSSL 3.x support needed for Fedora, CentOS, RHEL

[major]

• UAMQP Package Version: 1.4.2
• Operating System: Fedora 36 (Rawhide)
• Python Version: 3.10.0rc2

Describe the bug
Fedora 36 contains [OpenSSL 3|https://fedoraproject.org/wiki/Changes/OpenSSL3.0] and uamqp fails to compile with it. (Full error pasted below.)

To Reproduce
Steps to reproduce the behavior:

1. podman run --rm -it registry.fedoraproject.org/fedora:rawhide
2. In the container, install packages: dnf -y install cmake make gcc-c++ openssl-devel cython
3. In the container, install uamqp: pip -vvv install --no-binary :all: uamqp

Expected behavior
I expected the module to be built.

Additional context

    [ 51%] Building C object deps/azure-c-shared-utility/CMakeFiles/aziotsharedutil.dir/adapters/tlsio_openssl.c.o
    /tmp/pip-req-build-n02l9k5x/src/vendor/azure-uamqp-c/deps/azure-c-shared-utility/adapters/tlsio_openssl.c: In function ‘engine_destroy’:
    /tmp/pip-req-build-n02l9k5x/src/vendor/azure-uamqp-c/deps/azure-c-shared-utility/adapters/tlsio_openssl.c:752:9: warning: ‘ENGINE_free’ is deprecated: Since OpenSSL 3.0 [-Wdeprecated-declarations]
      752 |         ENGINE_free(tls->engine); // Release structural reference.
          |         ^~~~~~~~~~~
    In file included from /tmp/pip-req-build-n02l9k5x/src/vendor/azure-uamqp-c/deps/azure-c-shared-utility/adapters/tlsio_openssl.c:9:
    /usr/include/openssl/engine.h:493:27: note: declared here
      493 | OSSL_DEPRECATEDIN_3_0 int ENGINE_free(ENGINE *e);
          |                           ^~~~~~~~~~~
    /tmp/pip-req-build-n02l9k5x/src/vendor/azure-uamqp-c/deps/azure-c-shared-utility/adapters/tlsio_openssl.c: In function ‘engine_load’:
    /tmp/pip-req-build-n02l9k5x/src/vendor/azure-uamqp-c/deps/azure-c-shared-utility/adapters/tlsio_openssl.c:762:5: warning: ‘ENGINE_by_id’ is deprecated: Since OpenSSL 3.0 [-Wdeprecated-declarations]
      762 |     tls->engine = ENGINE_by_id(tls->engine_id);
          |     ^~~
    In file included from /tmp/pip-req-build-n02l9k5x/src/vendor/azure-uamqp-c/deps/azure-c-shared-utility/adapters/tlsio_openssl.c:9:
    /usr/include/openssl/engine.h:336:31: note: declared here
      336 | OSSL_DEPRECATEDIN_3_0 ENGINE *ENGINE_by_id(const char *id);
          |                               ^~~~~~~~~~~~
    /tmp/pip-req-build-n02l9k5x/src/vendor/azure-uamqp-c/deps/azure-c-shared-utility/adapters/tlsio_openssl.c: In function ‘add_certificate_to_store’:
    /tmp/pip-req-build-n02l9k5x/src/vendor/azure-uamqp-c/deps/azure-c-shared-utility/adapters/tlsio_openssl.c:951:24: warning: assignment discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
      951 |             bio_method = BIO_s_mem();
          |                        ^
    /tmp/pip-req-build-n02l9k5x/src/vendor/azure-uamqp-c/deps/azure-c-shared-utility/adapters/tlsio_openssl.c: In function ‘create_openssl_instance’:
    /tmp/pip-req-build-n02l9k5x/src/vendor/azure-uamqp-c/deps/azure-c-shared-utility/adapters/tlsio_openssl.c:1047:9: warning: ‘TLSv1_2_method’ is deprecated: Since OpenSSL 1.1.0 [-Wdeprecated-declarations]
     1047 |         method = TLSv1_2_method();
          |         ^~~~~~
    In file included from /tmp/pip-req-build-n02l9k5x/src/vendor/azure-uamqp-c/deps/azure-c-shared-utility/adapters/tlsio_openssl.c:5:
    /usr/include/openssl/ssl.h:2024:50: note: declared here
     2024 | OSSL_DEPRECATEDIN_1_1_0 __owur const SSL_METHOD *TLSv1_2_method(void); /* TLSv1.2 */
          |                                                  ^~~~~~~~~~~~~~
    /tmp/pip-req-build-n02l9k5x/src/vendor/azure-uamqp-c/deps/azure-c-shared-utility/adapters/tlsio_openssl.c:1051:9: warning: ‘TLSv1_1_method’ is deprecated: Since OpenSSL 1.1.0 [-Wdeprecated-declarations]
     1051 |         method = TLSv1_1_method();
          |         ^~~~~~
    In file included from /tmp/pip-req-build-n02l9k5x/src/vendor/azure-uamqp-c/deps/azure-c-shared-utility/adapters/tlsio_openssl.c:5:
    /usr/include/openssl/ssl.h:2016:50: note: declared here
     2016 | OSSL_DEPRECATEDIN_1_1_0 __owur const SSL_METHOD *TLSv1_1_method(void); /* TLSv1.1 */
          |                                                  ^~~~~~~~~~~~~~
    /tmp/pip-req-build-n02l9k5x/src/vendor/azure-uamqp-c/deps/azure-c-shared-utility/adapters/tlsio_openssl.c:1055:9: warning: ‘TLSv1_method’ is deprecated: Since OpenSSL 1.1.0 [-Wdeprecated-declarations]
     1055 |         method = TLSv1_method();
          |         ^~~~~~
    In file included from /tmp/pip-req-build-n02l9k5x/src/vendor/azure-uamqp-c/deps/azure-c-shared-utility/adapters/tlsio_openssl.c:5:
    /usr/include/openssl/ssl.h:2008:50: note: declared here
     2008 | OSSL_DEPRECATEDIN_1_1_0 __owur const SSL_METHOD *TLSv1_method(void); /* TLSv1.0 */
          |                                                  ^~~~~~~~~~~~
    /tmp/pip-req-build-n02l9k5x/src/vendor/azure-uamqp-c/deps/azure-c-shared-utility/adapters/tlsio_openssl.c: In function ‘tlsio_openssl_init’:
    /tmp/pip-req-build-n02l9k5x/src/vendor/azure-uamqp-c/deps/azure-c-shared-utility/adapters/tlsio_openssl.c:1213:5: warning: ‘ERR_load_BIO_strings’ is deprecated: Since OpenSSL 3.0 [-Wdeprecated-declarations]
     1213 |     ERR_load_BIO_strings();
          |     ^~~~~~~~~~~~~~~~~~~~
    In file included from /usr/include/openssl/cryptoerr.h:17,
                     from /usr/include/openssl/crypto.h:38,
                     from /usr/include/openssl/comp.h:22,
                     from /usr/include/openssl/ssl.h:28,
                     from /tmp/pip-req-build-n02l9k5x/src/vendor/azure-uamqp-c/deps/azure-c-shared-utility/adapters/tlsio_openssl.c:5:
    /usr/include/openssl/cryptoerr_legacy.h:31:27: note: declared here
       31 | OSSL_DEPRECATEDIN_3_0 int ERR_load_BIO_strings(void);
          |                           ^~~~~~~~~~~~~~~~~~~~
    /tmp/pip-req-build-n02l9k5x/src/vendor/azure-uamqp-c/deps/azure-c-shared-utility/adapters/tlsio_openssl.c: In function ‘tlsio_openssl_deinit’:
    /tmp/pip-req-build-n02l9k5x/src/vendor/azure-uamqp-c/deps/azure-c-shared-utility/adapters/tlsio_openssl.c:1241:5: warning: ‘ERR_remove_thread_state’ is deprecated: Since OpenSSL 1.1.0 [-Wdeprecated-declarations]
     1241 |     ERR_remove_thread_state(NULL);
          |     ^~~~~~~~~~~~~~~~~~~~~~~
    In file included from /tmp/pip-req-build-n02l9k5x/src/vendor/azure-uamqp-c/deps/azure-c-shared-utility/adapters/tlsio_openssl.c:6:
    /usr/include/openssl/err.h:473:30: note: declared here
      473 | OSSL_DEPRECATEDIN_1_1_0 void ERR_remove_thread_state(void *);
          |                              ^~~~~~~~~~~~~~~~~~~~~~~
    /tmp/pip-req-build-n02l9k5x/src/vendor/azure-uamqp-c/deps/azure-c-shared-utility/adapters/tlsio_openssl.c: In function ‘tlsio_openssl_setoption’:
    /tmp/pip-req-build-n02l9k5x/src/vendor/azure-uamqp-c/deps/azure-c-shared-utility/adapters/tlsio_openssl.c:1713:13: warning: ‘ENGINE_load_builtin_engines’ is deprecated: Since OpenSSL 3.0 [-Wdeprecated-declarations]
     1713 |             ENGINE_load_builtin_engines();
          |             ^~~~~~~~~~~~~~~~~~~~~~~~~~~
    In file included from /tmp/pip-req-build-n02l9k5x/src/vendor/azure-uamqp-c/deps/azure-c-shared-utility/adapters/tlsio_openssl.c:9:
    /usr/include/openssl/engine.h:358:28: note: declared here
      358 | OSSL_DEPRECATEDIN_3_0 void ENGINE_load_builtin_engines(void);
          |                            ^~~~~~~~~~~~~~~~~~~~~~~~~~~
    [ 52%] Building C object deps/azure-c-shared-utility/CMakeFiles/aziotsharedutil.dir/adapters/x509_openssl.c.o
    /tmp/pip-req-build-n02l9k5x/src/vendor/azure-uamqp-c/deps/azure-c-shared-utility/adapters/x509_openssl.c: In function ‘load_certificate_chain’:
    /tmp/pip-req-build-n02l9k5x/src/vendor/azure-uamqp-c/deps/azure-c-shared-utility/adapters/x509_openssl.c:81:28: error: invalid use of incomplete typedef ‘SSL_CTX’ {aka ‘struct ssl_ctx_st’}
       81 |                 if (ssl_ctx->extra_certs != NULL)
          |                            ^~
    In file included from /usr/include/openssl/ssl.h:31,
                     from /tmp/pip-req-build-n02l9k5x/src/vendor/azure-uamqp-c/deps/azure-c-shared-utility/inc/azure_c_shared_utility/x509_openssl.h:7,
                     from /tmp/pip-req-build-n02l9k5x/src/vendor/azure-uamqp-c/deps/azure-c-shared-utility/adapters/x509_openssl.c:4:
    /tmp/pip-req-build-n02l9k5x/src/vendor/azure-uamqp-c/deps/azure-c-shared-utility/adapters/x509_openssl.c:83:45: error: invalid use of incomplete typedef ‘SSL_CTX’ {aka ‘struct ssl_ctx_st’}
       83 |                     sk_X509_pop_free(ssl_ctx->extra_certs, X509_free);
          |                                             ^~
    /tmp/pip-req-build-n02l9k5x/src/vendor/azure-uamqp-c/deps/azure-c-shared-utility/adapters/x509_openssl.c:84:28: error: invalid use of incomplete typedef ‘SSL_CTX’ {aka ‘struct ssl_ctx_st’}
       84 |                     ssl_ctx->extra_certs = NULL;
          |                            ^~
    /tmp/pip-req-build-n02l9k5x/src/vendor/azure-uamqp-c/deps/azure-c-shared-utility/adapters/x509_openssl.c: In function ‘load_key_RSA’:
    /tmp/pip-req-build-n02l9k5x/src/vendor/azure-uamqp-c/deps/azure-c-shared-utility/adapters/x509_openssl.c:140:5: warning: ‘EVP_PKEY_get1_RSA’ is deprecated: Since OpenSSL 3.0 [-Wdeprecated-declarations]
      140 |     RSA* privatekey = EVP_PKEY_get1_RSA(evp_key);
          |     ^~~
    In file included from /usr/include/openssl/x509.h:29,
                     from /usr/include/openssl/ssl.h:31,
                     from /tmp/pip-req-build-n02l9k5x/src/vendor/azure-uamqp-c/deps/azure-c-shared-utility/inc/azure_c_shared_utility/x509_openssl.h:7,
                     from /tmp/pip-req-build-n02l9k5x/src/vendor/azure-uamqp-c/deps/azure-c-shared-utility/adapters/x509_openssl.c:4:
    /usr/include/openssl/evp.h:1348:16: note: declared here
     1348 | struct rsa_st *EVP_PKEY_get1_RSA(EVP_PKEY *pkey);
          |                ^~~~~~~~~~~~~~~~~
    /tmp/pip-req-build-n02l9k5x/src/vendor/azure-uamqp-c/deps/azure-c-shared-utility/adapters/x509_openssl.c:150:9: warning: ‘SSL_CTX_use_RSAPrivateKey’ is deprecated: Since OpenSSL 3.0 [-Wdeprecated-declarations]
      150 |         if (SSL_CTX_use_RSAPrivateKey(ssl_ctx, privatekey) != 1)
          |         ^~
    In file included from /tmp/pip-req-build-n02l9k5x/src/vendor/azure-uamqp-c/deps/azure-c-shared-utility/inc/azure_c_shared_utility/x509_openssl.h:7,
                     from /tmp/pip-req-build-n02l9k5x/src/vendor/azure-uamqp-c/deps/azure-c-shared-utility/adapters/x509_openssl.c:4:
    /usr/include/openssl/ssl.h:1803:12: note: declared here
     1803 | __owur int SSL_CTX_use_RSAPrivateKey(SSL_CTX *ctx, RSA *rsa);
          |            ^~~~~~~~~~~~~~~~~~~~~~~~~
    /tmp/pip-req-build-n02l9k5x/src/vendor/azure-uamqp-c/deps/azure-c-shared-utility/adapters/x509_openssl.c:162:9: warning: ‘RSA_free’ is deprecated: Since OpenSSL 3.0 [-Wdeprecated-declarations]
      162 |         RSA_free(privatekey);
          |         ^~~~~~~~
    In file included from /usr/include/openssl/x509.h:36,
                     from /usr/include/openssl/ssl.h:31,
                     from /tmp/pip-req-build-n02l9k5x/src/vendor/azure-uamqp-c/deps/azure-c-shared-utility/inc/azure_c_shared_utility/x509_openssl.h:7,
                     from /tmp/pip-req-build-n02l9k5x/src/vendor/azure-uamqp-c/deps/azure-c-shared-utility/adapters/x509_openssl.c:4:
    /usr/include/openssl/rsa.h:293:28: note: declared here
      293 | OSSL_DEPRECATEDIN_3_0 void RSA_free(RSA *r);
          |                            ^~~~~~~~
    /tmp/pip-req-build-n02l9k5x/src/vendor/azure-uamqp-c/deps/azure-c-shared-utility/adapters/x509_openssl.c: In function ‘x509_openssl_add_engine_key’:
    /tmp/pip-req-build-n02l9k5x/src/vendor/azure-uamqp-c/deps/azure-c-shared-utility/adapters/x509_openssl.c:227:5: warning: ‘ENGINE_init’ is deprecated: Since OpenSSL 3.0 [-Wdeprecated-declarations]
      227 |     if (!ENGINE_init(engine))
          |     ^~
    In file included from /tmp/pip-req-build-n02l9k5x/src/vendor/azure-uamqp-c/deps/azure-c-shared-utility/adapters/x509_openssl.c:14:
    /usr/include/openssl/engine.h:620:27: note: declared here
      620 | OSSL_DEPRECATEDIN_3_0 int ENGINE_init(ENGINE *e);
          |                           ^~~~~~~~~~~
    /tmp/pip-req-build-n02l9k5x/src/vendor/azure-uamqp-c/deps/azure-c-shared-utility/adapters/x509_openssl.c:234:9: warning: ‘ENGINE_set_default’ is deprecated: Since OpenSSL 3.0 [-Wdeprecated-declarations]
      234 |         if (!ENGINE_set_default(engine, ENGINE_METHOD_ALL))
          |         ^~
    In file included from /tmp/pip-req-build-n02l9k5x/src/vendor/azure-uamqp-c/deps/azure-c-shared-utility/adapters/x509_openssl.c:14:
    /usr/include/openssl/engine.h:708:27: note: declared here
      708 | OSSL_DEPRECATEDIN_3_0 int ENGINE_set_default(ENGINE *e, unsigned int flags);
          |                           ^~~~~~~~~~~~~~~~~~
    /tmp/pip-req-build-n02l9k5x/src/vendor/azure-uamqp-c/deps/azure-c-shared-utility/adapters/x509_openssl.c:241:13: warning: ‘ENGINE_load_private_key’ is deprecated: Since OpenSSL 3.0 [-Wdeprecated-declarations]
      241 |             EVP_PKEY* evp_key = ENGINE_load_private_key(engine, x509privatekey_id, NULL, NULL);
          |             ^~~~~~~~
    In file included from /tmp/pip-req-build-n02l9k5x/src/vendor/azure-uamqp-c/deps/azure-c-shared-utility/adapters/x509_openssl.c:14:
    /usr/include/openssl/engine.h:638:11: note: declared here
      638 | EVP_PKEY *ENGINE_load_private_key(ENGINE *e, const char *key_id,
          |           ^~~~~~~~~~~~~~~~~~~~~~~
    /tmp/pip-req-build-n02l9k5x/src/vendor/azure-uamqp-c/deps/azure-c-shared-utility/adapters/x509_openssl.c:264:9: warning: ‘ENGINE_finish’ is deprecated: Since OpenSSL 3.0 [-Wdeprecated-declarations]
      264 |         if (!ENGINE_finish(engine))  // Release functional reference.
          |         ^~
    In file included from /tmp/pip-req-build-n02l9k5x/src/vendor/azure-uamqp-c/deps/azure-c-shared-utility/adapters/x509_openssl.c:14:
    /usr/include/openssl/engine.h:628:27: note: declared here
      628 | OSSL_DEPRECATEDIN_3_0 int ENGINE_finish(ENGINE *e);
          |                           ^~~~~~~~~~~~~
    /tmp/pip-req-build-n02l9k5x/src/vendor/azure-uamqp-c/deps/azure-c-shared-utility/adapters/x509_openssl.c: In function ‘x509_openssl_add_certificates’:
    /tmp/pip-req-build-n02l9k5x/src/vendor/azure-uamqp-c/deps/azure-c-shared-utility/adapters/x509_openssl.c:353:24: warning: assignment discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
      353 |             bio_method = BIO_s_mem();
          |                        ^
    gmake[2]: *** [deps/azure-c-shared-utility/CMakeFiles/aziotsharedutil.dir/build.make:804: deps/azure-c-shared-utility/CMakeFiles/aziotsharedutil.dir/adapters/x509_openssl.c.o] Error 1
    gmake[1]: *** [CMakeFiles/Makefile2:943: deps/azure-c-shared-utility/CMakeFiles/aziotsharedutil.dir/all] Error 2
    gmake: *** [Makefile:146: all] Error 2
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-req-build-n02l9k5x/setup.py", line 343, in <module>
        setup(
      File "/usr/local/lib/python3.10/site-packages/setuptools/__init__.py", line 153, in setup
        return distutils.core.setup(**attrs)
      File "/usr/lib64/python3.10/distutils/core.py", line 148, in setup
        dist.run_commands()
      File "/usr/lib64/python3.10/distutils/dist.py", line 966, in run_commands
        self.run_command(cmd)
      File "/usr/lib64/python3.10/distutils/dist.py", line 985, in run_command
        cmd_obj.run()
      File "/usr/local/lib/python3.10/site-packages/setuptools/command/install.py", line 61, in run
        return orig.install.run(self)
      File "/usr/lib64/python3.10/distutils/command/install.py", line 579, in run
        self.run_command('build')
      File "/usr/lib64/python3.10/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/usr/lib64/python3.10/distutils/dist.py", line 985, in run_command
        cmd_obj.run()
      File "/usr/lib64/python3.10/distutils/command/build.py", line 135, in run
        self.run_command(cmd_name)
      File "/usr/lib64/python3.10/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/usr/lib64/python3.10/distutils/dist.py", line 985, in run_command
        cmd_obj.run()
      File "/tmp/pip-req-build-n02l9k5x/setup.py", line 214, in run
        self.build_cmake(ext)
      File "/tmp/pip-req-build-n02l9k5x/setup.py", line 271, in build_cmake
        subprocess.check_call(joined_cmd, shell=True, universal_newlines=True, env=build_env)
      File "/usr/lib64/python3.10/subprocess.py", line 369, in check_call
        raise CalledProcessError(retcode, cmd)
    subprocess.CalledProcessError: Command 'cmake --build . --config Release' returned non-zero exit status 2.
    Running setup.py install for uamqp ... error

[major]

If you're in a hurry and you want to reproduce the issue quickly from Fedora 36, just build this container using docker or buildah:

FROM registry.fedoraproject.org/fedora:rawhide
RUN dnf -y install cmake cython gcc-c++ findutils git make openssl-devel patch python3-devel python3-pip
RUN git clone https://github.com/Azure/azure-uamqp-python
# Apply a small patch to compile with GCC 11.x.
RUN curl -sL https://src.fedoraproject.org/rpms/python-uamqp/raw/rawhide/f/python-uamqp-1.2.14-gcc11.patch | patch -p0 -d azure-uamqp-python
# Don't treat warnings as errors.
RUN grep -rl "\-Werror" azure-uamqp-python/ | xargs sed -i 's/-Werror//'
RUN pip install --no-binary :all: azure-uamqp-python/

[yunhaoling]

thanks for reaching out. openssl 3.0 is fairly new, we will take a look into it.

[major]

Thanks, @yunhaoling! 🎉

[yunhaoling]

hey @major , apologize for the late update.

As you might know, uamqp is depending on a C library: https://github.com/Azure/azure-c-shared-utility and from what I learned it doesn't yet support OpenSSL3.0, discussion could be found here: Azure/azure-c-shared-utility#566.

I feel sorry that currently there's nothing I could with the underlying C code except waiting for the C team to support it -- What I can do is that I'll leave a comment in the discussion adding a data point for the openssl 3.0 support, hope that could help to some extent :).

May I ask whether you're using uamqp library alone to build application or you're using azure-eventhub, azure-servicebus instead?

[major]

@yunhaoling Thanks for the reply! I'm packaging azure-cli on Fedora and this is one of the requirements within azure-cli or one of the Python SDKs. In the meantime, I can link to OpenSSL 1.1 and that works.

[jiapei-nexera]

Looking forward to openssl3 support

[major]

Red Hat Enterprise Linux (RHEL) 9 made the move to OpenSSL 3.0 as well and I'm unable to package uamqp there, either. That affects CentOS Stream 9, Rocky, Alma, etc.

Looks like there's a discussion underway here: Azure/azure-c-shared-utility#566

[dustinchilson]

azure-c-shared-utility merged support Azure/azure-c-shared-utility#604
as did
azure-uamqp-c, Azure/azure-uamqp-c#423

[Gunni]

Hey,

uamqp-1.6.0 from pip still affected.

[andrea-cassioli-maersk]

Hey, any update on this? still a problem for uampq 1.6.3 on Linux Alpine

[glaubitz]

A version bump would be useful so uamqp could be updated in the affected distributions.

openSUSE is now affected as well as it recently switched to OpenSSL 3.0.

[kashifkhan]

Hi All,

sorry for the delay in response. I'm working on a few changes to bring about openSSL 3.0 support and am planning to release in the next cycle based on internal testing etc.

I was able to build the module successfully on Fedora rawhide using the following:

FROM fedora:rawhide
RUN dnf -y install cmake cython gcc-c++ findutils git make openssl-devel patch python3-devel python3-pip
RUN git clone --single-branch --branch fix_openssl3 https://github.com/kashifkhan/azure-uamqp-python.git
RUN pip install --no-binary :all: azure-uamqp-python/