python 3.12 hashlib broken on arm64

[ibuildthecloud]

If you run the following Dockerfile you'll see that python3.12 on arm64 hashlib is somehow broken.

FROM cgr.dev/chainguard/wolfi-base
RUN apk add -U python-3.12
RUN python -c 'from hashlib import sha256'

Output

 => ERROR [3/3] RUN python -c 'from hashlib import sha256'                                                                                                                                                      0.2s
------                                                                                                                                                                                                               
 > [3/3] RUN python -c 'from hashlib import sha256':                                                                                                                                                                 
0.161 ERROR:root:code for hash md5 was not found.                                                                                                                                                                    
0.161 Traceback (most recent call last):                                                                                                                                                                             
0.161   File "/usr/lib/python3.12/hashlib.py", line 245, in <module>                                                                                                                                                 
0.161     globals()[__func_name] = __get_hash(__func_name)                                                                                                                                                           
0.161                              ^^^^^^^^^^^^^^^^^^^^^^^
0.161   File "/usr/lib/python3.12/hashlib.py", line 123, in __get_builtin_constructor
0.161     raise ValueError('unsupported hash type ' + name)
0.161 ValueError: unsupported hash type md5
0.162 ERROR:root:code for hash sha1 was not found.
0.162 Traceback (most recent call last):
0.162   File "/usr/lib/python3.12/hashlib.py", line 245, in <module>
0.162     globals()[__func_name] = __get_hash(__func_name)
0.162                              ^^^^^^^^^^^^^^^^^^^^^^^
0.162   File "/usr/lib/python3.12/hashlib.py", line 123, in __get_builtin_constructor
0.162     raise ValueError('unsupported hash type ' + name)
0.162 ValueError: unsupported hash type sha1
0.162 ERROR:root:code for hash sha224 was not found.
0.162 Traceback (most recent call last):
0.162   File "/usr/lib/python3.12/hashlib.py", line 245, in <module>
0.162     globals()[__func_name] = __get_hash(__func_name)
0.162                              ^^^^^^^^^^^^^^^^^^^^^^^
0.162   File "/usr/lib/python3.12/hashlib.py", line 123, in __get_builtin_constructor
0.162     raise ValueError('unsupported hash type ' + name)
0.162 ValueError: unsupported hash type sha224
0.162 ERROR:root:code for hash sha256 was not found.
0.162 Traceback (most recent call last):
0.162   File "/usr/lib/python3.12/hashlib.py", line 245, in <module>
0.162     globals()[__func_name] = __get_hash(__func_name)
0.162                              ^^^^^^^^^^^^^^^^^^^^^^^
0.162   File "/usr/lib/python3.12/hashlib.py", line 123, in __get_builtin_constructor
0.162     raise ValueError('unsupported hash type ' + name)
0.162 ValueError: unsupported hash type sha256
0.162 ERROR:root:code for hash sha384 was not found.
0.162 Traceback (most recent call last):
0.162   File "/usr/lib/python3.12/hashlib.py", line 245, in <module>
0.162     globals()[__func_name] = __get_hash(__func_name)
0.162                              ^^^^^^^^^^^^^^^^^^^^^^^
0.162   File "/usr/lib/python3.12/hashlib.py", line 123, in __get_builtin_constructor
0.162     raise ValueError('unsupported hash type ' + name)
0.162 ValueError: unsupported hash type sha384
0.162 ERROR:root:code for hash sha512 was not found.
0.162 Traceback (most recent call last):
0.162   File "/usr/lib/python3.12/hashlib.py", line 245, in <module>
0.162     globals()[__func_name] = __get_hash(__func_name)
0.162                              ^^^^^^^^^^^^^^^^^^^^^^^
0.162   File "/usr/lib/python3.12/hashlib.py", line 123, in __get_builtin_constructor
0.162     raise ValueError('unsupported hash type ' + name)
0.162 ValueError: unsupported hash type sha512
0.162 ERROR:root:code for hash sha3_224 was not found.
0.162 Traceback (most recent call last):
0.162   File "/usr/lib/python3.12/hashlib.py", line 245, in <module>
0.162     globals()[__func_name] = __get_hash(__func_name)
0.162                              ^^^^^^^^^^^^^^^^^^^^^^^
0.162   File "/usr/lib/python3.12/hashlib.py", line 123, in __get_builtin_constructor
0.162     raise ValueError('unsupported hash type ' + name)
0.162 ValueError: unsupported hash type sha3_224
0.162 ERROR:root:code for hash sha3_256 was not found.
0.162 Traceback (most recent call last):
0.162   File "/usr/lib/python3.12/hashlib.py", line 245, in <module>
0.162     globals()[__func_name] = __get_hash(__func_name)
0.162                              ^^^^^^^^^^^^^^^^^^^^^^^
0.162   File "/usr/lib/python3.12/hashlib.py", line 123, in __get_builtin_constructor
0.162     raise ValueError('unsupported hash type ' + name)
0.162 ValueError: unsupported hash type sha3_256
0.163 ERROR:root:code for hash sha3_384 was not found.
0.163 Traceback (most recent call last):
0.163   File "/usr/lib/python3.12/hashlib.py", line 245, in <module>
0.163     globals()[__func_name] = __get_hash(__func_name)
0.163                              ^^^^^^^^^^^^^^^^^^^^^^^
0.163   File "/usr/lib/python3.12/hashlib.py", line 123, in __get_builtin_constructor
0.163     raise ValueError('unsupported hash type ' + name)
0.163 ValueError: unsupported hash type sha3_384
0.163 ERROR:root:code for hash sha3_512 was not found.
0.163 Traceback (most recent call last):
0.163   File "/usr/lib/python3.12/hashlib.py", line 245, in <module>
0.163     globals()[__func_name] = __get_hash(__func_name)
0.163                              ^^^^^^^^^^^^^^^^^^^^^^^
0.163   File "/usr/lib/python3.12/hashlib.py", line 123, in __get_builtin_constructor
0.163     raise ValueError('unsupported hash type ' + name)
0.163 ValueError: unsupported hash type sha3_512
0.163 ERROR:root:code for hash shake_128 was not found.
0.163 Traceback (most recent call last):
0.163   File "/usr/lib/python3.12/hashlib.py", line 245, in <module>
0.163     globals()[__func_name] = __get_hash(__func_name)
0.163                              ^^^^^^^^^^^^^^^^^^^^^^^
0.163   File "/usr/lib/python3.12/hashlib.py", line 123, in __get_builtin_constructor
0.163     raise ValueError('unsupported hash type ' + name)
0.163 ValueError: unsupported hash type shake_128
0.163 ERROR:root:code for hash shake_256 was not found.
0.163 Traceback (most recent call last):
0.163   File "/usr/lib/python3.12/hashlib.py", line 245, in <module>
0.163     globals()[__func_name] = __get_hash(__func_name)
0.163                              ^^^^^^^^^^^^^^^^^^^^^^^
0.163   File "/usr/lib/python3.12/hashlib.py", line 123, in __get_builtin_constructor
0.163     raise ValueError('unsupported hash type ' + name)
0.163 ValueError: unsupported hash type shake_256
0.163 Traceback (most recent call last):
0.163   File "<string>", line 1, in <module>
0.163 ImportError: cannot import name 'sha256' from 'hashlib' (/usr/lib/python3.12/hashlib.py)

[ibuildthecloud]

FWIW, python 3.13 does not exhibit this behavior, it works fine. I've just switched to python 3.13 for the short term.

[ajayk]

Can you check if you have the latest Wolfi-base

[xnox]

@ibuildthecloud did you re-pull latest cgr.dev/chainguard/wolfi-base? (no cache, to get openssl 3.4.0) as I expect sha256 to be always available..... from openssl...... which is where python gets those hash implementations from....

But will retest all combinations too.

[smoser]

I reproduced exactly the above error locally. Then a docker system prune --volumes --all followed by another attempt made it go away.

The wolfi-base I had cached locally was cgr.dev/chainguard/wolfi-base@sha256:d6b37317ae7cb5c0864189e9e5acd825386ae226a413e7c19370f5f87d150f92

$ img=cgr.dev/chainguard/wolfi-base@sha256:d6b37317ae7cb5c0864189e9e5acd825386ae226a413e7c19370f5f87d150f92
$ docker run -it --rm --entrypoint=""  "$img" \
  sh -c 'apk add python-3.12-base && python3.12 -c "import hashlib"'
...
(14/14) Installing python-3.12-base (3.12.7-r1)
...
ERROR:root:code for hash md5 was not found.
Traceback (most recent call last):
  File "/usr/lib/python3.12/hashlib.py", line 245, in <module>
    globals()[__func_name] = __get_hash(__func_name)
                             ^^^^^^^^^^^^^^^^^^^^^^^
...

[xnox]

libssl3 and libcrypto3 are backwards compatible, but not forwards compatible.

This is easiest to demonstrate with just the openssl command in the base image that @smoser posted

$ docker run --rm -ti --entrypoint sh cgr.dev/chainguard/wolfi-base@sha256:d6b37317ae7cb5c0864189e9e5acd825386ae226a413e7c19370f5f87d150f92
/ # apk add openssl~3.0
fetch https://packages.wolfi.dev/os/x86_64/APKINDEX.tar.gz
(1/1) Installing openssl (3.0.7-r3)
WARNING: Support for packages with multiple data parts will be dropped in apk-tools 3.
OK: 16 MiB in 15 packages
/ # openssl version
OpenSSL 3.0.7 1 Nov 2022 (Library: OpenSSL 3.3.1 4 Jun 2024)
/ # apk add openssl~3.1
(1/2) Installing openssl-provider-legacy (3.4.0-r1)
(2/2) Upgrading openssl (3.0.7-r3 -> 3.1.4-r1)
OK: 16 MiB in 16 packages
/ # openssl version
OpenSSL 3.1.4 24 Oct 2023 (Library: OpenSSL 3.3.1 4 Jun 2024)
/ # apk add openssl~3.2
(1/1) Upgrading openssl (3.1.4-r1 -> 3.2.1-r0)
OK: 16 MiB in 16 packages
/ # openssl version
OpenSSL 3.2.1 30 Jan 2024 (Library: OpenSSL 3.3.1 4 Jun 2024)
/ # apk add openssl~3.3
(1/1) Upgrading openssl (3.2.1-r0 -> 3.3.2-r2)
OK: 16 MiB in 16 packages
/ # openssl version
OpenSSL 3.3.2 3 Sep 2024 (Library: OpenSSL 3.3.1 4 Jun 2024)
/ # apk add openssl~3.4
(1/1) Upgrading openssl (3.3.2-r2 -> 3.4.0-r1)
OK: 16 MiB in 16 packages
/ # openssl version
openssl: /usr/lib/libssl.so.3: version `OPENSSL_3.4.0' not found (required by openssl)
openssl: /usr/lib/libcrypto.so.3: version `OPENSSL_3.4.0' not found (required by openssl)
/ # apk add -u libcrypto3 libssl3
(1/2) Upgrading libcrypto3 (3.3.1-r4 -> 3.4.0-r1)
(2/2) Upgrading libssl3 (3.3.1-r4 -> 3.4.0-r1)
OK: 15 MiB in 16 packages
/ # openssl version
OpenSSL 3.4.0 22 Oct 2024 (Library: OpenSSL 3.4.0 22 Oct 2024)
/ # exit 0

As you can see openssl binary 3.0 through to 3.3 work correctly against 3.3.1 libraries.
However openssl binary 3.4 fails against 3.3.1 libraries, unless they are upgraded.

Unfortunately, with python the runtime traceback is incomprehensible, to understand that libcrypto/libssl is out of date in the image.

The solution is to either upgrade the libcrypto3 and libssl3; or to pull a newer base image.

@ibuildthecloud Can you please confirm that executing docker pull cgr.dev/chainguard/wolfi-base:latest resolve the compatibility with python-3.12 for you? Since you show dockerfile, please ensure you always use docker build --no-cache --pull such that no cached results are used; and that base image is always attempted to be pulled to avoid such issues ever again.

[xnox]

To make this better in the future, opened melange issue chainguard-dev/melange#1621