aarch64 release compiled with different glibc version?

[AlistairB]

Hi,

Thanks for all the great recent work on stack! I am quite excited to see a new aarch64 compatible release. I have attempted to include this release in the official haskell docker images, however it has failed (The amd64 version is working fine). Notably: /usr/local/bin/stack: /lib/aarch64-linux-gnu/libm.so.6: version GLIBC_2.29 not found (required by /usr/local/bin/stack). (We use emulation for aarch64 but seems to work well so far.)

I think this suggests that the aarch64 version of stack is being compiled with a different version of glibc?

It may work with something like debian bullseye, but I think amd64 and aarch64 should be compiled with the same version of glibc. In terms of usefulness debian buster compatibility is much more useful than bullseye at this point. This is because neither ghc nor cabal currently support aarch64 debian bullseye.

[mpilgrem]

Hi @AlistairB, thanks for reporting that. The following are mostly notes for me and may well not answer directly your issue.

Stack 2.9.1 for the Linux/AArch64 platform is built with GHC 9.2.4 (https://www.haskell.org/ghc/download_ghc_9_2_4.html#linux_aarch64) - and the GHC developers' notes refer to 'a Debian 10 system'. Debian 10 is known as 'Buster'. Debian 10 comes with libc6 '2.28-10+deb10u1' (source: https://packages.debian.org/buster/libs/).

The Stack executable is built in the linux-arm64 job of the integration-tests.yml GitHub workflow. That workflow refers to the arm64.Dockerfile. That docker file starts with FROM: ubuntu: 20.04 and then downloads various Ubuntu packages using apt-get. Ubuntu 20.04 is known as 'focal'.

The latest version of libc6 for Ubuntu 20.04.04/ARM64 is 'Ubuntu GLIBC 2.31-0ubuntu9' (source: https://packages.ubuntu.com/focal/libc6).

[AlistairB]

Thanks for the info. I've done a bit more testing on using an ubuntu 22.04 AMI AWS EC2 instance running on ARM (t4g.small with 50GB storage. If you go too small they can run out of disk).

I installed docker using these steps being sure to do the post-install steps.

Then I executed the following:

$ docker run -it debian:buster bash
$ apt update && apt install -y curl
$ curl -sSL https://get.haskellstack.org/ | sh
$ stack --version
stack: /lib/aarch64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by stack)

Doing the above with docker run -it debian:bullseye bash does work. For amd64 both buster / bullseye do work.

I guess really this is the same as this issue. What I don't understand is why amd64 buster works.

Hmm it seems that the docker image being used to build amd64 is in fact ubuntu 16.04? I guess ultimately both amd64 and aarch64 should both be built on the same OS version? I'm assuming if both of them are ubuntu 18.04 and older the libc6 should be older than 2.28 and it should work for debian buster and up.

[mpilgrem]

It seems to me sensible to build Stack on Linux/AArch64 using the same 'system' as the GHC developers' build of GHC for Linux/AArch64 that is then used to build Stack. If that is right, then I assume the solution is to have different contents for arm64.Dockerfile, which starts:

FROM ubuntu:20.04

RUN DEBIAN_FRONTEND=noninteractive apt-get update && apt-get install -y \
    curl build-essential curl libffi-dev libffi7 libgmp-dev libgmp10 libncurses-dev libncurses5 libtinfo5 libnuma-dev xz-utils \
    g++ gcc libc6-dev libffi-dev libgmp-dev make zlib1g-dev git gnupg netbase

RUN cd /tmp && \
    curl -L https://github.com/llvm/llvm-project/releases/download/llvmorg-9.0.1/clang+llvm-9.0.1-aarch64-linux-gnu.tar.xz --output /tmp/llvm.tar.xz && \
    unxz /tmp/llvm.tar.xz && \
    tar xfv /tmp/llvm.tar --strip-components 1 -C /usr && \
    rm /tmp/llvm.tar

My experience with Linux and Docker is limited - essentially to the GitHub workflows on this repository - but I'll open a pull request and experiment.

[AlistairB]

I think there are 2 strategies. The current strategy for stack (amd64) seems to be to build with Ubuntu 16.04 and produce a single linux binary. As Ubuntu 16.04 is very old, this binary works on newer OSes such as Debian buster / bullseye and newer Ubuntus as I guess the glibc stuff is backwards compatible (I'm also not experienced with C toolchain stuff). If stack wants to stick with this approach which has some merit, then I think the aarch64 stack binary should be also be built with Ubuntu 16.04.

The approach taken by cabal and GHC is to produce multiple specific binaries built with and for specific OS / version combos. See:

• https://downloads.haskell.org/~cabal/cabal-install-3.8.1.0/
• https://downloads.haskell.org/~ghc/9.4.2/

Stack could alternatively try this approach instead. I think this strategy is more effort but more reliable as you build the binaries with the OS / version they are targeting. (In the case of alpine I believe they are statically linked binaries so do not care what versions of stuff exist on the OS running them)

Whichever approach is taken, I think changing the processor architecture should not change the OS / toolchain used to build the binary. So I guess in terms of your PR I would try using Ubuntu 16.04 with a similar configuration to the amd64 docker image as the potential quick fix.

Also, no particular rush on this as the docker official Haskell images have never included stack aarch64 support since it has not worked in the past. They have already been updated to 2.9.1 for amd64.

[mpilgrem]

@AlistairB, I thought, perhaps naively, that the Linux/x86_64 versions of Stack were being built under ubuntu-latest (ie Ubuntu 20.04), given the GitHub runner in the integration-tests job of the integration-tests.yml workflow. How does Ubuntu 16.04 come in? Is it something to do with the Nix lines in the workflow?

[mpilgrem]

@AlistairB, I have a request - could you possibly look at the Linux-ARM64 artifacts generated by the integration-tests workflow trigged by my pull request fix5871 at the bottom of https://github.com/commercialhaskell/stack/actions/runs/3109038128. I am hoping they include the bindist that you need on Debian 10.

Stack is built by Stack. In the case of Stack 2.9.1 on Linux/AArch64 the builder was the 'official' Stack 2.7.1 on Linux/AArch64. However, that Stack 2.7.1 also does not run on Debian 10! So, I 'borrowed' GHCup's 'unofficial' Stack 2.7.5 on Linux/AArch64, which - it turns out - does run on Debian 10.

[AlistairB]

Yeah sorry, to be fair this is me making a fair amount of inference without actually understanding exactly how it is built. My assumption for this is based on:

• The fpco/stack-build images which as I understand it are used by the --docker feature are based on ubuntu 16.04.
• The amd64 stack binary is working on debian 10 buster. My limited understanding of glibc is that it is closely tied to the OS and you can't simply run ubuntu 20.04 then downgrade glibc as it would break the OS. Therefore I assume it must be built with something older than ubuntu 16.04.

[AlistairB]

Hmm perhaps it is built in 'alpine' mode which does trigger it to attempt to build a static binary via --docker (which thus uses the 16.04 based images)?

EDIT: Or perhaps it actually is building a static binary which does not not care what glibc version is available on the OS.

[AlistairB]

could you possibly look at the Linux-ARM64 artifacts generated by the integration-tests workflow trigged by my pull request

will do

[AlistairB]

I did a bit of testing with this version and it works (doing a very basic stack --version check) for debian buster, bullseye and ubuntu 20.04. So I guess this could be a solution?

[mpilgrem]

I'll tap the Slack group for advice about the best way forward.

[mpilgrem]

I am hoping that the first release candidate for Stack 2.9.3 (see here) will not have the problem that affected Stack 2.9.1 on Linux/AArch64.

[AlistairB]

Looks good! haskell/docker-haskell#93