rustup: rustup-init hangs in armv7 docker container running on an arm64 Linux with `reqwest` backend

Problem

Running curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh in armv7 docker container running on an arm64 Linux hangs.

Steps

Run a docker container with --platform linux/arm/v7 on an arm64 Linux, install curl and run the sh.rustup.sh script

$ docker run --rm -it --platform linux/arm/v7 ubuntu:22.04 bash
root@668fa2822782:/# apt update && apt install curl -y
root@668fa2822782:/# curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --verbose
info: downloading installer
info: profile set to 'default'
info: default host triple is aarch64-unknown-linux-gnu
verbose: installing toolchain 'stable-aarch64-unknown-linux-gnu'
verbose: toolchain directory: '/root/.rustup/toolchains/stable-aarch64-unknown-linux-gnu'
info: syncing channel updates for 'stable-aarch64-unknown-linux-gnu'
verbose: creating temp file: /root/.rustup/tmp/dqlfqns1_reba5ms_file
verbose: downloading file from: 'https://static.rust-lang.org/dist/channel-rust-stable.toml.sha256'
verbose: downloading with reqwest

hangs forever at the verbose: downloading with reqwest step, use RUSTUP_USE_CURL=1 works fine

root@668fa2822782:/# RUSTUP_USE_CURL=1 curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | RUSTUP_USE_CURL=1 sh -s -- -y --verbose
info: downloading installer
info: profile set to 'default'
info: default host triple is aarch64-unknown-linux-gnu
verbose: installing toolchain 'stable-aarch64-unknown-linux-gnu'
verbose: toolchain directory: '/root/.rustup/toolchains/stable-aarch64-unknown-linux-gnu'
info: syncing channel updates for 'stable-aarch64-unknown-linux-gnu'
verbose: creating temp file: /root/.rustup/tmp/0lfud7winasgeiob_file
verbose: downloading file from: 'https://static.rust-lang.org/dist/channel-rust-stable.toml.sha256'
verbose: downloading with curl
verbose: deleted temp file: /root/.rustup/tmp/0lfud7winasgeiob_file
verbose: no update hash at: '/root/.rustup/update-hashes/stable-aarch64-unknown-linux-gnu'
verbose: creating temp file: /root/.rustup/tmp/zs3_t8gv_pyprmk6_file.toml
verbose: downloading file from: 'https://static.rust-lang.org/dist/channel-rust-stable.toml'
verbose: downloading with curl
verbose: checksum passed
verbose: creating temp file: /root/.rustup/tmp/zwsok2uoflekbroj_file
verbose: downloading file from: 'https://static.rust-lang.org/dist/channel-rust-stable.toml.asc'
verbose: downloading with curl
verbose: deleted temp file: /root/.rustup/tmp/zwsok2uoflekbroj_file
verbose: Good signature from on https://static.rust-lang.org/dist/channel-rust-stable.toml from:
verbose: from builtin Rust release key
verbose:   RSAEncryptSign/85AB96E6-FA1BE5FE - Rust Language (Tag and Release Signing Key) <rust-key@rust-lang.org>
verbose:   Fingerprint: 108F 6620 5EAE B0AA A8DD 5E1C 85AB 96E6 FA1B E5FE
verbose: deleted temp file: /root/.rustup/tmp/zs3_t8gv_pyprmk6_file.toml
info: latest update on 2022-11-03, rust version 1.65.0 (897e37553 2022-11-02)
info: downloading component 'cargo'
verbose: creating Download Directory directory: '/root/.rustup/downloads'
verbose: downloading file from: 'https://static.rust-lang.org/dist/2022-11-03/cargo-1.65.0-aarch64-unknown-linux-gnu.tar.xz'
verbose: downloading with curl
verbose: checksum passed
info: downloading component 'clippy'
verbose: downloading file from: 'https://static.rust-lang.org/dist/2022-11-03/clippy-1.65.0-aarch64-unknown-linux-gnu.tar.xz'
verbose: downloading with curl
verbose: checksum passed
info: downloading component 'rust-docs'
verbose: downloading file from: 'https://static.rust-lang.org/dist/2022-11-03/rust-docs-1.65.0-aarch64-unknown-linux-gnu.tar.xz'
verbose: downloading with curl
verbose: checksum passed
info: downloading component 'rust-std'
verbose: downloading file from: 'https://static.rust-lang.org/dist/2022-11-03/rust-std-1.65.0-aarch64-unknown-linux-gnu.tar.xz'
verbose: downloading with curl
verbose: checksum passed
info: downloading component 'rustc'
verbose: downloading file from: 'https://static.rust-lang.org/dist/2022-11-03/rustc-1.65.0-aarch64-unknown-linux-gnu.tar.xz'
verbose: downloading with curl
 55.9 MiB /  79.4 MiB ( 70 %)   0 B/s in  1s ETA: Unknown^C

Possible Solution(s)

No response

Notes

No response

Rustup version

rustup-init 1.25.1 (bb60b1e89 2022-07-12)

Installed toolchains

N/A

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 40 (23 by maintainers)

Most upvoted comments

I found the underlying issue: https://github.com/rust-lang/rust/issues/60605 armv6l LLVM uses the CP15 barrier instruction, which are deprecated since ARMv7. Linux by default is compiled to emulate and trace these calls, to emit a warning:

[ 3392.600164] cp15barrier_handler: 26605006 callbacks suppressed
[ 3392.600172] "reqwest-interna" (4115) uses deprecated CP15 Barrier instruction at 0xb12d14
[ 3392.600178] "reqwest-interna" (4115) uses deprecated CP15 Barrier instruction at 0xb12d38
...
[ 4393.668156] cp15barrier_handler: 25438047 callbacks suppressed
[ 4393.668164] "rustup-init" (4234) uses deprecated CP15 Barrier instruction at 0x8dc5d0
[ 4393.668170] "rustup-init" (4234) uses deprecated CP15 Barrier instruction at 0x8dc5d8
...

Above shows that reqwest is not the only thing which emits this call, but rustup-init does as well, later on when installing the downloaded component, leading to the same hang. So RUSTUP_USE_CURL=1 and similar help only with one step, to hang with the next, as also reflected by OP logs.

We even faced this a year earlier in a different context: doing Rust builds on Raspberry Pi 4 with 64-bit kernel on a 32-bit Raspberry Pi OS. It did not appear anymore/was not needed as we do not ship/promote the Raspbian/armv6l userland images for 64-bit capable Raspberry Pi models anymore, but have Debian/armv7l userland images, if really 32-bit is required for some reason. So I forgot about it.

It also explains why I face the issue on armv6l userland only, as the armv7l LLVM seems to not make use of this instruction anymore, as it is deprecated for this architecture already. As this was different for OP, it might have changed within the last years.

Since aarch64 CPUs usually still support the deprecated CP15 barrier instruction, Linux can be configured to make use of the hardware instruction instead of emulating it, which bypasses all the tracing and warning and solves the issue:

sudo sysctl -w 'abi.cp15_barrier=2'

Or permanently:

echo 'abi.cp15_barrier=2' | sudo tee /etc/sysctl.d/99-cp15_barrier.conf
sudo sysctl -p /etc/sysctl.d/99-cp15_barrier.conf

The default value is 1, which means to emulate the instruction, causing the issue.

Alternatively the kernel can be compiled without the support to emulate this instruction:

CONFIG_CP15_BARRIER_EMULATION=n

Let’s hope this gets some more attention at LLVM, since future ARM versions won’t support abi.cp15_barrier=2 (hardware instruction) anymore.

I tried to disable cURL and use the default reqwest backend instead with the 1.27 beta, and it also didn’t change anything compared to before.

Here is the output:

info: downloading installer
info: profile set to 'default'
info: setting default host triple to armv7-unknown-linux-gnueabihf
verbose: creating update-hash directory: '/root/.rustup/update-hashes'
verbose: installing toolchain 'nightly-armv7-unknown-linux-gnueabihf'
verbose: toolchain directory: '/root/.rustup/toolchains/nightly-armv7-unknown-linux-gnueabihf'
info: syncing channel updates for 'nightly-armv7-unknown-linux-gnueabihf'
verbose: creating temp root: /root/.rustup/tmp
verbose: creating temp file: /root/.rustup/tmp/rmn_jarvvywfunwl_file
verbose: downloading file from: 'https://static.rust-lang.org/dist/channel-rust-nightly.toml.sha256'
verbose: downloading with reqwest

Here a script to replicate via Docker Raspbian container:

cat > entrypoint.sh << 'EOF'
#!/bin/sh
echo '#/bin/sh\n[ "$*" = -m ] && echo armv6l || /bin/uname "$@"' > /usr/local/bin/uname
chmod +x /usr/local/bin/uname
uname -a
uname -m
apt-get -q update
apt-get -qq install curl
curl -sSf https://sh.rustup.rs | sh -s -- -y -v --default-host arm-unknown-linux-gnueabihf
EOF
chmod +x entrypoint.sh
docker run --rm -v './entrypoint.sh:/entrypoint.sh' --entrypoint '/entrypoint.sh' tianon/raspbian:bookworm-slim

Alternatively to test with a specific rustup-init build and use strace:

cat > entrypoint.sh << 'EOF'
#!/bin/sh
uname -a
uname -m
apt-get -q update
apt-get -qq install curl strace
curl -sSfO https://static.rust-lang.org/rustup/dist/arm-unknown-linux-gnueabihf/rustup-init
chmod +x rustup-init
strace ./rustup-init -y -v --default-host arm-unknown-linux-gnueabihf
EOF
chmod +x entrypoint.sh
docker run --rm -v './entrypoint.sh:/entrypoint.sh' --entrypoint '/entrypoint.sh' tianon/raspbian:bookworm-slim

--default-host arm-unknown-linux-gnueabihf is irrelevant here, as the toolchain architecture does not matter: it fails at rustup-init. I just added it to avoid possible confusion, as rustup-init selects by default armv7-unknown-linux-gnueabihf for the toolchain in this case, even if uname -m returns armv6l, while rustup-init.sh correctly selects the arm-unknown-linux-gnueabihf build of rustup-init. Some inconsistency, but a different topic.

When not using the arm-unknown-linux-gnueabihf build of rustup-init, but the armv7-unknown-linux-gnueabihf one instead (skipping the uname swapper or downloading it explicitly), for me it works, other than OP here. Maybe something has changed in the meantime.

Testing the new rustup-init dev version:

cat > entrypoint.sh << 'EOF'
#!/bin/sh
echo '#/bin/sh\n[ "$*" = -m ] && echo armv6l || /bin/uname "$@"' > /usr/local/bin/uname
chmod +x /usr/local/bin/uname
uname -a
uname -m
apt-get -q update
apt-get -qq install curl
export RUSTUP_UPDATE_ROOT=https://dev-static.rust-lang.org/rustup
curl -sSf https://sh.rustup.rs | sh -s -- -y -v --default-host arm-unknown-linux-gnueabihf
EOF
chmod +x entrypoint.sh
docker run --rm -v './entrypoint.sh:/entrypoint.sh' --entrypoint '/entrypoint.sh' tianon/raspbian:bookworm-slim

Has this issue been reproduced already with a modern 1.28.x build?

@djc Yes, if I have correctly understood @MichaIng’s report:

In the container they first issue:

root@container:~# ./rustup-init.sh -y -v
info: downloading installer
info: profile set to 'default'
info: default host triple is armv7-unknown-linux-gnueabihf
verbose: installing toolchain 'stable-armv7-unknown-linux-gnueabihf'
verbose: toolchain directory: '/root/.rustup/toolchains/stable-armv7-unknown-linux-gnueabihf'
info: syncing channel updates for 'stable-armv7-unknown-linux-gnueabihf'
verbose: creating temp file: /root/.rustup/tmp/zc_4ju74ta6in8lw_file
verbose: downloading file from: 'https://static.rust-lang.org/dist/channel-rust-stable.toml.sha256'
verbose: downloading with reqwest
^C

… note the default target was detected as armv7. I then suggested (in https://github.com/rust-lang/rustup/issues/4189#issuecomment-2664443987) trying out our beta build, but got negative feedback:

RUSTUP_UPDATE_ROOT=https://dev-static.rust-lang.org/rustup hence does not fix it for me. _https://github.com/rust-lang/rustup/issues/4189#issuecomment-2665984110_

This is why I’m asking for a minimal reproduction to make sure we are on the same page.

For good measure I also tried the same Dockerfile but with Ubuntu 18.04, 20.04 and 22.04 and I see the exact same behaviour across all of them.

That certainly looks like docker is providing the host’s cpuinfo through. This is the issue we had on reqwest - https://github.com/seanmonstar/reqwest/issues/642 which eventually boiled down to https://github.com/lxc/lxcfs/issues/553 which is what eventually fixed it in that case - I wonder if Docker needs equivalent help.