cross: Build fails with `sh: 1: cargo: not found`

The problem is that the image uses the host’s own Rust install, which in some cases (e.g. NixOS) cannot be run on Ubuntu. You can work around it by building an image with its own Rust install, for example:

FROM japaric/x86_64-unknown-freebsd:v0.1.4
RUN apt-get update && apt-get install -y wget
RUN mkdir -m777 /opt/rust /opt/cargo
ENV RUSTUP_HOME=/opt/rust CARGO_HOME=/opt/cargo PATH=/opt/cargo/bin:$PATH
RUN wget --https-only --secure-protocol=TLSv1_2 -O- https://sh.rustup.rs | sh /dev/stdin -y
RUN rustup target add x86_64-unknown-freebsd
RUN printf '#!/bin/sh\nexport CARGO_HOME=/opt/cargo\nexec /bin/sh "$@"\n' >/usr/local/bin/sh
RUN chmod +x /usr/local/bin/sh

Rust is installed outside ~ so the user the build runs as can access it. About the last two lines, I wasn’t able to override CARGO_HOME from the shell profile so I overrode the shell instead. Not pretty, but gets the job done.

Using something from the host breakes many of the otherwise strong guarantees alike “runs fine with me, will run fine with you” docker has to offer. IMO this should be at least considered to be changed.

I can confirm this issue, it also happens on armv7-unknown-linux-musleabihf-0.2.0

shouldn’t rust be installed in those container images ? or I misunderstood the way this tool works

FROM rustembedded/cross:armv7-unknown-linux-musleabihf-0.2.0

ENV PATH="/root/.cargo/bin:$PATH"

RUN curl https://sh.rustup.rs -sSf | sh -s -- -y --default-toolchain stable

@Emilgardis I just took a look at #524 . It seems to rely on a NIX_STORE environment variable to determine the location of the nix store. The NIX_STORE variable is not set by nix normally as far as I can tell. The default location is /nix/store so I’ll try setting the NIX_STORE variable and see if it fixes it.

I have the same issue in WSL2 (Ubuntu 20.04) running in Windows 11. Running Docker via Rancher Desktop in Windows and confirmed DinD works with docker run --rm -i hello-world.

Reproducing it like this:

Start build container that includes Docker

docker run --rm -it --entrypoint bash \
  -e CROSS_DOCKER_IN_DOCKER=true \
  -v /var/run/docker.sock:/var/run/docker.sock \
  gcr.io/cloud-builders/docker

Install Rust and cross, create project

apt update && apt install build-essential -y
curl https://sh.rustup.rs -sSf | sh -s -- -y --default-toolchain stable
source $HOME/.cargo/env
cargo install cross
cargo init hello-world && cd hello-world

Try to build with cross

cross build
info: downloading component 'rust-src'
info: installing component 'rust-src'
sh: 1: cargo: not found

---

EDIT: The exact same steps works on Windows 11 and Manjaro 21.2.6 with Linux 5.17.

+1 on current NixOS unstable 21.11pre304626.8ecc61c91a5 (Porcupine)

I’m getting this issue as well. What’s the point of the Docker image if the tool isn’t preinstalled? Is this project dead?

For those running cross from within a Docker image, see https://github.com/rust-embedded/cross#docker-in-docker.

An example of doing this interactively using the official rust image:

docker-host$ docker run -it -e CROSS_DOCKER_IN_DOCKER=true -v /var/run/docker.sock:/var/run/docker.sock -v $(pwd):/usr/src/myproj -w /usr/src/myproj --rm rust bash
root@c7fddc07d311:/usr/src/myproj# wget https://download.docker.com/linux/static/stable/x86_64/docker-19.03.12.tgz
root@c7fddc07d311:/usr/src/myproj# tar -C /usr/local/bin --strip-components=1 -xf docker-19.03.12.tgz docker/docker
root@c7fddc07d311:/usr/src/myproj# cargo install cross
root@c7fddc07d311:/usr/src/myproj# cross build --target aarch64-unknown-linux-gnu

Running cross with the NIX_STORE environment variable set to /nix/store seems to work for me on NixOS.

I hit the same issue but with a different root cause. I just want to share some notes that might help to people seeing this problem.

1. The docker images used by cross doesn’t include rust/cargo or other tool. These image provides the libraries required for cross-compiling.
2. The cross tool installs the requested toolchain in the system and then when docker is run some volumes are mounted to make rust visible inside the container.
3. The cargo: not found error is because some of these volume mounts is not working correctly.

Adding the --verbose flag to the cross build command helps to get the docker command used to launch the container. Try to run the same command from your terminal and start checking which of the volumes is failing.

Just created a pull request that should fix the NIX_STORE problem.

This feels like a very WSL specific issue so let me know if you want me to open a new issue to track this.

Yes please do!

Some more context to the WSL issue without really having a solution.

I had a look at the code for when cross starts a docker container and if you use DOCKER_IN_DOCKER=true it inspects the current container (docker inspect $HOSTNAME) to determine the volumes and tries to mount them, something like /var/lib/docker/overlay2/xxx/merged/root/.cargo:/cargo:Z. This is needed since afaik you won’t be able to mount volumes from Docker to Docker but instead have to reference the host machine/volume (ref).

However, when running Rancher Desktop (and I think Docker Desktop as well?) in combination with WSL, it sets up two different distros to host Docker:

$ wsl --list -v
  NAME                    STATE           VERSION
* Ubuntu                  Running         2
  rancher-desktop         Running         2
  rancher-desktop-data    Running         2

The container data volume does not exist in Ubuntu or rancher-desktop, however it exists in rancher-desktop-data. This means that if I look at the output for docker inspect $HOSTNAME and .[0].GraphDriver.Data, it will refer to paths which does not exist and is not accessible in the current container that tries to run cross.

I usually don’t use Windows or WSL so I need to read up more on how volume mounts + Windows + WSL + Docker in Docker works but right now it looks as simple as that the cross container that gets launched from the Docker container running in WSL basically can’t mount any toolchain successfully which also explains why cargo is not found.

This feels like a very WSL specific issue so let me know if you want me to open a new issue to track this.

@IllustratedMan-code good that it worked for you, but that solution is not how cross is supposed to work (as I alluded to here and mentioned here), cross is supposed to mount your system sysroot for x86_64-unknown-linux-gnu into /

For nix specifically, the issue should have been fixed with #524

We have to continue and look into this, but its hard when we’re not able to reproduce the issue

this solution worked

FROM ghcr.io/cross-rs/arm-unknown-linux-gnueabihf:main
RUN apt-get update && apt-get install -y wget
RUN mkdir -m777 /opt/rust /opt/cargo
ENV RUSTUP_HOME=/opt/rust CARGO_HOME=/opt/cargo PATH=/opt/cargo/bin:$PATH
RUN wget --https-only --secure-protocol=TLSv1_2 -O- https://sh.rustup.rs | sh /dev/stdin -y
RUN rustup target add arm-unknown-linux-gnueabihf
RUN printf '#!/bin/sh\nexport CARGO_HOME=/opt/cargo\nexec /bin/sh "$@"\n' >/usr/local/bin/sh
RUN chmod +x /usr/local/bin/sh

As am i, also NixOS 20.09:

Unable to find image 'rustembedded/cross:x86_64-pc-windows-gnu-0.2.1' locally
x86_64-pc-windows-gnu-0.2.1: Pulling from rustembedded/cross
d7c3167c320d: Pull complete 
131f805ec7fd: Pull complete 
322ed380e680: Pull complete 
6ac240b13098: Pull complete 
d7bf0568dabc: Pull complete 
981aecd3bf48: Pull complete 
6697c741376d: Pull complete 
cc1566a6c043: Pull complete 
7e4da0436841: Pull complete 
6a8d255df934: Pull complete 
8928e758bdd0: Pull complete 
f5186e3ba9b6: Pull complete 
8f352a8fa374: Pull complete 
4ad1b6ba757d: Pull complete 
Digest: sha256:5b195dffc3876e155bb95aac347be1440d1cd5bc85fb46c7d9b3feeed2b16d8c
Status: Downloaded newer image for rustembedded/cross:x86_64-pc-windows-gnu-0.2.1
sh: 1: cargo: not found

@andresv rustembedded/cross:aarch64-unknown-linux-musl doesn’t contain a cross executable, only the corresponding cross toolchain. You already installed cross in your cimg/rust:1.47.0 container, so just replace the whole docker run... command with cross build --target aarch64-unknown-linux-musl.

I had still to make some workaround for using cross in gitlab-ci even when using the master branch. But I managed to do it, you can find information in this comment: https://github.com/rust-embedded/cross/issues/273#issuecomment-528300479

This probably does not apply for nixOS users. But for the OP surely 😃

I am also encountering this issue. Nothing special about my setup, though.

[evanjs@nixentoo:~/src/scratch/sdl2test]$ cross build --target x86_64-pc-windows-gnu
Unable to find image 'japaric/x86_64-pc-windows-gnu:v0.1.14' locally
v0.1.14: Pulling from japaric/x86_64-pc-windows-gnu
b8a43e1b7983: Pull complete 
5c5185a06822: Pull complete 
8458c4a4e317: Pull complete 
6d0388b275d4: Pull complete 
56603a33e04b: Pull complete 
2cedd67f6904: Pull complete 
5d40de413ec5: Pull complete 
fbd3dda3b3f5: Pull complete 
ee23f3a81425: Pull complete 
22d70475d56a: Pull complete 
cc2c4c043bf3: Pull complete 
ace8770c3426: Pull complete 
Digest: sha256:b164f497e18f89133de814898d838603b07a6b1feb80b06271ddb3fb86128cc8
Status: Downloaded newer image for japaric/x86_64-pc-windows-gnu:v0.1.14
sh: 1: cargo: not found

Using NixOS 19.03 on 4.19.24 Docker: 18.09.2 Cross: 0.1.14 cargo 1.34.0-nightly (5c6aa46e6 2019-02-22)