cargo-chef: cargo chef prepare failed using cargo-chef:0.1.56

When using lukemathwalker/cargo-chef:0.1.56-rust-slim-bullseye (or, more specifically, lukemathwalker/cargo-chef:latest-rust-slim-bullseye), I’m now getting the following error:

 > [planner 2/2] RUN cargo chef prepare --recipe-path recipe.json:
#17 0.181 Error: Failed to compute recipe
#17 0.181
#17 0.181 Caused by:
#17 0.181     0: Cannot extract Cargo metadata
#17 0.181     1: `cargo metadata` exited with an error: error: failed to parse manifest at `/nexalon/Cargo.toml`
#17 0.181
#17 0.181        Caused by:
#17 0.181          can't find library `nexalon_lib`, rename file to `src/lib.rs` or specify lib.path
#17 0.181

If I change back to using lukemathwalker/cargo-chef:0.1.55-rust-slim-bullseye then this all works fine.

My Dockerfile is as follows:

FROM lukemathwalker/cargo-chef:0.1.56-rust-slim-bullseye AS chef
WORKDIR /nexalon
RUN rustc --version



FROM chef AS modules
COPY . .
RUN find . -name src | xargs rm -rf



FROM chef AS planner
COPY --from=modules /nexalon .
RUN cargo chef prepare --recipe-path recipe.json



FROM chef AS builder
RUN apt update && apt install --yes ca-certificates openssl libssl-dev pkg-config

COPY --from=planner /nexalon/recipe.json recipe.json
RUN cargo chef cook --release --recipe-path recipe.json

RUN find . -name src | xargs rm -rf
COPY . .

RUN cargo build --release



FROM debian:bullseye-slim AS runtime
WORKDIR /nexalon
RUN apt update
RUN apt install --yes ca-certificates

COPY --from=builder /nexalon/target/release/nexalon .
CMD "/nexalon/nexalon"

ENV RUST_LOG=info

And my Cargo.toml file is:

[package]
name = "nexalon"
version = "0.1.0"
edition = "2021"

[[bin]]
name = "nexalon"

[lib]
name = "nexalon_lib"

[dependencies]
async-trait = "0.1.68"
axum = "0.6.15"
axum-macros = "0.3.7"
config = "0.13.3"
dotenv = "0.15.0"
env_logger = "0.10.0"
http = "0.2.9"
http-body = "0.4.5"
http_halforms = { version = "0.1.0", features = ["axum"] }
hyper = { version = "0.14.26", features = ["full"] }
problemdetails = { version = "0.2.1", features = ["axum"] }
serde = { version = "1.0.160", features = ["derive"] }
serde_json = { version = "1.0.96", features = ["preserve_order"] }
thiserror = "1.0.40"
time = { version = "0.3.20", features = ["formatting", "parsing", "macros", "serde-well-known"] }
tokio = { version = "1.27.0", features = ["full"] }
tower = { version = "0.4.13", features = ["full"] }
tower-http = { version = "0.4.0", features = ["full"] }
tracing = { version = "0.1.37", features = ["log", "async-await"] }
tracing-futures = { version = "0.2.5", features = ["tokio"] }
tracing-log = "0.1.3"
tracing-subscriber = { version = "0.3.16", features = ["time", "json", "env-filter"] }

[dev-dependencies]
assert2 = "0.3.10"
insta = { version = "1.29.0", features = ["json"] }
test-case = "3.1.0"
test-log = { version = "0.2.11", features = ["trace"] }

[profile.dev]
split-debuginfo = "unpacked"

Cheers

About this issue

Original URL
State: closed
Created a year ago
Comments: 19 (11 by maintainers)

Most upvoted comments

Fixing this was easy (just copy the credentials in earlier in the original image), but diagnosing the error was very difficult. Maybe 0.1.56 should be yanked and republished as 0.2 instead?

@jyn514: I’m inclined not to yank—this update is surfacing a lot of Hyrum law pain that went unnoticed in previous versions because we were not talking full advantage of the assumptions that were stated in the README (i.e. prepare takes the whole project as input). I’ll keep this issue open as a reference point in case other folks bump into issues and we’ll work through what comes 👍🏻 Unless an actual defect pops through, I’m not going to yank.

LukeMathWalker on Apr 18, 2023

It’s been a month and no new issues have been reported, so I’ll go ahead and close this. Thanks for the help folks!

LukeMathWalker on May 23, 2023

I’ve merged #211 after reviewing that it won’t cause any issues for the time being—this should allow your Dockerfiles to build unchanged @neoeinstein.

It’s going out as 0.1.59, the new batch of Docker images is currently building.

LukeMathWalker on Apr 21, 2023

@LukeMathWalker that worked 👍 . Thanks!

arusahni on Apr 20, 2023

To fix it you need to copy in your vendor folder:

# [...]
FROM builder AS rust-builder
COPY --from=rust-chef-planner /workdir/recipe.json recipe.json
# The new line!
COPY vendor/ vendor/
RUN cargo chef cook $CARGO_FEATURES $CARGO_RELEASE --recipe-path recipe.json

That should not be an issue for caching since the vendored folder will be unchanged if your dependencies are unchanged @arusahni I’m actually surprised that it was working before.

LukeMathWalker on Apr 19, 2023

If prepare is guaranteed to produce the same recipe with the same deps, I guess this doesn’t give us much. But I had some previous issues with everything being rebuilt more often than expected, and restricting the copy to Cargo.toml/Cargo.lock seems to have helped

@mirek26: This is precisely what prepare is supposed to do—you throw the entire project at it and prepare will extract the relevant info that will remain unchanged as long as your dependencies remain unchanged. If you notice any scenario where this doesn’t happen, report it as a bug and I’ll be more than happy to look at it.

However, changing it to copy all of the source files as well does fix things. Curiously, this also doesn’t cause it to need to rebuild every layer when a source file changes, which means that either it was never needed in the first place (wouldn’t surprise me!) or else something has been fixed since I first started doing that…

@sazzer: it’s possible that we had some bugs in previous versions that were causing unnecessary rebuilds, but “copy the whole project” has always been the recommended pattern (e.g. see the examples in the README). If you run into any of those unnecessary rebuilds, please file a bug report!

LukeMathWalker on Apr 18, 2023

cargo chef should serve as a solution to avoid the “Copy only Cargo.xxx” pattern. Now that it parses the project using cargo metadata, it will require the whole project to be present (as was intended originally).

If it rebuilds too often, that might be a bug in cargo chef. Hopefully with using cargo metadata we will be now also be able to reduce these kinds of unnecessary rebuilds.

Kobzol on Apr 18, 2023