containerd-shim-spin: Spin apps fail to run - compiled with incompatible wasmtime version

I built the shim from main as of writing and I’m seeing the following when I attempt to run seemingly any Spin app.

Steps I followed:

  1. Build the shim via eg cargo build --release; move it to where containerd expects it
  2. Run containerd via eg sudo containerd -c /path/to/containerd/config.toml a. Note: containerd version containerd github.com/containerd/containerd v1.7.14 dcf2847247e18caba8dce86522029642f60fe96b
  3. Pull Spin app (published with Spin v2.4.2 via spin registry push) via ctr eg sudo ctr image pull docker.io/vdice/static-fileserver:latest
  4. Run Spin app via eg sudo ctr run --rm --runtime io.containerd.spin.v2 docker.io/vdice/static-fileserver:latest static bogus

See the following in the containerd/runwasi/shim logs:

time="2024-04-12T02:19:00.310100534Z" level=info msg="starting instance: static"

time="2024-04-12T02:19:00.310223623Z" level=info msg="calling start function"

time="2024-04-12T02:19:00.310247832Z" level=info msg="setting up wasi"

time="2024-04-12T02:19:00.310726562Z" level=info msg=" >>> configuring spin oci application 4"

time="2024-04-12T02:19:00.310737395Z" level=info msg="<<< writing wasm artifact with length 5882984 config to cache, near "/.cache/registry/manifests""

time="2024-04-12T02:19:00.315409687Z" level=info msg="writing spin oci config to "/spin.json""

time="2024-04-12T02:19:00.346827672Z" level=error msg="run_wasi ERROR >>>  failed: failed to build spin trigger

Caused by:
    0: Failed to instantiate component 'static-fileserver'
    1: deserializing module "/.cache/registry/wasm/sha256:5f05b15f0f7cd353d390bc5ebffec7fe25c6a6d7a05b9366c86dcb1a346e9f0f"
    2: Module was compiled with incompatible Wasmtime version '19.0.1'"

time="2024-04-12T02:19:00.347279817Z" level=info msg="error running start function: failed to build spin trigger"

time="2024-04-12T02:19:00.350726929Z" level=info msg="deleting instance: static"

time="2024-04-12T02:19:00.350799974Z" level=info msg="cgroup manager V2 will be used"

INFO[2024-04-11T20:19:00.351348290-06:00] shim disconnected                             id=static namespace=default
WARN[2024-04-11T20:19:00.351442211-06:00] cleaning up after shim disconnected           id=static namespace=default
INFO[2024-04-11T20:19:00.351447753-06:00] cleaning up dead shim                         namespace=default
time="2024-04-12T02:19:00.351227785Z" level=info msg="Shutting down shim instance"

I noticed the shim’s wasmtime version was recently bumped to 19.0.1 in https://github.com/spinkube/containerd-shim-spin/pull/71 – unintended consequence? Perhaps there are other deps needing to be bumped as well for compatibility?

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 16 (13 by maintainers)

Most upvoted comments

(An update to mention that Spin apps now run on the shim built from main after the revert, as expected. Thanks again!)

Yeah, I think we need to revert the Wasmtime bump for now.

The issue we’re seeing here seems to come out of a mismatch between the compiler output and Wasm engine — and I’m really confused by it, since I assumed we were doing a compatibility check somewhere (see

https://github.com/spinkube/containerd-shim-spin/blob/1f44c1e5eeb5cedb3faf73fd659a70a48f41c6af/containerd-shim-spin/src/engine.rs#L434

). But perhaps this is a larger issue with the caching feature that we need to revisit.

~This seems it might be related to the fact that the Spin compiled on version 0.19 but we are using linker and other functions from 0.17.0: https://github.com/containerd/runwasi/blob/5309512c6dac6d03df9e67190f560689790c04f7/crates/containerd-shim-wasmtime/src/instance.rs#L153-L166~

~The pre-compile and the checks we perform are all handled by the shim its self, but then when we set everything up we are linking with a different version.~

Monday morning 🤦, that is a different shim all together so there must be something else going on.

I believe we should consider adding containerd config options for the shim to enable feature code paths. I don’t believe Rust compilation flags would be desirable in cases like this since the user is likely to not have compiled the shim in the first place. Adding containerd config options for the shim would allow for on node configuration and debugging. It could also provide a path for turning on and off newly release / unstable features w/o recompilation.

+1

Thanks for the additional details, @kingdonb!

For context, the issue is coming from the ahead of time compilation feature that was added to the shim in #32. This is an optimization that ensures a Wasm component is only compiled once on any given node, so scaling up is significantly faster compared to when having to compile (especially when a pod only is allowed little CPU resources).

The issue we’re seeing here seems to come out of a mismatch between the compiler output and Wasm engine — and I’m really confused by it, since I assumed we were doing a compatibility check somewhere (see https://github.com/spinkube/containerd-shim-spin/blob/1f44c1e5eeb5cedb3faf73fd659a70a48f41c6af/containerd-shim-spin/src/engine.rs#L434). But perhaps this is a larger issue with the caching feature that we need to revisit.

Having a test that catches this scenario should be the resolution of this issue.

Yup, definitely a test needed to be added. I also think the dependabot should disable wasmtime bump since it’s a core dependency of Spin.

Thanks @devigned! Meanwhile, curious as to what we can add in CI to catch this? Did it slip by the tests due to the tested Spin Apps still being wrapped in Docker images, hence not hitting the precompilation flow?

That is exactly where my mind went too. Having a test that catches this scenario should be the resolution of this issue.