containerd-shim-spin: Spin apps fail to run - compiled with incompatible wasmtime version
I built the shim from main as of writing and I’m seeing the following when I attempt to run seemingly any Spin app.
Steps I followed:
- Build the shim via eg
cargo build --release; move it to where containerd expects it - Run containerd via eg
sudo containerd -c /path/to/containerd/config.tomla. Note: containerd versioncontainerd github.com/containerd/containerd v1.7.14 dcf2847247e18caba8dce86522029642f60fe96b - Pull Spin app (published with Spin v2.4.2 via
spin registry push) via ctr egsudo ctr image pull docker.io/vdice/static-fileserver:latest - Run Spin app via eg
sudo ctr run --rm --runtime io.containerd.spin.v2 docker.io/vdice/static-fileserver:latest static bogus
See the following in the containerd/runwasi/shim logs:
time="2024-04-12T02:19:00.310100534Z" level=info msg="starting instance: static"
time="2024-04-12T02:19:00.310223623Z" level=info msg="calling start function"
time="2024-04-12T02:19:00.310247832Z" level=info msg="setting up wasi"
time="2024-04-12T02:19:00.310726562Z" level=info msg=" >>> configuring spin oci application 4"
time="2024-04-12T02:19:00.310737395Z" level=info msg="<<< writing wasm artifact with length 5882984 config to cache, near "/.cache/registry/manifests""
time="2024-04-12T02:19:00.315409687Z" level=info msg="writing spin oci config to "/spin.json""
time="2024-04-12T02:19:00.346827672Z" level=error msg="run_wasi ERROR >>> failed: failed to build spin trigger
Caused by:
0: Failed to instantiate component 'static-fileserver'
1: deserializing module "/.cache/registry/wasm/sha256:5f05b15f0f7cd353d390bc5ebffec7fe25c6a6d7a05b9366c86dcb1a346e9f0f"
2: Module was compiled with incompatible Wasmtime version '19.0.1'"
time="2024-04-12T02:19:00.347279817Z" level=info msg="error running start function: failed to build spin trigger"
time="2024-04-12T02:19:00.350726929Z" level=info msg="deleting instance: static"
time="2024-04-12T02:19:00.350799974Z" level=info msg="cgroup manager V2 will be used"
INFO[2024-04-11T20:19:00.351348290-06:00] shim disconnected id=static namespace=default
WARN[2024-04-11T20:19:00.351442211-06:00] cleaning up after shim disconnected id=static namespace=default
INFO[2024-04-11T20:19:00.351447753-06:00] cleaning up dead shim namespace=default
time="2024-04-12T02:19:00.351227785Z" level=info msg="Shutting down shim instance"
I noticed the shim’s wasmtime version was recently bumped to 19.0.1 in https://github.com/spinkube/containerd-shim-spin/pull/71 – unintended consequence? Perhaps there are other deps needing to be bumped as well for compatibility?
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 16 (13 by maintainers)
(An update to mention that Spin apps now run on the shim built from main after the revert, as expected. Thanks again!)
Yeah, I think we need to revert the Wasmtime bump for now.
~This seems it might be related to the fact that the Spin compiled on version 0.19 but we are using linker and other functions from 0.17.0: https://github.com/containerd/runwasi/blob/5309512c6dac6d03df9e67190f560689790c04f7/crates/containerd-shim-wasmtime/src/instance.rs#L153-L166~
~The pre-compile and the checks we perform are all handled by the shim its self, but then when we set everything up we are linking with a different version.~
Monday morning 🤦, that is a different shim all together so there must be something else going on.
+1
Thanks for the additional details, @kingdonb!
For context, the issue is coming from the ahead of time compilation feature that was added to the shim in #32. This is an optimization that ensures a Wasm component is only compiled once on any given node, so scaling up is significantly faster compared to when having to compile (especially when a pod only is allowed little CPU resources).
The issue we’re seeing here seems to come out of a mismatch between the compiler output and Wasm engine — and I’m really confused by it, since I assumed we were doing a compatibility check somewhere (see https://github.com/spinkube/containerd-shim-spin/blob/1f44c1e5eeb5cedb3faf73fd659a70a48f41c6af/containerd-shim-spin/src/engine.rs#L434). But perhaps this is a larger issue with the caching feature that we need to revisit.
Yup, definitely a test needed to be added. I also think the dependabot should disable
wasmtimebump since it’s a core dependency of Spin.That is exactly where my mind went too. Having a test that catches this scenario should be the resolution of this issue.
https://github.com/spinkube/containerd-shim-spin/pull/80