woodpecker: Cannot run pipeline on Kubernetes: CreateContainerError
Component
agent
Describe the bug
I’m trying to get started with the Kubernetes backend (installed via the official helm chart). When I try to run a pipeline, the container gets stuck with CreateContainerError.
Error: container create failed: chdir: No such file or directory
After some debugging, it seems like the issue is related to the spec.workingDir being set to a directory that doesn’t exist at the time that the container is starting. I’m not exactly an expert here, but maybe we could leave the working dir unset and then change directories after the repo is cloned?
I can provide more debugging information, if that would be help.
System Info
{"source":"https://github.com/woodpecker-ci/woodpecker","version":"next-24204ecdeb"}
This is a vanilla Kubernetes 1.28.2 cluster installed via kubeadm. The nodes are Debian 12. The container runtime is CRI-O which was installed following the install guide here.
Additional context
I’m running CRI-O (I’m not sure if that’s relevant), and I discussed this issue a bit on the CRI-O repo here.
I’m using Forgejo as my forge. If I comment out the spec.workingDir in the Pod yaml and attempt to run the Pod manually I get an error like:
+ git fetch --no-tags --depth=1 --filter=tree:0 origin +master:
error: RPC failed; HTTP 403 curl 22 The requested URL returned error: 403
fatal: expected flush after ref listing
exit status 128
I’m not sure if that’s another issue or somehow related, but I thought it was worth mentioning.
Validations
- Read the Contributing Guidelines.
- Read the docs.
- Check that there isn’t already an issue that reports the same bug to avoid creating a duplicate.
- Checked that the bug isn’t fixed in the
nextversion already [https://woodpecker-ci.org/faq#which-version-of-woodpecker-should-i-use] - Check that this is a concrete bug. For Q&A join our Discord Chat Server or the Matrix room.
About this issue
- Original URL
- State: open
- Created 3 years ago
- Comments: 16 (16 by maintainers)
Commits related to this issue
- Add notes about CRI-O specific config CRI-O currently requires some additional config for pipelines to run. See #2510 — committed to dominic-p/woodpecker by dominic-p 3 years ago
- Add notes about CRI-O specific config (#2546) CRI-O currently requires some additional config for pipelines to run. This adds some basic documentation to the kubernetes section explaining the issue... — committed to woodpecker-ci/woodpecker by dominic-p 3 years ago
Interesting update from the CRI-O devs here. It appears that this behavior is not by design and they would be open to a PR to fix it. Again, go isn’t my thing, so this might be out of reach for me. But, at the very least we know a fix would be welcome.
Yahtzee! the
workspaceconfig did the trick. I didn’t bother testing an initContainer given that the initial approach worked, but I imagine that would solve the problem as well. I also updated the OP as requested with more system info.I’m not sure if this should be considered a documentation issue (e.g. CRI-O users need to configure the workspace manually). Or, if some change can/should be made to the kubernetes backend. It seems like out-of-the-box CRI-O support would be nice, but, unfortunately, I’m not a go programmer, so I wouldn’t be able to help much with a PR.
Secondary question: Now that I’m unstuck, I’m running into a new error trying to get my buildah container to run. I think it might be due to the
securityContextof the container. I tried configuring it like this (building off of the docs here), but it doesn’t seem to be working. If I dump the Pod YAML there is nosecurityContextconfig.Is it not possible to set the security context via the pipeline config? Or, am I just doing it wrong? I can also open a separate issue about this if that would be preferrable.