woodpecker: Cannot run pipeline on Kubernetes: CreateContainerError

Component

agent

Describe the bug

I’m trying to get started with the Kubernetes backend (installed via the official helm chart). When I try to run a pipeline, the container gets stuck with CreateContainerError.

Error: container create failed: chdir: No such file or directory

After some debugging, it seems like the issue is related to the spec.workingDir being set to a directory that doesn’t exist at the time that the container is starting. I’m not exactly an expert here, but maybe we could leave the working dir unset and then change directories after the repo is cloned?

I can provide more debugging information, if that would be help.

System Info

{"source":"https://github.com/woodpecker-ci/woodpecker","version":"next-24204ecdeb"}

This is a vanilla Kubernetes 1.28.2 cluster installed via kubeadm. The nodes are Debian 12. The container runtime is CRI-O which was installed following the install guide here.

Additional context

I’m running CRI-O (I’m not sure if that’s relevant), and I discussed this issue a bit on the CRI-O repo here.

I’m using Forgejo as my forge. If I comment out the spec.workingDir in the Pod yaml and attempt to run the Pod manually I get an error like:

+ git fetch --no-tags --depth=1 --filter=tree:0 origin +master:
error: RPC failed; HTTP 403 curl 22 The requested URL returned error: 403
fatal: expected flush after ref listing
exit status 128

I’m not sure if that’s another issue or somehow related, but I thought it was worth mentioning.

Validations

About this issue

  • Original URL
  • State: open
  • Created 3 years ago
  • Comments: 16 (16 by maintainers)

Commits related to this issue

Most upvoted comments

Interesting update from the CRI-O devs here. It appears that this behavior is not by design and they would be open to a PR to fix it. Again, go isn’t my thing, so this might be out of reach for me. But, at the very least we know a fix would be welcome.

Yahtzee! the workspace config did the trick. I didn’t bother testing an initContainer given that the initial approach worked, but I imagine that would solve the problem as well. I also updated the OP as requested with more system info.

I’m not sure if this should be considered a documentation issue (e.g. CRI-O users need to configure the workspace manually). Or, if some change can/should be made to the kubernetes backend. It seems like out-of-the-box CRI-O support would be nice, but, unfortunately, I’m not a go programmer, so I wouldn’t be able to help much with a PR.


Secondary question: Now that I’m unstuck, I’m running into a new error trying to get my buildah container to run. I think it might be due to the securityContext of the container. I tried configuring it like this (building off of the docs here), but it doesn’t seem to be working. If I dump the Pod YAML there is no securityContext config.

workspace:
  base: "/woodpecker"
  path: "/"

steps:

  # We'll remove this eventually, but for now it's nice just to make sure that the most basic
  # pipeline step works
  test:
    image: public.ecr.aws/docker/library/alpine
    commands:
      - echo Hello from test

  # The real work is done here. Build and push the container image
  build:
    image: quay.io/buildah/stable:v1.31
    commands:
      - /bin/sh ./build.sh
    backend_options:
      kubernetes:
        securityContext:
          runAsNonRoot: true
          runAsUser: 1000
          runAsGroup: 1000
          fsGroup: 1000
          capabilities:
            add: ["SETFCAP"]

Is it not possible to set the security context via the pipeline config? Or, am I just doing it wrong? I can also open a separate issue about this if that would be preferrable.