actions-runner-controller: containers create by Github workflow have wrong dockerMTU

Hello,

I’ve come across this issue as well since the Kubernetes network I’m setting up uses a smaller MTU (1420 instead of the usual 1500).

Despite the MTU being configurable with the dockerMTU setting on the Runner spec, it only affects the default docker network and does not propagate to networks created by the Github runner. This poses a problem when running workflows that use containers such as the actions/checkout@v2 one.

So, I followed-up on the workaround proposed by @FalconerTC here and changed it to support the sidecar dind container since /etc/docker/daemon.json is not available:

#!/usr/bin/env bash

# Inspired by https://github.com/actions-runner-controller/actions-runner-controller/issues/848#issuecomment-929394653
# Inspired by https://github.com/actions/runner/issues/775#issuecomment-927826684

if [[ $1 = "network" ]] && [[ $2 = "create" ]] ; then
    shift; shift #pop 2 first parameters

    MTU=$(docker network inspect bridge --format '{{index .Options "com.docker.network.driver.mtu"}}' 2>/dev/null); 
    if [[ ! -z "$MTU" ]]; then 
        /usr/local/bin/docker.bin network create --opt com.docker.network.driver.mtu=$MTU "${@}"  
    else
        /usr/local/bin/docker.bin network create "${@}"
    fi
else
    #just call docker as normal if not network create
    /usr/local/bin/docker.bin "${@}"
fi

This has the added benefit of propagating whatever MTU is set on the default docker bridge, which works in both dind and docker in runner situations.

I’ve pushed a docker image with the change to tiagomelo/github-actions-runner which basically places the script shim the docker container and propagate the docker MTU to newly created networks.

However, since using containers in Github Actions is pretty common and - apparently - having Kubernetes overlay networks with MTUs smaller than 1500, I think this should be integrated in the official summerwind/actions-runner image. I’m willing to open a PR with the change if you wish.

Meanwhile, this can be tested by creating a runner with the custom runner image:

apiVersion: actions.summerwind.dev/v1alpha1
kind: Runner
metadata:
  name: test-runner
spec:
  dockerEnabled: true
  dockerMTU: 1420
  image: tiagomelo/docker-actions-runner
  repository: <github-repository>

You can validate this is working with:

kubectl exec -it -c runner test-runner -- /bin/bash

within the runner:

docker network inspect bridge --format '{{index .Options "com.docker.network.driver.mtu"}}' 2>/dev/null
# returns 1420 (or whatever MTU is set)

docker run --rm rancher/curl https://github.com >/dev/null
# should complete

docker network create test

docker network inspect bridge --format '{{index .Options "com.docker.network.driver.mtu"}}' 2>/dev/null
# returns 1420 (or whatever MTU is set)

docker run --rm --network test rancher/curl https://github.com >/dev/null
# should complete

docker network rm test

Edit: Published a repository with the image under tiagoblackcode/github-actions-runner.

Hello,

I finally had some time to work on this. Please check out #1201 which adds a new docker image to the repository that propagates the MTU setting to networks created in Github workflows.

I’m willing to help you with support.

I’m gonna prepare a PR with the secondary image and regarding it in the next couple of days.

I agree with the general sentiment that the issue is not in ARC but instead in actions/runner or even in docker, honestly. Since docker supports the --mtu flag, it should use its MTU value as default to any networks created unless specified otherwise.

However, since it seems we’re kind of stuck on where to implement this, I suggest the following:

• Document in the Troubleshooting section the issue, and the workaround.
• Optionally, create a secondary image in ARC with the fix, and a USE AT YOUR OWN RISK advert. Alternatively, point to my image, but I’d rather have this in the official repository since it has more visibility. I suggest a secondary image because this can have unintended consequences we’re not aware of at the moment.
• Bring visibility to the actions/runner#775 issue.

This way ARC users have a documented way to fix the issue while we wait for actions/runner to support this.

@mumoshu what are your thoughts on this?

@NicklasWallgren regarding https://github.com/actions/runner/pull/1650, instead of having an env variable I think it’s better to inspect the bridge network, figure out its MTU and use it to create the network.

Edit:

There’s an open PR in https://github.com/moby/moby/pull/43197 to propagate the MTU set in --mtu to networks created without an explicit MTU.

Can this be reopened since the issue still persist? We’re running into the same issue where we use dind and buildah to create container images.

@tiagoblackcode Were you able to create a PR for this?

@erichorwath Hey. Please set dockerMTU appropriately as documented in https://github.com/actions-runner-controller/actions-runner-controller#additional-tweaks. Reading through all the docker-mtu related issues we had experienced https://github.com/actions-runner-controller/actions-runner-controller/issues?q=label%3A"docker+mtu+issue"+is%3Aclosed would also help.