crossplane: XFN doesn't run on AWS Bottlerocket
What happened?
xfn container doesn’t start when running on the AWS Bottlerocket nodes - returning a possibly misleading message of ‘no space left on device’
How can we reproduce it?
create an EKS cluster running Bottlerocket and start with xfn enabled.
to further debug I tried to give it as many privileges as possible:
xfn:
args:
- --debug
cache:
sizeLimit: 2Gi
enabled: true
securityContext:
readOnlyRootFilesystem: false
allowPrivilegeEscalation: true
privileged: true
hostPID: true
hostIPC: true
hostNetwork: true
capabilities:
add: ["SETUID", "SETGID", "SYS_ADMIN", "NET_ADMIN"]
runAsUser: 0
runAsGroup: 0
an strace of the xfn binary in the container results in:
from https://manpages.ubuntu.com/manpages/impish/man7/user_namespaces.7.html
ENOSPC (since Linux 4.9; beforehand EUSERS)
CLONE_NEWUSER was specified in the flags mask, and the call would cause the limit
on the number of nested user namespaces to be exceeded. See [user_namespaces](https://manpages.ubuntu.com/manpages/impish/man7/user_namespaces.7.html)(7).
the max_user_namespaces is set to 0:
from https://github.com/bottlerocket-os/bottlerocket/blob/develop/SECURITY_GUIDANCE.md
Bottlerocket does not currently support user namespaces. This means that UID 0 (root) inside the container is the same as UID 0 on the host.
I’m not sure if my lead is correct and if this actually relates to the user namespace so I could use a hand to further investigate the strace or any other leads.
What environment did it happen in?
Crossplane version: v1.11.2-stable Kube version:
Client Version: version.Info{Major:"1", Minor:"23+", GitVersion:"v1.23.13-eks-fb459a0", GitCommit:"55bd5d5cb7d32bc35e4e050f536181196fb8c6f7", GitTreeState:"clean", BuildDate:"2022-10-24T20:38:50Z", GoVersion:"go1.17.13", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"23+", GitVersion:"v1.23.16-eks-48e63af", GitCommit:"e6332a8a3feb9e0fe3db851878f88cb73d49dd7a", GitTreeState:"clean", BuildDate:"2023-01-24T19:18:15Z", GoVersion:"go1.19.5", Compiler:"gc", Platform:"linux/amd64"}
Thank you.
About this issue
- Original URL
- State: closed
- Created a year ago
- Reactions: 1
- Comments: 16 (9 by maintainers)
It does seem to work. Maybe others can test further.
No new log messages with
no space left on deviceBottlerocket (AMI ID ami-0a49771f6772955e6)
Just confirming: it does seem to work fine with the adjustment to the kernel configuration, setting a non-zero limit of user namespaces.
In the TOML settings (launch template’s user data), for example:
Careful with the quotes on the left side, they’re necessary. Otherwise, the dot in the setting name will be interpreted as a new section within the TOML document.
Tested it out with a silly echo function, that just spits out stdin to stdout (needed a new image to make
cat -the entrypoint, maybe configuring “command” and “args” would be a nice luxury once we get more important stuff out of the way during alpha/beta).Again thanks @nabuskey for the insight, sorry for missing it the first time around.
@LCaparelli thanks tested this today in one of our clusters - its working 😉
Folks, I opened a discussion in Bottlerocket’s repo: https://github.com/bottlerocket-os/bottlerocket/discussions/3318
From what I’ve been told there, it seems that the distro does support user namespaces, but limits it to 0 by default. This can be changed via user data, when launching the instance.
I still need to test this, but we might be able to make it work.
EDIT: just realized that’s what @nabuskey suggested. My bad 🙇
Looks like it is related to the number of configured available namespaces. Thank you very much for digging deep and providing relevant information.
This can be configured with
Looks like you can do it via cli too. See: https://github.com/bottlerocket-os/bottlerocket/pull/1158
If Bottlerocket doesn’t allow user namespaces this unfortunately isn’t likely to be something we can fix. This is part of why we think of
xfnas the “reference” Composition Function runner and expect that other implementations that make different tradeoffs will exist.@nabuskey I quickly checked the Bottlerocket doc
https://github.com/bottlerocket-os/bottlerocket/blob/develop/SECURITY_GUIDANCE.md#do-not-run-containers-as-uid-0
It says that
Bottlerocket does not currently support user namespaces.I am not sure if the info is up to date, but it matches the debug output from the Issue description.There is a good doc on security requirements for xfn runner https://docs.crossplane.io/knowledge-base/guides/composition-functions/#the-xfn-runner .
It says
So it looks like if user namespaces are not supported by Bottlerocket in general then xfn runner will not work