moby: docker service create doesn't allow --privileged flag

Damn, forgot to come back and congratulate this issue on its second birthday.

Missing --cap-add to use swarm mode in production, my management push me to move towards kubernetes if this option is not added soon. Do you have some plan and agenda for adding this feature please?

+1

--cap-add and --cap-drop Is a must. --privileged would be nice. Are there any plans implementing it?

Folks, please avoid filling up this issue with +1 😭 You could click the 👍 button instead.

Gave up on this and switched to Kubernetes.

On Jun 15, 2018, at 19:02, Denis notifications@github.com wrote:

+1

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

+1

--cap-add would be a huge help!

@purplesrl and @knick-burns sum it up.

At this point it is not likely that --privileged will be added as-is to docker service, as this option nullifies all security that containers provide

I don’t care about the security that containers provide. So what if a Docker container has unrestricted privileges when the host it’s running on is a £30 throw-away Raspberry Pi? Why not just permit people to use --privileged and if they shoot themselves in the foot with it, that’s on them?

What I actually do want is the service discovery, self-healing, self-containment and modularity you get from using Docker swarm.

If I may add … --privileged and/or --device* are quite critical for the case you need to run containers using GPU/CUDA calculations … Placement rules to allow these kind of containers to run on specific hosts can be used … but not being able to actually use the GPU … kind of renders the swarmmode useless for us … 😦

Please don’t +1 this issue, it’s not helpful. If you have specific requirements to need this feature, please describe the use case, which is more helpful.

At this point it is not likely that --privileged will be added as-is to docker service, as this option nullifies all security that containers provide (TL;DR; running a service/container with --privileged effectively gives root access to the host).

Instead, more fine-grained permissions are needed; while --cap-add / --cap-drop is an option, it doesn’t solve everything; for example, security profiles (SecComp, SELinux, AppArmor) are also related. Setting the right combination of options is complex (and cumbersome), which is why often people “just set --privileged” to make things work (but doing so, running highly insecure - see above).

Security should be usable, therefore a proposal is opened to work towards a design that both allows people to give services the privileges they need in a more fine-grained approach, without introducing the complexity of setting all options manually.

The proposal can be found here: Proposal: Entitlements in Moby, and the POC https://github.com/docker/libentitlement repository, also there’s a Google document describing the proposal in more detail.

We welcome feedback on the proposal; either by commenting on that issue, or taking part in the Orchestration Security SIG meetups (see https://forums.mobyproject.org/c/sig/orchestration-security)

@2stacks @blop @dalefwillis @danmanners @dfresh613 @dl00 @Farfad @galindro @getvivekv @gklbiti @joaquin386 @Kent-H @matansag @MingShyanWei @pinkal-vansia @realcbb @siavash9000 @soar @spfannemueller @TAGC @vingrad @voron3x @wangsquirrel

I removed your +1 comments, as they don’t add useful input to the discussion; if you don’t have a particular use-case / additional information that can help getting the design right, use the 👍 emoji in the top comment to let others know you’re interested in this (see @AkihiroSuda’s comment https://github.com/moby/moby/issues/24862#issuecomment-296577597). If you commented just to get notified on updates; use the subscribe button on the right side of this page.

It surprises me that security is thrown around as the reason why this has not been implemented, yet there is no inherent way to restrict egress traffic other than an external device, or by manipulating the ip tables.

I don’t really need you to be concerned with how I run my applications.

This issue is open since 2016 is there a chance that this gets added to Docker in the next years… ? This is a must have feature, for running Docker in Docker on a swarm. Please Docker team.

Sorry but this is very frustrating for me. I can’t switch to a HA setup because of this 😦

the news is all k8!

I’d really like to see it implemented soon - there are more solutions that require --privileged flag to function properly, e.g. cAdvisor which I’m using for containers performance monitoring.

Pleeeeaaaaase don’t send useless messages here. It just slowdown the process as lot of people get notified and that time is away from actual implementation.

My plan is to get #38380 released as part of 19.03 and then actual Swarm side changes on version which comes after that (19.06 I guess). Anyone who want to help plz test that and tell comments on PR so then hopefully we can stay on that schedule.

Pleeeeaaaaase add this feature in 😦

Hi all. I have a workaround. I create a swarm service from my image with next entrypoint: #!/bin/bash exec docker run --init --rm --privileged=true --cap-add=ALL …etc

It’s all. When service tasks are created they run my privileged container and when tasks are destroyed SIGTERM kills child privileged container thanks to --init flag.

And of course we need to map docker.sock and install docker inside image for parent service.

Dockerfile:

FROM docker RUN apk --no-cache add bash COPY entrypoint.sh / RUN chmod +x /entrypoint.sh ENTRYPOINT [“/entrypoint.sh”]

entrypoint.sh: #!/bin/bash exec docker run --init --rm --privileged=true --cap-add=ALL …etc

service-run.sh: #!/bin/bash docker service create --name myservice --detach=false
–mount “type=bind,source=/var/run/docker.sock,target=/var/run/docker.sock” myimage

Waiting for your thoughts!

@thaJeztah

The proposal looks great, but that looks like a complete change of the way privileges work, which will take a loooooong time.

Meanwhile docker allows you to run --privileged, or to use --cap-add for containers, so why not provide the same facility for services and let the users decide if they want to run containers securely or insecurely until that proposal is implemented. This step would be simple I think, it is just a matter of passing variables when creating the containers on the other hosts.

FYI. I started implementing capabilities feature. You can see plan and status on https://github.com/moby/moby/issues/25885#issuecomment-447657852

Hey guys, I’ve been maintaining my own poorly-patched fork of docker for a while with support for --cap-add and --privileged in docker stack deploy, maybe we should create a proper fork?

I want to deploy GlusterFS in swarm, but glusterfs dont work without --privileged flag ((( Please help!

I’d like to add that Kubernetes does support adding capabilities. For an example, check out https://caveofcode.com/2017/06/how-to-setup-a-vpn-connection-from-inside-a-pod-in-kubernetes/ (this is the exact same use case I have, and that I’d like to run on docker swarm).

IPSec VPN with strongSwan/Openswan is another use-case for privileged support.

Another use case: Using a Container-as-a-service environment (so no ability to install anything on the host). We want to deploy an image (–mode=global) that can report back the stats we care about on all other docker containers running on the same host, as well as on the host itself. (Similar to @wannabesrevenge’s case )

Here’s a specific (admittedly edge) use case: I distribute DinD containers around a cluster of host nodes (the DinD containers allow us to run mini-isolated test swarms). If I were a able to use privileged services on an outer host swarm, I could take advantage of swarm for the automatic distribution of these workloads (and named access) without having to manage these details manually.

Trying to deploy dell openmanage exporter and it doesn’t work. I understand that --priv / --cap-add can be a security issue, but if I want to shoot myself in the foot, I should be able to do so.

@edward-of-cit you can always use a volume to mount nfs.

Use case : Distributed embedded environment with multiple nodes (>100) running headless in the wild. Need to be able to remotely update applications / environments. The embedded systems operate external electrical devices (e.g. smart home / industrial automation). Example devices:

• USB to serial (either the USB or the TTY device could be shared)
• UARTs (TTY)
• GPIO pins
• I2C / SPI
• On-board LED’s

Current plan is to run without docker service (i.e. ‘manual swarm’) until a workable solution exists.

Any news on --device for services?

+1

I want to be able to run a dockerised web service on an RPi that interfaces with a microcontroller over USB and provides web-based access to it over a network. I’m blocked by this and dotnet/dotnet-docker#223.

Just another use case, I want to run keepalived with vrrp on a swarm and it needs net=host and --cap-add=NET_ADMIN, so cap-add would be great.

@manvalls the PR you linked to was closed because that implementation did --cap-add / --cap-drop. The proposal was to have a --capabilities flag (which would override the defaults) https://github.com/moby/moby/pull/26849#issuecomment-249176129 :

I think it is better long term if we switch to a --capabilities interface which says which you actually need, even if you might get more by default for compatibility.

If someone wants to work on that, that’s a change that will likely get accepted.

@manvalls I think --cap-add (or --capabilities, i.e., not merging defaults, but require the full set to be specified) is something that would be accepted; have you considered contributing, and opening a pull request to discuss that option?

@efocht actually, you need to split the subnet of the swarmed macvlan for each worker its seems that the ip allocation for macvlan networks are not distributed at the moment.

I do something like (splitting /16 into sub /24 per worker) :

# on worker 1
docker network create --config-only --subnet 10.140.0.0/16 --gateway 10.140.0.1 -o parent=ens160 --ip-range 10.140.1.0/24 local-network-name
# on worker 2
docker network create --config-only --subnet 10.140.0.0/16 --gateway 10.140.0.1 -o parent=ens160 --ip-range 10.140.2.0/24 local-network-name
# on worker 3
docker network create --config-only --subnet 10.140.0.0/16 --gateway 10.140.0.1 -o parent=ens160 --ip-range 10.140.3.0/24 local-network-name

# on manager
docker network create -d macvlan --scope swarm --attachable --config-from local-network-name swarm-network-name

I’m in desperate needs for this feature to run a mode: global in swarm using docker-compose v3 for monitoring on the host level…

My current workaround is to drop the docker-compose.yml and use compose v2 instead and execute it with privileged: true.

Drive-by observation: you should do --cap-add before --privileged, to encourage people to be more granular in what they need. #25885 relates.

I am very interested in this, because as far as I know the Oracle DB cannot run in a container without either the --privileged option or specifying the --shm flag when running a container.

Given that they are both not supported yet in API 1.24 for services AFAICS, it would be impossible to replace a Docker Swarm (standalone) by Docker 1.12 Swarm to run such services.

Edit: Oracle just published Docker files https://github.com/oracle/docker-images/tree/master/OracleDatabase, so this big hurdle is resolved for us.

@thaJeztah Another consideration - I have different needs between a replicated service and a global one. If a jobs service type is introduced, which has been discussed, those needs might be different too.

The global one I may expect to have more flags/options around, just given the nature of “other things” I might be doing with them (monitoring, networking, running containers, etc.). I suppose I could have a global service that mounts the docker socket that then runs a privileged container on each node, but that seems messy (now my the tasks in my global service are managing the lifecycle of a container on each engine separately).

Hopefully that helps with some of that discussion.

FYI https://github.com/moby/moby/issues/25885#issuecomment-557790402

@mwaeckerlin

We need to get nested runc to work without root privileges first. You may want to follow this issue: https://github.com/opencontainers/runc/issues/1658

@mwaeckerlin Dito, don’t care about it either, but was in your situation even before Swarm and that solution solved all my perceived problems. Just let me know for my info if it was successful.

Thanks.

I have an ASP.NET Core web service that I’ve dockerised. Part of the operation of this web service involves communicating with a microcontroller over USB. I can run it on my local Windows 7 machine using Docker Toolbox by passing the USB device through to Virtualbox (so the docker VM has access to it).

On Windows 10 deployment machines it would be preferable to use Docker For Windows, which runs Docker on top of Hyper-V. However, Hyper-V apparently doesn’t support USB pass-through like Virtualbox does. One idea I had to work around this would be to split my webservice into two separate ones; one that handles all the USB communication logic and the other handling everything else. That would mean I could run the former (the “device comms” service) on a Raspberry Pi and the latter would be free to run anywhere, even in a non-privileged container. The idea would be to have a Docker swarm with these two services linked to each other.

However, how can I create a “device comms” service which requires privileged access to the filesystem if --privileged is not supported? How do I make any sort of Dockerised application that needs to perform USB communication on a Windows 10 machine?

This issue has been open for 14 months and it doesn’t seem like any progress is being made.

I’d also like to see the --cap-add on docker services. I’ve written a workaround, similar to @seiferteric’s if anyone would like to try it out: https://github.com/calh/docker_priv_proxy

I’m using signal traps, socat, and the docker socket to pair swarm mode service containers with a local privileged mode container. It seems to work well so far!