azure-pipelines-agent: Can't acquire root on common container distros

Is there any updates on this? What’s the currently accepted/recommended workaround for a simple use-case where I need to run an application’s tests in a default Ubuntu container?

At the moment I am doing the same workaround as @esteve above with the following container config:

container:
  image: <your image here, ubuntu:latest for example>
  options:  "--name ci-container -v /usr/bin/docker:/tmp/docker:ro"

And then add this as the first step before you do anything else:

- script: |
      /tmp/docker exec -t -u 0 ci-container \
      sh -c "apt-get update && DEBIAN_FRONTEND=noninteractive apt-get -o Dpkg::Options::="--force-confold" -y install sudo"
    displayName: 'Install Sudo in container (thanks Microsoft!)'

This works and subsequent steps can use sudo but it still feels like a terrible hack and something that shouldn’t have to be done especially if it breaks the conventions that the majority of Docker Hub images are built around (the assumption that you are already root thus no need for sudo). No other CI service that I’m familiar with requires such workarounds.

Since standard, popular docker images simply won’t work because of this, it might be a good idea to revisit this design choice IMHO.

On the other hand, why is azure-pipelines even trying to execute anything inside the running container? I think other CI providers do not do anything like that. There could also be an option to disable that “feature”?

I am also facing this issue. I simply need to add an external deb repo - which is impossible to do in the hosted agent, and subsequently impossible to do in a docker container due to this behavior. Thanks to the many suggestions above, this is the solution that worked for me:

  - job: myjob
    pool:
      vmImage: 'ubuntu-20.04'
    container:
      image: debian:bullseye
      options: '--name mycontainer -v /usr/bin/docker:/tmp/docker:ro'
    steps:
    # Hack to install sudo into the container.
    # The sudoers file already exists and subsequently apt-get will wait infinitely for input
    # (despite futile attempts to use `-y` or `yes | apt-get`)
    - script: |
        /tmp/docker exec -t -u root mycontainer mv /etc/sudoers /etc/sudoers.bak
        /tmp/docker exec -t -u root mycontainer apt-get -qq update
        /tmp/docker exec -t -u root mycontainer apt-get -qq install sudo
        /tmp/docker exec -t -u root mycontainer mv /etc/sudoers.bak /etc/sudoers
    - script: |
        sudo ...

@danwalmsley if it’s of any use, I managed to install sudo by running docker exec -u 0 inside the container:

https://github.com/ApexAI/performance_test/blob/master/azure-pipelines.yml#L9-L17

Containers in Azure are configured so that you can run Docker inside them, so I just exported the Docker executable as a volume and then access the running container as root via docker exec. The only requirement is to name the container (by passing --name NAME in options), so you can access it via docker exec. The other thing is to not overwrite the sudo config files that the Azure agent generates, but I think it’d be better if the agent wrote them separately to a file /etc/sudoers.d/ instead of /etc/sudoers

Still an issue. Would be great if this got some attention.

Could you please provide a workaround for this?

Try to create a user with UID '1001' inside the container.
/usr/bin/docker exec  b76b29190ab25216e4e99fd12ec57375501125c3d19a59c05bffb2d7036e483e bash -c "getent passwd 1001 | cut -d: -f1 "
/usr/bin/docker exec  b76b29190ab25216e4e99fd12ec57375501125c3d19a59c05bffb2d7036e483e useradd -m -u 1001 vsts_azpcontainer
useradd: Permission denied.
useradd: cannot lock /etc/passwd; try again later.
##[error]Docker exec fail with exit code 1

This line should never run by default IMHO, and it should be at the very least configurable. Otherwise you make a whole lot of assumptions on the images of your users, and I don’t think that’s a good thing.

Today, I would recommend people to build a small layered image (no matter which Linux distro), tuck it in quay.io or dockerhub, and pull from there.

This is probably where we’ll end up eventually if this isn’t fixed, but having to create a separate repo and maintain and update custom containers is a lot of extra operational complexity for your average open source project, compared to just writing a few lines of yaml… Suddenly we have to figure out credential handling for our container repository, etc.

Also, is it possible to have the first step in Azure Pipelines just install sudo (I am assuming not).

Azure pipelines already starts by injecting some custom software into the image (a copy of nodejs that it uses to run the pipelines agent, which lets it run further commands inside the agent). If they injected a copy of sudo as well, in some well-known location, that would pretty much solve this for any container image.

There are really two issues here, that are mostly unrelated.

For the problem the LinuxBrew folks are hitting, where the agent initialization is assuming that plain docker exec will have root, I think the solution is just for the agent initialization code to use docker exec -u 0:0 to override any USER directive in the dockerfile. Docker has root to start off with; there’s no point in going root -> some other user -> sudo back to root.

For the problem I’m having, where there’s no way for my user code to get root without sudo, the best solution I can think of is to add a way to mark particular tasks as being executed as root. Then it would be the agent’s job to make this happen. For example, it might use sudo when running in a regular environment, and docker exec -u 0:0 or some other user-switching helper when running in an arbitrary container. Usage might look like:

- bash: "apt update && apt install -y some package"
  runAsRoot: true

I’m also hoping for a proper fix for this issue…

There was a related discussion in this issue in the dotnet-docker repo where it was decided to not include sudo in the SDK images. The closing comment was:

Closing as we are not going to make this change. Azure DevOps should reconsider their implementation to better support “bring your own image”.

It seems many agree with that comment.

I can see this issues is still present and its good to see this is really dragging… I mean, its actually 2 years old by now. Can we expect any changes in the future?

@njsmith after some trial an errors, I got a sudo installed in some containers this way: https://dev.azure.com/nexB/license-expression/_build/results?buildId=79 https://github.com/nexB/license-expression/blob/3fe3f9359c34b6e6e31e6b3454e450ca8e9e9d6e/azure-pipelines.yml#L80

This is incredibly hackish as it involves running first a command line docker as root that runs docker in docker to install sudo? (or something along these lines). Somehow it works and I am able to get sudo-less containers (such as the official Ubuntu, Debian and CentOS images) to gain sudo access such that I can then install a Python interpreter and eventually run the tests at last.

It looks like this has been first crafted by @esteve for https://github.com/ApexAI/performance_test/blame/6ae8375fa1e3111cb6fa60bdd1d42b9b9f370372/azure-pipelines.yml#L11

There are variations in https://github.com/quamotion/lz4.nativebinaries/blob/4030ff9d97259b05df84c080d494971b62931363/azure-pipelines.yml#L77 and https://github.com/dlemstra/Magick.Native/blob/3c83b2e7d06ded8f052bb5b282c5a79e27b2d6b7/azure-pipelines.yml

I have found a work around that seems to do the job. Add this command line task to the beginning. It gets the docker container id then runs a docker exec command to run apt and install sudo. It runs this on the host, not in the container. The key is the target: host line. The dpkg options is needed because they are editing the /etc/sudoers file, and the sudo package has its own version, this keeps the existing file.

# Azure container jobs run as non-root user which has sudo permissions, but sudo is missing.  Install sudo package...
- task: CmdLine@2
  target: host
  inputs:
    script: |
      cid=`docker container ls -q`
      echo "ID = $cid"
      /usr/bin/docker exec  $cid su -c "apt -y update && apt -o DPkg::Options::="--force-confold" -y install sudo"

This other way should also work to give a name to the docker container, and use that name in the command.

pool:
  vmImage: ubuntu-latest

container: 
  image: debian:10
  options: --name mycontainer

steps:
# Azure container jobs run as non-root user which has sudo permissions, but sudo is missing.  Install sudo package...
- task: CmdLine@2
  target: host
  inputs:
    script: |
      /usr/bin/docker exec  mycontainer su -c "apt -y update && apt -o DPkg::Options::="--force-confold" -y install sudo"

Still a problem in 2023.

Hi everyone! We are currently working on more prioritizing issues, but will get back to this one once be able to. This possible enhancement would require additional testing to avoid any possible regressions around it.

Whoah. I’m impressed that there are issues which are more important that pipeline not working at all 😃 (just bumping thread up to show that it’s not stale)

Hi everyone! We are currently working on more prioritizing issues, but will get back to this one once be able to. This possible enhancement would require additional testing to avoid any possible regressions around it.

Not for all possible use cases, but important/widespread ones: test results upload, pipeline artifacts, and caching.

Other CI systems definitely work differently than Azure Pipelines in this respect. AFAIK no one else tries to abstract the work (task) from the execution environment (container or VM).

Red Hat might consider creating a custom version of Universal Base Image which comes pre configured for CI/CD pipelines with sudo installed. We are tracking it here, but don’t have any plans yet.

Today, I would recommend people to build a small layered image (no matter which Linux distro), tuck it in quay.io or dockerhub, and pull from there. Maintaining a small layered image shouldn’t be that difficult.

Also, is it possible to have the first step in Azure Pipelines just install sudo (I am assuming not). Sorry, I have never used Azure Pipelines and don’t have time to test, but I am the product manager for UBI, so I find this use case interesting from a base image perspective.

To add a hair more background, there is constant tension when building base images (no matter which Linux distro). If we (Red Hat UBI team in this case, but same goes for any base image maintainer/architect) add more packages for every use case on the planet, then base images will eventually be 1GB+. Partners, customers, and cloud providers all need “special” things in their base images, and this use case is so similar.