compose: Not able to use private git repo for build context in Docker Compose 1.6.2

I have a private repo with src and Dockerfile to build from, and I am connecting to a remote docker host using docker-machine. I am able to run the following:

docker build -t MYCONTAINER git@github.com:myrepo/myrepo.git#master

However, if I try to run docker-compose, with the following build command:

  MYCONTAINER:
    build: git@github.com:MYREPO/myrepo.git#master

It fails due to github authentication, although I believed that the git cloning happens locally in the same way as docker build? Here is the verbose output form docker-compose:

compose.config.config.find: Using configuration files: ./production-dc.yml
docker.auth.auth.load_config: File doesn't exist
compose.cli.command.get_client: docker-compose version 1.6.2, build 4d72027
docker-py version: 1.7.2
CPython version: 2.7.9
OpenSSL version: OpenSSL 1.0.1j 15 Oct 2014
compose.cli.command.get_client: Docker base_url: https://REMOTEIP:2376
compose.cli.command.get_client: Docker version: KernelVersion=3.13.0-58-generic, Os=linux, BuildTime=2016-02-22T21:37:01.910365059+00:00, ApiVersion=1.22, Version=1.10.2, GitCommit=c3959b1, Arch=amd64, GoVersion=go1.5.3
compose.service.build: Building MYCONTAINER
compose.cli.verbose_proxy.proxy_callable: docker build <- (pull=False, stream=True, nocache=True, tag=u'containers_fablr', buildargs=None, rm=True, forcerm=False, path='git@github.com:MYREPO/myrepo.git#master', dockerfile=None)
docker.api.build._set_auth_headers: Looking for auth config
docker.api.build._set_auth_headers: No auth config in memory - loading from filesystem
docker.auth.auth.load_config: File doesn't exist
docker.api.build._set_auth_headers: No auth config found
compose.cli.verbose_proxy.proxy_callable: docker build -> <generator object _stream_helper at 0x104d01d70>
ERROR: compose.cli.main.log_api_error: Error trying to use git: exit status 128 (Cloning into '/var/lib/docker/tmp/docker-build-git575327272'...
Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
)

I replaced my URLs and repo names, as you can see. Not sure what the auth_headers errors relate to.

About this issue

Original URL
State: closed
Created 8 years ago
Reactions: 24
Comments: 22

Most upvoted comments

Is there any possibility of this issue being reopened? It would be ideal if docker-compose used the current user’s git / ssh credentials.

+49

charlieparkes on Feb 25, 2020

for gitlab "https://oauth2:${TOKEN}@gitlab.com/xxx/xxx.git" works

+26

fcce on Oct 9, 2018

too bad this issue was closed 😦 store token & credentials in docker-compose is insane !

+21

bdecarne on Jan 20, 2020

Can this issue get reopened please? It’s rather jarring for docker build git@gitserver:org/repo.git#branch to work as expected when docker-compose build service does work when this is the docker-compose.yaml file:

version: '3.8'
services:
  service:
    build: git@gitserver:org/repo.git#branch

I believe docker-compose build should have feature parity with the normal docker build CLI

+20

nerdkid93 on Aug 19, 2020

So I can report a step forward on this one. It’s possible to access a private repo through the url: https://username:password@github.com/username/repo.git instead of the ssh type url (git@github.com:…) . Additionally, instead of password in cleartext you can generate a token in Github and use that instead. And this works with docker-compose!

As @jmahowald stated, the build command is run from within the docker environment in docker-compose (e.g. on the VM, which lacks my SSH credentials), whereas in docker build it is run locally on my Mac (which has the credentials). However, I still consider this a bug as the behaviour is different between docker build and docker-compose build.

+19

ripperdoc on May 15, 2016

Creating personal tokens won’t work in certain contexts of distributed workflows with private repos. Documentation states building from git over ssh should work in docker compose and it does not.

+16

kwidholm-tm on Jun 25, 2018

Based on @ripperdoc’s insight, we can use a Github Personal Token that has read access to the github repo. Use the token as the username with an empty password:

https://${TOKEN}:@github.company.com/org/repo.git

Here’s an example:

services:
  reference:
    build: "https://f4d24eed3909ad8766e29fd2500e80cb33d153a8:@github.company.com/services-configuration/spring-cloud-config-reference-service.git"
    command: ["./wait-for-it.sh", "--timeout=60", "config-server:8888", "--", "java","-Djava.security.egd=file:/dev/./urandom","-jar","/app.jar"]

This works! At this point, you can define an environment variable for the token such your OPS team can provide their own token.

services:
  reference:
    build: "https://${TOKEN:-f4d24eed3909ad8766e29fd2500e80cb33d153a8}:@github.company.com/services-configuration/spring-cloud-config-reference-service.git"
    command: ["./wait-for-it.sh", "--timeout=60", "config-server:8888", "--", "java","-Djava.security.egd=file:/dev/./urandom","-jar","/app.jar"]

Finally, you can specify which branch to use with a suffix #branch in the URL. The example below forces the client to clone from the develop branch.

services:
  reference:
    build: "https://${TOKEN:-f4d24eed3909ad8766e29fd2500e80cb33d153a8}:@github.company.com/services-configuration/spring-cloud-config-reference-service.git#develop"
    command: ["./wait-for-it.sh", "--timeout=60", "config-server:8888", "--", "java","-Djava.security.egd=file:/dev/./urandom","-jar","/app.jar"]

+15

marcellodesales on Apr 6, 2018

If someone is using Bitbucket the equivalent to Personal access tokens are App passwords. This can be found under Bitbucket Settings --> Access Management --> App passwords

+12

chribsen on Aug 8, 2017

Same issue, very important feature

+11

goshander on Nov 30, 2020

I would love to see this work out of the box without using github tokens for auth. It’s unfortunate to have everyone create personal tokens to pull down a private repo image.

dan-kez on Mar 21, 2018

Is there a way to use azure devops private git repo for build context in Docker Compose?

I’m getting the following error ERROR: error downloading remote context https://MY-ORG@dev.azure.com/MY-ORG/PROJECTNAME/_git/reponame: failed to GET https://MY-ORG@dev.azure.com/MY-ORG/PROJECTNAME/_git/reponame with status 401 Unauthorized:

docker build https://MY-ORG@dev.azure.com/MY-ORG/PROJECTNAME/_git/reponame resuls with same error failed to resolve httpcontext: invalid response status 401

I’m using windows 10. Correct credentials are set in the windows credentials store so it is possible to pull changes in cmd with git.

shupoval on Mar 14, 2019

Any update on this? Any of the suggested ways to do it doesn’t really work on a dedicated private bitbucket which runs on port 7999. I’ve tried personal access tokens without luck. Please fix this!

Tried on both MacOS Mojave and Windows 10

MBeijer on Dec 6, 2018

+1 My assumption is that unlike docker-build the pull is coming from the docker-machine, which doesn’t have the correct credentials (or in my case, the correct CA chain to the server)

jmahowald on Mar 15, 2016