cloud-builders: Can't get .git directory to show up in remote builds

@bendory is there a different logic applied when using the GCB GitHub App? I’m trying to get private submodules working, but the extracted tarball doesn’t contain a .git folder

I think it’s worth reopening this issue. I appreciate the thorough answer, @bendory, but it seems the .gcloudignore isn’t used when builds are triggered from GitHub.

Or is there a separate issue tracking that in particular?

Really the solution here is just for the Cloud Build app to send .git/ with the rest of the files when a build is triggered, it’s clearly something that people need and its unintuitive that only this one directory is missing.

Where would be the right place to file an issue about this?

Indeed, if you look at the code for gcloudignore, you’ll see that we generally ignore a .git directory, since it is rather unusual to want to upload this as part of your build context.

If you want it, you have several options:

• create an empty .gcloudignore (touch .gcloudignore)
• create a .gcloudignore (or .gitignore) that ignores the files you don’t want but doesn’t ignore .git
• create a .gcloudignore that explicitly includes .git (echo '!.git' > .gcloudignore)
• turn off the ignore behavior completely: gcloud config set gcloudignore/enabled false

See the gcloudignore documentation for further details.

@saminiemi Your workaround doesn’t work if you want to take advantage of the “Build on PR” feature that’s only available with the Cloud Build GitHub App repo 😦

Also experiencing this issue with the new Cloud Build GitHub App triggers. Have attempted to explicitly add !.git to the .gcloudignore file with no success.

@bendory @wlynch - have you been able to recreate this on your end?

Why is this closed? Having a .git directory is still required for github triggered builds.

we generally ignore a .git directory, since it is rather unusual to want to upload this as part of your build context.

That assumption is false. We need the .git directory for several reasons, including git describe, git submodule, and git lfs

Here’s a cloudbuild.yaml file I made to hack around this. This checks out the .git directory and checks out a private submodule with git-lfs files.

steps:
- name: 'ubuntu+git-lfs'
  entrypoint: 'bash'
  args:
  - '-c'
  - |
    set -eux
    
    # cloudbuild doesn't check out the .git directory, so I have to
    # check it out again to get the .git directory, submodules, and
    # LFS files
    eval $(ssh-agent -s)
    mkdir -p -m 0700 /root/.ssh
    cat ci/host-github.com >> /root/.ssh/known_hosts

    # MYREPO1.key was made from ssh-keygen then added to the github
    # repo's Deploy Keys https://github.com/USER/REPO/settings/keys
    chmod 0600 ci/MYREPO1.key
    ssh-add ci/MYREPO1.key
    git init
    git lfs install
    git remote add origin git@github.com:/GITHUB_USER/$REPO_NAME.git
    git fetch --depth=1 origin $COMMIT_SHA
    git reset --hard FETCH_HEAD
    # have to switch to MYSUBREPO2.key to check out the MYSUBREPO2 submodule
    ssh-add -D
    chmod 0600 ci/MYSUBREPO2.key
    ssh-add ci/MYSUBREPO2.key
    # if there were other private repos here, I'd have to switch keys
    # and update them individually
    git submodule update --init --recursive
    # finally the .git repo is rebuild and checked out
        
    make test

We do this instead of moving the repo:

  # checkout git submodules
  - name: 'gcr.io/cloud-builders/git'
    entrypoint: 'bash'
    args:
    - '-c'
    - |
      git init
      git remote add origin git@github.com:/orgname/$REPO_NAME.git
      git fetch --depth=1 origin $COMMIT_SHA
      git reset --hard FETCH_HEAD
      git config -f .gitmodules submodule.core.url git@github.com:/orgname/$REPO_NAME.git
      git submodule update --init

And for authentication we use this project: https://github.com/connectedcars/auth-wrapper

I’m also running into this: with the .git directory is not being added to my image in Cloud Build triggered by GitHub, despite having an empty .gcloudignore present.