checkout: Git LFS with a token doesn't work with GitHub Enterprise Server without subdomain isolation

We have found an issue regarding GitHub checkout action v2, Git LFS and GitHub Enterprise Server (on Azure).

We have a very simply workflow which doesn’t work:

- uses: actions/checkout@v2
  with:
    lfs: 'true'

All git lfs request will be rejected with following error: HTTP/1.1 400 Bad Request By enabling GIT_TRACE=1 and GIT_CURL_VERBOSE=1 we see following:

> GET /storage/lfs/3/objects/d5c5871801d62c64f453462558c3a4697ac162730e49d48461ce87bafa83684c HTTP/1.1
> Host: ***.westeurope.cloudapp.azure.com
> Authorization: RemoteAuth AAAAAF72C****
> Authorization: Basic * * * * *
> User-Agent: git-lfs/2.12.1 (GitHub; windows amd64; go 1.14.10; git 85b28e06)
....
> HTTP/1.1 400 Bad Request
> Content-Length: 150
> Content-Type: text/html
> Date: Thu, 17 Dec 2020 10:25:15 GMT
> Server: GitHub.com

My current understanding: The GitHub checkout action is using the extraheader option with basic authorization in the local git config. So for each request this basic authorization header will be used.

In additional a remoteAuth authorization header will be added as result of the git lfs batch api reponse /info/lfs/objects/batch:

{
    "objects": [
        {
            "oid": "fcc622faad3b44962e9211cc2fd478e7c0480d516098fab011ccdb1d29fbde81",
            "size": 4612119,
            "actions": {
                "download": {
                    "href": "...",
                    "header": {
                        "Authorization": "RemoteAuth AAAAAMHDN3KJWR****"
                    }
                }
            }
        }
     ]
 }

Now we have two authorization headers.

In this comment https://github.com/git-lfs/git-lfs/issues/4031#issuecomment-589254543 one of the git lfs maintainer mentions that the git lfs server only allows request with one authorization header. So all requests with two authorization headers will be rejected.

About this issue

  • Original URL
  • State: open
  • Created 3 years ago
  • Reactions: 11
  • Comments: 20 (2 by maintainers)

Most upvoted comments

An obvious solution is to simply stop using git lfs.

@Roda83 I have a suspicion what is going on:

On your machine subdomain isolation is not enabled. Therefore, the Git Authorization header is automatically applied since it is the very same domain as the Git clone URL. In my test case subdomain isolation is enabled. Therefore, the LFS files are downloaded from a different domain (media.foo.bar) and no additional header is applied.

In general it is highly recommended to enable subdomain isolation as this is an important GHES security feature. More info here: https://docs.github.com/en/enterprise-server@2.22/admin/configuration/enabling-subdomain-isolation

We will try to fix this problem. Can you try to enable subdomain isolation in the meantime?

I think , the workaround means setup ssh config to connect GHES.

work flow

- uses: actions/checkout@v3
   with:
     lfs: false
     persist-credentials: false
- name: setup private key
  shell: bash
  run: |
    echo "${{ secrets.your_register_private_key }}" > ~/.ssh/id_rsa
- name: git lfs pull
  shell: bash
  run: |
    git remote set-url origin git@your-ghes-domain.com:foo/foo.git
    git lfs pull

@ericsciple Yes currently we are using the checkout-action v2 with lfs: false and persist-credentials: false and a second action to perform a git-lfs pull over SSH. This works as workaround at the moment until we will enable subdomain isolation on our system.

I’d like to point out that the correct way to handle this is by Actions using a custom credential helper as outlined in #162. If Apple Git doesn’t work properly, please detect that and reject using that version instead of using the extra header.

The behavior of Git LFS in this case is that it sends two Authorization headers and the GitHub LFS server rejects that. While not compliant with the HTTP spec, this is also the behavior that Git has, but the GitHub Git server happens to silently accept that, which is a bug and will likely change soon. Git LFS in this case is behaving in a bug-for-bug compatible way with Git, and this problem really needs to be fixed correctly on the Actions side.

Thanks @dassencio !

@Roda83 I 💯 agree with @dassencio . One more note: you likely need a new TLS certificate if you enable subdomain isolation.

I proposed a change in Git LFS to handle that situation better in the future: https://github.com/git-lfs/git-lfs/pull/4369

@Roda83: Subdomain isolation is not enabled by default. While we strongly recommend enabling it to prevent cross-site scripting attacks, it requires additional DNS configuration (in your case, on Azure) to work.

@larsxschneider: Thank you very much for sharing your expertise on this topic with us!

@Roda83: Today I tried to reproduce this issue without success. This is what I did (on both GitHub Enterprise Cloud and GitHub Enterprise Server):

  1. Created a private repository with files stored via Git LFS (about 30 files / 200MB in total).
  2. Created a workflow file (contents below).
  3. Set up a self-hosted runner with Windows 2019 Server => workflow succeeded.
  4. Added 10000 small files (1KB each) to the repository (all stored via Git LFS) => workflow succeeded.

Contents of workflow file:

name: Git LFS test
on:
 push:
   branches: [ main ]
jobs:
 checkout:
   name: Checkout
   runs-on: self-hosted
   steps:
   - name: Checkout with Git LFS
     uses: actions/checkout@v2
     with:
       token: ${{ github.token }}
       lfs: 'true'
       clean: 'true'
       fetch-depth: 0
   - name: List files
     run: ls