checkout: private submodule checkout fails @v2

What worked: create new personal access token (PAT)

• Generate PAT: https://github.com/settings/tokens
• And add the secret: https://github.com/<-- username -->/<-- repo -->/settings/secrets/new, give it a unique name say MY_REPO_PAT.
• Configure action as follows:

    steps:
    - name: Checkout
      uses: actions/checkout@v2
      with:
        token: ${{ secrets.MY_REPO_PAT }}
        submodules: recursive

@jleni here’s a version that will checkout all repositories defined in .gitmodules 😃

It works for both ssh and https:

- name: Checkout submodules using a PAT
  run: |
    git config --file .gitmodules --get-regexp url | while read url; do
      git config --file=.gitmodules $(echo "$url" | sed -E "s/git@github.com:|https:\/\/github.com\//https:\/\/${{ secrets.CI_PAT }}:${{ secrets.CI_PAT }}@github.com\//")
    done
    git submodule sync
    git submodule update --init --recursive

Thanks, this is what we’ve ended up by doing, but it just seems strange that such a simple operation (fully checking out the repository with its submodules) doesn’t work out of the box for private repositories. IMO it should at least be clearly mentioned in the README as I lost quite some time trying to understand why it didn’t work.

I have the exact same issue but I just made it work using the Checkout multiple repos (private) example

    - name: Checkout submodule
      uses: actions/checkout@v2.0.0
      with:
        token: '${{ secrets.GITHUB_PAT }}'
        repository: myOrg/myRepo
        path: path/to/submodule
        ref: 'master'

This is working just as a workaround and might not be practical for a project with a lot of submodules, but I hope it helps until #72 is resolved

What worked: create new personal access token (PAT)

• Generate PAT: https://github.com/settings/tokens
• And add the secret: https://github.com/<-- username -->/<-- repo -->/settings/secrets/new, give it a unique name say MY_REPO_PAT.
• Configure action as follows:

    steps:
    - name: Checkout
      uses: actions/checkout@v2
      with:
        token: ${{ secrets.MY_REPO_PAT }}
        submodules: recursive

This worked for me!! Thanks @ashwinvis

It’s a little concerning that the only solution here is to create a PAT. The point of a Github Action is to centralise the deployment process for an organisation, that organisation may have multiple private repositories, for example defining graphql or protobuf contracts. They need importing into each repo during the build process as submodules. To rely on PAT for this process is a security concern as well as an availability risk. If the person who creates the PAT leaves the company, the whole company will now be unable to deploy due to a quite deep down and hidden error within this process.

What is the recommended solution for allowing an organisation to deploy code that references submodules from within their own organisation?

This is actually blocking us from using GitHub actions. I hope it can be prioritized…

Sub-modules support was removed in v2, I’m not sure why though

https://github.com/actions/checkout/releases/tag/v2.0.0

I have 1 repo that contains 2 private submodules.

The next workaround worked for me:

1. Create 2 SSH keys by running ssh-keygen -t ed25519 twice.
2. Put each public key as deployment key into each private submodule repo.
3. Put 2 private keys as a secret into the main repo.
4. Change your action a bit

      - uses: shaunco/ssh-agent@git-repo-mapping # this action will configure git to use the right SSH key per each repo. 
        with:
          ssh-private-key: |
                ${{ secrets.SUBMODULE_1_SSH_PRIVATE_KEY}}
                ${{ secrets.SUBMODULE_2_SSH_PRIVATE_KEY}}
          repo-mappings: |
            github.com/username_or_organization/submodule1_repo_name
            github.com/username_or_organization/submodule2_repo_name
      - uses: actions/checkout@v2
        with:
          submodules: recursive # this is important not to forget

Similar to some of the comments above, I got it working with the following, where GITHUB_ACCESS_TOKEN is a personal access token in the format username:token, that is base64 encoded. Would be great for submodules and private submodules to be supported more directly.

git config --global url."https://github.com/".insteadOf "git@github.com:"
git submodule sync --recursive
git -c "http.extraheader=Authorization: basic ${GITHUB_ACCESS_TOKEN}" -c protocol.version=2 submodule update --init --force --recursive --depth=1

Maybe it will help somebody. This solution works when you want to keep flexibility of URL repos and still use GitHub Actions with Deploy Keys to access private submodules:

  - name: Checkout
    uses: actions/checkout@v2

  - name: Clone Submodule
    run: |
        mkdir -p $HOME/.ssh
        echo '${{ secrets.SUBMODULE_REPO_DEPLOY_KEY }}' > $HOME/.ssh/ssh.key
        chmod 600 $HOME/.ssh/ssh.key
        export GIT_SSH_COMMAND="ssh -i $HOME/.ssh/ssh.key"
        git submodule set-url <path-to-submodule> git@github.com:<organization/submodule>.git
        git submodule update --init --recursive
        git submodule set-url <path-to-submodule> https://github.com/<organization/submodule>.git
        unset GIT_SSH_COMMAND

Hi @stephencweiss,

You can find the article at https://ehlers.berlin/blog/private-submodules-in-github-ci/ 😃

@b-m-f can you elaborate? What does your .yml look like?

Something like this?

jobs:
    build:
        runs-on: ubuntu-latest
        steps:
            - name: Fetch submodules
              shell: bash
              env:
                  SSH_KEY: ${{secrets.SSH_KEY}}
              run: |
                  mkdir $HOME/.ssh && echo "$SSH_KEY" > $HOME/.ssh/id_rsa && chmod 600 $HOME/.ssh/id_rsa 
                  git submodule update --init --recursive

Or are you using actions/checkout@v2.3.1?

Below is my configuration. It works smoothly to fetch a theme from another repository. It uses the THEME_SSH key provided via the secrets and added as a deploy key in the theme repository.

Give me a few days and I will follow up this solution with a blog post.

# A workflow run is made up of one or more jobs that can run sequentially or in parallel
jobs:
  build:
    runs-on: ubuntu-latest

    steps:
      # Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
      - uses: actions/checkout@v2
        with:
          repo-token: ${{ secrets.GITHUB_TOKEN }}
      - name: Trigger release build
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          THEME_SSH: ${{secrets.THEME_SSH}}
        run: |
          mkdir $HOME/.ssh && echo "$THEME_SSH" > $HOME/.ssh/id_rsa && chmod 600 $HOME/.ssh/id_rsa && git submodule update --init --recursive && docker login docker.pkg.github.com -u $GITHUB_ACTOR --password $GITHUB_TOKEN && docker build . -t docker.pkg.github.com/b-m-f/blog

I also need to have sub-module support. When I use v2, there’s no .git directory so I cannot perform any commands like checking out sub-modules. For now I’m using v1 to continue my development.

Can’t believe there’s still no better way to do this

Read only version:

• create SSH keypair on your computer
• add public part as deploy key to private submodule
• add private key as secret in repo where action is triggered. E.g SSH_KEY
• add Secret to env with

 env:
          SSH_KEY: ${{secrets.SSH_KEY}}

• do this before pulling the submodules in the run configuration

mkdir $HOME/.ssh && echo "$SSH_KEY" > $HOME/.ssh/id_rsa && chmod 600 $HOME/.ssh/id_rsa

late to the party, but seems to be working with more recent versions, e.g. 2.3.1

    - uses: actions/checkout@v2.3.1
      with:
        token: ${{secrets.PAT_MY_TOKEN}}
        submodules: recursive
        #lfs: true

assuming that you have added PAT_MY_TOKEN to your Secrets section

Actions does not create a PAT in your account. The token we generate for each job is a GitHub App per repository token and can only read the repo that the workflow is defined in.

@tvainika the default token embedded in the main repo does not have access to fetch other private repositories. Instead if you supply token: ${{ secrets.MY_GITHUB_PAT }} then it should work.

Sub-modules support was removed in v2, I’m not sure why though

I understood that it refers to another syntax with: submodules as the README I linked gives this new syntax I used.

For anyone who gets here in the future, adding to @fbernaly’s comment, when creating a PAT:

1. Repository access -> Only select repositories -> Give access to both main repo and submodule repo.
2. Permissions -> Repository permissions -> Give Read-only access to Contents and Metadata.

For me, the issue was that the default token used by the github action didn’t have access to clone the submodule repository. I provided access by creating a new secret in my private repo that contains a copy of my SSH github-private key. This private key provides access to both repositories. For more security, you could also use a more restrictive PAT and the token a keyword. Note that the key used needs to have access to both repositories cos it will be used for cloning the parent repository and the submodule repository.

  steps:
    - uses: actions/checkout@v2
      with: 
        submodules: true
        ssh-key: ${{ secrets.xxx}}

I keep getting the same error no matter what I try.

fatal: could not read Username for 'https://github.com': No such device or address

Thanks all for the feedback! I updated V2 to include the recent submodule/SSH changes from master.

An input submodules has been added now. I merged to master, try it out and let me know. Collecting feedback and will update the v2 tag next week.

Also I’m hoping to merge support for SSH soon.

I got frustrated switching my submodules from SSH to HTTPS and dealing with PATs so I decided to build my own action for those wishing to checkout private or public submodules via SSH in their workflows: submodule-checkout

thanks to @alicia 's work, we also got private ssh submodules working! However, we ended up with a slightly different result:

on: push
jobs:
  check-elm:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
    - name: Checkout submodules
      shell: bash
      run: |
        # From https://github.com/actions/checkout/issues/116#issuecomment-583221947
        git config --global url."https://github.com/".insteadOf "git@github.com:"
        git submodule sync --recursive
        git -c "http.extraheader=Authorization: basic ${{secrets.GITHUB_ACCESS_TOKEN}}" -c protocol.version=2 submodule update --init --force --recursive --depth=1
    - uses: actions/setup-node@v1
      with:
        node-version: '8.16.0'
    - run: npm run test

The main difference being ${{secrets.GITHUB_ACCESS_TOKEN}} rather than ${GITHUB_ACCESS_TOKEN}.

@samkit-jain yeah, that is why I ended up using my code above: https://github.com/actions/checkout/issues/116#issuecomment-573880976, as my private repositories use ssh and I do not want to change all my submodules just because of the CI.

@tvainika the default token embedded in the main repo does not have access to fetch other private repositories. Instead if you supply token: ${{ secrets.MY_GITHUB_PAT }} then it should work.

The url in .gitmodules needs to be the https one for this to work.

Example .gitmodules

[submodule "module"]
	path = module
	url = https://github.com/example/module

Example action.yml

name: Test

on: [push]

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v2
        with:
          token: ${{ secrets.MY_GITHUB_PAT }}

      - name: Checkout submodules
        shell: bash
        run: |
          auth_header="$(git config --local --get http.https://github.com/.extraheader)"
          git submodule sync --recursive
          git -c "http.extraheader=$auth_header" -c protocol.version=2 submodule update --init --force --recursive --depth=1

Thanks @chrispat for clarifying this, I was wondering why we didn’t see this automatically created PAT in our organization. But as long as I’m writing this, I’d also like to say that, if you’re taking enhancement suggestions, it would be great if it were (optionally) possible to generate these per-job tokens with access to all organization repositories and not just the current one too, as it’s still very surprising that something as basic as checking out a repository with its submodules doesn’t work out of the box when using actions.

@vadz I’ve personally set up an extra account in our organization, so it has access to all the private repositories, but without having access to my personal repositories.

I also use this account for different IoT devices to be able to access Github via ssh and tokend.

BTW, sorry if this is off-topic here, but what kind of tokens do people use in this case? AFAICS tokens are always associated with an account, i.e. give access to all the repositories the person has access to, which is not ideal, as it implies that the CI scripts of any of the private organizations I work with have access to all my other repositories, including those of the different private organizations.

We also looked at the deploy keys, but those can’t be reused for the different repositories and we have recursive submodules in our submodule, so maintaining them all and using the correct key for each checkout would be a lot of fun that we’d rather avoid.

Is creating a specialized “CI” pseudo-user in each organization the only solution to be able to easily check out a private repository with its submodules?

oh wow! that’s awesome!! thanks!!

@ericsciplec @rodrigorn is there a better alternative? I would like to avoid maintaining submodule hashes/branches in two different places. This is very error prone…

I agree completely but haven’t found any better solution yet

It’s a little concerning that the only solution here is to create a PAT. The point of a Github Action is to centralise the deployment process for an organisation, that organisation may have multiple private repositories, for example defining graphql or protobuf contracts. They need importing into each repo during the build process as submodules. To rely on PAT for this process is a security concern as well as an availability risk. If the person who creates the PAT leaves the company, the whole company will now be unable to deploy due to a quite deep down and hidden error within this process.

What is the recommended solution for allowing an organisation to deploy code that references submodules from within their own organisation?

Because of that, I use GitHub App Installation access tokens which are not directly tied to a user but instead tied to the GitHub App installed in your organization. I use this action to use a previously configured and installed GitHub App to get a special token with more privileges (like access to other organization repos). This other action also seems to do the same. Then, my action would look like:

name: Some test
on:
  pull_request:
jobs:
  some_job:
    runs-on: ubuntu-22.04
    name: Some action
    steps:
      - name: Get token
        id: get_token
        uses: machine-learning-apps/actions-app-token@master
        with:
          APP_PEM: ${{ secrets.AUTH_APP_PEM }}
          APP_ID: ${{ secrets.AUTH_APP_ID }}
      - name: Checkout repository and submodules
        uses: actions/checkout@v3
        with:
          submodules: recursive
          token: ${{ steps.get_token.outputs.app_token }}

@jleni here’s a version that will checkout all repositories defined in .gitmodules 😃 It works for both ssh and https:

- name: Checkout submodules using a PAT
  run: |
    git config --file .gitmodules --get-regexp url | while read url; do
      git config --file=.gitmodules $(echo "$url" | sed -E "s/git@github.com:|https:\/\/github.com\//https:\/\/${{ secrets.CI_PAT }}:${{ secrets.CI_PAT }}@github.com\//")
    done
    git submodule sync
    git submodule update --init --recursive

I get this error when trying to use your script…

Run git config --file .gitmodules --get-regexp url | while read url; do
sed: -e expression #1, char 86: unterminated `s' command

The script look fine though. Do you know what the issue is @Lauszus ?

From your error output it looks like you are running Windows. I don’t think my script works with Windows.

@vavsab when using shaunco/ssh-agent@git-repo-mapping have you faced the need of pointing to a submodule using repo-mappings with the need to point to specific branch? with .gitmodules file there is the possibility of defining branch…

@TiagoGalvaoChange no, I haven’t. My .gitmodules only contain links to specific commits. So you may fork and add more logic to it to cover your case.

@jleni here’s a version that will checkout all repositories defined in .gitmodules 😃

It works for both ssh and https:

- name: Checkout submodules using a PAT
  run: |
    git config --file .gitmodules --get-regexp url | while read url; do
      git config --file=.gitmodules $(echo "$url" | sed -E "s/git@github.com:|https:\/\/github.com\//https:\/\/${{ secrets.CI_PAT }}:${{ secrets.CI_PAT }}@github.com\//")
    done
    git submodule sync
    git submodule update --init --recursive

This worked for me. Thanks a lot! 👍

@b-m-f Tried out the steps in your blog post but it doesn’t seem to work. Always get an error: git@github.com: Permission denied (publickey).

@ericsciple Any ETA when the submodules checkout will be merged, and become available under the v2 tag?

As an alternative to @Lauszus solution to rewrite the .gitmodules file with the HTTP authentication info and a PAT, you could use this Action

https://github.com/webfactory/ssh-agent

You can generate an SSH keypair and add the public key to the repository you need access to (a normal Deploy Key) and then add the private key as a secret in your repo that’s running the GH Action

I’m not suggesting this is great, but until there’s native support for at least something resembling same-org mutual authentication for actions, this may be as good as it gets.

@ashwinvis yes it is. I have updated my comment to make it more clear.

@neutrinog no .git folder indicates git 2.18 or higher is not in your PATH. I updated the readme earlier today to make that more clear.