What happened:

I configured a “postsubmit” job for a private repo.

my-owner/my-repo:
  - name: test-job
    decorate: true
    decoration_config:
    ssh_key_secrets:
    - my-private-ssh-key
    clone_uri: git@github.com/my-owner/my-repo.git
    spec:
      containers:
        - image: alpine
          command: [ "/bin/printenv" ]

That works fine. When adding an extra_refs item, then it cant’s clone the new repo

my-owner/my-repo:
  - name: test-job
    decorate: true
    decoration_config:
    ssh_key_secrets:
    - my-private-ssh-key
    - my-private-ssh-key-2
    clone_uri: git@github.com/my-owner/my-repo.git
    extra_refs:
    - org: my-owner
      repo: my-repo-2
      base_ref: master
      workdir: false
      clone_uri: git@github.com/my-owner/my-repo-2.git
    spec:
      containers:
        - image: alpine
          command: [ "/bin/printenv" ]

I get the errors:

 clonerefs {"command":"git fetch git@github.com:my-org/my-repo-2.git --tags --prune","component":"clonerefs","error":"exit status 128","file":"prow/pod-utils/c 
 lone/clone.go:61","func":"k8s.io/test-infra/prow/pod-utils/clone.Run.func2","level":"info","msg":"Ran command","output":"ERROR: Repository not found.\nfatal: Could not read f 
 rom remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists.\n","severity":"info","time":"2021-04-19T04:37:52Z"}  

It’s as if clonerefs was unable to use the new ssh key.

I can confirm that the ssh-key works, because if I create a postsubmit job for that second repository:

      my-org/my-repo-2:
      - name: test-postsubmit
        decorate: true
        decoration_config:
          ssh_key_secrets:
            - my-private-ssh-key-2
        clone_uri: "git@github.com:my-org/my-repo-2.git"
        spec:
          containers:
          - image: alpine
            command: ["/bin/printenv"]

Then cloneref can clone the repo successfully.

What you expected to happen:

That it should be able to clone multiple private repositories.

It may be due to the fact that ssh does not play well with multiple ssh keys pointing to the same domain (github.com). I’ve used tekton in the past, and I had to hack around the /etc/ssh/ssh.conf file.

How to reproduce it (as minimally and precisely as possible):

See the first section

Please provide links to example occurrences, if any:

Anything else we need to know?:

Perhaps a per-repo ssh config could help 👍

git config core.sshCommand "ssh -i ~/.ssh/id_rsa_example -o 'IdentitiesOnly yes'"

https://gist.github.com/jexchan/2351996

Workaround

The workaround I found:

1. Create a preset for loading the extra-repo ssh-key manually inside the job. You cannot add the extra-repo key to the ssh_key_secrets, because it will fail to clone the main repo (even if you change the order of the keys doesn’t matter (I tried))

    presets:
    - labels:
        preset-extra-repo: "true"
      volumes:
      - name: "ssh-keys-my-private-ssh-key-2"
        secret:
          secretName: "my-private-ssh-key-2"
          defaultMode: 256
      volumeMounts:
      - name: "ssh-keys-my-private-ssh-key-2"
        mountPath: "/secrets/ssh/my-private-ssh-key-2"
        readOnly: true

1. Add a label to the job config to apply the preset

    postsubmits:
      my-org/my-repo:
      - name: test-postsubmit
        decorate: true
        decoration_config:
          ssh_key_secrets:
            - my-private-ssh-key
        clone_uri: "git@github.com:my-org/my-repo.git"
        labels:
          # THIS line.
          preset-extra-repo: "true"

1. In the job script, clone the repository manually

    postsubmits:
      my-org/my-repo:
      - name: test-postsubmit
        decorate: true
        decoration_config:
          ssh_key_secrets:
            - my-private-ssh-key
        clone_uri: "git@github.com:my-org/my-repo.git"
        labels:
          # THIS line.
          preset-extra-repo: "true"
        containers:
        - name: test
          command: [ "bash", "-c" ]
          args: 
          - >
            echo "github.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa+PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84KezmD5QlWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+weqqUUmpaaasXVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28G3skua2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ==" > /etc/.ssh/known_hosts &&
              mkdir -p /home/prow/go/src/github.com/my-org/my-repo-2 &&
              ssh-agent bash -c 'ssh-add /secrets/ssh/my-private-ssh-key-2/ssh-privatekey; git clone git@github.com:my-org/my-repo-2.git /home/prow/go/src/github.com/my-org/my-repo-2' &&
            ... Do some stuff

If you need to pull/push stuff from the extra repository, you have to wrap the command with

ssh-agent bash -c 'ssh-add /secrets/ssh/my-private-key-2/ssh-privatekey; git push'

Personally, I wrapped this in a bash script that I put in /bin

/bin/git_repo2

#!/usr/bin/env bash

set -e
mkdir -p ~/.ssh
echo "github.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa+PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84KezmD5QlWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+weqqUUmpaaasXVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28G3skua2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ==" > ~/.ssh/known_hosts
ssh-agent bash -c "ssh-add /secrets/ssh/my-private-key-2/ssh-privatekey; git $*"

Then, I can

containers: 
- name: test
  command: [ "bash", "-c" ]
  args: 
  - >
    mkdir -p /home/prow/go/src/github.com/my-org/my-repo-2 &&
    cd /home/prow/go/src/github.com/my-org/my-repo-2 &&
    git_repo2 clone git@github.com:my-org/my-repo.git . &&
    kustomize edit set image nginx=bla &&
    git add . && git commit -m "updated nginx image" && git_repo2 push