concourse: Could not read from remote repository.

Summary

I upgraded from version 6.5.5 to version 7.0.0.

Since then, I am unable to clone a git resource. I use a private SSH key for repo authorization.

resources:
- icon: gitlab
  name: concourse-ci-pipelines
  source:
    branch: develop
    private_key: (("gitlab.xyz.com.ssh_private_key))
    uri: git@gitlab.xyz.com:summit/common/tools/concourse-ci-pipelines.git

I redeployed Concourse v6.5.5 with the same deployment scripts. It works very well. I start the procedure again by changing only the version number to v7.0.0 and I still get this result:

selected worker: prd-concourse-worker-67449cbb77-ct77j Identity added: /tmp/git-resource-private-key Cloning into ‘/tmp/git-resource-repo-cache’… fatal: Could not read from remote repository.

Please make sure you have the correct access rights and the repository exists.

Do you have an idea how to diagnose this problem?

About this issue

Original URL
State: closed
Created 3 years ago
Reactions: 1
Comments: 22 (8 by maintainers)

Most upvoted comments

Um my broken pipelines are now magically working again I did nothing except perhaps restart the worker (I checked right after restarting, broken, went to lunch, and when I came back they were working fine, fetching git repos like nothing).

Maybe I should go out for lunch more often 😉

srheaume on Feb 17, 2021

Thanks for all the new info! That’s really helpful. I’ll try and reproduce with minikube and/or one of our GKE clusters. I have some questions that if you can answer will help me:

k8 version this is happening on?

To clarify:

this issue started with the garden/guardian runtime? And…

The issue started with release 7.0.0 with default runtime.

containerd kinda fixes the issue, the caveat being that containerd isn’t enabling local k8s dns addresses to resolve?

Exactly

Any special config you had to do for the helm chart or did default values reproduce the issue?

Default values will reproduce the issue

srheaume on Feb 18, 2021

Maybe try this:

I followed the recommendation to enable containerd: CONCOURSE_RUNTIME=containerd

then I remembered I need to force the DNS, which was previously done in garden-config.ini, but the release notes suggest doing this now: CONCOURSE_CONTAINERD_DNS_SERVER=192.168.1.1 so I think my sequence was:

set CONCOURSE_RUNTIME=containerd
restart, things don’t work
set the CONCOURSE_CONTAINERD_DNS_SERVER to what I had previously in garden-config.ini
Somehow bork the worker restart in a way that made me think things weren’t yet working
go to lunch
worker restart completed properly, things are now working because of the DNS setting

for clarity, the reason I’m setting the DNS server this way is because of what’s mentioned in https://concourse-ci.org/concourse-worker.html#troubleshooting-and-fixing-dns-resolution (which might need to be updated for containerd now?)

Roadmaster on Feb 17, 2021

@taylorsilva With the tag set to 1.12.0, it does not work. It may be related to one of the following two :

SSH_ASKPASS_REQUIRE=force
alpine:3 to alpine:latest

This has been introduced in version 1.12.0

srheaume on Feb 17, 2021

@taylorsilva Thank you for your reply. I think we’re starting to narrow down the problem. It works fine with the following:

resource_types:
- name: git
  type: registry-image
  source:
    repository: concourse/git-resource
    tag: 1.11.0

Version 1.11.0 works well. It seems that something was introduced in version 1.12.0 that creates a problem.

srheaume on Feb 17, 2021