source-controller: Github Enterprise SSH handshake failing

Describe the bug

We use flux2 with our github enterprise instance. This worked fine until the 0.28.3 Release. Since the Update we cannot fetch our repos anymore and get the following error message:

k get gitrepositories -n flux-system

NAME         URL                                                              AGE   READY   STATUS
flux2-sync   ssh://git@foo/abc/def   64d   False   failed to checkout and determine revision: unable to clone ' ssh://git@foo/abc/def': ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain

With version 0.27.0 everything is fine again, without any other change to the environment (same keys and so on):

k get gitrepositories -n flux-system 

NAME         URL                                                              AGE   READY   STATUS
flux2-sync   ssh://git@foo/abc/def   64d   True    Fetched revision: main/4afa5937b48b12a42c0739c13870c282826b3979

If you need more information, please let me know. Unfortunately the source-controller log also doesnt give me more than the above error message

Steps to reproduce

We installed flux with the community helm charts without any relevant change to the values.yaml

Expected behavior

It is working again ^^

Screenshots and recordings

No response

OS / Distro

N/A

Flux version

N/A

Flux check

► checking prerequisites ✗ flux 0.24.1 <0.28.5 (new version is available, please upgrade) //cli is not used ✔ Kubernetes 1.21.5-eks-bc4871b >=1.19.0-0 ► checking controllers ✔ helm-controller: deployment ready ► ghcr.io/fluxcd/helm-controller:v0.18.2 ✔ image-automation-controller: deployment ready ► ghcr.io/fluxcd/image-automation-controller:v0.21.1 ✔ image-reflector-controller: deployment ready ► ghcr.io/fluxcd/image-reflector-controller:v0.17.1 ✔ kustomize-controller: deployment ready ► ghcr.io/fluxcd/kustomize-controller:v0.22.2 ✔ notification-controller: deployment ready ► ghcr.io/fluxcd/notification-controller:v0.23.1 ✔ source-controller: deployment ready ► ghcr.io/fluxcd/source-controller:v0.22.3 ✔ all checks passed

Git provider

GitHub Enterprise

Container Registry provider

No response

Additional context

No response

Code of Conduct

I agree to follow this project’s Code of Conduct

About this issue

Original URL
State: open
Created 2 years ago
Reactions: 3
Comments: 44 (14 by maintainers)

Most upvoted comments

Hi is there any update or potential timeline for this issue?

hazim1093 on Jul 26, 2022

i meet same issue. My flux env :

flux check
► checking prerequisites
✗ flux 0.24.1 <0.31.1 (new version is available, please upgrade)
✔ Kubernetes 1.21.9 >=1.19.0-0
► checking controllers
✔ helm-controller: deployment ready
► ghcr.io/fluxcd/helm-controller:v0.12.0
✔ kustomize-controller: deployment ready
► ghcr.io/fluxcd/kustomize-controller:v0.15.5
✔ notification-controller: deployment ready
► ghcr.io/fluxcd/notification-controller:v0.17.1
✔ source-controller: deployment ready
► ghcr.io/fluxcd/source-controller:v0.16.0
✔ all checks passed

kubectl get gitrepositories -n flux-system
NAME          URL                                                                 READY   STATUS                                                                                                                                                                                                            AGE
flux-system   ssh://git@github.com/xxxx/xxx-xxx.git   False   unable to clone 'ssh://git@github.com/xxxx/xxx-xxx.git', error: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain   98d

flux reconcile source git flux-system
► annotating GitRepository flux-system in flux-system namespace
✔ GitRepository annotated
◎ waiting for GitRepository reconciliation
✗ GitRepository reconciliation failed: 'unable to clone 'ssh://git@github.com/xxxx/xxx-xxx.git', error: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain'

Kevinwoolworth on Jun 14, 2022

About the auth-N:

ssh -fNvv steveizzle@SECRET_ENTERPRISE_URL
.....
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp256,ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa
...

That should be a yes. And the rsa key is still working with the old version of the source controller…

Regarding the new rc:

k get deploy -n flux-system source-controller -o wide
source-controller   1/1     1            1           142d   manager      ghcr.io/fluxcd/source-controller:rc-b877bc21   app=source-controller

# ecdsa-sha2-nistp384:
k apply -f flux-secret-new.yaml 
flux reconcile source git -n testing-cmo-flux flux-app-test

► annotating GitRepository flux-app-test in testing-cmo-flux namespace
✔ GitRepository annotated
◎ waiting for GitRepository reconciliation
✔ fetched revision master/bd3e6d2d8b24c97aa57dd8cb66db6fa9c113d022

# ssh-rsa 4096:
k delete -f flux-secret-new.yaml
k apply -f flux-secret-new-rsa.yaml

flux reconcile source git -n testing-cmo-flux flux-app-test
► annotating GitRepository flux-app-test in testing-cmo-flux namespace
✔ GitRepository annotated
◎ waiting for GitRepository reconciliation
✗ GitRepository reconciliation failed: 'failed to checkout and determine revision: unable to list remote for 'ssh://git@'SECRET_ENTERPRISE_URL: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain'


# Relevant Reconcilliation Loop errors: 
k logs -n flux-system source-controller-67f4567c-7bz5z
{"level":"error","ts":"2022-06-13T08:36:38.301Z","logger":"controller.gitrepository","msg":"failed to checkout and determine revision: unable to list remote for 'ssh://git@SECRET_ENTERPRISE_URL/flux-app-test.git': ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain","reconciler group":"source.toolkit.fluxcd.io","reconciler kind":"GitRepository","name":"flux-app-test","namespace":"testing-cmo-flux","cid":"83666aeb-96e2-4a90-aaf1-6b3abe5f5bb1","error":"failed to checkout and determine revision: unable to list remote for 'ssh://git@SECRET_ENTERPRISE_URL/flux-app-test.git': ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain","stacktrace":"github.com/fluxcd/source-controller/internal/reconcile/summarize.ErrorActionHandler\n\tgithub.com/fluxcd/source-controller/internal/reconcile/summarize/processor.go:78\ngithub.com/fluxcd/source-controller/internal/reconcile/summarize.(*Helper).SummarizeAndPatch\n\tgithub.com/fluxcd/source-controller/internal/reconcile/summarize/summary.go:182\ngithub.com/fluxcd/source-controller/controllers.(*GitRepositoryReconciler).Reconcile.func1\n\tgithub.com/fluxcd/source-controller/controllers/gitrepository_controller.go:211\ngithub.com/fluxcd/source-controller/controllers.(*GitRepositoryReconciler).Reconcile\n\tgithub.com/fluxcd/source-controller/controllers/gitrepository_controller.go:240\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\tsigs.k8s.io/controller-runtime@v0.11.2/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.11.2/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.11.2/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.11.2/pkg/internal/controller/controller.go:227"}
{"level":"error","ts":"2022-06-13T08:36:38.301Z","msg":"failed to checkout and determine revision: unable to list remote for 'ssh://git@SECRET_ENTERPRISE_URL/flux-app-test.git': ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain","name":"flux-app-test","namespace":"testing-cmo-flux","reconciler kind":"GitRepository","annotations":null,"error":"GitOperationFailed","stacktrace":"github.com/fluxcd/pkg/runtime/events.(*Recorder).Eventf\n\tgithub.com/fluxcd/pkg/runtime@v0.16.2/events/recorder.go:113\ngithub.com/fluxcd/source-controller/internal/reconcile/summarize.recordEvent\n\tgithub.com/fluxcd/source-controller/internal/reconcile/summarize/processor.go:123\ngithub.com/fluxcd/source-controller/internal/reconcile/summarize.ErrorActionHandler\n\tgithub.com/fluxcd/source-controller/internal/reconcile/summarize/processor.go:80\ngithub.com/fluxcd/source-controller/internal/reconcile/summarize.(*Helper).SummarizeAndPatch\n\tgithub.com/fluxcd/source-controller/internal/reconcile/summarize/summary.go:182\ngithub.com/fluxcd/source-controller/controllers.(*GitRepositoryReconciler).Reconcile.func1\n\tgithub.com/fluxcd/source-controller/controllers/gitrepository_controller.go:211\ngithub.com/fluxcd/source-controller/controllers.(*GitRepositoryReconciler).Reconcile\n\tgithub.com/fluxcd/source-controller/controllers/gitrepository_controller.go:240\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\tsigs.k8s.io/controller-runtime@v0.11.2/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.11.2/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.11.2/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.11.2/pkg/internal/controller/controller.go:227"}
{"level":"debug","ts":"2022-06-13T08:36:38.302Z","logger":"events","msg":"Warning","object":{"kind":"GitRepository","namespace":"testing-cmo-flux","name":"flux-app-test","uid":"7e4d321f-b7b9-4370-bd92-70f825d88f42","apiVersion":"source.toolkit.fluxcd.io/v1beta2","resourceVersion":"567995259"},"reason":"GitOperationFailed","message":"failed to checkout and determine revision: unable to list remote for 'ssh://git@SECRET_ENTERPRISE_URL/flux-app-test.git': ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain"}
{"level":"error","ts":"2022-06-13T08:36:38.306Z","logger":"controller.gitrepository","msg":"Reconciler error","reconciler group":"source.toolkit.fluxcd.io","reconciler kind":"GitRepository","name":"flux-app-test","namespace":"testing-cmo-flux","error":"failed to checkout and determine revision: unable to list remote for 'ssh://git@SECRET_ENTERPRISE_URL/flux-app-test.git': ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.11.2/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.11.2/pkg/internal/controller/controller.go:227"}

# v0.21.2: 
k edit deploy -n flux-system source-controller 
deployment.apps/source-controller edited

k get deploy -n flux-system source-controller -o wide
NAME                READY   UP-TO-DATE   AVAILABLE   AGE    CONTAINERS   IMAGES                                     SELECTOR
source-controller   1/1     1            1           142d   manager      ghcr.io/fluxcd/source-controller:v0.21.2   app=source-controller

flux reconcile source git -n testing-cmo-flux flux-app-test
► annotating GitRepository flux-app-test in testing-cmo-flux namespace
✔ GitRepository annotated
◎ waiting for GitRepository reconciliation
✔ fetched revision master/bd3e6d2d8b24c97aa57dd8cb66db6fa9c113d022

steveizzle on Jun 13, 2022