argo-workflows: Quickstart: transport: authentication handshake failed

Summary

What happened/what you expected to happen? When connecting to the UI of Argo Workflows (after port forwarding), I receive this message and virtually no components load:

Service Unavailable: connection error: desc = "transport: authentication handshake failed: x509: certificate signed by unknown authority"

What version of Argo Workflows are you running? latest (ie 3.2.6)

Diagnostics

Quickstart Guide - follow steps in guide to reproduce

What Kubernetes provider are you using? EKS/Minikube

Configuration:

    spec:
      containers:
      - args:
        - server
        - --namespaced
        - --auth-mode
        - server
        - --auth-mode
        - client
        - --insecure-skip-verify
        - "true"

kustomization.yaml

resources:
  - namespace.yml
  - https://raw.githubusercontent.com/argoproj/argo-workflows/v3.2.6/manifests/quick-start-minimal.yaml

namespace: argo-workflows

Note: This is reproducible w/ the minimal quickstart via minikube as well

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

About this issue

Original URL
State: closed
Created 2 years ago
Reactions: 11
Comments: 18 (6 by maintainers)

Commits related to this issue

fix: insecureSkipVerify needed. Fixes #7632 (#7651) Signed-off-by: Tom Meadows <thomas.meadows@jetstack.io> — committed to argoproj/argo-workflows by ChaosInTheCRD 2 years ago
fix: insecureSkipVerify needed. Fixes #7632 (#7651) Signed-off-by: Tom Meadows <thomas.meadows@jetstack.io> — committed to yriveiro/argo-workflows by ChaosInTheCRD 2 years ago

Most upvoted comments

@materkey I have done this in the past…its just kind of annoying to update the readiness probe, add a new flag to server start, update ingress; This definitely needs to be resolve by the Argo team bc any new user of workflows is going to be put off by these errors. I have used the quickstart in the past without any issues…so this seems a bit wonky.

jmclean-starburst on Jan 25, 2022

Please, you must use the manifests attached to the release:

kubectl create namespace argo
kubectl apply -n argo -f https://github.com/argoproj/argo-workflows/releases/download/v3.2.6/install.yaml

alexec on Jan 25, 2022

I was able to fix this by patching the image off of latest

resources:
  - namespace.yml
  - https://raw.githubusercontent.com/argoproj/argo-workflows/v3.2.6/manifests/quick-start-minimal.yaml

images:
- name: quay.io/argoproj/argocli
  newTag: v3.2.6

namespace: argo-workflows

jmclean-starburst on Jan 25, 2022

I believe I found the issue; the argo CLI is using latest for the image tag, instead of specifying the version tag (ie v3.2.6). I have tested this w/ v3.1.15 and after changing the image tag, it works. I will now validate same for version 3.2.6

jmclean-starburst on Jan 25, 2022

UPD: After @jmclean-starburst comment https://github.com/argoproj/argo-workflows/issues/7632#issuecomment-1021252593 I’m using v3.2.6 for quay.io/argoproj/argocli container image and for my kustomization.yaml to solve this issue

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- github.com/argoproj/argo-workflows/manifests/base?ref=v3.2.6
- github.com/argoproj/argo-workflows/manifests/cluster-install/workflow-controller-rbac?ref=v3.2.6
- github.com/argoproj/argo-workflows/manifests/cluster-install/argo-server-rbac?ref=v3.2.6
...

Old comment text

I’ve ended up by using --secure=false and scheme: HTTP for argo-server probe as a workaround. Also deleted nginx.ingress.kubernetes.io/backend-protocol: https for ingress. Https for argo workflows worked for months and now it is broken for an unknown reason.

materkey on Jan 25, 2022