argo-workflows: Argo CLI does not work with load balancer (e.g. AWS ALB) that do not support HTTP/2

Summary

I followed the docs to use argo CLI with the access token.

API Request

It works with API.

> curl https://$HOST/api/v1/workflow-templates/$NAMESPACE -H "Authorization: $ARGO_TOKEN"
{"metadata":{"se ...}

CLI Request

It doesn’t work with CLI.

> argo template list -n $NAMESPACE -v -e -k
DEBU[0000] CLI version                                   version="{latest+f8e750d.dirty 2020-09-17T22:11:28Z f8e750de5ebab6f3c494c972889b31ef24c73c9b v2.11.0 dirty go1.15.2 gc darwin/amd64}"
DEBU[0000] Client options                                opts="{{$HOST:443 true true}  0x236c500 <nil> 0x236c560}"
FATA[0000] could not find a token

> argo template list -n $NAMESPACE -v --token $ARGO_TOKEN
DEBU[0000] CLI version                                   version="{latest+f8e750d.dirty 2020-09-17T22:11:28Z f8e750de5ebab6f3c494c972889b31ef24c73c9b v2.11.0 dirty go1.15.2 gc darwin/amd64}"
DEBU[0000] Client options                                opts="{{$HOST:443 false false}  0x236c500 <nil> 0x236c560}"
2020/10/28 00:10:32 rpc error: code = Unavailable desc = connection closed

> argo template list -n $NAMESPACE -v -e --token $ARGO_TOKEN
DEBU[0000] CLI version                                   version="{latest+f8e750d.dirty 2020-09-17T22:11:28Z f8e750de5ebab6f3c494c972889b31ef24c73c9b v2.11.0 dirty go1.15.2 gc darwin/amd64}"
DEBU[0000] Client options                                opts="{{$HOST:443 true false}  0x236c500 <nil> 0x236c560}"
2020/10/28 00:10:43 rpc error: code = Unknown desc = : HTTP status code 464; transport: missing content-type field

> argo template list -n $NAMESPACE -v -e -k --token $ARGO_TOKEN
DEBU[0000] CLI version                                   version="{latest+f8e750d.dirty 2020-09-17T22:11:28Z f8e750de5ebab6f3c494c972889b31ef24c73c9b v2.11.0 dirty go1.15.2 gc darwin/amd64}"
DEBU[0000] Client options                                opts="{{$HOST:443 true true}  0x236c500 <nil> 0x236c560}"
2020/10/28 00:10:51 rpc error: code = Unknown desc = : HTTP status code 464; transport: missing content-type field

Diagnostics

What Kubernetes provider are you using?

EKS 1.17.9

What version of Argo Workflows are you running?

> argo version
argo: latest+5eebce9.dirty
  BuildDate: 2020-10-19T20:59:29Z
  GitCommit: 5eebce9af4409da9de536f189877542dd88692e0
  GitTreeState: dirty
  GitTag: v2.11.6
  GoVersion: go1.15.3
  Compiler: gc
  Platform: darwin/amd64

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 27 (27 by maintainers)

Commits related to this issue

docs: Update access token docs. See #4394 — committed to argoproj/argo-workflows by alexec 4 years ago
feat(cli): Support HTTP CLI requests. Fixes #4394 Signed-off-by: Alex Collins <alex_collins@intuit.com> — committed to alexec/argo-workflows by alexec 4 years ago
feat(server): Add WorkflowLogs API. See #4394 — committed to alexec/argo-workflows by alexec 4 years ago
feat(server): Add WorkflowLogs API. See #4394 (#4450) Signed-off-by: Alex Collins <alex_collins@intuit.com> — committed to argoproj/argo-workflows by alexec 4 years ago
feat(cli): Support ARGO_HTTP1 for HTTP/1 CLI requests. Fixes #4394 (#4416) Signed-off-by: Alex Collins <alex_collins@intuit.com> — committed to argoproj/argo-workflows by alexec 4 years ago
feat(cli): Support ARGO_HTTP1 for HTTP/1 CLI requests. Fixes #4394 (#4416) Signed-off-by: Alex Collins <alex_collins@intuit.com> — committed to argoproj/argo-workflows by alexec 4 years ago
feat(server): Add WorkflowLogs API. See #4394 (#4450) Signed-off-by: Alex Collins <alex_collins@intuit.com> Signed-off-by: Alex Capras <alexcapras@gmail.com> — committed to alexcapras/argo by alexec 4 years ago
feat(cli): Support ARGO_HTTP1 for HTTP/1 CLI requests. Fixes #4394 (#4416) Signed-off-by: Alex Collins <alex_collins@intuit.com> Signed-off-by: Alex Capras <alexcapras@gmail.com> — committed to alexcapras/argo by alexec 4 years ago

Most upvoted comments

You can use the CLI in one of three modes:

# Kubernetes API Mode (default)

Requests are sent directly to the Kubernetes API. No Argo Server is needs. Large workflows and the workflow archive are not supported.

Use when you have direct access to the Kubernetes API, and don't need large workflow or workflow archive support.

If you're using instance ID (which is very unlikely), you'll need to set it:

	ARGO_INSTANCEID=your-instance-ic

# Argo Server GRPC Mode 

Requests are sent to the Argo Server API via GRPC (using HTTP/2). Large workflows and the workflow archive are supported. Network load-balancers that do not support HTTP/2 are not supported. 

Use if you do not have access to the Kubernetes API (e.g. you're in another cluster), and you're running the Argo Server using a network load-balancer that support HTTP/2.

To enable, set ARGO_SERVER. This is in the format  "host:port" - do not prefix with "http" or "https":

	ARGO_SERVER=localhost:2746 ;

If you're have transport-layer security (TLS) enabled (i.e. you are running "argo server --secure"):

	ARGO_SECURE=true

If your server is running with self-signed certificates. Do not use in production:

	ARGO_INSECURE_SKIP_VERIFY=true 

If your server is behind an ingress with a path (you'll be running "arg server --bashhref /...) or "BASH_HREF=/... argo server""):

	ARGO_PATH=/argo

By default, the CLI uses your KUBECONFIG to determine default for ARGO_TOKEN and ARGO_NAMESPACE. You probably don't want this, so to prevent it:

	KUBECONFIG=/dev/null

You will then need to set:
 
	ARGO_NAMESPACE=argo 

And 

	ARGO_TOKEN='Bearer ****' ;# Should always start with "Bearer " or "Basic ". 

# Argo Server HTTP Mode

As per GRPC mode, but uses HTTP. Can be used with ALB that does not support HTTP/2. The command "argo logs --since-time=2020...." will not work (due to time-type).

Use this when your network load-balancer does not support HTTP/2.

Use the same configuration as GRPC mode, but also set:

	ARGO_HTTP=true

alexec on Nov 2, 2020

I’ve created binaries to test:

https://github.com/argoproj/argo/releases/tag/http-rc1

alexec on Nov 5, 2020

excellent news!

alexec on Nov 3, 2020

I’ve pushed argoproj/argocli:http for testing. You’ll need to set ARGO_HTTP=true to use it. I’ll upload a Mac binary shortly.

alexec on Oct 29, 2020