argo-workflows: Error in UI: `Connection closed to api/v1/workflow-events/argo?...`
Summary
What happened/what you expected to happen?
I exposed Argo Workflow UI using Ambassador under /argo/ subpath. I was able to access the UI. However, I got Unable to load data error.

After that I got recurring “Connection closed to api/v1/workflow-events/argo?listOptions.resourceVersion=30237…” error.
Connection closed to api/v1/workflow-events/argo?listOptions.resourceVersion=7273&fields=result.object.metadata.name,result.object.metadata.namespace,result.object.metadata.resourceVersion,result.object.metadata.uid,result.object.status.finishedAt,result.object.status.phase,result.object.status.startedAt,result.object.status.estimatedDuration,result.object.status.progress,result.type,result.object.metadata.labels,result.object.spec.suspend:
Diagnostics
What Kubernetes provider are you using?
1.19.4
What version of Argo Workflows are you running?
2.12.2
Steps to reproduce
- Create K3D cluster
k3d cluster create dev-ld --api-port 6443 -p 8080:80@loadbalancer --agents 2 \
--k3s-server-arg '--flannel-backend=none' --k3s-server-arg '--no-deploy=traefik' \
--volume "$(CURRENT_DIR)/k3d_custom/calico.yaml:/var/lib/rancher/k3s/server/manifests/calico.yaml"
- Install Ambassador Edge Stack
kubectl apply -f https://www.getambassador.io/yaml/aes-crds.yaml && \
kubectl wait --for condition=established --timeout=90s crd -lproduct=aes && \
kubectl apply -f https://www.getambassador.io/yaml/aes.yaml && \
kubectl -n ambassador wait --for condition=available --timeout=90s deploy -lproduct=aes
- Install metallb metal-lb-layer2-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: metallb-system
name: config
data:
config: |
address-pools:
- name: default
protocol: layer2
addresses:
- 172.18.0.3-172.18.0.254
# https://kubernetes.github.io/ingress-nginx/deploy/baremetal/
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.5/manifests/namespace.yaml
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.5/manifests/metallb.yaml
# On first install only
kubectl create secret generic -n metallb-system memberlist --from-literal=secretkey="$(openssl rand -base64 128)"
kubectl apply -f $(CURRENT_DIR)/k3d_custom/metal-lb-layer2-config.yaml
- Configure tuntap - Mac only
# Install tuntap:
brew install Caskroom/cask/tuntap
# https://blog.kubernauts.io/k3s-with-k3d-and-metallb-on-mac-923a3255c36e
# only needed for Mac
# create a bridge interface between the Physical Machine and the Host Virtual Machine.
# download scripts form https://github.com/arashkaffamanesh/k3d-k3s-metallb
k3d cluster stop dev-ld
./k3d_custom/docker-tuntap-osx/sbin/docker_tap_install.sh
echo "Wait till docker is restarted."
sleep 40
echo "..done"
./k3d_custom/docker-tuntap-osx/sbin/docker_tap_up.sh
sudo route -v add -net 172.18.0.0 -netmask 255.255.255.0 10.0.75.2
k3d cluster start dev-ld
- Install argo
kubectl create namespace argo || true
kubectl -n argo apply -f https://raw.githubusercontent.com/argoproj/argo/master/manifests/quick-start-postgres.yaml
# Since K3S uses containerd, you need to configure the workflow controller to use the PNS (Pod Namespace Sharing) executor
kubectl -n argo patch cm workflow-controller-configmap -p '{"data": {"containerRuntimeExecutor": "pns"}}';
kubectl -n argo wait pod -lapp=argo-server --for condition=available --timeout=180s
- Patch argo argocd-patch.yaml
spec:
template:
spec:
containers:
- args:
- server
- --namespaced
- --auth-mode
- server
- --auth-mode
- client
- --basehref
- /argo/
image: argoproj/argocli:latest
name: argo-server
kubectl -n argo patch deployment/argo-server -p "$(cat argocd-patch.yaml)"
- Create Ambassador mapping argo-ingress-ambassador.yaml
apiVersion: getambassador.io/v2
kind: Mapping
metadata:
name: argo-server
namespace: argo
spec:
prefix: /argo/
regex_rewrite:
pattern: '/argo(/|$)(.*)'
substitution: '/\2'
service: argo-server:2746
kubectl -n argo apply -f resources/argo-ingress-ambassador.yaml
- argo-server log
❯ kubectl -n argo logs argo-server-db794976-gchd2
time="2020-12-27T05:06:17.892Z" level=info authModes="[server client]" baseHRef=/argo/ managedNamespace=argo namespace=argo secure=false
time="2020-12-27T05:06:17.893Z" level=warning msg="You are running in insecure mode. Learn how to enable transport layer security: https://argoproj.github.io/argo/tls/"
time="2020-12-27T05:06:17.893Z" level=info msg="config map" name=workflow-controller-configmap
time="2020-12-27T05:06:17.893Z" level=info msg="SSO disabled"
time="2020-12-27T05:06:18.061Z" level=info msg="Starting Argo Server" instanceID= version=764f118ca4180b476a9b053873e55eeccbc5e202
time="2020-12-27T05:06:18.062Z" level=info msg="Creating DB session"
time="2020-12-27T05:06:18.121Z" level=info msg="Node status offloading config" ttl=5m0s
time="2020-12-27T05:06:18.122Z" level=info msg="Creating event controller" operationQueueSize=16 workerCount=4
time="2020-12-27T05:06:18.125Z" level=info msg="Argo Server started successfully on http://localhost:2746"
time="2020-12-27T05:10:23.551Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=GetVersion grpc.service=info.InfoService grpc.start_time="2020-12-27T05:10:23Z" grpc.time_ms=37.122 span.kind=server system=grpc
time="2020-12-27T05:10:23.661Z" level=warning msg="finished unary call with code PermissionDenied" error="rpc error: code = PermissionDenied desc = workflowtemplates.argoproj.io is forbidden: User \"system:serviceaccount:argo:argo-server\" cannot list resource \"workflowtemplates\" in API group \"argoproj.io\" at the cluster scope" grpc.code=PermissionDenied grpc.method=ListWorkflowTemplates grpc.service=workflowtemplate.WorkflowTemplateService grpc.start_time="2020-12-27T05:10:23Z" grpc.time_ms=81.194 span.kind=server system=grpc
time="2020-12-27T05:10:23.665Z" level=warning msg="finished unary call with code PermissionDenied" error="rpc error: code = PermissionDenied desc = cronworkflows.argoproj.io is forbidden: User \"system:serviceaccount:argo:argo-server\" cannot list resource \"cronworkflows\" in API group \"argoproj.io\" at the cluster scope" grpc.code=PermissionDenied grpc.method=ListCronWorkflows grpc.service=cronworkflow.CronWorkflowService grpc.start_time="2020-12-27T05:10:23Z" grpc.time_ms=81.08 span.kind=server system=grpc
time="2020-12-27T05:10:23.757Z" level=warning msg="finished unary call with code PermissionDenied" error="rpc error: code = PermissionDenied desc = workflows.argoproj.io is forbidden: User \"system:serviceaccount:argo:argo-server\" cannot list resource \"workflows\" in API group \"argoproj.io\" at the cluster scope" grpc.code=PermissionDenied grpc.method=ListWorkflows grpc.service=workflow.WorkflowService grpc.start_time="2020-12-27T05:10:23Z" grpc.time_ms=5.809 span.kind=server system=grpc
time="2020-12-27T05:10:23.917Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=GetInfo grpc.service=info.InfoService grpc.start_time="2020-12-27T05:10:23Z" grpc.time_ms=0.886 span.kind=server system=grpc
time="2020-12-27T05:10:24.080Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=ListWorkflowTemplates grpc.service=workflowtemplate.WorkflowTemplateService grpc.start_time="2020-12-27T05:10:23Z" grpc.time_ms=120.036 span.kind=server system=grpc
time="2020-12-27T05:10:24.090Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=ListCronWorkflows grpc.service=cronworkflow.CronWorkflowService grpc.start_time="2020-12-27T05:10:23Z" grpc.time_ms=124.877 span.kind=server system=grpc
time="2020-12-27T05:10:24.150Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=ListWorkflows grpc.service=workflow.WorkflowService grpc.start_time="2020-12-27T05:10:24Z" grpc.time_ms=132.258 span.kind=server system=grpc
time="2020-12-27T05:10:27.242Z" level=info msg="finished streaming call with code OK" grpc.code=OK grpc.method=WatchWorkflows grpc.service=workflow.WorkflowService grpc.start_time="2020-12-27T05:10:24Z" grpc.time_ms=3000.936 span.kind=server system=grpc
time="2020-12-27T05:10:37.329Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=ListWorkflows grpc.service=workflow.WorkflowService grpc.start_time="2020-12-27T05:10:37Z" grpc.time_ms=15.356 span.kind=server system=grpc
time="2020-12-27T05:10:40.387Z" level=info msg="finished streaming call with code OK" grpc.code=OK grpc.method=WatchWorkflows grpc.service=workflow.WorkflowService grpc.start_time="2020-12-27T05:10:37Z" grpc.time_ms=2999.413 span.kind=server system=grpc
- workflow-controller log
❯ kubectl logs -n argo $(kubectl get pods -l app=workflow-controller -n argo -o name) | more
time="2020-12-27T05:08:33.189Z" level=info msg="config map" name=workflow-controller-configmap
time="2020-12-27T05:08:33.233Z" level=info msg="Configuration:\nartifactRepository:\n archiveLogs: true\n s3:\n accessKeySecret:\n key: accesskey\n name: my-minio-cred\n bucket: my-bucket\n endpoint: minio:9000\n insecure: true\n secretKeySecret:\n key: secretkey\n name: my-minio-cred\ncontainerRuntimeExecutor: pns\ninitialDelay: 0s\nlinks:\n- name: Example Workflow Link\n scope: workflow\n url: http://logging-facility?namespace=${metadata.namespace}&workflowName=${metadata.name}&startedAt=${status.startedAt}&finishedAt=${status.finishedAt}\n- name: Example Pod Link\n scope: pod\n url: http://logging-facility?namespace=${metadata.namespace}&podName=${metadata.name}&startedAt=${status.startedAt}&finishedAt=${status.finishedAt}\nmetricsConfig:\n disableLegacy: true\n enabled: true\n path: /metrics\n port: 9090\nnodeEvents: {}\npersistence:\n archive: true\n archiveTTL: 168h0m0s\n connectionPool:\n maxIdleConns: 100\n nodeStatusOffLoad: true\n postgresql:\n database: postgres\n host: postgres\n passwordSecret:\n key: password\n name: argo-postgres-config\n port: 5432\n tableName: argo_workflows\n userNameSecret:\n key: username\n name: argo-postgres-config\npodSpecLogStrategy: {}\ntelemetryConfig: {}\n"
:...skipping...
time="2020-12-27T05:08:33.189Z" level=info msg="config map" name=workflow-controller-configmap
time="2020-12-27T05:08:33.233Z" level=info msg="Configuration:\nartifactRepository:\n archiveLogs: true\n s3:\n accessKeySecret:\n key: accesskey\n name: my-minio-cred\n bucket: my-bucket\n endpoint: minio:9000\n insecure: true\n secretKeySecret:\n key: secretkey\n name: my-minio-cred\ncontainerRuntimeExecutor: pns\ninitialDelay: 0s\nlinks:\n- name: Example Workflow Link\n scope: workflow\n url: http://logging-facility?namespace=${metadata.namespace}&workflowName=${metadata.name}&startedAt=${status.startedAt}&finishedAt=${status.finishedAt}\n- name: Example Pod Link\n scope: pod\n url: http://logging-facility?namespace=${metadata.namespace}&podName=${metadata.name}&startedAt=${status.startedAt}&finishedAt=${status.finishedAt}\nmetricsConfig:\n disableLegacy: true\n enabled: true\n path: /metrics\n port: 9090\nnodeEvents: {}\npersistence:\n archive: true\n archiveTTL: 168h0m0s\n connectionPool:\n maxIdleConns: 100\n nodeStatusOffLoad: true\n postgresql:\n database: postgres\n host: postgres\n passwordSecret:\n key: password\n name: argo-postgres-config\n port: 5432\n tableName: argo_workflows\n userNameSecret:\n key: username\n name: argo-postgres-config\npodSpecLogStrategy: {}\ntelemetryConfig: {}\n"
time="2020-12-27T05:08:33.233Z" level=info msg="Persistence configuration enabled"
time="2020-12-27T05:08:33.234Z" level=info msg="Creating DB session"
time="2020-12-27T05:08:33.257Z" level=info msg="Persistence Session created successfully"
time="2020-12-27T05:08:33.265Z" level=info msg="Migrating database schema" clusterName=default dbType=postgres
time="2020-12-27T05:08:33.275Z" level=info msg="applying database change" change="create table if not exists argo_workflows (\n id varchar(128) ,\n name varchar(256),\n phase varchar(25),\n namespace varchar(256),\n workflow text,\n startedat timestamp default CURRENT_TIMESTAMP,\n finishedat timestamp default CURRENT_TIMESTAMP,\n primary key (id, namespace)\n)" changeSchemaVersion=0
time="2020-12-27T05:08:33.286Z" level=info msg="applying database change" change="create unique index idx_name on argo_workflows (name)" changeSchemaVersion=1
time="2020-12-27T05:08:33.295Z" level=info msg="applying database change" change="create table if not exists argo_workflow_history (\n id varchar(128) ,\n name varchar(256),\n phase varchar(25),\n namespace varchar(256),\n workflow text,\n startedat timestamp default CURRENT_TIMESTAMP,\n finishedat timestamp default CURRENT_TIMESTAMP,\n primary key (id, namespace)\n)" changeSchemaVersion=2
time="2020-12-27T05:08:33.304Z" level=info msg="applying database change" change="alter table argo_workflow_history rename to argo_archived_workflows" changeSchemaVersion=3
time="2020-12-27T05:08:33.310Z" level=info msg="applying database change" change="drop index idx_name" changeSchemaVersion=4
time="2020-12-27T05:08:33.315Z" level=info msg="applying database change" change="create unique index idx_name on argo_workflows(name, namespace)" changeSchemaVersion=5
time="2020-12-27T05:08:33.321Z" level=info msg="applying database change" change="alter table argo_workflows drop constraint argo_workflows_pkey" changeSchemaVersion=6
time="2020-12-27T05:08:33.326Z" level=info msg="applying database change" change="alter table argo_workflows add primary key(name,namespace)" changeSchemaVersion=7
time="2020-12-27T05:08:33.331Z" level=info msg="applying database change" change="alter table argo_archived_workflows drop constraint argo_workflow_history_pkey" changeSchemaVersion=8
time="2020-12-27T05:08:33.335Z" level=info msg="applying database change" change="alter table argo_archived_workflows add primary key(id)" changeSchemaVersion=9
time="2020-12-27T05:08:33.342Z" level=info msg="applying database change" change="alter table argo_archived_workflows rename column id to uid" changeSchemaVersion=10
time="2020-12-27T05:08:33.345Z" level=info msg="applying database change" change="alter table argo_archived_workflows alter column uid set not null" changeSchemaVersion=11
time="2020-12-27T05:08:33.351Z" level=info msg="applying database change" change="alter table argo_archived_workflows alter column phase set not null" changeSchemaVersion=12
time="2020-12-27T05:08:33.356Z" level=info msg="applying database change" change="alter table argo_archived_workflows alter column namespace set not null" changeSchemaVersion=13
time="2020-12-27T05:08:33.360Z" level=info msg="applying database change" change="alter table argo_archived_workflows alter column workflow set not null" changeSchemaVersion=14
time="2020-12-27T05:08:33.365Z" level=info msg="applying database change" change="alter table argo_archived_workflows alter column startedat set not null" changeSchemaVersion=15
time="2020-12-27T05:08:33.370Z" level=info msg="applying database change" change="alter table argo_archived_workflows alter column finishedat set not null" changeSchemaVersion=16
time="2020-12-27T05:08:33.374Z" level=info msg="applying database change" change="alter table argo_archived_workflows add clustername varchar(64)" changeSchemaVersion=17
time="2020-12-27T05:08:33.378Z" level=info msg="applying database change" change="update argo_archived_workflows set clustername = 'default' where clustername is null" changeSchemaVersion=18
time="2020-12-27T05:08:33.381Z" level=info msg="applying database change" change="alter table argo_archived_workflows alter column clustername set not null" changeSchemaVersion=19
time="2020-12-27T05:08:33.384Z" level=info msg="applying database change" change="alter table argo_archived_workflows drop constraint argo_archived_workflows_pkey" changeSchemaVersion=20
time="2020-12-27T05:08:33.390Z" level=info msg="applying database change" change="alter table argo_archived_workflows add primary key(clustername,uid)" changeSchemaVersion=21
time="2020-12-27T05:08:33.395Z" level=info msg="applying database change" change="create index argo_archived_workflows_i1 on argo_archived_workflows (clustername,namespace)" changeSchemaVersion=22
time="2020-12-27T05:08:33.405Z" level=info msg="applying database change" change="alter table argo_workflows drop column phase" changeSchemaVersion=23
time="2020-12-27T05:08:33.409Z" level=info msg="applying database change" change="alter table argo_workflows drop column startedat" changeSchemaVersion=24
time="2020-12-27T05:08:33.412Z" level=info msg="applying database change" change="alter table argo_workflows drop column finishedat" changeSchemaVersion=25
time="2020-12-27T05:08:33.415Z" level=info msg="applying database change" change="alter table argo_workflows rename column id to uid" changeSchemaVersion=26
time="2020-12-27T05:08:33.419Z" level=info msg="applying database change" change="alter table argo_workflows alter column uid set not null" changeSchemaVersion=27
time="2020-12-27T05:08:33.422Z" level=info msg="applying database change" change="alter table argo_workflows alter column namespace set not null" changeSchemaVersion=28
time="2020-12-27T05:08:33.425Z" level=info msg="applying database change" change="alter table argo_workflows add column clustername varchar(64)" changeSchemaVersion=29
time="2020-12-27T05:08:33.429Z" level=info msg="applying database change" change="update argo_workflows set clustername = 'default' where clustername is null" changeSchemaVersion=30
time="2020-12-27T05:08:33.432Z" level=info msg="applying database change" change="alter table argo_workflows alter column clustername set not null" changeSchemaVersion=31
time="2020-12-27T05:08:33.436Z" level=info msg="applying database change" change="alter table argo_workflows add column version varchar(64)" changeSchemaVersion=32
time="2020-12-27T05:08:33.440Z" level=info msg="applying database change" change="alter table argo_workflows add column nodes text" changeSchemaVersion=33
time="2020-12-27T05:08:33.443Z" level=info msg="applying database change" change="backfillNodes{argo_workflows}" changeSchemaVersion=34
time="2020-12-27T05:08:33.443Z" level=info msg="Backfill node status"
time="2020-12-27T05:08:33.447Z" level=info msg="applying database change" change="alter table argo_workflows alter column nodes set not null" changeSchemaVersion=35
time="2020-12-27T05:08:33.450Z" level=info msg="applying database change" change="alter table argo_workflows drop column workflow" changeSchemaVersion=36
time="2020-12-27T05:08:33.454Z" level=info msg="applying database change" change="alter table argo_workflows add column updatedat timestamp not null default current_timestamp" changeSchemaVersion=37
time="2020-12-27T05:08:33.457Z" level=info msg="applying database change" change="alter table argo_workflows drop constraint argo_workflows_pkey" changeSchemaVersion=38
time="2020-12-27T05:08:33.462Z" level=info msg="applying database change" change="drop index idx_name" changeSchemaVersion=39
time="2020-12-27T05:08:33.466Z" level=info msg="applying database change" change="alter table argo_workflows drop column name" changeSchemaVersion=40
time="2020-12-27T05:08:33.470Z" level=info msg="applying database change" change="alter table argo_workflows add primary key(clustername,uid,version)" changeSchemaVersion=41
time="2020-12-27T05:08:33.475Z" level=info msg="applying database change" change="create index argo_workflows_i1 on argo_workflows (clustername,namespace)" changeSchemaVersion=42
time="2020-12-27T05:08:33.482Z" level=info msg="applying database change" change="alter table argo_archived_workflows alter column workflow type json using workflow::json" changeSchemaVersion=43
time="2020-12-27T05:08:33.496Z" level=info msg="applying database change" change="alter table argo_archived_workflows alter column name set not null" changeSchemaVersion=44
time="2020-12-27T05:08:33.506Z" level=info msg="applying database change" change="create index argo_workflows_i2 on argo_workflows (clustername,namespace,updatedat)" changeSchemaVersion=45
time="2020-12-27T05:08:33.512Z" level=info msg="applying database change" change="create table if not exists argo_archived_workflows_labels (\n\tclustername varchar(64) not null,\n\tuid varchar(128) not null,\n name varchar(317) not null,\n value varchar(63) not null,\n primary key (clustername, uid, name),\n \tforeign key (clustername, uid) references argo_archived_workflows(clustername, uid) on delete cascade\n)" changeSchemaVersion=46
time="2020-12-27T05:08:33.523Z" level=info msg="applying database change" change="alter table argo_workflows alter column nodes type json using nodes::json" changeSchemaVersion=47
time="2020-12-27T05:08:33.536Z" level=info msg="applying database change" change="alter table argo_archived_workflows add column instanceid varchar(64)" changeSchemaVersion=48
time="2020-12-27T05:08:33.540Z" level=info msg="applying database change" change="update argo_archived_workflows set instanceid = '' where instanceid is null" changeSchemaVersion=49
time="2020-12-27T05:08:33.545Z" level=info msg="applying database change" change="alter table argo_archived_workflows alter column instanceid set not null" changeSchemaVersion=50
time="2020-12-27T05:08:33.550Z" level=info msg="applying database change" change="drop index argo_archived_workflows_i1" changeSchemaVersion=51
time="2020-12-27T05:08:33.555Z" level=info msg="applying database change" change="create index argo_archived_workflows_i1 on argo_archived_workflows (clustername,instanceid,namespace)" changeSchemaVersion=52
time="2020-12-27T05:08:33.562Z" level=info msg="applying database change" change="drop index argo_workflows_i1" changeSchemaVersion=53
time="2020-12-27T05:08:33.566Z" level=info msg="applying database change" change="drop index argo_workflows_i2" changeSchemaVersion=54
time="2020-12-27T05:08:33.571Z" level=info msg="applying database change" change="create index argo_workflows_i1 on argo_workflows (clustername,namespace,updatedat)" changeSchemaVersion=55
time="2020-12-27T05:08:33.577Z" level=info msg="applying database change" change="create index argo_archived_workflows_i2 on argo_archived_workflows (clustername,instanceid,finishedat)" changeSchemaVersion=56
time="2020-12-27T05:08:33.582Z" level=info msg="Node status offloading config" ttl=5m0s
time="2020-12-27T05:08:33.582Z" level=info msg="Node status offloading is enabled"
time="2020-12-27T05:08:33.583Z" level=info msg="Workflow archiving is enabled"
time="2020-12-27T05:08:33.585Z" level=info msg="Starting Workflow Controller" version=764f118ca4180b476a9b053873e55eeccbc5e202
time="2020-12-27T05:08:33.585Z" level=info msg="Workers: workflow: 32, pod: 32, pod cleanup: 4"
time="2020-12-27T05:08:33.771Z" level=info msg="Manager initialized successfully"
I1227 05:08:33.774426 1 leaderelection.go:242] attempting to acquire leader lease argo/workflow-controller...
I1227 05:08:33.792510 1 leaderelection.go:252] successfully acquired lease argo/workflow-controller
time="2020-12-27T05:08:33.792Z" level=info msg="new leader" id=workflow-controller-75cb95b89d-6l49r leader=workflow-controller-75cb95b89d-6l49r
time="2020-12-27T05:08:33.793Z" level=info msg="started leading" id=workflow-controller-75cb95b89d-6l49r
time="2020-12-27T05:08:33.793Z" level=info msg="Performing periodic GC every 5m0s"
time="2020-12-27T05:08:33.793Z" level=info msg="Performing archived workflow GC" periodicity=24h0m0s ttl=604800000000000
time="2020-12-27T05:08:33.799Z" level=info msg="Starting workflow TTL controller (resync 20m0s, workflowTTLWorkers 4)"
time="2020-12-27T05:08:33.799Z" level=info msg="Started workflow TTL worker"
time="2020-12-27T05:08:33.801Z" level=info msg="Starting prometheus metrics server at localhost:9090/metrics"
time="2020-12-27T05:08:33.805Z" level=info msg="Starting CronWorkflow controller"
time="2020-12-27T05:13:32.973Z" level=info msg="Alloc=5797 TotalAlloc=22882 Sys=70080 NumGC=8 Goroutines=201"
time="2020-12-27T05:13:33.581Z" level=info msg="Performing periodic workflow GC"
time="2020-12-27T05:13:33.593Z" level=info msg="Zero old offloads, nothing to do"
Message from the maintainers:
Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Reactions: 34
- Comments: 45 (13 by maintainers)
for GKE to honor this timeout, you have to annotate your service to indicate the backendconfig to use. See the Docs
This was difficult for me to find so I hope it helps someone!
@alexec you are correct that this happens in scenarios where there is some sort of middleware (load balancer, etc.) present. However, this is a lot of deployments in production I would assume. I did some research and the underlying cause is in the way Argo deals with streaming events.
Here you open an event source stream but will only send back something if the K8S watch returns anything. Which could be a long time - or never. This will cause timeouts to trigger as no data is exchanged on that connection. This leaves the only workaround/hack right now to increase such timeouts in all your middleware - that is very bad though, since it also makes scalability and being open to attack more of a problem as you end up holding much more state. Also, it never really resolves the problem - it just pushes the problem out far enough for most users to not notice.
Websockets, etc. deal with this on the protocol level. But since you are using EventSource it is on you to ensure such “keep alives” are performed on the application data channel. The event stream format allows you to quite easily send a comment to achieve just that (see: https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events#event_stream_format).
Can we reopen this issue? I am facing the same issue on EKS using AWS load balancer controller @alexec
I was missing
watchonworkflows/workflowtemplateson my sideRan into this issue. Turned out to be that the service account created for logging into Workflows didn’t have access to get and list pods + pod logs.
I am still getting the issue when using Emissary and Argo Workflows version 3.4.0
@luckyswede in case you are still on it, got it working by adding the configuration mentioned in the previous comment in my reverse proxy!
I believe this is fixed in v3.0. Closing.
I have the same issue in
v3.3.8with Ambassador3.0.0. If it can help, I’ve temporarily “patched” it by increasing the timeout tooI’ve opened a feature request 🤞🏻 https://github.com/emissary-ingress/emissary/issues/4323
I encountered this issue with Argo 3.3.1 while using Contour with its support for the Gateway API. I was unable to figure out how to make it work while using the
HTTPRouteresource from the Gateway API specification, because it makes configuring timeouts implementation specific, and I don’t think Contour has implemented anything yet. However, it turns out that Contour lets you mix and matchHTTPRoutes from the Gateway API spec and their ownHTTPProxyresources, and I was able to get rid of this message with anHTTPProxylike this:Edit: I was able to add option
timeout: 0sto gloo api gateway virtual service route which stops EventSource calls from erroring/retrying. Hope that helps someone else.Running v3.2.2 - these use to cause the entire UI to go away and need a reload, seeing the modal still pop up, just not as annoying now. Can we maybe get some guidance on configuring auth gateways/proxies like gloo/ambassador/nginx/contour to work with argo workflows use of event-source?
I’m getting this error with the QuickStart guide not sure what I’m missing here.
Is there any solution for those who use GCE ingress controller on GKE?
Any pointers where this was fixed in v3?
@alexec Do you have the link to the fix? Also, when will we see any 3.0 branch?