argo-workflows: Error in UI: `Connection closed to api/v1/workflow-events/argo?...`

Summary

What happened/what you expected to happen?

I exposed Argo Workflow UI using Ambassador under /argo/ subpath. I was able to access the UI. However, I got Unable to load data error. 01 forbidden

After that I got recurring “Connection closed to api/v1/workflow-events/argo?listOptions.resourceVersion=30237…” error. 02 connection-closed Connection closed to api/v1/workflow-events/argo?listOptions.resourceVersion=7273&fields=result.object.metadata.name,result.object.metadata.namespace,result.object.metadata.resourceVersion,result.object.metadata.uid,result.object.status.finishedAt,result.object.status.phase,result.object.status.startedAt,result.object.status.estimatedDuration,result.object.status.progress,result.type,result.object.metadata.labels,result.object.spec.suspend:

Diagnostics

What Kubernetes provider are you using?
1.19.4 What version of Argo Workflows are you running? 2.12.2

Steps to reproduce

  1. Create K3D cluster
k3d cluster create dev-ld --api-port 6443 -p 8080:80@loadbalancer --agents 2 \
    --k3s-server-arg '--flannel-backend=none' --k3s-server-arg '--no-deploy=traefik' \
    --volume "$(CURRENT_DIR)/k3d_custom/calico.yaml:/var/lib/rancher/k3s/server/manifests/calico.yaml"
  1. Install Ambassador Edge Stack
kubectl apply -f https://www.getambassador.io/yaml/aes-crds.yaml && \
kubectl wait --for condition=established --timeout=90s crd -lproduct=aes && \
kubectl apply -f https://www.getambassador.io/yaml/aes.yaml && \
kubectl -n ambassador wait --for condition=available --timeout=90s deploy -lproduct=aes
  1. Install metallb metal-lb-layer2-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses:
      - 172.18.0.3-172.18.0.254
# https://kubernetes.github.io/ingress-nginx/deploy/baremetal/
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.5/manifests/namespace.yaml
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.5/manifests/metallb.yaml
# On first install only
kubectl create secret generic -n metallb-system memberlist --from-literal=secretkey="$(openssl rand -base64 128)"
kubectl apply -f $(CURRENT_DIR)/k3d_custom/metal-lb-layer2-config.yaml
  1. Configure tuntap - Mac only
# Install tuntap:
brew install Caskroom/cask/tuntap
# https://blog.kubernauts.io/k3s-with-k3d-and-metallb-on-mac-923a3255c36e
# only needed for Mac
# create a bridge interface between the Physical Machine and the Host Virtual Machine.
# download scripts form https://github.com/arashkaffamanesh/k3d-k3s-metallb
k3d cluster stop dev-ld
./k3d_custom/docker-tuntap-osx/sbin/docker_tap_install.sh
echo "Wait till docker is restarted."
sleep 40
echo "..done"
./k3d_custom/docker-tuntap-osx/sbin/docker_tap_up.sh
sudo route -v add -net 172.18.0.0 -netmask 255.255.255.0 10.0.75.2
k3d cluster start dev-ld
  1. Install argo
kubectl create namespace argo || true
kubectl -n argo apply -f https://raw.githubusercontent.com/argoproj/argo/master/manifests/quick-start-postgres.yaml
# Since K3S uses containerd, you need to configure the workflow controller to use the PNS (Pod Namespace Sharing) executor
kubectl -n argo patch cm workflow-controller-configmap -p '{"data": {"containerRuntimeExecutor": "pns"}}';
kubectl -n argo wait pod -lapp=argo-server --for condition=available --timeout=180s
  1. Patch argo argocd-patch.yaml
spec:
  template:
    spec:
      containers:
      - args:
        - server
        - --namespaced
        - --auth-mode
        - server
        - --auth-mode
        - client
        - --basehref
        - /argo/
        image: argoproj/argocli:latest
        name: argo-server
kubectl -n argo patch deployment/argo-server -p "$(cat argocd-patch.yaml)"
  1. Create Ambassador mapping argo-ingress-ambassador.yaml
apiVersion: getambassador.io/v2
kind: Mapping
metadata:
  name: argo-server
  namespace: argo
spec:
  prefix: /argo/
  regex_rewrite:
    pattern: '/argo(/|$)(.*)'
    substitution: '/\2'
  service: argo-server:2746
kubectl -n argo apply -f resources/argo-ingress-ambassador.yaml
  1. argo-server log
❯ kubectl -n argo logs argo-server-db794976-gchd2                                           
time="2020-12-27T05:06:17.892Z" level=info authModes="[server client]" baseHRef=/argo/ managedNamespace=argo namespace=argo secure=false
time="2020-12-27T05:06:17.893Z" level=warning msg="You are running in insecure mode. Learn how to enable transport layer security: https://argoproj.github.io/argo/tls/"
time="2020-12-27T05:06:17.893Z" level=info msg="config map" name=workflow-controller-configmap
time="2020-12-27T05:06:17.893Z" level=info msg="SSO disabled"
time="2020-12-27T05:06:18.061Z" level=info msg="Starting Argo Server" instanceID= version=764f118ca4180b476a9b053873e55eeccbc5e202
time="2020-12-27T05:06:18.062Z" level=info msg="Creating DB session"
time="2020-12-27T05:06:18.121Z" level=info msg="Node status offloading config" ttl=5m0s
time="2020-12-27T05:06:18.122Z" level=info msg="Creating event controller" operationQueueSize=16 workerCount=4
time="2020-12-27T05:06:18.125Z" level=info msg="Argo Server started successfully on http://localhost:2746"
time="2020-12-27T05:10:23.551Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=GetVersion grpc.service=info.InfoService grpc.start_time="2020-12-27T05:10:23Z" grpc.time_ms=37.122 span.kind=server system=grpc
time="2020-12-27T05:10:23.661Z" level=warning msg="finished unary call with code PermissionDenied" error="rpc error: code = PermissionDenied desc = workflowtemplates.argoproj.io is forbidden: User \"system:serviceaccount:argo:argo-server\" cannot list resource \"workflowtemplates\" in API group \"argoproj.io\" at the cluster scope" grpc.code=PermissionDenied grpc.method=ListWorkflowTemplates grpc.service=workflowtemplate.WorkflowTemplateService grpc.start_time="2020-12-27T05:10:23Z" grpc.time_ms=81.194 span.kind=server system=grpc
time="2020-12-27T05:10:23.665Z" level=warning msg="finished unary call with code PermissionDenied" error="rpc error: code = PermissionDenied desc = cronworkflows.argoproj.io is forbidden: User \"system:serviceaccount:argo:argo-server\" cannot list resource \"cronworkflows\" in API group \"argoproj.io\" at the cluster scope" grpc.code=PermissionDenied grpc.method=ListCronWorkflows grpc.service=cronworkflow.CronWorkflowService grpc.start_time="2020-12-27T05:10:23Z" grpc.time_ms=81.08 span.kind=server system=grpc
time="2020-12-27T05:10:23.757Z" level=warning msg="finished unary call with code PermissionDenied" error="rpc error: code = PermissionDenied desc = workflows.argoproj.io is forbidden: User \"system:serviceaccount:argo:argo-server\" cannot list resource \"workflows\" in API group \"argoproj.io\" at the cluster scope" grpc.code=PermissionDenied grpc.method=ListWorkflows grpc.service=workflow.WorkflowService grpc.start_time="2020-12-27T05:10:23Z" grpc.time_ms=5.809 span.kind=server system=grpc
time="2020-12-27T05:10:23.917Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=GetInfo grpc.service=info.InfoService grpc.start_time="2020-12-27T05:10:23Z" grpc.time_ms=0.886 span.kind=server system=grpc
time="2020-12-27T05:10:24.080Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=ListWorkflowTemplates grpc.service=workflowtemplate.WorkflowTemplateService grpc.start_time="2020-12-27T05:10:23Z" grpc.time_ms=120.036 span.kind=server system=grpc
time="2020-12-27T05:10:24.090Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=ListCronWorkflows grpc.service=cronworkflow.CronWorkflowService grpc.start_time="2020-12-27T05:10:23Z" grpc.time_ms=124.877 span.kind=server system=grpc
time="2020-12-27T05:10:24.150Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=ListWorkflows grpc.service=workflow.WorkflowService grpc.start_time="2020-12-27T05:10:24Z" grpc.time_ms=132.258 span.kind=server system=grpc
time="2020-12-27T05:10:27.242Z" level=info msg="finished streaming call with code OK" grpc.code=OK grpc.method=WatchWorkflows grpc.service=workflow.WorkflowService grpc.start_time="2020-12-27T05:10:24Z" grpc.time_ms=3000.936 span.kind=server system=grpc
time="2020-12-27T05:10:37.329Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=ListWorkflows grpc.service=workflow.WorkflowService grpc.start_time="2020-12-27T05:10:37Z" grpc.time_ms=15.356 span.kind=server system=grpc
time="2020-12-27T05:10:40.387Z" level=info msg="finished streaming call with code OK" grpc.code=OK grpc.method=WatchWorkflows grpc.service=workflow.WorkflowService grpc.start_time="2020-12-27T05:10:37Z" grpc.time_ms=2999.413 span.kind=server system=grpc
  1. workflow-controller log
❯ kubectl logs -n argo $(kubectl get pods -l app=workflow-controller -n argo -o name) | more            
time="2020-12-27T05:08:33.189Z" level=info msg="config map" name=workflow-controller-configmap
time="2020-12-27T05:08:33.233Z" level=info msg="Configuration:\nartifactRepository:\n  archiveLogs: true\n  s3:\n    accessKeySecret:\n      key: accesskey\n      name: my-minio-cred\n    bucket: my-bucket\n    endpoint: minio:9000\n    insecure: true\n    secretKeySecret:\n      key: secretkey\n      name: my-minio-cred\ncontainerRuntimeExecutor: pns\ninitialDelay: 0s\nlinks:\n- name: Example Workflow Link\n  scope: workflow\n  url: http://logging-facility?namespace=${metadata.namespace}&workflowName=${metadata.name}&startedAt=${status.startedAt}&finishedAt=${status.finishedAt}\n- name: Example Pod Link\n  scope: pod\n  url: http://logging-facility?namespace=${metadata.namespace}&podName=${metadata.name}&startedAt=${status.startedAt}&finishedAt=${status.finishedAt}\nmetricsConfig:\n  disableLegacy: true\n  enabled: true\n  path: /metrics\n  port: 9090\nnodeEvents: {}\npersistence:\n  archive: true\n  archiveTTL: 168h0m0s\n  connectionPool:\n    maxIdleConns: 100\n  nodeStatusOffLoad: true\n  postgresql:\n    database: postgres\n    host: postgres\n    passwordSecret:\n      key: password\n      name: argo-postgres-config\n    port: 5432\n    tableName: argo_workflows\n    userNameSecret:\n      key: username\n      name: argo-postgres-config\npodSpecLogStrategy: {}\ntelemetryConfig: {}\n"
:...skipping...
time="2020-12-27T05:08:33.189Z" level=info msg="config map" name=workflow-controller-configmap
time="2020-12-27T05:08:33.233Z" level=info msg="Configuration:\nartifactRepository:\n  archiveLogs: true\n  s3:\n    accessKeySecret:\n      key: accesskey\n      name: my-minio-cred\n    bucket: my-bucket\n    endpoint: minio:9000\n    insecure: true\n    secretKeySecret:\n      key: secretkey\n      name: my-minio-cred\ncontainerRuntimeExecutor: pns\ninitialDelay: 0s\nlinks:\n- name: Example Workflow Link\n  scope: workflow\n  url: http://logging-facility?namespace=${metadata.namespace}&workflowName=${metadata.name}&startedAt=${status.startedAt}&finishedAt=${status.finishedAt}\n- name: Example Pod Link\n  scope: pod\n  url: http://logging-facility?namespace=${metadata.namespace}&podName=${metadata.name}&startedAt=${status.startedAt}&finishedAt=${status.finishedAt}\nmetricsConfig:\n  disableLegacy: true\n  enabled: true\n  path: /metrics\n  port: 9090\nnodeEvents: {}\npersistence:\n  archive: true\n  archiveTTL: 168h0m0s\n  connectionPool:\n    maxIdleConns: 100\n  nodeStatusOffLoad: true\n  postgresql:\n    database: postgres\n    host: postgres\n    passwordSecret:\n      key: password\n      name: argo-postgres-config\n    port: 5432\n    tableName: argo_workflows\n    userNameSecret:\n      key: username\n      name: argo-postgres-config\npodSpecLogStrategy: {}\ntelemetryConfig: {}\n"
time="2020-12-27T05:08:33.233Z" level=info msg="Persistence configuration enabled"
time="2020-12-27T05:08:33.234Z" level=info msg="Creating DB session"
time="2020-12-27T05:08:33.257Z" level=info msg="Persistence Session created successfully"
time="2020-12-27T05:08:33.265Z" level=info msg="Migrating database schema" clusterName=default dbType=postgres
time="2020-12-27T05:08:33.275Z" level=info msg="applying database change" change="create table if not exists argo_workflows (\n    id varchar(128) ,\n    name varchar(256),\n    phase varchar(25),\n    namespace varchar(256),\n    workflow text,\n    startedat timestamp default CURRENT_TIMESTAMP,\n    finishedat timestamp default CURRENT_TIMESTAMP,\n    primary key (id, namespace)\n)" changeSchemaVersion=0
time="2020-12-27T05:08:33.286Z" level=info msg="applying database change" change="create unique index idx_name on argo_workflows (name)" changeSchemaVersion=1
time="2020-12-27T05:08:33.295Z" level=info msg="applying database change" change="create table if not exists argo_workflow_history (\n    id varchar(128) ,\n    name varchar(256),\n    phase varchar(25),\n    namespace varchar(256),\n    workflow text,\n    startedat timestamp default CURRENT_TIMESTAMP,\n    finishedat timestamp default CURRENT_TIMESTAMP,\n    primary key (id, namespace)\n)" changeSchemaVersion=2
time="2020-12-27T05:08:33.304Z" level=info msg="applying database change" change="alter table argo_workflow_history rename to argo_archived_workflows" changeSchemaVersion=3
time="2020-12-27T05:08:33.310Z" level=info msg="applying database change" change="drop index idx_name" changeSchemaVersion=4
time="2020-12-27T05:08:33.315Z" level=info msg="applying database change" change="create unique index idx_name on argo_workflows(name, namespace)" changeSchemaVersion=5
time="2020-12-27T05:08:33.321Z" level=info msg="applying database change" change="alter table argo_workflows drop constraint argo_workflows_pkey" changeSchemaVersion=6
time="2020-12-27T05:08:33.326Z" level=info msg="applying database change" change="alter table argo_workflows add primary key(name,namespace)" changeSchemaVersion=7
time="2020-12-27T05:08:33.331Z" level=info msg="applying database change" change="alter table argo_archived_workflows drop constraint argo_workflow_history_pkey" changeSchemaVersion=8
time="2020-12-27T05:08:33.335Z" level=info msg="applying database change" change="alter table argo_archived_workflows add primary key(id)" changeSchemaVersion=9
time="2020-12-27T05:08:33.342Z" level=info msg="applying database change" change="alter table argo_archived_workflows rename column id to uid" changeSchemaVersion=10
time="2020-12-27T05:08:33.345Z" level=info msg="applying database change" change="alter table argo_archived_workflows alter column uid set not null" changeSchemaVersion=11
time="2020-12-27T05:08:33.351Z" level=info msg="applying database change" change="alter table argo_archived_workflows alter column phase set not null" changeSchemaVersion=12
time="2020-12-27T05:08:33.356Z" level=info msg="applying database change" change="alter table argo_archived_workflows alter column namespace set not null" changeSchemaVersion=13
time="2020-12-27T05:08:33.360Z" level=info msg="applying database change" change="alter table argo_archived_workflows alter column workflow set not null" changeSchemaVersion=14
time="2020-12-27T05:08:33.365Z" level=info msg="applying database change" change="alter table argo_archived_workflows alter column startedat set not null" changeSchemaVersion=15
time="2020-12-27T05:08:33.370Z" level=info msg="applying database change" change="alter table argo_archived_workflows alter column finishedat set not null" changeSchemaVersion=16
time="2020-12-27T05:08:33.374Z" level=info msg="applying database change" change="alter table argo_archived_workflows add clustername varchar(64)" changeSchemaVersion=17
time="2020-12-27T05:08:33.378Z" level=info msg="applying database change" change="update argo_archived_workflows set clustername = 'default' where clustername is null" changeSchemaVersion=18
time="2020-12-27T05:08:33.381Z" level=info msg="applying database change" change="alter table argo_archived_workflows alter column clustername set not null" changeSchemaVersion=19
time="2020-12-27T05:08:33.384Z" level=info msg="applying database change" change="alter table argo_archived_workflows drop constraint argo_archived_workflows_pkey" changeSchemaVersion=20
time="2020-12-27T05:08:33.390Z" level=info msg="applying database change" change="alter table argo_archived_workflows add primary key(clustername,uid)" changeSchemaVersion=21
time="2020-12-27T05:08:33.395Z" level=info msg="applying database change" change="create index argo_archived_workflows_i1 on argo_archived_workflows (clustername,namespace)" changeSchemaVersion=22
time="2020-12-27T05:08:33.405Z" level=info msg="applying database change" change="alter table argo_workflows drop column phase" changeSchemaVersion=23
time="2020-12-27T05:08:33.409Z" level=info msg="applying database change" change="alter table argo_workflows drop column startedat" changeSchemaVersion=24
time="2020-12-27T05:08:33.412Z" level=info msg="applying database change" change="alter table argo_workflows drop column finishedat" changeSchemaVersion=25
time="2020-12-27T05:08:33.415Z" level=info msg="applying database change" change="alter table argo_workflows rename column id to uid" changeSchemaVersion=26
time="2020-12-27T05:08:33.419Z" level=info msg="applying database change" change="alter table argo_workflows alter column uid set not null" changeSchemaVersion=27
time="2020-12-27T05:08:33.422Z" level=info msg="applying database change" change="alter table argo_workflows alter column namespace set not null" changeSchemaVersion=28
time="2020-12-27T05:08:33.425Z" level=info msg="applying database change" change="alter table argo_workflows add column clustername varchar(64)" changeSchemaVersion=29
time="2020-12-27T05:08:33.429Z" level=info msg="applying database change" change="update argo_workflows set clustername = 'default' where clustername is null" changeSchemaVersion=30
time="2020-12-27T05:08:33.432Z" level=info msg="applying database change" change="alter table argo_workflows alter column clustername set not null" changeSchemaVersion=31
time="2020-12-27T05:08:33.436Z" level=info msg="applying database change" change="alter table argo_workflows add column version varchar(64)" changeSchemaVersion=32
time="2020-12-27T05:08:33.440Z" level=info msg="applying database change" change="alter table argo_workflows add column nodes text" changeSchemaVersion=33
time="2020-12-27T05:08:33.443Z" level=info msg="applying database change" change="backfillNodes{argo_workflows}" changeSchemaVersion=34
time="2020-12-27T05:08:33.443Z" level=info msg="Backfill node status"
time="2020-12-27T05:08:33.447Z" level=info msg="applying database change" change="alter table argo_workflows alter column nodes set not null" changeSchemaVersion=35
time="2020-12-27T05:08:33.450Z" level=info msg="applying database change" change="alter table argo_workflows drop column workflow" changeSchemaVersion=36
time="2020-12-27T05:08:33.454Z" level=info msg="applying database change" change="alter table argo_workflows add column updatedat timestamp not null default current_timestamp" changeSchemaVersion=37
time="2020-12-27T05:08:33.457Z" level=info msg="applying database change" change="alter table argo_workflows drop constraint argo_workflows_pkey" changeSchemaVersion=38
time="2020-12-27T05:08:33.462Z" level=info msg="applying database change" change="drop index idx_name" changeSchemaVersion=39
time="2020-12-27T05:08:33.466Z" level=info msg="applying database change" change="alter table argo_workflows drop column name" changeSchemaVersion=40
time="2020-12-27T05:08:33.470Z" level=info msg="applying database change" change="alter table argo_workflows add primary key(clustername,uid,version)" changeSchemaVersion=41
time="2020-12-27T05:08:33.475Z" level=info msg="applying database change" change="create index argo_workflows_i1 on argo_workflows (clustername,namespace)" changeSchemaVersion=42
time="2020-12-27T05:08:33.482Z" level=info msg="applying database change" change="alter table argo_archived_workflows alter column workflow type json using workflow::json" changeSchemaVersion=43
time="2020-12-27T05:08:33.496Z" level=info msg="applying database change" change="alter table argo_archived_workflows alter column name set not null" changeSchemaVersion=44
time="2020-12-27T05:08:33.506Z" level=info msg="applying database change" change="create index argo_workflows_i2 on argo_workflows (clustername,namespace,updatedat)" changeSchemaVersion=45
time="2020-12-27T05:08:33.512Z" level=info msg="applying database change" change="create table if not exists argo_archived_workflows_labels (\n\tclustername varchar(64) not null,\n\tuid varchar(128) not null,\n    name varchar(317) not null,\n    value varchar(63) not null,\n    primary key (clustername, uid, name),\n \tforeign key (clustername, uid) references argo_archived_workflows(clustername, uid) on delete cascade\n)" changeSchemaVersion=46
time="2020-12-27T05:08:33.523Z" level=info msg="applying database change" change="alter table argo_workflows alter column nodes type json using nodes::json" changeSchemaVersion=47
time="2020-12-27T05:08:33.536Z" level=info msg="applying database change" change="alter table argo_archived_workflows add column instanceid varchar(64)" changeSchemaVersion=48
time="2020-12-27T05:08:33.540Z" level=info msg="applying database change" change="update argo_archived_workflows set instanceid = '' where instanceid is null" changeSchemaVersion=49
time="2020-12-27T05:08:33.545Z" level=info msg="applying database change" change="alter table argo_archived_workflows alter column instanceid set not null" changeSchemaVersion=50
time="2020-12-27T05:08:33.550Z" level=info msg="applying database change" change="drop index argo_archived_workflows_i1" changeSchemaVersion=51
time="2020-12-27T05:08:33.555Z" level=info msg="applying database change" change="create index argo_archived_workflows_i1 on argo_archived_workflows (clustername,instanceid,namespace)" changeSchemaVersion=52
time="2020-12-27T05:08:33.562Z" level=info msg="applying database change" change="drop index argo_workflows_i1" changeSchemaVersion=53
time="2020-12-27T05:08:33.566Z" level=info msg="applying database change" change="drop index argo_workflows_i2" changeSchemaVersion=54
time="2020-12-27T05:08:33.571Z" level=info msg="applying database change" change="create index argo_workflows_i1 on argo_workflows (clustername,namespace,updatedat)" changeSchemaVersion=55
time="2020-12-27T05:08:33.577Z" level=info msg="applying database change" change="create index argo_archived_workflows_i2 on argo_archived_workflows (clustername,instanceid,finishedat)" changeSchemaVersion=56
time="2020-12-27T05:08:33.582Z" level=info msg="Node status offloading config" ttl=5m0s
time="2020-12-27T05:08:33.582Z" level=info msg="Node status offloading is enabled"
time="2020-12-27T05:08:33.583Z" level=info msg="Workflow archiving is enabled"
time="2020-12-27T05:08:33.585Z" level=info msg="Starting Workflow Controller" version=764f118ca4180b476a9b053873e55eeccbc5e202
time="2020-12-27T05:08:33.585Z" level=info msg="Workers: workflow: 32, pod: 32, pod cleanup: 4"
time="2020-12-27T05:08:33.771Z" level=info msg="Manager initialized successfully"
I1227 05:08:33.774426       1 leaderelection.go:242] attempting to acquire leader lease  argo/workflow-controller...
I1227 05:08:33.792510       1 leaderelection.go:252] successfully acquired lease argo/workflow-controller
time="2020-12-27T05:08:33.792Z" level=info msg="new leader" id=workflow-controller-75cb95b89d-6l49r leader=workflow-controller-75cb95b89d-6l49r
time="2020-12-27T05:08:33.793Z" level=info msg="started leading" id=workflow-controller-75cb95b89d-6l49r
time="2020-12-27T05:08:33.793Z" level=info msg="Performing periodic GC every 5m0s"
time="2020-12-27T05:08:33.793Z" level=info msg="Performing archived workflow GC" periodicity=24h0m0s ttl=604800000000000
time="2020-12-27T05:08:33.799Z" level=info msg="Starting workflow TTL controller (resync 20m0s, workflowTTLWorkers 4)"
time="2020-12-27T05:08:33.799Z" level=info msg="Started workflow TTL worker"
time="2020-12-27T05:08:33.801Z" level=info msg="Starting prometheus metrics server at localhost:9090/metrics"
time="2020-12-27T05:08:33.805Z" level=info msg="Starting CronWorkflow controller"
time="2020-12-27T05:13:32.973Z" level=info msg="Alloc=5797 TotalAlloc=22882 Sys=70080 NumGC=8 Goroutines=201"
time="2020-12-27T05:13:33.581Z" level=info msg="Performing periodic workflow GC"
time="2020-12-27T05:13:33.593Z" level=info msg="Zero old offloads, nothing to do"

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Reactions: 34
  • Comments: 45 (13 by maintainers)

Commits related to this issue

Most upvoted comments

for GKE to honor this timeout, you have to annotate your service to indicate the backendconfig to use. See the Docs

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: argo-server
  annotations:
    networking.gke.io/managed-certificates: argo
spec:
  defaultBackend:
    service:
      name: argo-server
      port:
        number: 2746
---
kind: BackendConfig
metadata:
  name: config-argo-server
spec:
  timeoutSec: 86400
  connectionDraining:
    drainingTimeoutSec: 60
---
apiVersion: v1
kind: Service
metadata:
  name: argo-server
  annotations:
    cloud.google.com/backend-config: '{"default": "config-argo-server"}' # this is the annotation needed
spec:
  selector:
    app: argo-server
  ports:
  - name: web
    port: 2746
    targetPort: 2746

This was difficult for me to find so I hope it helps someone!

@alexec you are correct that this happens in scenarios where there is some sort of middleware (load balancer, etc.) present. However, this is a lot of deployments in production I would assume. I did some research and the underlying cause is in the way Argo deals with streaming events.

Here you open an event source stream but will only send back something if the K8S watch returns anything. Which could be a long time - or never. This will cause timeouts to trigger as no data is exchanged on that connection. This leaves the only workaround/hack right now to increase such timeouts in all your middleware - that is very bad though, since it also makes scalability and being open to attack more of a problem as you end up holding much more state. Also, it never really resolves the problem - it just pushes the problem out far enough for most users to not notice.

Websockets, etc. deal with this on the protocol level. But since you are using EventSource it is on you to ensure such “keep alives” are performed on the application data channel. The event stream format allows you to quite easily send a comment to achieve just that (see: https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events#event_stream_format).

Can we reopen this issue? I am facing the same issue on EKS using AWS load balancer controller @alexec

I was missing watch on workflows/workflowtemplates on my side

Ran into this issue. Turned out to be that the service account created for logging into Workflows didn’t have access to get and list pods + pod logs.

I am still getting the issue when using Emissary and Argo Workflows version 3.4.0

@luckyswede in case you are still on it, got it working by adding the configuration mentioned in the previous comment in my reverse proxy!

This is not fixed in v3.0, but it appears the problem only occurs when a proxy is in front of the server.

Typically proxies (e.g. Nginx/Apache) disconnect the event-stream and need configuring with longer keep-alive timeouts.

https://stackoverflow.com/questions/13672743/eventsource-server-sent-events-through-nginx

I believe this is fixed in v3.0. Closing.

I have the same issue in v3.3.8 with Ambassador 3.0.0. If it can help, I’ve temporarily “patched” it by increasing the timeout too

---
apiVersion: getambassador.io/v3alpha1
kind: Mapping
metadata:
  name: argo-workflows-mapping
  labels:
    emissaryHost: emissary-host
spec:
  hostname: "argo-workflows.{{ .Values.hostname }}"
  prefix: /
  service: argo-workflows-server.argo:2746
  # argo-workflows uses eventsource https://developer.mozilla.org/en-US/docs/Web/API/EventSource
  # see https://www.getambassador.io/docs/emissary/latest/topics/using/timeouts
  timeout_ms: 300000 # 5 minutes, default is 3000 (3 seconds)
  connect_timeout_ms: 300000 # 5 minutes, cannot be disabled

I’ve opened a feature request 🤞🏻 https://github.com/emissary-ingress/emissary/issues/4323

I encountered this issue with Argo 3.3.1 while using Contour with its support for the Gateway API. I was unable to figure out how to make it work while using the HTTPRoute resource from the Gateway API specification, because it makes configuring timeouts implementation specific, and I don’t think Contour has implemented anything yet. However, it turns out that Contour lets you mix and match HTTPRoutes from the Gateway API spec and their own HTTPProxy resources, and I was able to get rid of this message with an HTTPProxy like this:

---
apiVersion: projectcontour.io/v1
kind: HTTPProxy
metadata:
  name: argo-server
  namespace: argo
spec:
  routes:
  - services:
    - name: argo-server
      port: 2746
    timeoutPolicy:
      response: 300s
      idle: 300s

Edit: I was able to add option timeout: 0s to gloo api gateway virtual service route which stops EventSource calls from erroring/retrying. Hope that helps someone else.

Running v3.2.2 - these use to cause the entire UI to go away and need a reload, seeing the modal still pop up, just not as annoying now. Can we maybe get some guidance on configuring auth gateways/proxies like gloo/ambassador/nginx/contour to work with argo workflows use of event-source?

I’m getting this error with the QuickStart guide not sure what I’m missing here.

Is there any solution for those who use GCE ingress controller on GKE?

Any pointers where this was fixed in v3?

@alexec Do you have the link to the fix? Also, when will we see any 3.0 branch?