argo-events: failed to start eventsource server: timed out waiting for the condition

Describe the bug I’m unable to start github-eventsource pod on a Kubernetes with istio enabled.

Environment (please complete the following information):

• Kubernetes: [v1.19.11]
• Argo: [v3.1.11]
• Argo Events: [v1.4.1]

Additional context Originally I was unable to access eventbus-default-stan service, but as suggested in https://github.com/argoproj/argo-events/issues/1311 I updated the service to:

$ k get svc -o yaml eventbus-default-stan-svc 
apiVersion: v1
kind: Service
metadata:
  annotations:
    resource-spec-hash: "744782701"
  creationTimestamp: "2021-09-16T14:37:29Z"
  labels:
    controller: eventbus-controller
    eventbus-name: default
    owner-name: default
    stan: "yes"
  name: eventbus-default-stan-svc
  namespace: argo-events
  ownerReferences:
  - apiVersion: argoproj.io/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: EventBus
    name: default
    uid: 7cd4b511-ebdc-474a-8f42-e93c1dbc0e70
  resourceVersion: "17087296"
  selfLink: /api/v1/namespaces/argo-events/services/eventbus-default-stan-svc
  uid: 525d6589-ea83-42b9-a80c-86aa45d051c9
spec:
  clusterIP: None
  ports:
  - name: tcp-client
    port: 4222
    protocol: TCP
    targetPort: 4222
  - name: cluster
    port: 6222
    protocol: TCP
    targetPort: 6222
  - name: monitor
    port: 8222
    protocol: TCP
    targetPort: 8222
  selector:
    controller: eventbus-controller
    eventbus-name: default
    owner-name: default
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

Now I’m able to access it using curl from testing container:

/ $ curl eventbus-default-stan-svc:4222
curl: (1) Received HTTP/0.9 when not allowed

but github-eventsource is still unable to start:

nicon@5CG10725TV-W10:~/.../prod/airflow$ k logs github-eventsource-8w5n2-65b768f64b-dg7dm main
{"level":"info","ts":1632155363.8100798,"logger":"argo-events.eventsource","caller":"cmd/start.go:63","msg":"starting eventsource server","eventSourceName":"github","version":"v1.4.1"}
{"level":"info","ts":1632155363.8101642,"logger":"argo-events.eventsource","caller":"eventsources/eventing.go:309","msg":"Starting event source server...","eventSourceName":"github"}
{"level":"info","ts":1632155363.8110776,"logger":"argo-events.eventsource","caller":"metrics/metrics.go:172","msg":"starting metrics server","eventSourceName":"github"}
{"level":"info","ts":1632155363.8148913,"logger":"argo-events.eventsource","caller":"driver/nats.go:93","msg":"NATS auth strategy: Token","eventSourceName":"github","clientID":"client-github-eventsource-8w5n2-65b768f64b-dg7dm-481"}   
{"level":"error","ts":1632155365.820021,"logger":"argo-events.eventsource","caller":"driver/nats.go:102","msg":"Failed to connect to NATS server","eventSourceName":"github","clientID":"client-github-eventsource-8w5n2-65b768f64b-dg7dm-481","error":"read tcp 10.99.1.185:33986->10.99.1.191:4222: i/o timeout","stacktrace":"github.com/argoproj/argo-events/eventbus/driver.(*natsStreaming).Connect\n\t/home/runner/work/argo-events/argo-events/eventbus/driver/nats.go:102\ngithub.com/argoproj/argo-events/eventsources.(*EventSourceAdaptor).run.func1\n\t/home/runner/work/argo-events/argo-events/eventsources/eventing.go:317\ngithub.com/argoproj/argo-events/common.Connect.func1\n\t/home/runner/work/argo-events/argo-events/common/retry.go:107\nk8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtection\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.19.7-rc.0/pkg/util/wait/wait.go:211\nk8s.io/apimachinery/pkg/util/wait.ExponentialBackoff\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.19.7-rc.0/pkg/util/wait/wait.go:399\ngithub.com/argoproj/argo-events/common.Connect\n\t/home/runner/work/argo-events/argo-events/common/retry.go:106\ngithub.com/argoproj/argo-events/eventsources.(*EventSourceAdaptor).run\n\t/home/runner/work/argo-events/argo-events/eventsources/eventing.go:316\ngithub.com/argoproj/argo-events/eventsources.(*EventSourceAdaptor).Start\n\t/home/runner/work/argo-events/argo-events/eventsources/eventing.go:284\ngithub.com/argoproj/argo-events/eventsources/cmd.Start\n\t/home/runner/work/argo-events/argo-events/eventsources/cmd/start.go:65\ngithub.com/argoproj/argo-events/cmd/commands.NewEventSourceCommand.func1\n\t/home/runner/work/argo-events/argo-events/cmd/commands/eventsource.go:14\ngithub.com/spf13/cobra.(*Command).execute\n\t/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:846\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:950\ngithub.com/spf13/cobra.(*Command).Execute\n\t/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:887\ngithub.com/argoproj/argo-events/cmd/commands.Execute\n\t/home/runner/work/argo-events/argo-events/cmd/commands/root.go:19\nmain.main\n\t/home/runner/work/argo-events/argo-events/cmd/main.go:8\nruntime.main\n\t/opt/hostedtoolcache/go/1.15.15/x64/src/runtime/proc.go:204"}  

Can you please help me with that?

---

Message from the maintainers:

If you wish to see this enhancement implemented please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.