apisix-ingress-controller: bug: Upstream HealthCheck Issue - Unhealthy Upstream doesn't be excluded temporarily

Current Behavior

There is a problem with the unhealthy external service being delivered as it is without being excluded from routing targets.

Mar-06-2024 17-37-44

Expected Behavior

Two external services (ALB configured in front of each) are configured as upstream nodes and should be temporarily excluded from routing if a 5XX error occurs through health check configuration.

Error Logs

No response

Steps to Reproduce

For reproducing the issue, one service is deployed, and the other one is not deployed (only ALB’s are set up.)

apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
  name: route
spec:
  http:
    - match:
        hosts:
          - service.corp.com
        methods:
          - POST
        paths:
          - /svc/*
      name: route
      plugins:
        - enable: true
          name: traffic-split
        - config:
            regex_uri:
              - ^\/svc\/(.+)$
              - /$1
          enable: true
          name: proxy-rewrite
      upstreams:
        - name: upstream
          weight: 100
apiVersion: apisix.apache.org/v2
kind: ApisixUpstream
metadata:
  name: upstream
  namespace: apisix
spec:
  externalNodes:
    - name: svc01.corp.com
      port: 443
      type: Domain
      weight: 50
    - name: svc02.corp.com
      port: 443
      type: Domain
      weight: 50
  healthCheck:
    active:
      healthy:
        httpCodes:
          - 200
          - 404
        interval: 3s
        successes: 1
      httpPath: /
      type: https
      unhealthy:
        httpCodes:
          - 503
          - 504
        httpFailures: 1
        interval: 3s
        timeouts: 3
    passive:
      healthy:
        httpCodes:
          - 200
          - 404
        successes: 1
      type: https
      unhealthy:
        httpCodes:
          - 503
          - 504
        httpFailures: 1
        timeouts: 3
  loadbalancer:
    type: roundrobin
  passHost: node
  scheme: https

Environment

  • APISIX Ingress controller version (run apisix-ingress-controller version --long)
  • Kubernetes cluster version (run kubectl version)
  • OS version if running APISIX Ingress controller in a bare-metal environment (run uname -a)

Runs on an AWS EKS Cluster (Kubernetes v1.25). Uses APISIX Helm Chart (1.11.0, App 3.8.0).

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 41 (18 by maintainers)

Most upvoted comments

@kworkbee Thanks for reporting. I’ll take a look

@Revolyssup It will take some time to check. I will mention it again as soon as I check. Thank you for your quick response.

The following specific logs are visible in the APISIX Log. Did SSL Handshake Failure Affect?

2024/03/13 15:42:17 [error] 50#50: *33439 [lua] healthcheck.lua:1383: log(): [healthcheck] (upstream#/apisix/upstreams/32eb11c7) failed SSL handshake with 'X.X.X.X (X.X.X.X:443)', using server name (sni) 'svc01.corp.com': 19: self-signed certificate in certificate chain, context: ngx.timer, client: X.X.X.X, server: 0.0.0.0:9080

@Revolyssup As you just mentioned, this is the result of erasing and re-applying both ApisixRoute and ApisixUpstream.

2024-03-13T13:49:24+08:00	warn	apisix/cluster.go:599	failed to check health for cluster default: dial tcp 172.20.44.139:9180: connect: connection refused, will retry
2024-03-13T13:49:29+08:00	warn	apisix/cluster.go:599	failed to check health for cluster default: dial tcp 172.20.44.139:9180: connect: connection refused, will retry
2024-03-13T13:49:34+08:00	warn	apisix/cluster.go:599	failed to check health for cluster default: dial tcp 172.20.44.139:9180: connect: connection refused, will retry
2024-03-13T13:49:34+08:00	warn	providers/controller.go:598	failed to check health for default cluster: timed out waiting for the condition, give up leader
2024-03-13T13:49:34+08:00	warn	apisix/cluster.go:599	failed to check health for cluster default: dial tcp 172.20.44.139:9180: connect: connection refused, will retry
2024-03-13T13:49:39+08:00	warn	apisix/cluster.go:599	failed to check health for cluster default: dial tcp 172.20.44.139:9180: connect: connection refused, will retry
2024-03-13T13:49:44+08:00	warn	apisix/cluster.go:599	failed to check health for cluster default: dial tcp 172.20.44.139:9180: connect: connection refused, will retry
2024-03-13T13:49:44+08:00	warn	providers/controller.go:598	failed to check health for default cluster: timed out waiting for the condition, give up leader
2024-03-13T13:49:44+08:00	warn	apisix/cluster.go:599	failed to check health for cluster default: dial tcp 172.20.44.139:9180: connect: connection refused, will retry
2024-03-13T13:49:44+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "971e4f30", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/971e4f30", "cluster": "default"}
2024-03-13T13:49:44+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "9756d1ef", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/9756d1ef", "cluster": "default"}
2024-03-13T13:50:42+08:00	error	translation/apisix_route.go:238	failed to translate ApisixUpstream at Upstream[0]	{"error": "apisixupstream.apisix.apache.org \"feature-flags-nodes-upstream\" not found", "apisix_upstream": "apisix/feature-flags-nodes-upstream"}
2024-03-13T13:50:42+08:00	error	apisix/route.go:138	failed to create route: unexpected status code 400; error message: {"error_msg":"failed to fetch upstream info by upstream id [32eb11c7], response code: 404"}
2024-03-13T13:50:42+08:00	error	apisix/apisix_route.go:380	failed to sync ApisixRoute to apisix	{"error": "1 error occurred:\n\t* unexpected status code 400; error message: {\"error_msg\":\"failed to fetch upstream info by upstream id [32eb11c7], response code: 404\"}\n\n\n"}
2024-03-13T13:50:42+08:00	warn	apisix/apisix_route.go:482	sync ApisixRoute failed, will retry	{"object": {"Type":1,"Object":{"Key":"apisix/feature-flags-route","OldObject":null,"GroupVersion":"apisix.apache.org/v2"},"OldObject":null,"Tombstone":null}, "error": "1 error occurred:\n\t* unexpected status code 400; error message: {\"error_msg\":\"failed to fetch upstream info by upstream id [32eb11c7], response code: 404\"}\n\n\n"}
W0313 13:50:42.477556       1 warnings.go:70] unknown field "status.conditions[0].lastTransitionTime"
2024-03-13T13:50:43+08:00	error	translation/apisix_route.go:238	failed to translate ApisixUpstream at Upstream[0]	{"error": "apisixupstream.apisix.apache.org \"feature-flags-nodes-upstream\" not found", "apisix_upstream": "apisix/feature-flags-nodes-upstream"}
2024-03-13T13:50:43+08:00	error	apisix/route.go:138	failed to create route: unexpected status code 400; error message: {"error_msg":"failed to fetch upstream info by upstream id [32eb11c7], response code: 404"}
2024-03-13T13:50:43+08:00	error	apisix/apisix_route.go:380	failed to sync ApisixRoute to apisix	{"error": "1 error occurred:\n\t* unexpected status code 400; error message: {\"error_msg\":\"failed to fetch upstream info by upstream id [32eb11c7], response code: 404\"}\n\n\n"}
2024-03-13T13:50:43+08:00	warn	apisix/apisix_route.go:482	sync ApisixRoute failed, will retry	{"object": {"Type":1,"Object":{"Key":"apisix/feature-flags-route","OldObject":null,"GroupVersion":"apisix.apache.org/v2"},"OldObject":null,"Tombstone":null}, "error": "1 error occurred:\n\t* unexpected status code 400; error message: {\"error_msg\":\"failed to fetch upstream info by upstream id [32eb11c7], response code: 404\"}\n\n\n"}
2024-03-13T13:50:44+08:00	error	translation/apisix_route.go:238	failed to translate ApisixUpstream at Upstream[0]	{"error": "apisixupstream.apisix.apache.org \"feature-flags-nodes-upstream\" not found", "apisix_upstream": "apisix/feature-flags-nodes-upstream"}
2024-03-13T13:50:44+08:00	error	apisix/route.go:138	failed to create route: unexpected status code 400; error message: {"error_msg":"failed to fetch upstream info by upstream id [32eb11c7], response code: 404"}
2024-03-13T13:50:44+08:00	error	apisix/apisix_route.go:380	failed to sync ApisixRoute to apisix	{"error": "1 error occurred:\n\t* unexpected status code 400; error message: {\"error_msg\":\"failed to fetch upstream info by upstream id [32eb11c7], response code: 404\"}\n\n\n"}
2024-03-13T13:50:44+08:00	warn	apisix/apisix_route.go:482	sync ApisixRoute failed, will retry	{"object": {"Type":1,"Object":{"Key":"apisix/feature-flags-route","OldObject":null,"GroupVersion":"apisix.apache.org/v2"},"OldObject":null,"Tombstone":null}, "error": "1 error occurred:\n\t* unexpected status code 400; error message: {\"error_msg\":\"failed to fetch upstream info by upstream id [32eb11c7], response code: 404\"}\n\n\n"}
2024-03-13T13:50:45+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "32eb11c7", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/32eb11c7", "cluster": "default"}
2024-03-13T13:50:45+08:00	error	apisix/apisix_upstream.go:376	failed to get upstream apisix_feature-flags-nodes-upstream: not found
2024-03-13T13:50:45+08:00	error	apisix/apisix_upstream.go:845	update status	{"status": {"conditions":[{"type":"ResourcesAvailable","status":"False","observedGeneration":1,"lastTransitionTime":"2024-03-13T05:50:45Z","reason":"ResourceSyncAborted","message":"not found"}]}}
W0313 13:50:45.220898       1 warnings.go:70] unknown field "status.conditions[0].lastTransitionTime"
2024-03-13T13:50:45+08:00	warn	apisix/apisix_upstream.go:489	sync ApisixUpstream failed, will retry	{"object": {"Type":1,"Object":{"Key":"apisix/feature-flags-nodes-upstream","OldObject":null,"GroupVersion":"apisix.apache.org/v2"},"OldObject":null,"Tombstone":null}, "error": "not found"}
W0313 13:50:45.273798       1 warnings.go:70] unknown field "status.conditions[0].lastTransitionTime"
2024-03-13T13:50:46+08:00	error	apisix/apisix_upstream.go:845	update status	{"status": {"conditions":[{"type":"ResourcesAvailable","status":"True","observedGeneration":1,"lastTransitionTime":"2024-03-13T05:50:46Z","reason":"ResourcesSynced","message":"Sync Successfully"}]}}
W0313 13:50:46.274203       1 warnings.go:70] unknown field "status.conditions[0].lastTransitionTime"

@kworkbee What logs do you get immediately after kubectl apply -f <the above provided upstream> ?

Can you delete both ApisixRoute and ApisixUpstream, reapply and show logs?