apisix-ingress-controller: request help: Frequent prompts warn apisix/upstream.go:69 upstream not found

Issue description

Why does the Apisix ingress controller frequently request upstream of Apisix, resulting in a large number of 404 requests on my Apisix;

2023-06-09T21:24:19+08:00       warn    apisix/upstream.go:69   upstream not found      {"name": "default_js-design-server_3002_service", "url": "http://js-apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/upstreams/88c42dc3", "cluster": "default"}
2023-06-09T21:24:19+08:00       warn    apisix/upstream.go:69   upstream not found      {"name": "default_js-design-server_50000_service", "url": "http://js-apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/upstreams/d8c6437f", "cluster": "default"}

Steps to Reproduce

kubectl apply -f svc.yaml
---
apiVersion: v1
kind: Service
metadata:
  name: backend-svc
  namespace: demo
spec:
  type: ClusterIP
  selector:
    component: backend
  ports:
    - name: http
      port: 3002
      targetPort: 3002
    - name: grpc
      port: 50000
      targetPort: 50000
---
apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
  name: js-design-server
  namespace: js-design-test
spec:
  http:
    - name: serverV1
      # websocket: true
      match:
        paths:
        - /serverV1*
      backends:
      - serviceName: js-design-svc-test
        servicePort: 3002
    - name: server/socket
      websocket: true
      match:
        hosts:
        - ds.js.design
        paths:
        - /*
        - /socket.io*
      backends:
      - serviceName: js-design-svc-test
        servicePort: 50000
---
apiVersion:  apisix.apache.org/v2
kind: ApisixUpstream
metadata:
  name: js-design-svc-test
  namespace: js-design-test
spec:
  loadbalancer:
    type: roundrobin
  portLevelSettings:
  - port: 50000
    scheme: grpc
  - port: 3002
    scheme: http

i saw #1655 This issue is because ApisixRoute was not associated with ApisixUpstream, and I correctly associated it. Why are there still many warnings

As shown in the figure, there are many 404s in the log collection system, which is very unfriendly

image

Also, I see if there is any new progress on the issue #864, and I really need this feature as well

Because I have to configure apisixupstream for forwarding on GRPC

@tao12345666333

Environment

  • your apisix-ingress-controller version (output of apisix-ingress-controller version --long): 1.6.1
  • your Kubernetes cluster version (output of kubectl version): 1.20
  • if you run apisix-ingress-controller in Bare-metal environment, also show your OS version (uname -a):

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 1
  • Comments: 19 (3 by maintainers)

Most upvoted comments

I’m also observing the same behavior in my setup.

@tao12345666333 This happens because the apisix ingress controller tries to sync Kubernetes endpoints (services) as upstream. So as long as you have Kubernetes service in the watched namespace - ingress will try to sync it as an upstream.

Is that an expected behaviour? Are there any ways to disable such a behaviour?

Example logs

2024-01-18T15:35:18+08:00	debug	types/types.go:211	sync upstream nodes change	{"cluster": "name=default; base_url=.../apisix/admin", "upstream_name": "...", "nodes": [{"host":"10.161.5.20","port":8080,"weight":100}]}
2024-01-18T15:35:18+08:00	debug	apisix/upstream.go:43	try to look up upstream	{"name": "...", "url": ".../apisix/admin/upstreams", "cluster": "default"}
2024-01-18T15:35:18+08:00	debug	apisix/upstream.go:59	failed to find upstream in cache, will try to lookup from APISIX	{"name": "...", "error": "not found"}
2024-01-18T15:35:18+08:00	debug	apisix/cluster.go:645	get resource in cluster	{"cluster_name": "default", "name": "upstream", "url": ".../apisix/admin/upstreams/ed2f2b93"}
2024-01-18T15:35:18+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "ed2f2b93", "url": ".../apisix/admin/upstreams/ed2f2b93", "cluster": "default"}

Hi, Are there any updates on this ? Can we reopen issue ? I think Apisix should be updated to only watch specific services (not all)

Are there any updates to this? It is really annoying to get logs flooded with warns and errors when you don’t want to have all of the services in the watched namespace to be used as upstream.

The only way I’ve found making it tolerable is to put all apisixresources to their own namespace and use externalNodes to point to service in another namespace but it is not ideal…

I have same problems: having service, upstream, route with same name, but it also log “not found”

And another big problem is “log is increase overtime” 20231003-172455

Search with one service name: also show that it log more frequently overtime So there are maybe a bug somewhere

In fact, some services do not need to be configured as upstream types because they are not an API. The excessive number of alerts has already burdened me, so there is absolutely no need to generate so many alerts.