apisix-ingress-controller: request help: Frequent prompts warn apisix/upstream.go:69 upstream not found
Issue description
Why does the Apisix ingress controller frequently request upstream of Apisix, resulting in a large number of 404 requests on my Apisix;
2023-06-09T21:24:19+08:00 warn apisix/upstream.go:69 upstream not found {"name": "default_js-design-server_3002_service", "url": "http://js-apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/upstreams/88c42dc3", "cluster": "default"}
2023-06-09T21:24:19+08:00 warn apisix/upstream.go:69 upstream not found {"name": "default_js-design-server_50000_service", "url": "http://js-apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/upstreams/d8c6437f", "cluster": "default"}
Steps to Reproduce
kubectl apply -f svc.yaml
---
apiVersion: v1
kind: Service
metadata:
name: backend-svc
namespace: demo
spec:
type: ClusterIP
selector:
component: backend
ports:
- name: http
port: 3002
targetPort: 3002
- name: grpc
port: 50000
targetPort: 50000
---
apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
name: js-design-server
namespace: js-design-test
spec:
http:
- name: serverV1
# websocket: true
match:
paths:
- /serverV1*
backends:
- serviceName: js-design-svc-test
servicePort: 3002
- name: server/socket
websocket: true
match:
hosts:
- ds.js.design
paths:
- /*
- /socket.io*
backends:
- serviceName: js-design-svc-test
servicePort: 50000
---
apiVersion: apisix.apache.org/v2
kind: ApisixUpstream
metadata:
name: js-design-svc-test
namespace: js-design-test
spec:
loadbalancer:
type: roundrobin
portLevelSettings:
- port: 50000
scheme: grpc
- port: 3002
scheme: http
i saw #1655 This issue is because ApisixRoute was not associated with ApisixUpstream, and I correctly associated it. Why are there still many warnings
As shown in the figure, there are many 404s in the log collection system, which is very unfriendly
Also, I see if there is any new progress on the issue #864, and I really need this feature as well
Because I have to configure apisixupstream for forwarding on GRPC
Environment
- your apisix-ingress-controller version (output of apisix-ingress-controller version --long): 1.6.1
- your Kubernetes cluster version (output of kubectl version): 1.20
- if you run apisix-ingress-controller in Bare-metal environment, also show your OS version (uname -a):
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 1
- Comments: 19 (3 by maintainers)
I’m also observing the same behavior in my setup.
@tao12345666333 This happens because the apisix ingress controller tries to sync Kubernetes endpoints (services) as upstream. So as long as you have Kubernetes service in the watched namespace - ingress will try to sync it as an upstream.
Is that an expected behaviour? Are there any ways to disable such a behaviour?
Example logs
Hi, Are there any updates on this ? Can we reopen issue ? I think Apisix should be updated to only watch specific
services(not all)Are there any updates to this? It is really annoying to get logs flooded with warns and errors when you don’t want to have all of the services in the watched namespace to be used as upstream.
The only way I’ve found making it tolerable is to put all apisixresources to their own namespace and use externalNodes to point to service in another namespace but it is not ideal…
I have same problems: having service, upstream, route with same name, but it also log “not found”
And another big problem is “log is increase overtime”
Search with one service name: also show that it log more frequently overtime So there are maybe a bug somewhere
In fact, some services do not need to be configured as upstream types because they are not an API. The excessive number of alerts has already burdened me, so there is absolutely no need to generate so many alerts.