apisix-ingress-controller: bug: ApisixIngress synced failed, with error: not found
Current Behavior
When use ApisixUpstream with apisix ingress controller 1.7
It show this status:
apiVersion: apisix.apache.org/v2
kind: ApisixUpstream
...
status:
conditions:
- message: not found
observedGeneration: 1
reason: ResourceSyncAborted
status: "False"
type: ResourcesAvailable
event log:
Normal ResourcesSynced 18m (x2 over 18m) ApisixIngress ApisixIngress synced successfully
Normal ResourcesSynced 15m ApisixIngress ApisixIngress synced successfully
Warning ResourceSyncAborted 11m (x7 over 14m) ApisixIngress ApisixIngress synced failed, with error: not found
Warning ResourceSyncAborted 16s (x13 over 10m) ApisixIngress ApisixIngress synced failed, with error: not found
But when I change image to 1.6.1, it was successful
apiVersion: apisix.apache.org/v2
kind: ApisixUpstream
...
status:
conditions:
- message: Sync Successfully
observedGeneration: 1
reason: ResourcesSynced
status: "True"
type: ResourcesAvailable
If I change back to 1.7.0, it show not found again
Expected Behavior
It should be successful like before version
Error Logs
2023-10-06T11:52:36+08:00 debug apisix/upstream.go:43 try to look up upstream {"name": "<NAMESPACE>_<SERVICENAME>_8088", "url": "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/upstreams", "cluster": "default"}
2023-10-06T11:52:36+08:00 debug apisix/apisix_upstream.go:351 updating upstream since ApisixUpstream changed {"upstream": {"id":"ad53b4e6","name":"<NAMESPACE>_<SERVICENAME>_8088","desc":"Created by apisix-ingress-controller, DO NOT modify it manually","labels":{"managed-by":"apisix-ingress-controller"},"type":"roundrobin","nodes":[{"host":"10.205.17.189","port":8088,"weight":100}],"scheme":"http"}, "ApisixUpstream name": "<NAMESPACE>_<SERVICENAME>_8088"}
2023-10-06T11:52:36+08:00 debug apisix/upstream.go:189 try to update upstream {"id": "ad53b4e6", "name": "<NAMESPACE>_<SERVICENAME>_8088", "cluster": "default", "url": "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/upstreams"}
2023-10-06T11:52:36+08:00 debug apisix/upstreamservicerelation.go:91 try to create upstreamService in cache {"cluster": "default"}
2023-10-06T11:52:36+08:00 debug apisix/upstreamservicerelation.go:47 try to get upstreamService in cache {"service_name": "<NAMESPACE>_<SERVICENAME>", "cluster": "default"}
2023-10-06T11:52:36+08:00 debug apisix/cluster.go:798 updating resource in cluster {"cluster_name": "default", "name": "upstream", "url": "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/upstreams/ad53b4e6", "body": "{\"id\":\"ad53b4e6\",\"name\":\"<NAMESPACE>_<SERVICENAME>_8088\",\"desc\":\"Created by apisix-ingress-controller, DO NOT modify it manually\",\"labels\":{\"managed-by\":\"apisix-ingress-controller\"},\"type\":\"roundrobin\",\"nodes\":[{\"host\":\"10.205.17.189\",\"port\":8088,\"weight\":100}],\"scheme\":\"http\"}"}
2023-10-06T11:52:36+08:00 debug apisix/resource.go:135 got upstream: {"desc":"Created by apisix-ingress-controller, DO NOT modify it manually","labels":{"managed-by":"apisix-ingress-controller"},"create_time":1696389535,"scheme":"http","pass_host":"pass","type":"roundrobin","id":"ad53b4e6","name":"<NAMESPACE>_<SERVICENAME>_8088","update_time":1696564356,"nodes":[{"host":"10.205.17.189","weight":100,"priority":0,"port":8088}],"hash_on":"vars"}
2023-10-06T11:52:36+08:00 debug apisix/upstream.go:43 try to look up upstream {"name": "<NAMESPACE>_<SERVICENAME>_8088_service", "url": "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/upstreams", "cluster": "default"}
2023-10-06T11:52:36+08:00 debug apisix/upstream.go:59 failed to find upstream in cache, will try to lookup from APISIX {"name": "<NAMESPACE>_<SERVICENAME>_8088_service", "error": "not found"}
2023-10-06T11:52:36+08:00 debug apisix/cluster.go:645 get resource in cluster {"cluster_name": "default", "name": "upstream", "url": "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/upstreams/6fd1d84"}
2023-10-06T11:52:36+08:00 warn apisix/cluster.go:1164 upstream not found {"id": "6fd1d84", "url": "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/upstreams/6fd1d84", "cluster": "default"}
2023-10-06T11:52:36+08:00 error apisix/apisix_upstream.go:332 failed to get upstream <NAMESPACE>_<SERVICENAME>_8088_service: not found
2023-10-06T11:52:36+08:00 warn apisix/apisix_upstream.go:488 sync ApisixUpstream failed, will retry {"object": {"Type":1,"Object":{"Key":"<NAMESPACE>/<SERVICENAME>","OldObject":null,"GroupVersion":"apisix.apache.org/v2"},"OldObject":null,"Tombstone":null}, "error": "not found"}
Steps to Reproduce
- Install apisix helm chart
- Create any ApisixIngress with ApisixUpstream
Environment
Docker image: apache/apisix-ingress-controller:1.7.0
Version: 1.7.0 Git SHA: no-git-module Go Version: go1.20.8 Building OS/Arch: linux/amd64 Running OS/Arch: linux/amd64
About this issue
- Original URL
- State: open
- Created 3 years ago
- Reactions: 4
- Comments: 16 (5 by maintainers)
I’m also having the same issue. when will this be solved?
version 1.8.0(latest) the same issue on AKS : ApisixIngress synced failed, with error: not found
I’m having the same error as well. version 1.8.0 on EKS: ApisixIngress synced failed, with error: not found
API7 told us they’re shifting towards development of a replacement. Probably due to the sorry state of the ingress controller in particular; we also experienced the controller simply stalling and not updating endpoints at scale, so we frequently kill our instance every few hours with mayfly as a dirty fix. We partially migrated back to ingress-nginx to have some stable footing.
So I wouldn’t hold my breath for this version to be fixed.
Any update on this problem? I am facing the same issue.
Is deploying the ingress controller without etcd a workaround? No sync needed?
This issue has been marked as stale due to 90 days of inactivity. It will be closed in 30 days if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the dev@apisix.apache.org list. Thank you for your contributions.