apisix-ingress-controller: bug: ApisixIngress synced failed, with error: not found

Current Behavior

When use ApisixUpstream with apisix ingress controller 1.7

It show this status:

apiVersion: apisix.apache.org/v2
kind: ApisixUpstream
...
status:
  conditions:
  - message: not found
    observedGeneration: 1
    reason: ResourceSyncAborted
    status: "False"
    type: ResourcesAvailable

event log:

  Normal   ResourcesSynced      18m (x2 over 18m)     ApisixIngress  ApisixIngress synced successfully
  Normal   ResourcesSynced      15m                   ApisixIngress  ApisixIngress synced successfully
  Warning  ResourceSyncAborted  11m (x7 over 14m)     ApisixIngress  ApisixIngress synced failed, with error: not found
  Warning  ResourceSyncAborted  16s (x13 over 10m)    ApisixIngress  ApisixIngress synced failed, with error: not found

But when I change image to 1.6.1, it was successful

apiVersion: apisix.apache.org/v2
kind: ApisixUpstream
...
status:
  conditions:
  - message: Sync Successfully
    observedGeneration: 1
    reason: ResourcesSynced
    status: "True"
    type: ResourcesAvailable

If I change back to 1.7.0, it show not found again

Expected Behavior

It should be successful like before version

Error Logs

2023-10-06T11:52:36+08:00    debug    apisix/upstream.go:43    try to look up upstream    {"name": "<NAMESPACE>_<SERVICENAME>_8088", "url": "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/upstreams", "cluster": "default"}
2023-10-06T11:52:36+08:00    debug    apisix/apisix_upstream.go:351    updating upstream since ApisixUpstream changed    {"upstream": {"id":"ad53b4e6","name":"<NAMESPACE>_<SERVICENAME>_8088","desc":"Created by apisix-ingress-controller, DO NOT modify it manually","labels":{"managed-by":"apisix-ingress-controller"},"type":"roundrobin","nodes":[{"host":"10.205.17.189","port":8088,"weight":100}],"scheme":"http"}, "ApisixUpstream name": "<NAMESPACE>_<SERVICENAME>_8088"}
2023-10-06T11:52:36+08:00    debug    apisix/upstream.go:189    try to update upstream    {"id": "ad53b4e6", "name": "<NAMESPACE>_<SERVICENAME>_8088", "cluster": "default", "url": "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/upstreams"}
2023-10-06T11:52:36+08:00    debug    apisix/upstreamservicerelation.go:91    try to create upstreamService in cache    {"cluster": "default"}
2023-10-06T11:52:36+08:00    debug    apisix/upstreamservicerelation.go:47    try to get upstreamService in cache    {"service_name": "<NAMESPACE>_<SERVICENAME>", "cluster": "default"}
2023-10-06T11:52:36+08:00    debug    apisix/cluster.go:798    updating resource in cluster    {"cluster_name": "default", "name": "upstream", "url": "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/upstreams/ad53b4e6", "body": "{\"id\":\"ad53b4e6\",\"name\":\"<NAMESPACE>_<SERVICENAME>_8088\",\"desc\":\"Created by apisix-ingress-controller, DO NOT modify it manually\",\"labels\":{\"managed-by\":\"apisix-ingress-controller\"},\"type\":\"roundrobin\",\"nodes\":[{\"host\":\"10.205.17.189\",\"port\":8088,\"weight\":100}],\"scheme\":\"http\"}"}
2023-10-06T11:52:36+08:00    debug    apisix/resource.go:135    got upstream: {"desc":"Created by apisix-ingress-controller, DO NOT modify it manually","labels":{"managed-by":"apisix-ingress-controller"},"create_time":1696389535,"scheme":"http","pass_host":"pass","type":"roundrobin","id":"ad53b4e6","name":"<NAMESPACE>_<SERVICENAME>_8088","update_time":1696564356,"nodes":[{"host":"10.205.17.189","weight":100,"priority":0,"port":8088}],"hash_on":"vars"}
2023-10-06T11:52:36+08:00    debug    apisix/upstream.go:43    try to look up upstream    {"name": "<NAMESPACE>_<SERVICENAME>_8088_service", "url": "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/upstreams", "cluster": "default"}
2023-10-06T11:52:36+08:00    debug    apisix/upstream.go:59    failed to find upstream in cache, will try to lookup from APISIX    {"name": "<NAMESPACE>_<SERVICENAME>_8088_service", "error": "not found"}
2023-10-06T11:52:36+08:00    debug    apisix/cluster.go:645    get resource in cluster    {"cluster_name": "default", "name": "upstream", "url": "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/upstreams/6fd1d84"}
2023-10-06T11:52:36+08:00    warn    apisix/cluster.go:1164    upstream not found    {"id": "6fd1d84", "url": "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/upstreams/6fd1d84", "cluster": "default"}
2023-10-06T11:52:36+08:00    error    apisix/apisix_upstream.go:332    failed to get upstream <NAMESPACE>_<SERVICENAME>_8088_service: not found
2023-10-06T11:52:36+08:00    warn    apisix/apisix_upstream.go:488    sync ApisixUpstream failed, will retry    {"object": {"Type":1,"Object":{"Key":"<NAMESPACE>/<SERVICENAME>","OldObject":null,"GroupVersion":"apisix.apache.org/v2"},"OldObject":null,"Tombstone":null}, "error": "not found"}

Steps to Reproduce

  1. Install apisix helm chart
  2. Create any ApisixIngress with ApisixUpstream

Environment

Docker image: apache/apisix-ingress-controller:1.7.0

Version: 1.7.0 Git SHA: no-git-module Go Version: go1.20.8 Building OS/Arch: linux/amd64 Running OS/Arch: linux/amd64

About this issue

  • Original URL
  • State: open
  • Created 3 years ago
  • Reactions: 4
  • Comments: 16 (5 by maintainers)

Most upvoted comments

I’m also having the same issue. when will this be solved?

version 1.8.0(latest) the same issue on AKS : ApisixIngress synced failed, with error: not found

I’m having the same error as well. version 1.8.0 on EKS: ApisixIngress synced failed, with error: not found

API7 told us they’re shifting towards development of a replacement. Probably due to the sorry state of the ingress controller in particular; we also experienced the controller simply stalling and not updating endpoints at scale, so we frequently kill our instance every few hours with mayfly as a dirty fix. We partially migrated back to ingress-nginx to have some stable footing.

So I wouldn’t hold my breath for this version to be fixed.

Any update on this problem? I am facing the same issue.

Is deploying the ingress controller without etcd a workaround? No sync needed?

This issue has been marked as stale due to 90 days of inactivity. It will be closed in 30 days if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the dev@apisix.apache.org list. Thank you for your contributions.