karmada: After the failure cluster is recovered, the residual resources are not deleted
What happened:
As described in comment: https://github.com/karmada-io/karmada/pull/3808#discussion_r1295679640
What you expected to happen:
After the failure cluster is recovered, these residual resources can be deleted normally.
How to reproduce it (as minimally and precisely as possible):
Anything else we need to know?:
Environment:
- Karmada version:
- kubectl-karmada or karmadactl version (the result of
kubectl-karmada versionorkarmadactl version): - Others:
About this issue
- Original URL
- State: closed
- Created 10 months ago
- Comments: 17 (17 by maintainers)
Hi @RainbowMango @chaunceyjiang @XiShanYongYe-Chang
I’m sorry for replying late.
After the discussion among our team, we think that the core issue is that why we need
max retryinAsyncWorker. We could have a further discussion on this later.And for #3808, I think it’s okay to revert it for the new version coming soon. But the core thought of it we think is still right. We hope it could be brought back when we solve the promblem of
max retry.@XiShanYongYe-Chang @RainbowMango Thanks for your replying.
I’m quite glad to help update it. And I will try to bring #3808 back after it. 😃
I’ll do it as soon as possible
So I got a thought for a while, that make objectWatcher use
DeepEqual()checks if there is content changes or not(mutating webhook and API compatibility between karmada and member clusters might be the problem), then it can reduce the frequency of updating member clusters when execution-controller reconciling.This makes every affected items ratelimited, clearing items from the ratelimiter may take a lot of time. And if one day we might let user control the options of ratelimiter and number of async worker retries (or just alter to controller-runtime). In that case, recovery mechanism that rely on dynamic parameters is unreliable.
On the other hand, there’s no much difference between watch cluster objects with async worker/controller-runtime’s retry mechanism, they both trigger the execution-controller reconciling.