kubernetes: Inconceivable: dispatch before arrival - apparent APF error
What happened?
Running a load test on K8s 1.23.5, and I got some messages like this:
\"Inconceivable!\" err=\"dispatch before arrival\" QS=\"system\" queue=0 dispatchR=\"6570.26436833ss\" request=... flowDistinguisher:system:node:[redacted] fsName:system-nodes ... workEstimate:{WorkEstimate:{InitialSeats:1 FinalSeats:0 AdditionalLatency:0} totalWork:300000 finalWork:0} ... arrivalTime:{wall:13887921938884605587 ext:18745363038248 ... arrivalR:657026745599 startTime:{wall:0 ext:0 loc:<nil>} waitStarted:false}
What did you expect to happen?
It runs without inconceivable errors
How can we reproduce it (as minimally and precisely as possible)?
I don’t have a good repro, sorry. Output is from an internal load test that we run here at AKS. There were other errors logged around the same time (so maybe there’s some load-related issue in the cluster that caused the problem - I haven’t checked that at this time). Just figured I’d better log this issue because, given the wording of the error message, I figured someone might like to know.
Kubernetes version
1.23.5
Cloud provider
AKS
About this issue
- Original URL
- State: open
- Created 2 years ago
- Comments: 22 (12 by maintainers)
Commits related to this issue
- Call queueSet::boundNextDispatchLocked enough Fix the one path where boundNextDispatchLocked was not being called after modifying a queue. Also check for negative work in a request. These are motiv... — committed to MikeSpreitzer/kubernetes by MikeSpreitzer 2 years ago
- Call queueSet::boundNextDispatchLocked enough Fix the one path where boundNextDispatchLocked was not being called after modifying a queue. Also check for negative work in a request. These are motiv... — committed to kubernetes/apiserver by MikeSpreitzer 2 years ago
- Call queueSet::boundNextDispatchLocked enough Fix the one path where boundNextDispatchLocked was not being called after modifying a queue. Also check for negative work in a request. These are motiv... — committed to MikeSpreitzer/kubernetes by MikeSpreitzer 2 years ago
- Call queueSet::boundNextDispatchLocked enough Fix the one path where boundNextDispatchLocked was not being called after modifying a queue. Also check for negative work in a request. These are motiv... — committed to MikeSpreitzer/kubernetes by MikeSpreitzer 2 years ago
- Call queueSet::boundNextDispatchLocked enough Fix the one path where boundNextDispatchLocked was not being called after modifying a queue. Also check for negative work in a request. These are motiv... — committed to MikeSpreitzer/kubernetes by MikeSpreitzer 2 years ago
- Call queueSet::boundNextDispatchLocked enough Fix the one path where boundNextDispatchLocked was not being called after modifying a queue. Also check for negative work in a request. These are motiv... — committed to danielye11/kubernetes by MikeSpreitzer 2 years ago
- Call queueSet::boundNextDispatchLocked enough Fix the one path where boundNextDispatchLocked was not being called after modifying a queue. Also check for negative work in a request. These are motiv... — committed to kubernetes/apiserver by MikeSpreitzer 2 years ago
- Call queueSet::boundNextDispatchLocked enough Fix the one path where boundNextDispatchLocked was not being called after modifying a queue. Also check for negative work in a request. These are motiv... — committed to kubernetes/apiserver by MikeSpreitzer 2 years ago
- Call queueSet::boundNextDispatchLocked enough Fix the one path where boundNextDispatchLocked was not being called after modifying a queue. Also check for negative work in a request. These are motiv... — committed to kubernetes/apiserver by MikeSpreitzer 2 years ago
Can you retry, with verbosity targeted specifically at https://github.com/kubernetes/apiserver/blob/v0.23.5/pkg/util/flowcontrol/fairqueuing/queueset/queueset.go#L948 ?
That, and the metric, would help confirm or deny two hypotheses about what’s going wrong.