containerized-data-importer: It looks like the cdi controller is not eventually consistent
It looks like the DataVolume will only try to import an image once and then just goes to a failed state. I think that such a behaviour is fine for workloads (Pod, VMI) but for infrastructure components it is very unusual (I actually can’t remember to have seen such a behaviour).
The main issue with that is, that all other controllers which want to use data volumes will also have to manage retry-logic for them to make the system eventually consistent again. Normally controller include retry-logic (including back-off) themselves. Also the whole way on how controller are intended to be written with e.g. client-go are designed with that assumption in mind.
Also PVCs work that way. If a storage provisioner does not have enough space left to back the PVC with a PV, the PVC does not go to a final failed state. Instead, as soon as enough space is left, the provisioner will eventually create a PV and the PVC will go to a bound state.
It is not impossible that a controller can add a proper-retry logic for a DataVolume to itself (see for instance https://github.com/kubevirt/kubevirt/pull/1475), but it significantly increases the complexity of every consumer.
What do you think?
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Comments: 24 (24 by maintainers)
@rmohr @aglitke I think the closest thing our import and clone pods can be compared to is a k8s “job”
For example, with a k8s job we can define that a pod needs to execute to completion once. The job controller will ensure that the pod is retried until it eventually runs to completion successfully.
k8s jobs handle failures in two ways.
spec.backOffLimit which defines how many times a pod will get retried before the job is retried. There’s an exponential backoff here where it goes from 10s, 20s, 40s, etc… all the way to a time cap of 6 minutes.
spec.activeDeadlineSeconds A job fails if it doesn’t complete after ‘x’ amount of time regardless of how many retries occur during that time.
For CDI, I’m not sure it makes sense to use k8s jobs directly. For situations like the clone controller where multiple pods have to be coordinated, having a job controller performing retry logic on our behalf might be unwieldy. However, the k8s jobs API does give us a good precedence for how to perform retry logic for failed import/clone workloads.
Here are my thoughts on how we should proceed.
Default Behavior: retry imports/clone pod execution with exponential back off starting at 10s and capping at 6 minutes just like the Jobs api. This means we’ll need the ability to either clobber previous files, or resume writing previous files. If we don’t already, It would also be nice if we could trickle up information related to why an import job is failing and post that to the event recorder for the pvc.
User Deadline Tuneable Given how vastly different the import/clone times might be depending on file size and network speed, we’ll never be able to come up with an acceptable deadline tuneable that works for everyone. Users should be able to define how long they’d like to wait for the CDI job to complete before marking it as failed.
User Backoff Tuneable When configuration errors occur (unreachable or invalid http import) the import/clone will fail regardless of how many times CDI retries. The default behavior is that CDI will retry forever. Users should be able to specify a custom limit.
thoughts?