kubernetes: imagePullPolicy to pull latest image OR use existing image if repository unavailable
What would you like to be added?
Add another option for a container’s imagePullPolicy, which attempts to pull the image, and falls back to using a local cached copy if the image could not be pulled.
Why is this needed?
In development, images are frequently updated, so it’s essential to grab the latest version of the image to test against.
In production, single points of failure should be minimized, so if the image is available through the repository OR cached locally, then the container should be able to start up (i.e. neither the repo nor the local cache is a single point of failure).
Currently, imagePullPolicy only offers three options:
Always: Only tries to pull the image; ignores cache and fails if repository is unavailableIfNotPresent: First tries to use a cached image; falls back to pulling the imageNever: Only tries to use cached image; ignores repository
From these options, Always is the only one that grabs the latest changes, so it’s the only suitable option for development, but it turns the repository into a single point of failure, so it’s not suitable for production unless you’re okay with everything breaking if the repository goes offline.
In addition, using the latest tag (or any other tag that changes, like a v1 tag that gives the latest 1.x.y version) only works if the imagePullPolicy is Always, or the image cache on each node is cleared whenever the tag is updated, so if those tags are used in production, and you don’t want the repository to be a single point of failure, then there is NO suitable option for production at all.
A fourth option, which first tries to pull the image, and falls back to using a cached copy (the inverse of IfNotPresent), would be well-suited to both development and production, and would fit in very nicely in between Always and IfNotPresent.
Honestly, I think this would be the only imagePullPolicy I would ever use for ANYTHING I do (work and personal projects) if it were available, and I don’t understand why it’s missing.
About this issue
- Original URL
- State: open
- Created 2 years ago
- Reactions: 4
- Comments: 21 (5 by maintainers)
I had really hoped we could avoid turning this into an argument about the validity of using mutable tags, but here goes.
According to the Kubernetes website, “Kubernetes aims to support an extremely diverse variety of workloads”. I recognize that not everyone would have any use for this new
imagePullPolicy, or mutable tags, but that’s okay; nobody is going to force you to use it.I know the typical recommendation is to use immutable tags so you can pin down the exact version you want to use, but that does NOT suit all use cases. For example, it may be desired to use a tag like
v1.2which will update to the latest1.2.xversion, so you automatically get bug fixes (especially security bug fixes) whenever a pod restarts. For a small business with a small IT team hosting their corporate website, file server, et cetera, it may be preferable to have automatic updates rather than have to constantly check for updates and manually change the configuration to a new version in order to keep the system secure.And then there’s situations like mine.
We were initially using immutable versioned tags, but we’ve switched to using
latestexcept during development because we realized all we were doing by using immutable tags in testing and demo environments was forcing people to make more configuration changes, because there was no use case for running an outdated version.The image I’ve been developing is just a monitoring sidecar, which gets added to pods created by over a dozen other companies, and deployed to a private cloud as part of a larger system-of-systems. The system-of-systems as a whole will eventually be installed on production hardware, formally tested (again), and then put into operation for multiple years, where it needs to keep working without system administrators to tend to it. There’s no risk of outdated images in production, because the ONLY version of the image in the entire cloud is the version in the registry we delivered along with the rest of our subsystem. So in production, it doesn’t matter whether we use immutable versioned tags, or
latest, or even an SHA hash.In testing or demos, it’s a different story. Over a dozen companies need to add our sidecar to their pods. With immutable versioned tags, we’d have to tell them all to update their YAML every time we update our image, so it matches what’s in the registry we give them as part of our next build. And of course, until they GET our next build (which is due at the same time as theirs is), that tag wouldn’t work for them. And that’s not even getting into what happens if compatibility issues forced someone to hold back the version of another subsystem in their integration testing environment. It was enough of a pain just going through this with our own test team. By just using
latest, the pods will grab whatever version is in our registry, which is also the version that corresponds to the rest of our build they installed.If, for some reason, the new version of our sidecar interferes with the rest of the pod, they can just remove the sidecar until we’ve fixed the compatibility issue, so there’s no need to roll back to an earlier version. And an earlier version may not work with our latest build, anyway, because we may have made breaking API changes.
So, for us, using
latestis what makes the most sense.