cli: volumes wont be deleted since update to docker 23
Since the update to docker 23 unused volumes will not be deleted anymore with docker volume prune
nor docker system prune --volumes
.
The answer is always Total reclaimed space: 0B
.
When I delete the volumes manually I even get these warning when running docker-compose.yml
Error response from daemon: open /var/lib/docker/volumes/docker_volume1/_data: no such file or directory
Whats happening?
About this issue
- Original URL
- State: closed
- Created a year ago
- Reactions: 31
- Comments: 50 (11 by maintainers)
Links to this issue
Commits related to this issue
- fix broken prune command. Seems like system prune will not delete any named volumes after the recent update due to change in default. More on this here - https://github.com/docker/cli/issues/4028. Fo... — committed to subspace/infra by vedhavyas a year ago
- docker prune by default does not remove named volumes See https://github.com/docker/cli/issues/4028. — committed to outsinre/outsinre.github.com by outsinre a year ago
- Update docs/command output for volume pruning In previous versions of the Docker API, `system prune --volumes` and `volume prune` would remove all dangling volumes. With API v1.42, this was changed s... — committed to edmorley/docker-cli by edmorley a year ago
- Update docs/command output for volume pruning In previous versions of the Docker API, `system prune --volumes` and `volume prune` would remove all dangling volumes. With API v1.42, this was changed s... — committed to edmorley/docker-cli by edmorley a year ago
- Update docs/command output for volume pruning In previous versions of the Docker API, `system prune --volumes` and `volume prune` would remove all dangling volumes. With API v1.42, this was changed s... — committed to thaJeztah/cli by edmorley a year ago
- Remove volumes in docker-cleanup https://github.com/docker/cli/issues/4028 — committed to sdt/.dotfiles by sdt 8 months ago
- gitlab_runner: Update for changed volume pruning behavior in Docker 23.0 "docker system prune --volumes" does no longer prune named volumes in Docker 23.0[1][2], so use "docker volume prune --all"[3]... — committed to archlinux/infrastructure by klausenbusk 7 months ago
Sorry I gave the wrong command in the first line there,
docker volume prune --filter all=1
I guess that’s fine reasoning, but the execution of this change was very strange for multiple reasons:
-h
, and hopefully notice mention of named vs anonymous volumes.I have spent * days * trying to figure out why all of our gitlab runners are suddenly out of disk space, resorting to crude methods of stopping all containers manually in the middle of the night so I can loop through volumes and delete them. I expect a command like
docker volume prune -f
toRemove all unused local volumes
as the docs say and have said for years (the example usage on the docs page even explicitly has amy-named-vol
being removed).Regardless of doc updates being missed (we’re humans, it happens), a change with this big of an impact should have had deprecation/compatibility warnings for at least one version. “My logs are being polluted, what’s going on? Oh, I need to update a command, cool.” is a much easier problem to deal with than “Why is the entire fleet of servers all running out of disk at the same time?!”
The changelog has this listed under “bug fixes and enhancements” and has the prefix of
API:
with no mention whatsoever of this effecting the cli. Even the most vigilant changelog readers would have missed this without intimate knowledge of the cli/API interaction.@Emporea
docker volume prune --all
should give the old behavior. I understanddocker system prune
doesn’t support this yet (not intentionally).No, I don’t think that should be necessary, it would likely be simpler to use
docker volume prune --filter all=1
in addition todocker system prune
until your older volumes are no longer in use.You can also use
DOCKER_API_VERSION=1.40 docker system prune --volumes
to get the old behavior.Yes this sounds like it is related to the mentioned change. The default change allows us to:
docker volume prune
safer to execute. Given that named volumes typically are named so they can be easily referencedIt does mean that upgrading causes volumes which were created prior to 23.0.0 to not be considered for pruning except for when specifying
--all
. However “anonymous” volumes created after 23.0.0 will be considered for pruning… and of course again--all
gives the old behavior.Also if your client uses the older API version it will also get the old behavior.
It seems like it’s this change in 23 (mentioned in the 23 changelog). When I run this
all=true
filter ondocker volume prune
it works. Butdocker system prune
does not accept this filter, so now seems brokenNot clear why this default needed to change
Thanks for the reports. I found the bug and will post a patch momentarily.
If its a hex string then it is most likely not a named volume (unless you have something generating strings when you create the volume). You should be able to inspect the volume and check for the label:
com.docker.volume.anonymous
. If this key exists (value is unused) then it may be collected bydocker volume prune
.It would be nice if the change would be cascaded into
system prune
andsystem df
as raised by others.prune
still claims it removes all volumes not linked to any containers if passed--volumes
.df
, similarly list those volumes as reclaimable space, which should now be taken with a grain of salt.I spent half a day today on this, I expected that my volumes were being deleted… thank you for changing the behavior and not sending a message to the console that it no longer works as before (without negativity)
@neersighted just to be precise though, from what I know the
-a
,--all
in system prune affect only images. For volumes, there is another option/flag, which is--volumes
.Thinking about it, I could definitely see a
--volumes
that would default to--volumes anonymous
and a--volumes all
allowing to cascade the need to delete named volume to thedocker volume prune
command, here.But I also see the place where this “better safe than sorry” change is coming.
https://github.com/docker/cli/pull/4497 was accepted and cherry-picked, which addresses the docs/
--help
issue. It is intentional thatsystem prune -a
no longer affects named volumes; I think a--really-prune-everything
flag is out of scope for this issue, but feel free to open a feature request if you think that it is useful in the 90% case. My earlier comments are still quite relevant, I think:I’m going to close this for now, as the last set of sharp edges identified here are solved (and will be in the next patch release), but please feel free to continue the discussion or open that follow-up feature request.
As stated in this thread,
system prune -a
no longer prunes named volumes, by design. If you need to prune named volumes, the method to use is currentlyvolune prune -a
.system prune -a
is a command often fired indiscriminately, and has lead to much data loss and gnashing of teeth. While having to run two commands is a mild pain, it helps with preventing frustration and loss of data for new users copying commands out of tutorials.We can certainly explore a
system prune --all=with-named-volumes
or something in the future for users who understand exactly what they are doing, but currently the need to run a separate command is by design.We have not changed the behavior of
system prune -a
to implyvolume prune -a
.I just use
docker volume rm $(docker volume ls -q)
since I can’t remember the filter options.@sudo-bmitch I’m getting the same behavior as described by @Re4zOon with
docker compose
.Edit: With plain
docker
too. I can reproduce withdocker run -e MARIADB_ROOT_PASSWORD=root mariadb
for example. If I stop and remove the container, the volume stays and prune won’t delete it, unless I specify--filter all=1
.Output of
docker system info
:Yeah, the help needs to change to say ‘anonymous.’
Same as the help output, PRs welcome.
I’d like to point out you’re a tiny minority there – for the vast majority of users, “Docker deleted my data after I ran
system prune -a
” has been a sharp edge for years. Most users expectprune
to ‘clean up garbage,’ not ‘clean up the things I wanted Docker to keep.’The only part where this possibly needs to propagate is
docker system prune -a
and we’re still not sure what the better behavior is.Agreed, there should be an example of using the all filter in the help text.
Please keep in mind that this has been a persistent pain for educators, commercial support, and undercaffinated experts for years. People are here in this thread because they find the behavior change surprising, and yeah, it looks like review missed the docs updates needed (and this is because of the historical incorrect split of client/server logic we are still cleaning up) – however, please keep in mind that this thread represents the minority of users who find this behavior unexpected or problematic.
We certainly can improve here, and there are a lot of valid points, but the happy path for the majority of users is to stop pruning named volumes by default.