dvc: dvc queue status doesn't report active workers
Bug Report
Description
I have two experiments queued up in my dvc queue:
Now I do dvc queue start -j 2. The 2 experiments are running (I can see that via the CPU and GPU usages). Also the dvc queue status shows them as running. But the worker reported by dvc queue status don’t show up:
Reproduce
- dvc exp run --queue --name exp-1
- dvc exp run --queue --name exp-2
- dvc queue start -j 2
- dvc queue status
Expected
See the active running workers.
Environment information
Output of dvc doctor:
DVC version: 3.50.1 (pip)
-------------------------
Platform: Python 3.11.0rc1 on Linux-5.15.0-105-generic-x86_64-with-glibc2.35
Subprojects:
dvc_data = 3.15.1
dvc_objects = 5.1.0
dvc_render = 1.0.2
dvc_task = 0.4.0
scmrepo = 3.3.2
Supports:
azure (adlfs = 2024.4.1, knack = 0.11.0, azure-identity = 1.16.0),
http (aiohttp = 3.9.5, aiohttp-retry = 2.8.3),
https (aiohttp = 3.9.5, aiohttp-retry = 2.8.3)
Config:
Global: /home/raw/.config/dvc
System: /etc/xdg/dvc
Cache types: symlink
Cache directory: ext4 on /dev/sdb1
Caches: local
Remotes: azure
Workspace directory: ext4 on /dev/nvme0n1p3
Repo: dvc, git
Repo.site_cache_dir: /var/tmp/dvc/repo/1c67604caf3156e9fea7df37dda80d5f
Additional Information (if any):
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 1
- Comments: 25 (10 by maintainers)
Commits related to this issue
- dvc-10427 Add 2" delay to expirations check As per https://github.com/iterative/dvc/issues/10427 investigation we realised that kombu creates expiration dates for messages 1" in the future which affe... — committed to nablabits/dvc-task by nablabits 2 years ago
- dvc-10427 Update environmentafter the discoveries on 10427 Please see: https://github.com/iterative/dvc/issues/10427#issuecomment-2373554706 — committed to nablabits/example-get-started-experiments by nablabits 2 years ago
- dvc-10427 Send the shutdown message to the appropriate worker. Please check: https://github.com/iterative/dvc/issues/10427#issuecomment-2373554706 — committed to nablabits/dvc-task by nablabits 2 years ago
- Revert "dvc-10427 Add 2" delay to expirations check" This reverts commit f15acd796b9721e52cb012fb7fc4e706e06ae8f0 as per https://github.com/iterative/dvc/issues/10427#issuecomment-2387754170 — committed to nablabits/dvc-task by nablabits 2 years ago
- dvc-10427 Fix Celery Monitor Shutdowns (#142) * dvc-10427 Add 2" delay to expirations check As per https://github.com/iterative/dvc/issues/10427 investigation we realised that kombu creates expira... — committed to iterative/dvc-task by nablabits 2 years ago
Hi @shcheklein , so finally I got to the bottom of this issue 🐌 , the thing is as follows:
processedonce they are done.indirectory._gcwon’t clean them as_delete_expiredwill detect that the message is not expired yet (source). These are two sample expiration times in one of the experiments I ran:1725093643.3807564 > 1725093642.829151where we can see that one is 1" off which is precisely what this folk is adding as a timeout when the message is created._clean_pidboxwon’t clean them either as the shutdown messages belong to thecelery.pidboxexchange whereas the exchange that is being cleaned isreply.celery.pidboxdvc exp cleanwill naturally clean the files as it will find them expired which is also in agreement.So, the obvious solution is to add that extra second timeout to this check although I’m unsure of the side effects that that may have.
Do you want me to open an issue in
dvc-taskfor that?I’ve finally managed to reproduce the issue consistently at will. This is what I did:
Next up: I’ve checked
dvc-exp-worker-1.outfinding something promising, in failed runs we get abefore the task succeeds but the task seems to be there as it will eventually succeed which agrees with what was stated in the main description. In non-failing runs, this
Got shutdown from remotehappens after the task has succeeded. This may well be the key of the problem as we retrieve these active workers by means ofcelery.control.inspectwhich seems to me using thisremoteunder the hood. So I will investigate that avenue.Closed by https://github.com/iterative/dvc-task/pull/142
Yes, agreed, let’s ship the simplest version (w/o modifying the timestamps, GC time thresholds, etc) and then create a new issue / PR to keep the discussion / research going! Thanks @nablabits .
Hi @shcheklein
I’ve just tested your proposal and it worked like a charm ✨ , thanks for the suggestion. I will update https://github.com/iterative/dvc/pull/10552 and https://github.com/iterative/dvc-task/pull/142 accordingly.
Do we want to do something with extra 2"? as the messages kombu creates are 1" in the future. Not a big deal on itself but may save a few headaches in the future as debugging these folks is a bit time consuming.
Just another brief update here
TL;DR on what We Have So Far
dvc-taskascleanseems not cleaning everything or something is still created aftercleanis called. Removing the conditional logic so as to run this clean always and putting asleep(60)here fixes the issue. But, of course we don’t want this sloppy solution 😉.dvc/tmp/exps/celery/broker/in/, and there are residual files that remain there after the tasks are done. Removing these files solves the issue.-j 2as if one enqueues two experiments but only selects one job, then it behaves normally. Running for one job still creates those residuals inin, so I’m kind of exploring the differences between both approaches.Shared Data Directories
One of the things that I tried is to set different
outdirectory here so as to check if that had something to do and debug the inner workings of kombu and dvc-task, but this created all sorts of issues.I have more personal annotations for myself in case anyone is interested but above are the main findings.
Thanks for the patience 🙏
thanks for the update @nablabits !
Hi there, for a few weeks I couldn’t work on this issue, but this week has been somewhat fruitful.
TL;DR
shutdownfor the next command rundvc exp cleanbefore enqueuing and running a new experiment fixes the issuepidboxLogAfter enabling debug logs in for Celery,I realised that working runs got this after the task is accepted:
Whereas faulty ones got:
This is not because of
ThreadPoolExecutorandCmdQueueStopeither. These are the two explicit places whereshutdown()is called.Clean Experiments
I also realised that running
dvc exp cleanbefore running experiments solves the issue, so this may be a potential workaround. As per documentation I note that This is done automatically when running queued experiments. which led me to play a bit with this--cleanflag —source—. Unfortunately, adding it selectively to either worker or even to all of them didn’t seem to fix the issue. Still, I will keep investigating in this line unless someone advises otherwise.Thanks for the patience 🙏
@nablabits sure! please give it a try.