openpilot: deleter: inconsistency after AGNOS11 update
Describe the bug
I suspect that the deleter after AGNOS11 update is no more consistent. I haven’t tested it yet to confirm, but I’ve already caught four people who had to delete realdata/* manually (because a noEnter “out of storage” event)
Provide a route where the issue occurs
dongleIDs: 0b54d0594d924cd9, 075b133b6181e058, a0e5673b89222047
075b133b6181e058/000000cc--7d1fff3f36/42
openpilot version
13760968bbdec44c2bf234fde79f8ef0e5bbe254
Additional info
Not confirmed
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 1
- Comments: 38 (28 by maintainers)
Confirmed.
This reproduces the problem on AGNOS 11.4. Switching to system Python (
/usr/bin/python3) makes it work correctly.The system Python just happens to be exactly the same Python version, built differently.
In the Astral repo I can’t find any PRs or issues referencing the clock issue, but Astral makes extensive build and code changes. For starters, you can see above the UV build is done with an ancient GCC. It acts differently even in the output above, the Ctrl-D to exit is consistently echoed to screen in the UV build but not echoed in the system build.
It comes with its own threading library (which is right in our problem area):
Shipping both copies of Python 3.12.3 inflates the AGNOS image by almost 100MB:
Insert do-not-want GIF here.
After the move to Ubuntu 24.04, do we continue to have good reason to not use the system Python? UV can still maintain full control of the venv and dependencies while using the system Python binaries, just use
only-systemrather thanonly-managed. I get needing the separately controlled version for desktop and maybe CI (?) but I don’t see why we need it for the hand-crafted AGNOS image where we have full control of the system Python. Untested, but it sure looks like we can fix this bug with a two line PR that also reclaims 100MB of disk for free.Fixed in https://github.com/commaai/openpilot/pull/34413
It’s happening due to a time warp, which can be seen in @korhojoa’s logs. The clock is going back to 2015, trigger unknown but isn’t supposed to matter. Threading event.wait() is supposed to use a monotonic clock and therefore be robust to that, but it isn’t. @incognitojam found python/cpython#91394 which may describe the issue.
To reproduce on device, apply this patch:
Then launch
deleterin one terminal:Then tamper with the system time in another terminal:
You will stop seeing “Sleeping” output from deleter, but it will resume when the clock is fixed:
This explains why deleter recovers when we get Internet connectivity, because
timesyncdcan fix the time.The correct fix is not yet clear to me. There are a number of hacks we could apply, but ideally if Python is supposed to be using a monotonic clock for Event.wait(), we should find out why it isn’t and fix that.
Following up to myself:
The same problem cannot be reproduced on release; the threading wait() call doesn’t seem vulnerable to system time changes. It remains to be determined what changed in Python or AGNOS that broke monotonic timekeeping, but having identified that it’s a regression, it should be fixable somehow.
Correct behavior with:
Wrong behavior with:
41f4ae6d3b79a9b1is my dongle ID.i see this issue consistently. I have added a menu item that will show my remaining storage from the developer menu. I observed that the storage went to 38 and stayed there for the next couple hours until i stopped again, then it stopped at 21, then at 10. All of the routes that i drove appeared to be updated, but they were not being deleted from the c3. I do not have comma prime and take lots of long drives.
When i play with the deleter script to make it start deleting some stuff, it seems to try to delete only one route/seg then stops? there were LOTS of routes here.
Since that image, i sent an update with a syntax error and for some reason could not ssh back in, some service not started? ip changed? im not sure. i had to reset. I am confident that i will see this again. Ill monitor it closely and improve the logging as i go.
Should be very easy to reproduce, happens to me 100% of the time if there is no network connection. I’ll try the branch.
I’ve made a somewhat pinned branch of master-ci that makes deleter log like hell with rotation and an extra logging to see if deleter dies to something that’s not an
OSErrorto somewhere in/dataverbosely and intend to “daily it”:https://github.com/nelsonjchen/openpilot/tree/33993_master-ci
(I’ll rebase it onto a newer master-ci whenever I feel like it.)
It’ll be logging onto the SOM’s storage, but that’s good since it’ll be separate from the NVMe drive.
That, combined with
fallocate -l 400G /data/media/dummy.datto “trigger” it the storage being “full-ish” so it’ll be in the persistently deleting state on my 1TB C3 SSD, should help trace the issue when it breaks again. I had to put this weight back on since I cleared some of the the SSD to get it working again ASAP.Anyway, just mentioning my methodology if anyone else wants to follow along.
Just started looking into this:
075b133b6181e058/000000cc--7d1fff3f36/42fe18f736cb0d7813/0000040d--3a202cd3b9only some dongs, added to description
Got any routes or dongles?