openpilot: deleter: inconsistency after AGNOS11 update

Describe the bug

I suspect that the deleter after AGNOS11 update is no more consistent. I haven’t tested it yet to confirm, but I’ve already caught four people who had to delete realdata/* manually (because a noEnter “out of storage” event)

Provide a route where the issue occurs

dongleIDs: 0b54d0594d924cd9, 075b133b6181e058, a0e5673b89222047

075b133b6181e058/000000cc--7d1fff3f36/42

openpilot version

13760968bbdec44c2bf234fde79f8ef0e5bbe254

Additional info

Not confirmed

image

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 1
  • Comments: 38 (28 by maintainers)

Most upvoted comments

It’s possible it’s just an issue with uv’s Python builds (https://github.com/astral-sh/python-build-standalone).

Confirmed.

#!/usr/bin/env python3
import threading

def testing_thread(test_event: threading.Event):
  while True:
    print("Sleeping...")
    test_event.wait(1)

def main():
  testing_thread(threading.Event())

if __name__ == "__main__":
  main()

This reproduces the problem on AGNOS 11.4. Switching to system Python (/usr/bin/python3) makes it work correctly.

The system Python just happens to be exactly the same Python version, built differently.

comma@comma-2a02b9c8:/data/openpilot$ python3
Python 3.12.3 (main, Apr 15 2024, 17:09:49) [GCC 6.3.0 20170516] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> ^D
comma@comma-2a02b9c8:/data/openpilot$ /usr/bin/python3
Python 3.12.3 (main, Sep 11 2024, 14:17:37) [GCC 13.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 

In the Astral repo I can’t find any PRs or issues referencing the clock issue, but Astral makes extensive build and code changes. For starters, you can see above the UV build is done with an ancient GCC. It acts differently even in the output above, the Ctrl-D to exit is consistently echoed to screen in the UV build but not echoed in the system build.

It comes with its own threading library (which is right in our problem area):

comma@comma-2a02b9c8:/data/openpilot$ ls -al /usr/local/uv/python/cpython-3.12.3-linux-aarch64-gnu/lib
total 48340
drwxr-xr-x 10 root root     4096 Nov 13 18:02 .
drwxr-xr-x  6 root root     4096 Nov 13 18:02 ..
drwxr-xr-x  2 root root     4096 Nov 13 18:02 itcl4.2.2
lrwxrwxrwx  1 root root       20 Nov 13 18:02 libpython3.12.so -> libpython3.12.so.1.0
-rwxrwxr-x  1 root root 49387544 Nov 13 18:02 libpython3.12.so.1.0
-rwxrwxr-x  1 root root    66328 Nov 13 18:02 libpython3.so
drwxr-xr-x  2 root root     4096 Dec 17 00:14 pkgconfig
drwxr-xr-x 38 root root     4096 Dec 17 00:14 python3.12
drwxr-xr-x  5 root root     4096 Nov 13 18:02 tcl8
drwxr-xr-x  6 root root     4096 Nov 13 18:02 tcl8.6
drwxr-xr-x  2 root root     4096 Nov 13 18:02 thread2.8.7
drwxr-xr-x  4 root root     4096 Nov 13 18:02 Tix8.4.3
drwxr-xr-x  6 root root     4096 Nov 13 18:02 tk8.6

Shipping both copies of Python 3.12.3 inflates the AGNOS image by almost 100MB:

comma@comma-2a02b9c8:/data/openpilot$ du -sh /usr/local/uv/python/cpython-3.12.3-linux-aarch64-gnu
96M     /usr/local/uv/python/cpython-3.12.3-linux-aarch64-gnu

Insert do-not-want GIF here.

After the move to Ubuntu 24.04, do we continue to have good reason to not use the system Python? UV can still maintain full control of the venv and dependencies while using the system Python binaries, just use only-system rather than only-managed. I get needing the separately controlled version for desktop and maybe CI (?) but I don’t see why we need it for the hand-crafted AGNOS image where we have full control of the system Python. Untested, but it sure looks like we can fix this bug with a two line PR that also reclaims 100MB of disk for free.

It’s happening due to a time warp, which can be seen in @korhojoa’s logs. The clock is going back to 2015, trigger unknown but isn’t supposed to matter. Threading event.wait() is supposed to use a monotonic clock and therefore be robust to that, but it isn’t. @incognitojam found python/cpython#91394 which may describe the issue.

To reproduce on device, apply this patch:

diff --git a/system/loggerd/deleter.py b/system/loggerd/deleter.py
index 2a2bd5fe1..44e796a62 100755
--- a/system/loggerd/deleter.py
+++ b/system/loggerd/deleter.py
@@ -69,7 +69,8 @@ def deleter_thread(exit_event: threading.Event):
           cloudlog.exception(f"issue deleting {delete_path}")
       exit_event.wait(.1)
     else:
-      exit_event.wait(30)
+      cloudlog.info("Sleeping")
+      exit_event.wait(5)

Then launch deleter in one terminal:

comma@comma-2a02b9c8:/data/openpilot$ LOGPRINT=debug system/loggerd/deleter.py  
Sleeping                               
Sleeping                               
Sleeping       

Then tamper with the system time in another terminal:

sudo systemctl stop systemd-timesyncd
sudo date --set "1 Jan 2015 12:00:00"
date

You will stop seeing “Sleeping” output from deleter, but it will resume when the clock is fixed:

sudo systemctl start systemd-timesyncd

This explains why deleter recovers when we get Internet connectivity, because timesyncd can fix the time.

The correct fix is not yet clear to me. There are a number of hacks we could apply, but ideally if Python is supposed to be using a monotonic clock for Event.wait(), we should find out why it isn’t and fix that.

Following up to myself:

The same problem cannot be reproduced on release; the threading wait() call doesn’t seem vulnerable to system time changes. It remains to be determined what changed in Python or AGNOS that broke monotonic timekeeping, but having identified that it’s a regression, it should be fixable somehow.

Correct behavior with:

comma@comma-2a02b9c8:/data/openpilot$ python --version
Python 3.11.4
comma@comma-2a02b9c8:/data/openpilot$ cat /VERSION
10.1

Wrong behavior with:

comma@comma-2a02b9c8:/data/openpilot$ python --version
Python 3.12.3
comma@comma-2a02b9c8:/data/openpilot$ cat /VERSION 
11.4

41f4ae6d3b79a9b1 is my dongle ID.

i see this issue consistently. I have added a menu item that will show my remaining storage from the developer menu. I observed that the storage went to 38 and stayed there for the next couple hours until i stopped again, then it stopped at 21, then at 10. All of the routes that i drove appeared to be updated, but they were not being deleted from the c3. I do not have comma prime and take lots of long drives.

When i play with the deleter script to make it start deleting some stuff, it seems to try to delete only one route/seg then stops? there were LOTS of routes here. image

Since that image, i sent an update with a syntax error and for some reason could not ssh back in, some service not started? ip changed? im not sure. i had to reset. I am confident that i will see this again. Ill monitor it closely and improve the logging as i go.

Should be very easy to reproduce, happens to me 100% of the time if there is no network connection. I’ll try the branch.

I’ve made a somewhat pinned branch of master-ci that makes deleter log like hell with rotation and an extra logging to see if deleter dies to something that’s not an OSError to somewhere in /data verbosely and intend to “daily it”:

https://github.com/nelsonjchen/openpilot/tree/33993_master-ci

(I’ll rebase it onto a newer master-ci whenever I feel like it.)

It’ll be logging onto the SOM’s storage, but that’s good since it’ll be separate from the NVMe drive.

That, combined with fallocate -l 400G /data/media/dummy.dat to “trigger” it the storage being “full-ish” so it’ll be in the persistently deleting state on my 1TB C3 SSD, should help trace the issue when it breaks again. I had to put this weight back on since I cleared some of the the SSD to get it working again ASAP.

Anyway, just mentioning my methodology if anyone else wants to follow along.

Just started looking into this:

075b133b6181e058/000000cc--7d1fff3f36/42

  • eventually recovers towards the end of the drive, almost like the deleter was stuck
  • no rlogs
  • bootlog for this route is missing, not on the NVMe?

fe18f736cb0d7813/0000040d--3a202cd3b9

  • doesn’t recover like the other route did
  • thanks for uploading all the logs @nelsonjchen!
  • has the bootlog, and NVMe is mounted for that bootlog

Got any routes or dongles?

only some dongs, added to description

Got any routes or dongles?