depthai: [BUG] Encoder Problem With Latest Raspberry Pi OS

So we discovered in internal testing, thanks to Chris, that after upgrading our Raspberry Pi’s to the latest Raspberry Pi OS (with dist-upgrade, exact version below) that the DepthAI video encoder is not stable, so this example (04 - RGB Encoding) fails quickly. Prior to this upgrade to Raspberry Pi OS (the latest as of this writing), this instability did not happen. And we do not see the instability on other platforms (to our knowledge).

But on the latest Raspberry Pi OS, every Gen2 DepthAI version we tested has this instability.

The versions tested are below:

1.
depthai==2.0.0.1+4450dc3a7246f6d80f143c57db3f7f797f677882
segmentation fault after ~2 minutes
2.
depthai==2.0.0.1+31e14bdf1d2de39e85ab2d4806ad63f28970baf2
    h264Packet = q.get()  # blocking call, will wait until a new data has arrived
RuntimeError: Communication exception - possible device error/misconfiguration. Original message 'Couldn't read data from stream: 'h265' (X_LINK_ERROR)'
3.
depthai==0.0.2.1+c9a19df719cb668e438d6eafd193cdf60a0d9354
    h264Packet = q.get()  # blocking call, will wait until a new data has arrived
RuntimeError: Communication exception - possible device error/misconfiguration. Original message '[json.exception.parse_error.110] parse error at byte 1: syntax error while parsing MessagePack value: unexpected end of input'
4.
depthai==0.0.2.1+1adf28f17fc3c5409f8226f309eb7122a7eb24bb
    h264Packet = q.get()  # blocking call, will wait until a new data has arrived
RuntimeError: Communication exception - possible device error/misconfiguration. Original message 'Error while reading data from xlink channel: h265 (X_LINK_ERROR)'
5.
depthai==0.0.2.1+e9930fcf2bf2f6d81b7c5ba8455e9d2f6a8d8685
    h264Packet = q.get()  # blocking call, will wait until a new data has arrived
RuntimeError: Communication exception - possible device error/misconfiguration. Original message 'Error while reading data from xlink channel: h265 (X_LINK_ERROR)'
6.
depthai==0.0.2.1+9ad44703a6d25d42d1418cc8660f19a8ed8d5346
    h264Packet = q.get()  # blocking call, will wait until a new data has arrived
RuntimeError: Communication exception - possible device error/misconfiguration. Original message 'Error while reading data from xlink channel: h265 (X_LINK_ERROR)'

And we have found that Gen1 does not suffer from this issue on Raspberry Pi OS. We are continuing to debug.

Raspberry Pi OS version where we first noticed the problem:

OS version:
VERSION="10 (buster)"
kernel version:
Linux raspberrypi 5.10.17-v7+ #1403 SMP Mon Feb 22 11:29:51 GMT 2021 armv7l GNU/Linux

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 22 (10 by maintainers)

Most upvoted comments

I am also facing the same error on Raspberry Pi OS on gen2 API. Please let me know how to solve this error: ‘Error while reading data from xlink channel: (my_nn) (X_LINK_ERROR)’ and I am using Linux raspberry pi 5.10.17

I need some suggestions from you guys:

So should I downgrade from pi 5.10.17, and will it solve the error?
Or should I wait for the new update to come, and how long will it take?
Also, sometimes it crashes (throwing errors), and sometimes it’s not crashing at all.
One of the errors I m getting is the device not found even when the device is connected properly.

with dai.Device(pipeline) as device:
RuntimeError: Failed to find device after booting, error message: X_LINK_DEVICE_NOT_FOUND ---- [After running the code, it freeze my Raspberry Pi and gives this error, and also I m getting this error when I run my program first time, when I start my Raspberry Pi, or when I removed connect my OAC-D to Pi]

uchihatashi on Apr 16, 2021

So we are learning more as we dig in. We have 2 engineers trying to figure out the crash behavior here. One is full-time on this and the other wrapping up https://github.com/luxonis/depthai-core/issues/94 while testing.

What we’ve found so far is that it doesn’t crash with non-blocking queue IIRC, which was the default behaviour in gen1 (if we remember correctly). We were excited that we thought we found an is that was culprit, but it was from the previous PR. (So that was found/fixed at least.)

As @aviogit mentioned, it definitely something related to the fact that the packets are not consumed in time by the host, on RPI4 is working fine, while RPI3 queues get full.

And yes, @jasonopto 's case it was because we do not yet have Pi Zero wheels prebuilt (which is largely because the thing is so slow/hard to work with, ha, like @jasonopto mentioned - we’d need to figure out some reliable cross or cloud-compile for armv6/PiZero).

Luxonis-Brandon on Apr 14, 2021

I don’t know if it can be a useful information @jasonopto, but 04_rgb_encoding.py works perfectly on my RPi4. I’m using the official Raspbian 5.10.9-v7l+ kernel (but after that everything else is Ubuntu 20.04 x86).

aviogit on Apr 14, 2021

VLC fails to play .h264/.h265 videos but mplayer (or MX Player on Android) can reproduce them. However, for every frame, it spams:

No pts value from demuxer to use for frame! pts after filters MISSING

Also, it would be a real plus on the Pi to have video playable by Chrome/Firefox because this would allow serving videos via any web server, the ideal configuration for watching captured videos on the field.

aviogit on May 19, 2021

USB transfer is definitely more stable @Luxonis-Brandon; no XLINK crashes yet FYI the video is unwatchable on the rpi3 using VLC; had to transfer to Windows PC for playback to work There is a ffmpeg mux “fix your code to set timestamps properly” message but unclear if related to rpi3 playback failure Also, ./install_requirements.py might be missing ‘sudo apt-get install libatlas-base-dev’ package

chris-piekarski on May 19, 2021

We think we figured this one out. We’re testing now and it’s looking good. If anyone else is interested in testing it, see below:

https://github.com/luxonis/depthai-python/actions/runs/850855986 If someone has free rpi3/4 etc. when CI finishes (or build from sources), can you set up for test?

git fetch --all
git checkout origin/rpi_crash_mitigation
cd examples
./install_requirements.py
./04_rgb_encoding.py

So far running for more than 1 hour on rpi3 (although cpp, but shouldn’t matter). It will write a lot to the SD card, so optionally redirect output to /dev/null.

Luxonis-Brandon on May 17, 2021

Ok! It worked thanks! Didnt know what usb rules was. Thanks

diegomarvid on Apr 29, 2021

Ok sir, @Luxonis-Brandon I will try to downgrade it and check whether it’s running properly or not. I am also thinking of using ubuntu-OS on Pi because I m not getting an error when I run the same code on my laptop (ubuntu v18). @dhruvsheth-ai sure!!!

uchihatashi on Apr 21, 2021

Great question @jasonopto . So I didn’t write down what was on my Pi that was working, unfortunately. So we’ll need to test. But I do know that we tested both 5.4.83 and 5.10.17 and they both have the problem (see here for a great version chart). I think I was most likely running 5.4.51 (2020-08-20), but I’m checking. We have logs as to what was flashed. Will circle back.

Luxonis-Brandon on Apr 12, 2021

What’s the latest os verion that worked with Depthai? It’ll be great to have a link to it for download/install.

jasonopto on Apr 12, 2021