depthai-core: OAK POE: 'Couldn't read data from stream: 'right' (X_LINK_ERROR)'

We have stability issues running the OAK-FFC-PoE-3P in our test bench. After some hours, typically 12-18 hours, image delivery will crash with the error message: Communication exception - possible device error/misconfiguration. Original message ‘Couldn’t read data from stream: ‘right’ (X_LINK_ERROR)’

Catching the exception and initializing the pipeline again works 2-3 times, then a power cycle is always needed to connect again.

At booting it also have issues with finding the device: X_LINK_DEVICE_NOT_FOUND, see log below.

SW config: Testing with the latest 2.15 release candidate The test script just grabs images 800P from all 3 cameras: right, left, rgb and show the images on the screen. FPS: 10 FPS. Ubuntu 20.04

HW config: Board connected to standard Arducam cameras (OAK-FFC-OV9282-M12 (Mono, 22 pin), and OAK-FFC-IMX477-M12) The host also have a M2 card with two additional Movidius chip. (not used)

See log below

~/code$ python3 test_cameras.py 
2.15.0.0.dev+777440adcd592f9c5dadb15c86fdda8eade08e8b
2022-02-24 07:42:35.038265
Found:  [<depthai.DeviceInfo object at 0x7fd7bfa431f0>, <depthai.DeviceInfo object at 0x7fd7c0a95fb0>, <depthai.DeviceInfo object at 0x7fd7bfa6b430>]
Exception------------------------------------------
2022-02-24 07:43:13.137641
Failed to find device after booting, error message: X_LINK_DEVICE_NOT_FOUND
Found:  [<depthai.DeviceInfo object at 0x7fd7bfaa5830>, <depthai.DeviceInfo object at 0x7fd7c0a91530>, <depthai.DeviceInfo object at 0x7fd7bfaaad30>]
Exception------------------------------------------
2022-02-24 07:44:04.565841
Failed to find device after booting, error message: X_LINK_DEVICE_NOT_FOUND
Found:  [<depthai.DeviceInfo object at 0x7fd7c0ad26f0>, <depthai.DeviceInfo object at 0x7fd7bfa3c8f0>, <depthai.DeviceInfo object at 0x7fd7bfa68370>]
Exception------------------------------------------
2022-02-24 07:44:56.550247
Failed to find device after booting, error message: X_LINK_DEVICE_NOT_FOUND
Found:  [<depthai.DeviceInfo object at 0x7fd7c0a95fb0>, <depthai.DeviceInfo object at 0x7fd7c0a959b0>, <depthai.DeviceInfo object at 0x7fd7bfa68870>]
No image from Left
2022-02-24 07:45:32.321267
No image from right
2022-02-24 07:45:32.321292
No image from Left
2022-02-24 07:45:32.489247
No image from right
2022-02-24 07:45:32.489298
Exception------------------------------------------
2022-02-24 17:10:24.965580
Communication exception - possible device error/misconfiguration. Original message 'Couldn't read data from stream: 'right' (X_LINK_ERROR)'
Found:  [<depthai.DeviceInfo object at 0x7fd7bfa33570>, <depthai.DeviceInfo object at 0x7fd7bfaa5830>, <depthai.DeviceInfo object at 0x7fd7bfa43ab0>]
Exception------------------------------------------
2022-02-24 17:11:16.450010
Failed to find device after booting, error message: X_LINK_DEVICE_NOT_FOUND
Found:  [<depthai.DeviceInfo object at 0x7fd7c0ad26f0>, <depthai.DeviceInfo object at 0x7fd7bfae05b0>, <depthai.DeviceInfo object at 0x7fd7bfa33d70>]
No image from Left
2022-02-24 17:11:50.231063
No image from right
2022-02-24 17:11:50.231084
No image from right
2022-02-24 17:11:50.292075
No image from Left
2022-02-24 17:11:50.363400
Exception------------------------------------------
2022-02-24 23:34:23.171944
Communication exception - possible device error/misconfiguration. Original message 'Couldn't read data from stream: 'right' (X_LINK_ERROR)'
Found:  [<depthai.DeviceInfo object at 0x7fd7bfa434f0>, <depthai.DeviceInfo object at 0x7fd7c0a959b0>]
Found:  [<depthai.DeviceInfo object at 0x7fd7c0ad26f0>, <depthai.DeviceInfo object at 0x7fd7bfaaad30>]
Found:  [<depthai.DeviceInfo object at 0x7fd7bfa434f0>, <depthai.DeviceInfo object at 0x7fd7c0a959b0>]
Found:  [<depthai.DeviceInfo object at 0x7fd7c0ad26f0>, <depthai.DeviceInfo object at 0x7fd7bfaaad30>]

It will now forever try to find the OAK device.