depthai: [BUG] Multiple OAK-D devices not closing properly

From a mixture of AMD updates and latest DepthAI updates I can’t replicate this anymore - so for now I think this is safe to say is fixed, thanks for looking into this.

So after some cleanup of my code I have the following three python functions (it’s very similar to the https://github.com/luxonis/depthai-experiments/blob/master/gen2-multiple-devices/main.py code)

(Using cv2, depthai and contextlib as imports)

The create_pipeline function creates and returns a depthai pipeline that has colour and depth outputs called rgb and depth

def create_pipeline():
	# Start defining a pipeline
	pipeline = depthai.Pipeline()

	# Define a source - color camera
	cam_rgb = pipeline.createColorCamera()
	cam_rgb.setPreviewSize(600, 600)
	cam_rgb.setBoardSocket(depthai.CameraBoardSocket.RGB)
	cam_rgb.setResolution(depthai.ColorCameraProperties.SensorResolution.THE_1080_P)
	cam_rgb.setInterleaved(False)


	# Define a source - two mono (grayscale) cameras
	left = pipeline.createMonoCamera()
	left.setResolution(depthai.MonoCameraProperties.SensorResolution.THE_800_P)
	left.setBoardSocket(depthai.CameraBoardSocket.LEFT)

	right = pipeline.createMonoCamera()
	right.setResolution(depthai.MonoCameraProperties.SensorResolution.THE_800_P)
	right.setBoardSocket(depthai.CameraBoardSocket.RIGHT)

	left.setFps(60.0)
	right.setFps(60.0)


	# Define a source - depth camera
	depth = pipeline.createStereoDepth()
	depth.setConfidenceThreshold(200)
	left.out.link(depth.left)
	right.out.link(depth.right)
	depth.setOutputDepth(True)


	# Create output
	xout_rgb = pipeline.createXLinkOut()
	xout_rgb.setStreamName("rgb")
	cam_rgb.preview.link(xout_rgb.input)


	xout_depth = pipeline.createXLinkOut()
	xout_depth.setStreamName("depth")
	depth.depth.link(xout_depth.input)
	
	return pipeline

The get_all_cameras function takes a pipeline and returns a list of devices, a list of rgb source queues and a list of depth source queues.

def get_all_cameras(pipeline):
	devices = []
	
	rgb_sources = []
	depth_sources = []
	
	for device_info in depthai.Device.getAllAvailableDevices():
		device = depthai.Device(pipeline=pipeline, deviceDesc=device_info, usb2Mode=False)
		
		devices.append(device)
		
		print("Conected to " + device_info.getMxId())
		device.startPipeline()
		queue_rgb = device.getOutputQueue(name="rgb", maxSize=1, blocking=False)
		rgb_sources.append(queue_rgb)	
		queue_depth = device.getOutputQueue(name="depth", maxSize=1, blocking=False)
		depth_sources.append(queue_depth)
	
	return devices, rgb_sources, depth_sources

And finally the close_cameras function takes a list of devices and closes them (I was originally using the contextlib.ExitStack() approach described in the https://github.com/luxonis/depthai-experiments/blob/master/gen2-multiple-devices/main.py file, but I found this harder to troubleshoot which is why I swapped to this approach).

def close_cameras(devices):
	for device in devices:
		device.close()

Finally the wrapper code to run it all:

pipeline = create_pipeline()

devices, rgbs, depths = get_all_cameras(pipeline)

running = True
while running:	
	for i, q_rgb in enumerate(rgbs):
		in_rgb = q_rgb.tryGet()
		if in_rgb is not None:
			cv2.imshow("rgb-" + str(i + 1), in_rgb.getCvFrame())

	if cv2.waitKey(1) == ord('q'):
		running = False

close_cameras(devices)

In total the file looks like this:

import cv2
import depthai
import contextlib



def create_pipeline():
	# Start defining a pipeline
	pipeline = depthai.Pipeline()

	# Define a source - color camera
	cam_rgb = pipeline.createColorCamera()
	cam_rgb.setPreviewSize(600, 600)
	cam_rgb.setBoardSocket(depthai.CameraBoardSocket.RGB)
	cam_rgb.setResolution(depthai.ColorCameraProperties.SensorResolution.THE_1080_P)
	cam_rgb.setInterleaved(False)


	# Define a source - two mono (grayscale) cameras
	left = pipeline.createMonoCamera()
	left.setResolution(depthai.MonoCameraProperties.SensorResolution.THE_800_P)
	left.setBoardSocket(depthai.CameraBoardSocket.LEFT)

	right = pipeline.createMonoCamera()
	right.setResolution(depthai.MonoCameraProperties.SensorResolution.THE_800_P)
	right.setBoardSocket(depthai.CameraBoardSocket.RIGHT)

	left.setFps(60.0)
	right.setFps(60.0)


	# Define a source - depth camera
	depth = pipeline.createStereoDepth()
	depth.setConfidenceThreshold(200)
	left.out.link(depth.left)
	right.out.link(depth.right)
	depth.setOutputDepth(True)


	# Create output
	xout_rgb = pipeline.createXLinkOut()
	xout_rgb.setStreamName("rgb")
	cam_rgb.preview.link(xout_rgb.input)


	xout_depth = pipeline.createXLinkOut()
	xout_depth.setStreamName("depth")
	depth.depth.link(xout_depth.input)
	
	return pipeline


def get_all_cameras(pipeline):
	devices = []
	
	rgb_sources = []
	depth_sources = []
	
	for device_info in depthai.Device.getAllAvailableDevices():
		device = depthai.Device(pipeline=pipeline, deviceDesc=device_info, usb2Mode=True)
		
		devices.append(device)
		
		print("Conected to " + device_info.getMxId())
		device.startPipeline()
		# Output queue will be used to get the rgb frames from the output defined above
		queue_rgb = device.getOutputQueue(name="rgb", maxSize=1, blocking=False)
		rgb_sources.append(queue_rgb)	
		queue_depth = device.getOutputQueue(name="depth", maxSize=1, blocking=False)
		depth_sources.append(queue_depth)
	
	return devices, rgb_sources, depth_sources

def close_cameras(devices):
	for device in devices:
		device.close()
		

pipeline = create_pipeline()

devices, rgbs, depths = get_all_cameras(pipeline)

running = True
while running:	
	for i, q_rgb in enumerate(rgbs):
		in_rgb = q_rgb.tryGet()
		if in_rgb is not None:
			cv2.imshow("rgb-" + str(i + 1), in_rgb.getCvFrame())

	if cv2.waitKey(1) == ord('q'):
		running = False

close_cameras(devices)

I can confirm that it is primarily occuring when using devices on different USB controllers.

In this screenshot the first 3 executions were taken on my front connector USB ports (ones connected via a USB 3 header). At the red line I swap one of the cameras to use a USB directly attached to the motherboard and I have issues (the 2nd camera isn’t connecting when it’s run again). If I move them both to the motherboard USB then there is no issue.

Ideally they would be connected via different streams to not run into bandwidth limitations on a USB controller (this is a common issue with SteamVR basestations and tracker dongles overwhelming USB controllers), but I can accept running them this way.

Updated to the latest BIOS (including beta) and latest X570 chipset drivers and I can confirm the issue is still present. It does howerver go away now and both cameras connect after a couple of minutes of waiting - whereas before even after 30m they refused to reconnect.

From this I am guessing it is more than likely a compatiblity issue with X570 drivers and the Windows USB library you are using.

I have a open support ticket with AMD about other platform instabilities and I will raise this as an example if that’s okay with you? I can’t promise that it will go anywhere.

Thanks, John

Hi Brandon,

Testing on:

• AMD Ryzen 5950x
• Corsair Vengeance DDR4 2x16GB (for a total of 32GB) RAM CL16
• 2x Samsung 980 Pro 2TB SSD
• ASUS AMD Ryzen X570 ROG Crosshair VIII Dark Hero
• Nvidia RTX 3090 FE

Using combinations of the USB cables:

• USBs included with the cameras
• USB-C Oculus Link Cable (This cable validates in Oculus’ software with a bandwidth >2Gbps, and is in general very high quality fibre optic cable with copper wire power delivery)
• USB 2.0 1m USB-C to USB-C cable (I used this to validate that USB2.0 was error free even when in auto mode)

I’m using the Fractal Meshify 2 Compact case, which has 2 front panel USB 3.2Gen1 and a front type 3.2Gen2 Type C (I think the naming conventions on those are right). If I connect the both cameras using any assortments of cables to the motherboard USBs that are wired to the processor, both to the chipset USBs, both to the front panel USBs or to a front panel and a front panel USB-C I have very few issues (I would estimate similar to the 1:50 rate Erik mentioned above). If I connect so that one is via the processor and one via the chipset in any capacity (i.e. one on the motherboard USBs and one on the front panel) I have the issue almost every time, so I am leaning towards this being a chipset compatibility issue.

I’ll try to update to the latest chipset drivers and a beta BIOS tomorrow and see if that fixes the issues magically.

If you’re looking to build the system yourself I would advise against it:

The AM4 platform (specially X570) has USB driver issues (surprise surprise) which I suspect might be playing a role in this issue. From all the reading I have done on this it doesn’t sound like the USB driver issues that plague AM4 are the root cause (since I also validated it happening on an older Intel Z170 system), but it could be why its consistent for me. The driver issues should result in slower speed or disconnects, not the inability to connect.

Also, Samsung NVME drives can cause Bluescreens while on AM4 systems, so you have to manage your Windows PCIe power management settings to be high performance to avoid the issue - something that means if you have other PCIe devices attached (i.e. network cards, GPUs) it will cause them to also get this high performance power setting which can stop some of them from idling.

Finally, my biggest issue is that the 5950x, 5900x and to a lesser extent the 5800x all have issues with WHEA errors causing processor RMAs (which annoyingly is what plagues my current system). The ‘fix’ is to stop all boosting/overclocking/idling functionality (so the processor is locked at stock frequencies), which is what I am currently doing, and to request an RMA. I don’t think that is the cause of this issue as it just causes outright system crashes instead of driver or connected device instability.

Long story short I was using Intel Z170 as a work machine for the last 6ish years with literally 0 issues. I have a new workload that required much greater GPU and CPU resources so I went for the best in slot on both and have had significant issues since upgrading. I can’t recommend the 5000 series AMD platform as stable. It’s frustrating as if Intel had a competitive high core count processor I would have chosen that instead, but I am left with no option but AMD for my usecase.

I’ll grab the code I am using towards the end of the day (working my day job at the moment 😄).

My code is just trying to grab 2 cameras depth feeds at the moment.

Basically my end game goal is to grab the depth feed of 2/3 cameras in as close to real time and as high framerate (with API v2 I can’t seem to get a depth output at >60fps, but I haven’t really messed around with this either) as possible, then convert it to a point cloud.

I want to then run multiway registration (http://www.open3d.org/docs/latest/tutorial/pipelines/multiway_registration.html) on the pointclouds (I’m only using pointclouds here because the documentation is for pointclouds, but in the long run I want to skip this and just use depth images directly) to create a combined 3D point cloud akin to: https://www.youtube.com/watch?v=ZXEWylZ9srM

There are loads of applications for a real-time scanned 3D scene like this, use in fashion to preview clothes on people, use in VR (how I wish to use it) for better room scale tracking and 3D object identification and pose estimation (such as identifying and allowing you to pass through objects into VR like water bottles, tennis rackets, pets - so you don’t stand on them!) and if you could drone mount the cameras with this setup you could do full 3D scans of buildings and environments with ease.

Apologies, just saw the end of your reply:

One workaround would be to just wait for the amount of devices you have connected, or get IDs of all connected devices and try to “manually” connect to them one-by-one. We will continue to monitor this issue.

If I try this I get the following:

RuntimeError: Failed to find device (14442C1041A809D100-ma2480), error message: X_LINK_DEVICE_NOT_FOUND