moby: Application running in Docker container fails to reconnect after USB devices unplugged/replugged

Description

I’ve created an ASP.NET Core web-service that runs on a Raspberry Pi 3 Model B which acts as a proxy for USB devices, allowing clients to send/receive messages to/from USB devices connected to the RPi. Part of the job of this application is to be resilient to communication failures - if the device stops working or the USB cable is unplugged, the application waits for it to become available again and then reconnects to it. All USB communication is done through LibUsbDotNet, which delegates to libusb on Unix environments like an RPi.

This application works without issue when running natively on the RPi. However, I encounter issues with reconnecting to USB devices when running this application within a Docker container (running in privileged mode and mounting /dev/bus/usb as a volume).

It might be that this issue boils down to me just not mounting everything I need to within the Docker container for it to handle device reconnections using libusb.

Describe the results you received:

When running in a Docker container my application can see USB devices getting disconnected/reconnected (via device files appearing/disappearing under /dev/bus/usb) without any problems, but consistently fails to re-establish communication with a USB device after it is initially disconnected/reconnected.

An example of the log output I get when disconnecting/reconnecting the device with the application running within a container:

...
comms_1       | 2017-10-31 17:54:17.571 [VRB] [5] MyCompany.Communications.Web.Service.Monitoring.PeriodicBroadcaster   Broadcasted "Heartbeat (Up)"
comms_1       | 2017-10-31 17:54:17.756 [VRB] [16] MyCompany.Communications.UsbDevice.Monitoring.DevMonitor   USB device removed (VID: MyVid, PID: 1)
comms_1       | 2017-10-31 17:54:17.777 [DBG] [16] MyCompany.Communications.UsbDevice.RestartableUsbCommunicationService   USB event concerning device occurred
comms_1       | 2017-10-31 17:54:17.824 [VRB] [16] MyCompany.Communications.UsbDevice.UsbCommunicationService   Sending "Id: 32, Payload: 0" to "testcube"
comms_1       | 2017-10-31 17:54:17.854 [ERR] [16] MyCompany.Communications.TestCubeUsb.TestCubeCommunicationService   Failed to unlock comms on TestCube
comms_1       | 2017-10-31 17:54:17.911 [INF] [16] MyCompany.Communications.UsbDevice.UsbCommunicationService   Resetting "testcube" as USB device
comms_1       | 2017-10-31 17:54:17.918 [INF] [16] MyCompany.Communications.UsbDevice.UsbCommunicationService   Terminated communication with "testcube"
comms_1       | 2017-10-31 17:54:18.071 [DBG] [5] MyCompany.Utilities.AspNetCore.WebSockets.WebSocketBehavior   Published "{\"CommunicationState\":\"Down\",\"Id\":\"heartbeatEvent\"}" over WebSocket
comms_1       | 2017-10-31 17:54:18.072 [VRB] [5] MyCompany.Communications.Web.Service.Monitoring.PeriodicBroadcaster   Broadcasted "Heartbeat (Down)"
...
comms_1       | 2017-10-31 17:54:22.589 [VRB] [11] MyCompany.Communications.Web.Service.Monitoring.PeriodicBroadcaster   Broadcasted "Heartbeat (Down)"
comms_1       | 2017-10-31 17:54:22.640 [VRB] [16] MyCompany.Communications.UsbDevice.Monitoring.DevMonitor   USB device connected (VID: MyVid, PID: 1)
comms_1       | 2017-10-31 17:54:22.650 [DBG] [16] MyCompany.Communications.UsbDevice.RestartableUsbCommunicationService   USB event concerning device occurred
comms_1       | 2017-10-31 17:54:22.677 [DBG] [16] MyCompany.Communications.Core.RestartableCommunicationService   Attempting to start/restart communication.
comms_1       | 2017-10-31 17:54:22.707 [ERR] [19] MyCompany.Communications.UsbDevice.UsbCommunicationService   Cannot find "testcube"
comms_1       | 2017-10-31 17:54:22.715 [ERR] [19] MyCompany.Communications.Core.RestartableCommunicationService   Failed to start/restart communication.
...

Describe the results you expected:

I expect that my application functions identically when running in a Docker container as it does when running natively - when it notices a USB device gets reconnected, it is successfully able to re-establish communication with it.

Log output from the same scenario performed when running the application natively on my RPi:

...
2017-11-01 09:57:29.512 [VRB] [19] MyCompany.Communications.Web.Service.Monitoring.PeriodicBroadcaster   Broadcasted "Heartbeat (Up)"
2017-11-01 09:57:29.689 [VRB] [15] MyCompany.Communications.UsbDevice.Monitoring.DevMonitor   USB device removed (VID: MyVid, PID: 1)
2017-11-01 09:57:29.711 [DBG] [15] MyCompany.Communications.UsbDevice.RestartableUsbCommunicationService   USB event concerning device occurred
2017-11-01 09:57:29.757 [VRB] [15] MyCompany.Communications.UsbDevice.UsbCommunicationService   Sending "Id: 32, Payload: 0" to "testcube"
2017-11-01 09:57:29.784 [ERR] [15] MyCompany.Communications.TestCubeUsb.TestCubeCommunicationService   Failed to unlock comms on TestCube
2017-11-01 09:57:29.841 [INF] [15] MyCompany.Communications.UsbDevice.UsbCommunicationService   Resetting "testcube" as USB device
2017-11-01 09:57:29.848 [INF] [15] MyCompany.Communications.UsbDevice.UsbCommunicationService   Terminated communication with "testcube"
2017-11-01 09:57:30.018 [DBG] [5] MyCompany.Utilities.AspNetCore.WebSockets.WebSocketBehavior   Published "{\"CommunicationState\":\"Down\",\"Id\":\"heartbeatEvent\"}" over WebSocket
2017-11-01 09:57:30.023 [VRB] [5] MyCompany.Communications.Web.Service.Monitoring.PeriodicBroadcaster   Broadcasted "Heartbeat (Down)"
...
2017-11-01 09:57:33.050 [VRB] [5] MyCompany.Communications.Web.Service.Monitoring.PeriodicBroadcaster   Broadcasted "Heartbeat (Down)"
2017-11-01 09:57:33.301 [VRB] [15] MyCompany.Communications.UsbDevice.Monitoring.DevMonitor   USB device connected (VID: MyVid, PID: 1)
2017-11-01 09:57:33.309 [DBG] [15] MyCompany.Communications.UsbDevice.RestartableUsbCommunicationService   USB event concerning device occurred
2017-11-01 09:57:33.639 [DBG] [5] MyCompany.Communications.Core.RestartableCommunicationService   Attempting to start/restart communication.
2017-11-01 09:57:33.661 [INF] [5] MyCompany.Communications.UsbDevice.UsbCommunicationService   Successfully initiated communication with "testcube"
2017-11-01 09:57:33.663 [VRB] [5] MyCompany.Communications.UsbDevice.UsbCommunicationService   Sending "Id: 32, Payload: 131" to "testcube"
2017-11-01 09:57:34.060 [DBG] [5] MyCompany.Utilities.AspNetCore.WebSockets.WebSocketBehavior   Published "{\"CommunicationState\":\"Up\",\"Id\":\"heartbeatEvent\"}" over WebSocket
2017-11-01 09:57:34.062 [VRB] [5] MyCompany.Communications.Web.Service.Monitoring.PeriodicBroadcaster   Broadcasted "Heartbeat (Up)"
...

Steps to reproduce the issue: Due to the nature of this issue I don’t think it would be simple to reproduce this from scratch, but the general outline is:

Create an ASP.NET Core 2.0 web application that uses LibUsbDotNet for USB communication and USB system event notifications.
Publish the application targeting the architecture of a Raspberry Pi 3 Model B:
```
dotnet publish -c Release -r linux-arm -o /app/dist
```

Build a Docker image derived from microsoft/dotnet:2.0-runtime-stretch-arm32v7 to run this application:

FROM microsoft/dotnet:2.0-runtime-stretch-arm32v7
ENTRYPOINT ["dotnet", "Example.Web.Service.dll"]
WORKDIR /app
EXPOSE 36363
COPY dist ./
RUN apt-get update
RUN apt-get install -y libusb-1.0-0-dev usbutils
RUN ln -s /lib/arm-linux-gnueabihf/libusb-1.0.so.0 libusb-1.0.dll

Transfer this image to a Linux environment like a Raspberry Pi if not already on it. I can’t check if this issue occurs on my local Windows 10 development machine as my application does not support monitoring USB system events on Windows platforms.
Connect a USB device to the Linux machine.

Launch a container from this image on the Linux machine

docker run --privileged -v /dev/bus/usb:/dev/bus/usb example_web_service

Have the application establish communication with a USB device (ideally confirm this via logs from the Docker container).
Unplug the USB cable to disconnect the device. Confirm via logs that communication has dropped, and that the device file for the device has been removed from /dev/bus/usb.
Reconnect the USB cable. Confirm via logs that the Docker container sees the device file for the device reappear under /dev/bus/usb. Confirm that the application fails to reconnect to the device in spite of this.

Additional information you deem important (e.g. issue happens only occasionally):

There’s additional information available in LibUsbDotNet/LibUsbDotNet#24.

Output of docker version:

Client:
 Version:      17.05.0-ce
 API version:  1.29
 Go version:   go1.7.5
 Git commit:   89658be
 Built:        Thu May  4 22:28:29 2017
 OS/Arch:      linux/arm

Server:
 Version:      17.05.0-ce
 API version:  1.29 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   89658be
 Built:        Thu May  4 22:28:29 2017
 OS/Arch:      linux/arm
 Experimental: false

Output of docker info:

Containers: 2
 Running: 0
 Paused: 0
 Stopped: 2
Images: 3
Server Version: 17.05.0-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9048e5e50717ea4497b757314bad98ea3763c145
runc version: 9c2d8d184e5da67c95d601382adf14862e4f2228
init version: 949e6fa
Kernel Version: 4.1.19-v7+
Operating System: Ubuntu 16.04.3 LTS
OSType: linux
Architecture: armv7l
CPUs: 4
Total Memory: 925.8MiB
Name: [redacted]
ID: D2EX:NTM5:3KMF:IIEI:FZ5U:E3PA:AXHG:CRUF:XK4L:AUET:LPFX:SWRL
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Username: [redacted]
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No memory limit support
WARNING: No swap limit support
WARNING: No kernel memory limit support
WARNING: No oom kill disable support
WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support

Additional environment details (AWS, VirtualBox, physical, etc.):

Output of uname -a on Raspberry Pi:

Linux [redacted] 4.1.19-v7+ #858 SMP Tue Mar 15 15:56:00 GMT 2016 armv7l armv7l armv7l GNU/Linux

About this issue

Original URL
State: open
Created 7 years ago
Reactions: 1
Comments: 15 (1 by maintainers)

Most upvoted comments

same problem here.

jacknlliu on Nov 14, 2017

I ran across this issue while looking for a solution to a problem I’m having. Was any solution found?

I’m running a Ubuntu 18.04 container in a CI environment that interacts with USB hardware. As part of the test, the power to the USB device is turned off and back on again. Whenever this test takes place, the the container OS never rediscovers the device.

EDIT: The solution in our case was to add the following docker volume: -v /dev/bus/usb/:/dev/bus/usb/

daheise on Sep 26, 2019

In fact, we already used --privileged & /dev:/dev, meanwhile, what we used is just usb bus in /dev/bus/usb/, no /dev/.../by-id/... for the device, looks more like a scenario android adb use.

Finally, we found the answer here: https://stackoverflow.com/questions/49687378/how-to-get-hosts-udev-events-from-a-docker-container

In our application, we need udev/netlink support when reconnect happen, NETLINK is the way to communicate between kernel & user space, as container default not in the same space of host, so issues there.

So, the solution is as the answer suggested to use --net=host, also I post a new answer which works for us, that is enable udev in container.

Maybe others’ issue not same as mine, just put it here for new comers in case helpful.

atline on May 28, 2020

3 years passed. Any update on this？ We encountered the same issue, we have -v /dev:/dev --privileged added when start ubuntu:18.04 container. There is /dev/bus/usb/003/065 in container before device leave, then the device back, we could see it becomes /dev/bus/usb/003/066, but the application with libusb support no longer able to connect it again? This only happens in container, not in host environment, any suggestion? @thaJeztah

atline on May 27, 2020