firecracker: vsock: error reading from backing stream
While developing the vhost-user-vsock application for cloud-hypervisor, I had an issue that happens also with the vsock device emulation: https://github.com/cloud-hypervisor/cloud-hypervisor/issues/1001
As @sboeuf suggested, I tried with Firecracker and I had a very similar issue:
2020-04-06T11:52:33.632842533 [anonymous-instance:ERROR:src/devices/src/virtio/vsock/csm/connection.rs:242] vsock: error reading from backing stream: lp=5201, pp=1031, err=Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavailable" }
After the error, the throughput goes to 0 and the peers are able to communicate only restarting the connection, but the error happens every time I tried to run iperf-vsock (guest connecting to the host)
How to reproduce
I used iperf-vsock to stress the vsock connection.
Kernel: hello-vmlinux.bin (Linux (none) 4.14.55-84.37.amzn2.x86_64 #1 SMP Wed Jul 25 18:47:15 UTC 2018 x86_64 Linux) Firecracker: firecracker-v0.21.1-x86_64
VM config
{
"boot-source": {
"kernel_image_path": "./hello-vmlinux.bin",
"boot_args": "console=ttyS0 reboot=k panic=1 pci=off"
},
"drives": [
{
"drive_id": "rootfs",
"path_on_host": "rootfs.ext4",
"is_root_device": true,
"is_read_only": false
}
],
"machine-config": {
"vcpu_count": 2,
"mem_size_mib": 1024,
"ht_enabled": false
},
"vsock": {
"vsock_id": "1",
"guest_cid": 4,
"uds_path": "/tmp/vm4.vsock"
}
}
guest
~# mkdir /tmp # iperf3 uses mkstemp(3)
~# iperf3 --vsock -c 2
Connecting to host 2, port 5201
[ 5] local 4 port 1033 connected to 2 port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 91.0 MBytes 763 Mbits/sec
2020-04-06T11:55:51.001682775 [anonymous-instance:ERROR:src/devices/src/virtio/vsock/csm/connection.rs:242] vsock: error reading from backing stream: lp=5201, pp=1033, err=Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavailable" }
[ 5] 1.00-2.00 sec 422 MBytes 3.54 Gbits/sec
[ 5] 2.00-3.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 3.00-4.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 4.00-5.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 5.00-6.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 6.00-7.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 7.00-8.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 8.00-9.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 9.00-10.00 sec 0.00 Bytes 0.00 bits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 513 MBytes 431 Mbits/sec sender
[ 5] 0.00-10.01 sec 513 MBytes 430 Mbits/sec receiver
iperf Done.
host
> ./iperf3 --vsock -s -B /tmp/vm4.vsock
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 40:8c44:a220:e9bb:30b4:aa01::, port 43521
[ 5] local 6d34:2e76:736f:636b:5f35:3230:3100:0 port 12148 connected to :: port 0
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 431 MBytes 3.61 Gbits/sec
[ 5] 1.00-2.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 2.00-3.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 3.00-4.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 4.00-5.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 5.00-6.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 6.00-7.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 7.00-8.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 8.00-9.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 9.00-10.00 sec 0.00 Bytes 0.00 bits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 431 MBytes 361 Mbits/sec receiver
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 47 (12 by maintainers)
@stefano-garzarella sounds like we can close this issue for now, thanks for all your help 😃
@sboeuf yes, for testing the vhost-user-vsock application. When the patches will be merged, I’ll send a PR to Cloud-Hypervisor.
@stefano-garzarella, I pushed a new PR version which seems to work so far.
I started a test with all the combinations that you also ran -
host to guestandguest to hostin both reverse and normal mode:The microVMs have been running for a while on a Xeon 8175M based server without any crashes or stalls, but as I was mentioning before, there seem to be more frequent problems when running on processors that have better per-core performance than this one.
Could you try these patches on your side?
@stefano-garzarella at one point during the investigation, I rewrote that function to only be called once, but I still hit the stalling issue. But perhaps at that moment, I was hitting one of the previously fixed problems and eventually reverted it.
It’s a good catch to check its logic. I’ll look into it.
@stefano-garzarella yes, it seems that for some reason when using musl it triggers more frequently.
I was basically trying to figure out if using different setups has a higher incidence of triggering the issue, so I ran about 10 iterations of starting two Firecracker 0.21.1 VMs in parallel (one built with musl and the other with gnu) and started
iperf-vsockin the host as a server. In 8 out of the 10 runs, the iperf instance in the musl build stopped generating traffic long before the gnu build - less than 30s vs. about 5 minutes.Most probably both libc libraries work fine, but for some reason the musl build makes it easier to trigger this issue.
@stefano-garzarella we recently added musl support to Cloud-Hypervisor, so this might help reproducing the issue as well.
@stefano-garzarella, I also noticed slight differences when linking the Firecracker binary against musl/glibc, with the musl build showing more consistent failures.
@sboeuf, I’ve also started looking into the issue.
I managed to reproduce the broken pipe issue after a few tries, but so far it only happened after about 12 minutes when using the 5.6 kernel.
@stefano-garzarella I’m having trouble replicating the problem. Configurations that I’ve tried so far:
hello-vmlinux.binAll combinations seem to work fine, even though with a pretty low throughput. Haven’t fiddled with buffer sizes yet though, which seems likely to be the cause.
Back to the errors, both of them seem pretty peculiar. In the first case, you get an unexpected EAGAIN / EWOULDBLOCK. That suggests the vsock back end received an EPOLLIN event for the unix socket, attempted to read data from it, yet still received EAGAIN. Seems weird, and I don’t know what could cause this (i.e. EPOLLIN followed by EAGAIN).
The second error seems even more bizarre (
TxBufFull). This would suggest that one of the peers is not respecting flow control. I.e. either Firecracker or the guest vsock driver is mishandling vsock flow control.Both situations require a more in-depth look, so I’d appreciate some more help with reproducing the issue. Thanks!
Hi @stefano-garzarella! Thanks for logging this. We’ll get back to you after we look into it.