podman: pasta: port range forwarding: mismatch between data sent and received
not ok 434 podman networking with pasta(1) - TCP port range forwarding, IPv4, loopback
...
$ podman run --net=pasta -p [127.0.0.1]:5776-5778:5776-5778/tcp quay.io/libpod/testimage:20221018 sh -c for port in $(seq 5776 5778); do socat -u TCP4-LISTEN:${port},bind=[127.0.0.1] STDOUT & done; wait
2023/01/26 17:23:46 socat[119766] E connect(8, AF=2 127.0.0.1:5777, 16): Interrupted system call
xx
#/vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
#| FAIL: Mismatch between data sent and received
#| expected: = 'xxx'
#| actual: 'xx'
#\^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Also happens in a different test, same symptom (xxx vs xx), same socat EINTR
- fedora-36 : sys podman fedora-36 rootless host
- PR #17242
- 01-26 19:04 in [sys] podman networking with pasta(1) - TCP port range forwarding, IPv4, loopback
- PR #17242
- fedora-37 : sys remote fedora-37 rootless host [remote]
- PR #17242
- 01-26 19:10 in [sys] podman networking with pasta(1) - TCP translated port range forwarding, IPv4, tap
- PR #17242
About this issue
- Original URL
- State: closed
- Created a year ago
- Comments: 20 (3 by maintainers)
Commits related to this issue
- pasta: Workaround occasional socat failures in CI With a number of the port range forwarding tests, we've seen occasional failures where the sending socat fails with an EINTR on connect(). This was ... — committed to dgibson/podman by dgibson a year ago
- Revert "Revert "Revert "pasta: Use two connections instead of three in TCP range forward tests""" This reverts commit c2a24abc0d33cdb21681cfec63677ac29e92446c, which itself reverted 1c08f2edac3f9ecf1... — committed to dgibson/podman by dgibson a year ago
- pasta: Workaround occasional socat failures in CI With a number of the port range forwarding tests, we've seen occasional failures where the sending socat fails with an EINTR on connect(). This was ... — committed to dgibson/podman by dgibson a year ago
- Revert "Revert "Revert "pasta: Use two connections instead of three in TCP range forward tests""" This reverts commit c2a24abc0d33cdb21681cfec63677ac29e92446c, which itself reverted 1c08f2edac3f9ecf1... — committed to dgibson/podman by dgibson a year ago
- Revert^3 "pasta: Use two connections instead of three in TCP range forward tests" This reverts commit c2a24abc0d33cdb21681cfec63677ac29e92446c, which itself reverted 1c08f2edac3f9ecf128cf8da91276e963... — committed to dgibson/podman by dgibson a year ago
- pasta: Workaround occasional socat failures in CI With a number of the port range forwarding tests, we've seen occasional failures where the sending socat fails with an EINTR on connect(). This was ... — committed to cgiradkar/podman by dgibson a year ago
- Revert^3 "pasta: Use two connections instead of three in TCP range forward tests" This reverts commit c2a24abc0d33cdb21681cfec63677ac29e92446c, which itself reverted 1c08f2edac3f9ecf128cf8da91276e963... — committed to cgiradkar/podman by dgibson a year ago
- pasta: Workaround occasional socat failures in CI With a number of the port range forwarding tests, we've seen occasional failures where the sending socat fails with an EINTR on connect(). This was ... — committed to cgiradkar/podman by dgibson a year ago
- Revert^3 "pasta: Use two connections instead of three in TCP range forward tests" This reverts commit c2a24abc0d33cdb21681cfec63677ac29e92446c, which itself reverted 1c08f2edac3f9ecf128cf8da91276e963... — committed to cgiradkar/podman by dgibson a year ago
- pasta: Workaround occasional socat failures in CI With a number of the port range forwarding tests, we've seen occasional failures where the sending socat fails with an EINTR on connect(). This was ... — committed to cgiradkar/podman by dgibson a year ago
- Revert^3 "pasta: Use two connections instead of three in TCP range forward tests" This reverts commit c2a24abc0d33cdb21681cfec63677ac29e92446c, which itself reverted 1c08f2edac3f9ecf128cf8da91276e963... — committed to cgiradkar/podman by dgibson a year ago
- pasta: Workaround occasional socat failures in CI With a number of the port range forwarding tests, we've seen occasional failures where the sending socat fails with an EINTR on connect(). This was ... — committed to cgiradkar/podman by dgibson a year ago
- Revert^3 "pasta: Use two connections instead of three in TCP range forward tests" This reverts commit c2a24abc0d33cdb21681cfec63677ac29e92446c, which itself reverted 1c08f2edac3f9ecf128cf8da91276e963... — committed to cgiradkar/podman by dgibson a year ago
Ok, I think I know what’s going on. It looks like a bug in
socatto me, but there is a pretty easy workaround.Viewing the problem under strace proved surprisingly tricky: I wasn’t able to strace the entire bats run, because once I had a large enough number of ports that it reproduced frequently, strace slowed it down enough to just grind to a halt. But with a variety of tricks I eventually managed it.
It appears that the
connect()in the “client” socat is being interrupted from aSIGCHLDas theprintf xit invoked to generate the input exits. As I say, that seems like a bug insocat, I guess it’s some of race.The workaround is not to use an
EXEC:address for socat, but use anOPEN:address, creating a 1-byte temporary file to send instead of generating it on fly with eachsocatinvocation.I have a draft patch doing that, and I’m running it through some testing now. I have had the test (with 1000 ports) time out once, but I haven’t seen it hit the EINTR problem so far.
A friendly reminder that this issue had no activity for 30 days.
Sorry, I didn’t notice about this issue right away, I’m taking a look.
Somewhat interestingly, socat always fails (gets interrupted) on the third connection (out of three): the expected data includes a number of
xcharacters, each one echoed back by a separate connection, and we’re always getting two (xxinstead ofxxx).What interrupts socat here (and whether socat should just handle
EINTR) is still a mystery to me at the moment.