edgevpn: Intermittent connection drops with Edgevpn 0.10.0/libp2p 0.18.0-rc5 leaves disconnected peers
After about a 30min of usage, I started to notice constants connection drops by peers node. The issue seems to be persisting as connections doesn’t seems to be rebuilt between nodes automatically, leaving peers disconnected. the only workaround is restarting the service.
This seems to be tied with the recent libp2p bump to 0.18.0-rc5. I’m not sure if it’s due to rsmngr configuration or either something else. I can’t still trace it, but this is what I’m seeing now at a behavioral level:
while opening a bunch of multiple streams to a single connection the connection gets eventually killed and seems the node can’t recover and connect to it again.
Although this seems to be an issue even with small streams - where I was previously pushing GBs of traffic just fine between nodes, now doesn’t hold even for simple http requests.
@vyzo / @marten-seemann sorry guys to ping you directly again, and don’t want to sound annoying either. I’m seeing weird issues with 0.18.0 -rc5 here. I’m not sure if it’s due to rsmngr configuration or either something else. I can’t still trace it and give some helpful debug information, but this is what I’m seeing now at a behavioral level, the effect is quite noticeable.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 1
- Comments: 17 (11 by maintainers)
Commits related to this issue
- :arrow_down: Revert libp2p bump It seems there are issues with the new rc regarding connections. Meanwhile trying to figure out what's wrong downgrade to last good version. See #12 — committed to mudler/edgevpn by mudler 2 years ago
- :arrow_up: Bump internal version To downgrade libp2p, see https://github.com/mudler/edgevpn/issues/12 — committed to kairos-io/kairos by mudler 2 years ago
- :arrow_up: Bump libp2p to 0.18.0-rc6 See: https://github.com/mudler/edgevpn/issues/12 — committed to mudler/edgevpn by mudler 2 years ago
- :arrow_down: Revert libp2p bump Also reverts rcmgr configurations See https://github.com/mudler/edgevpn/issues/12 — committed to mudler/edgevpn by mudler 2 years ago
- :gear: Set rcmgr limits from maxconns Also disables it by default Fixes https://github.com/mudler/edgevpn/issues/12 — committed to mudler/edgevpn by mudler 2 years ago
can you try either disabling mplex or with libp2p/go-mplex#99 ?
yeah, the default inbound conn limit is very conservative.
you can also get logs with
GOLOG_LOG_LEVEL=debug, there should be some hints there.can you also check whether mplex is involved? You probably dont need it at all, can you try limiting the muxer to just yamux?