webmesh: err: sendto: destination address required
I’m following the https://webmeshproj.github.io/guides/personal-vpn/ guide. I’m able to connect and some ping works, but I get this on the server side:
{"time":"2023-08-07T02:02:13.95968374-04:00","level":"INFO","msg":"starting webmesh node","version":"unknown","commit":"unknown","buildDate":"unknown"}
{"time":"2023-08-07T02:02:13.976051984-04:00","level":"INFO","msg":"using CN as node ID","component":"mesh","node-id":"server"}
{"time":"2023-08-07T02:02:13.976094557-04:00","level":"INFO","msg":"loading plugin","name":"mtls"}
{"time":"2023-08-07T02:02:14.010115941-04:00","level":"INFO","msg":"All 0 tables opened in 0s","component":"raft","storage":"/var/lib/webmesh/store","component":"raftbadger"}
{"time":"2023-08-07T02:02:14.017333494-04:00","level":"INFO","msg":"Discard stats nextEmptySlot: 0","component":"raft","storage":"/var/lib/webmesh/store","component":"raftbadger"}
{"time":"2023-08-07T02:02:14.017356434-04:00","level":"INFO","msg":"Set nextTxnTs to 0","component":"raft","storage":"/var/lib/webmesh/store","component":"raftbadger"}
{"time":"2023-08-07T02:02:14.076359285-04:00","level":"INFO","msg":"All 0 tables opened in 0s","component":"badger"}
{"time":"2023-08-07T02:02:14.090821071-04:00","level":"INFO","msg":"Discard stats nextEmptySlot: 0","component":"badger"}
{"time":"2023-08-07T02:02:14.090842809-04:00","level":"INFO","msg":"Set nextTxnTs to 0","component":"badger"}
{"time":"2023-08-07T02:02:14.091883921-04:00","level":"INFO","msg":"starting raft instance","component":"raft","storage":"/var/lib/webmesh/store","listen-addr":"[::]:9443"}
{"time":"2023-08-07T02:02:14.091997004-04:00","level":"INFO","msg":"initial configuration","component":"raft","index":0,"servers":["%+v",null]}
{"time":"2023-08-07T02:02:14.092069274-04:00","level":"INFO","msg":"bootstrapping cluster","component":"mesh","node-id":"server"}
{"time":"2023-08-07T02:02:14.092073906-04:00","level":"INFO","msg":"entering follower state","component":"raft","follower":{},"leader-address":"","leader-id":""}
{"time":"2023-08-07T02:02:18.538521544-04:00","level":"WARN","msg":"heartbeat timeout reached, starting election","component":"raft","last-leader-addr":"","last-leader-id":""}
{"time":"2023-08-07T02:02:18.53858546-04:00","level":"INFO","msg":"entering candidate state","component":"raft","node":{},"term":2}
{"time":"2023-08-07T02:02:18.538669733-04:00","level":"INFO","msg":"election won","component":"raft","term":2,"tally":1}
{"time":"2023-08-07T02:02:18.538681113-04:00","level":"INFO","msg":"entering leader state","component":"raft","leader":{}}
{"time":"2023-08-07T02:02:18.597710471-04:00","level":"INFO","msg":"newly bootstrapped cluster, setting IPv4/IPv6 networks","component":"mesh","node-id":"server","ipv4-network":"172.16.0.0/12","ipv6-network":"fda1:420:b42e::/48"}
{"time":"2023-08-07T02:02:18.598660537-04:00","level":"INFO","msg":"registering ourselves as a node in the cluster","component":"mesh","node-id":"server","server-id":"server"}
{"time":"2023-08-07T02:02:18.598683097-04:00","level":"INFO","msg":"generating wireguard key for ourselves","component":"mesh","node-id":"server"}
{"time":"2023-08-07T02:02:18.598689159-04:00","level":"INFO","msg":"generating new wireguard key","component":"mesh","node-id":"server"}
{"time":"2023-08-07T02:02:18.598998931-04:00","level":"INFO","msg":"starting network manager","component":"mesh","node-id":"server"}
{"time":"2023-08-07T02:02:18.599013156-04:00","level":"INFO","msg":"Configuring firewall","component":"net-manager","opts":{"ID":"server","DefaultPolicy":"accept","WireguardPort":51821,"RaftPort":9443,"GRPCPort":8443}}
{"time":"2023-08-07T02:02:18.643151554-04:00","level":"INFO","msg":"Configuring wireguard","component":"net-manager","opts":{"NodeID":"server","ListenPort":51821,"Name":"","ForceName":false,"ForceTUN":false,"PersistentKeepAlive":0,"MTU":1350,"AddressV4":"172.16.0.1/32","AddressV6":"fda1:420:b42e:442::/64","Metrics":false,"MetricsInterval":15000000000,"DisableIPv4":false,"DisableIPv6":false}}
{"time":"2023-08-07T02:02:18.643335776-04:00","level":"INFO","msg":"creating wireguard interface","component":"wireguard","name":"webmesh0"}
{"time":"2023-08-07T02:02:18.657321514-04:00","level":"INFO","msg":"re-adding ourselves to the cluster with the acquired wireguard address","component":"mesh","node-id":"server"}
{"time":"2023-08-07T02:02:18.657365768-04:00","level":"INFO","msg":"updating configuration","component":"raft","command":0,"server-id":"server","server-addr":"172.16.0.1:9443","servers":["%+v",[{"Suffrage":0,"ID":"server","Address":"172.16.0.1:9443"}]]}
{"time":"2023-08-07T02:02:18.657447005-04:00","level":"INFO","msg":"initial bootstrap complete","component":"mesh","node-id":"server"}
{"time":"2023-08-07T02:02:18.657459953-04:00","level":"INFO","msg":"mesh connection is ready, starting services"}
{"time":"2023-08-07T02:02:18.657884285-04:00","level":"INFO","msg":"Starting gRPC server on [::]:8443","component":"server"}
{"time":"2023-08-07T02:02:38.015101241-04:00","level":"INFO","msg":"started call","component":"server","protocol":"grpc","grpc.component":"server","grpc.service":"v1.Node","grpc.method":"Join","grpc.method_type":"unary","peer.address":"100.127.110.11:36454","grpc.start_time":"2023-08-07T02:02:38-04:00","grpc.time_ms":"0.01"}
{"time":"2023-08-07T02:02:38.015353545-04:00","level":"INFO","msg":"join request received","component":"node-server","op":"join","id":"admin","request":{"id":"admin","public_key":"sDNLid5iqiuIrFJ4sCYoUgaE4Xlbdy2taZazqYdZBF0=","raft_port":9443,"grpc_port":8443,"assign_ipv4":true}}
{"time":"2023-08-07T02:02:38.016100312-04:00","level":"INFO","msg":"adding non-voter to cluster","component":"node-server","op":"join","id":"admin","raft_address":"172.16.0.2:9443"}
{"time":"2023-08-07T02:02:38.01612437-04:00","level":"INFO","msg":"updating configuration","component":"raft","command":1,"server-id":"admin","server-addr":"172.16.0.2:9443","servers":["%+v",[{"Suffrage":0,"ID":"server","Address":"172.16.0.1:9443"},{"Suffrage":1,"ID":"admin","Address":"172.16.0.2:9443"}]]}
{"time":"2023-08-07T02:02:38.016176729-04:00","level":"INFO","msg":"added peer, starting replication","component":"raft","peer":"admin"}
{"time":"2023-08-07T02:02:38.016396001-04:00","level":"ERROR","msg":"failed to appendEntries to","component":"raft","peer":{"Suffrage":1,"ID":"admin","Address":"172.16.0.2:9443"},"error":"dial tcp 172.16.0.2:9443: connect: no route to host"}
{"time":"2023-08-07T02:02:38.016547253-04:00","level":"INFO","msg":"sending join response","component":"node-server","op":"join","id":"admin","response":{"address_ipv4":"172.16.0.2/32","address_ipv6":"fda1:420:b42e:ceea::/64","network_ipv4":"172.16.0.0/12","network_ipv6":"fda1:420:b42e::/48","peers":[{"id":"server","public_key":"5sdQXSMQGUXWzj7Ri8ZyYPoom2++4ISdWk+9sWSZDjg=","primary_endpoint":"100.85.215.110:51821","wireguard_endpoints":["100.85.215.110:51821"],"address_ipv4":"172.16.0.1/32","address_ipv6":"fda1:420:b42e:442::/64","allowed_ips":["172.16.0.1/32","fda1:420:b42e:442::/64"]}],"mesh_domain":"webmesh.internal."}}
{"time":"2023-08-07T02:02:38.016636105-04:00","level":"INFO","msg":"finished call","component":"server","protocol":"grpc","grpc.component":"server","grpc.service":"v1.Node","grpc.method":"Join","grpc.method_type":"unary","peer.address":"100.127.110.11:36454","grpc.start_time":"2023-08-07T02:02:38-04:00","grpc.code":"OK","grpc.time_ms":"1.548"}
{"time":"2023-08-07T02:02:38.017910703-04:00","level":"WARN","msg":"could not ping descendant","component":"net-manager","descendant":"admin","error":"run pinger: write ip4 0.0.0.0->172.16.0.2: sendto: destination address required"}
{"time":"2023-08-07T02:02:38.018289245-04:00","level":"WARN","msg":"could not ping descendant","component":"net-manager","descendant":"admin","error":"run pinger: write ip4 0.0.0.0->172.16.0.2: sendto: destination address required"}
{"time":"2023-08-07T02:02:41.339334625-04:00","level":"ERROR","msg":"failed to heartbeat to","component":"raft","peer":"172.16.0.2:9443","backoff time":10000000,"error":"dial tcp 172.16.0.2:9443: i/o timeout"}
{"time":"2023-08-07T02:02:44.949781283-04:00","level":"ERROR","msg":"failed to heartbeat to","component":"raft","peer":"172.16.0.2:9443","backoff time":10000000,"error":"dial tcp 172.16.0.2:9443: i/o timeout"}
{"time":"2023-08-07T02:02:48.366660528-04:00","level":"ERROR","msg":"failed to heartbeat to","component":"raft","peer":"172.16.0.2:9443","backoff time":10000000,"error":"dial tcp 172.16.0.2:9443: i/o timeout"}
{"time":"2023-08-07T02:02:51.944757019-04:00","level":"ERROR","msg":"failed to heartbeat to","component":"raft","peer":"172.16.0.2:9443","backoff time":20000000,"error":"dial tcp 172.16.0.2:9443: i/o timeout"}
{"time":"2023-08-07T02:02:55.425564585-04:00","level":"ERROR","msg":"failed to heartbeat to","component":"raft","peer":"172.16.0.2:9443","backoff time":40000000,"error":"dial tcp 172.16.0.2:9443: i/o timeout"}
{"time":"2023-08-07T02:02:55.741667558-04:00","level":"INFO","msg":"shutting down gRPC server"}
{"time":"2023-08-07T02:02:55.741742112-04:00","level":"INFO","msg":"Shutting down gRPC server","component":"server"}
{"time":"2023-08-07T02:02:55.741822634-04:00","level":"INFO","msg":"shutting down mesh connection"}
{"time":"2023-08-07T02:02:55.741864178-04:00","level":"INFO","msg":"creating new db snapshot","component":"snapshots"}
{"time":"2023-08-07T02:02:55.741992909-04:00","level":"INFO","msg":"Number of ranges found: 2","component":"badger"}
{"time":"2023-08-07T02:02:55.742081389-04:00","level":"INFO","msg":"DB.Backup Streaming about 0 B of uncompressed data (0 B on disk)","component":"badger"}
{"time":"2023-08-07T02:02:55.742121461-04:00","level":"INFO","msg":"Sent range 0 for iteration: [, 2f72656769737472792f65646765732f61646d696e2f736572766572fffffffffffffff1) of size: 0 B","component":"badger"}
{"time":"2023-08-07T02:02:55.742132076-04:00","level":"INFO","msg":"Sent range 1 for iteration: [2f72656769737472792f65646765732f61646d696e2f736572766572fffffffffffffff1, ) of size: 0 B","component":"badger"}
{"time":"2023-08-07T02:02:55.754551205-04:00","level":"INFO","msg":"DB.Backup Sent data of size 2.2 KiB","component":"badger"}
{"time":"2023-08-07T02:02:55.754931487-04:00","level":"INFO","msg":"db snapshot complete","component":"snapshots","duration":"13.017689ms","size":"982 B"}
{"time":"2023-08-07T02:02:55.754985262-04:00","level":"INFO","msg":"starting snapshot up to","component":"raft","index":20}
{"time":"2023-08-07T02:02:55.755028353-04:00","level":"INFO","msg":"creating new snapshot","component":"raft","storage":"/var/lib/webmesh/store","component":"snapshotstore","path":"/var/lib/webmesh/store/snapshots/2-20-1691388175755.tmp"}
{"time":"2023-08-07T02:02:55.781345175-04:00","level":"INFO","msg":"snapshot complete up to","component":"raft","index":20}
{"time":"2023-08-07T02:02:55.781571056-04:00","level":"ERROR","msg":"failed to transfer leadership","component":"raft","storage":"/var/lib/webmesh/store","error":"cannot find peer"}
{"time":"2023-08-07T02:02:56.721167203-04:00","level":"ERROR","msg":"failed to appendEntries to","component":"raft","peer":{"Suffrage":1,"ID":"admin","Address":"172.16.0.2:9443"},"error":"dial tcp 172.16.0.2:9443: i/o timeout"}
{"time":"2023-08-07T02:02:58.988560091-04:00","level":"ERROR","msg":"failed to heartbeat to","component":"raft","peer":"172.16.0.2:9443","backoff time":80000000,"error":"dial tcp 172.16.0.2:9443: i/o timeout"}
{"time":"2023-08-07T02:02:58.988724645-04:00","level":"INFO","msg":"Lifetime L0 stalled for: 0s","component":"badger"}
{"time":"2023-08-07T02:02:59.016826222-04:00","level":"INFO","msg":"Level 0 [ ]: NumTables: 01. Size: 1.3 KiB of 0 B. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 64 MiB\nLevel 1 [ ]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiB\nLevel 2 [ ]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiB\nLevel 3 [ ]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiB\nLevel 4 [ ]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiB\nLevel 5 [ ]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiB\nLevel 6 [B]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiB\nLevel Done","component":"badger"}
{"time":"2023-08-07T02:02:59.031477119-04:00","level":"INFO","msg":"Lifetime L0 stalled for: 0s","component":"raft","storage":"/var/lib/webmesh/store","component":"raftbadger"}
{"time":"2023-08-07T02:02:59.099681848-04:00","level":"INFO","msg":"Level 0 [ ]: NumTables: 01. Size: 2.3 KiB of 0 B. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 64 MiB\nLevel 1 [ ]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiB\nLevel 2 [ ]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiB\nLevel 3 [ ]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiB\nLevel 4 [ ]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiB\nLevel 5 [ ]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiB\nLevel 6 [B]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiB\nLevel Done","component":"raft","storage":"/var/lib/webmesh/store","component":"raftbadger"}
[pid 219825] sendto(22, "\10\0L\322\270\356\0\0\27y\6rKN\376\214\356nZ\225\234\267O~\235K^Y\302B\227V", 32, 0, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("172.16.0.2")}, 16 <unfinished ...>
[pid 219825] <... sendto resumed>) = -1 EDESTADDRREQ (Destination address required)
I think this cause disconnections.
0.1.2
About this issue
- Original URL
- State: closed
- Created a year ago
- Comments: 20 (20 by maintainers)
Sweet - I fixed my other bug so about to tag a new release. Will close this, but feel free to open a new issue if anything arises.
I’m still working on the second issue - but if you are able to build from
mainyou can see if the first one was your problem. If you can’t build yourself, the CI will have an image in a little bit.Just to give you a quick update. I’m trying to hammer out one last bug and then I’ll tag a new release.
I think most related to your issue was in the storage update trigger on each node. It was doing some hacky logic to make sure it was fully up to date - and a recent fix had made that no longer necessary. I think that old hack was causing peer refreshes not to happen at the right times.
I’m having another issue where peer refreshes are sometimes returning the wrong internal IPs to be set to the allowed IPs. I’m not sure if this is something you are experiencing - but I hope to figure it out before I push this other fix out.
I think I’ve found an issue - not sure if it is related to yours. But in a similar setup the second client keeps dropping the peer. I’ll let you know what I figure out.
I’m gonna try to replicate it locally - and if I fail may reach out for more infoz
I hope to be able to look more closely tonight. One more thing that you can try (and is a really easy way to shoot yourself in the foot that I need to document better) - is that given this is a sort of “zero-trust” solution, the default behavior only lets people peer up if there is a Network ACL allowing it. Those are managed via the admin API, but you can set a default allow-all rule at bootstrap with
--bootstrap.default-network-policy=accept. You’ll see something similar in most of the examples.To the
wmctl connectquestion. There isn’t. That utility was originally just for testing, but decided to keep it around. It acts like a NATd VPN client by default. You just poke out.At least one node needs to accessible currently for peerings to work. But I am currently working on other methods of discovery.