metallb: MetalLB provides Services with IP Addresses but doesn't ARP for the address

Hi everyone - I installed MetalLB (v0.11.0) on a fresh RKE2 cluster (v1.22.5+rke2r1) with calico as CNI (docker.io/rancher/mirrored-calico-cni:v3.20.2).

When I deploy a service with Service Type Loadbalancer, the service gets an IP address from MetalLB, but no speaker actually ARPs for the address.

MetalLB ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: rancher-cluster-ingress-ip
      protocol: layer2
      addresses:
      - 131.159.88.8/32
    - name: general
      protocol: layer2
      addresses:
      - 131.159.88.9-131.159.88.10

Test Service:

apiVersion: v1
kind: Service
metadata:
  name: nginx
  annotations:
    metallb.universe.tf/address-pool: general
spec:
  ports:
  - port: 80
    targetPort: 80
  selector:
    app: nginx
  type: LoadBalancer

Kubectl get services:

k get services
NAME         TYPE           CLUSTER-IP     EXTERNAL-IP    PORT(S)        AGE
kubernetes   ClusterIP      10.43.0.1      <none>         443/TCP        4h58m
nginx        LoadBalancer   10.43.215.76   131.159.88.9   80:31159/TCP   3h14m

Logs from MetalLB controller (The log also shows the ingress controller - the allocation for the ingress has the same problem):

k logs -n metallb-system controller-7dcc8764f4-hvz28
{"branch":"HEAD","caller":"level.go:63","commit":"v0.11.0","goversion":"gc / go1.16.9 / amd64","level":"info","msg":"MetalLB controller starting version 0.11.0 (commit v0.11.0, branch HEAD)","ts":"2022-01-06T15:28:45.895719604Z","version":"0.11.0"}
{"caller":"level.go:63","level":"info","msg":"secret succesfully created","op":"CreateMlSecret","ts":"2022-01-06T15:28:45.95870185Z"}
{"caller":"level.go:63","configmap":"metallb-system/config","event":"configLoaded","level":"info","msg":"config (re)loaded","ts":"2022-01-06T15:28:46.060039534Z"}
{"caller":"level.go:63","error":"controller not synced","level":"error","msg":"controller not synced yet, cannot allocate IP; will retry after sync","op":"allocateIP","service":"ingress-nginx/ingress-nginx-controller","ts":"2022-01-06T15:28:46.060429455Z"}
{"caller":"level.go:63","event":"stateSynced","level":"info","msg":"controller synced, can allocate IPs now","ts":"2022-01-06T15:28:46.060888327Z"}
{"caller":"level.go:63","event":"ipAllocated","ip":"131.159.88.8","level":"info","msg":"IP address assigned by controller","service":"ingress-nginx/ingress-nginx-controller","ts":"2022-01-06T15:28:46.066451483Z"}
{"caller":"level.go:63","event":"serviceUpdated","level":"info","msg":"updated service object","service":"ingress-nginx/ingress-nginx-controller","ts":"2022-01-06T15:28:46.165715111Z"}
{"caller":"level.go:63","configmap":"metallb-system/config","event":"configLoaded","level":"info","msg":"config (re)loaded","ts":"2022-01-06T15:37:12.545217535Z"}
{"caller":"level.go:63","event":"ipAllocated","ip":"131.159.88.9","level":"info","msg":"IP address assigned by controller","service":"default/nginx","ts":"2022-01-06T15:39:30.668939857Z"}
{"caller":"level.go:63","event":"serviceUpdated","level":"info","msg":"updated service object","service":"default/nginx","ts":"2022-01-06T15:39:30.728002891Z"}

Logs from all speakers:

kubetail -l component=speaker -n metallb-system
Will tail 6 logs...
speaker-47s49
speaker-7jkf8
speaker-94jzz
speaker-qgjvg
speaker-r9cm6
speaker-tn7kn
[speaker-r9cm6] {"caller":"level.go:63","level":"info","msg":"triggering discovery","op":"memberDiscovery","ts":"2022-01-06T15:33:47.797857234Z"}
[speaker-tn7kn] {"caller":"level.go:63","level":"info","msg":"triggering discovery","op":"memberDiscovery","ts":"2022-01-06T15:33:56.922473237Z"}
[speaker-7jkf8] {"caller":"level.go:63","level":"info","msg":"triggering discovery","op":"memberDiscovery","ts":"2022-01-06T15:33:56.932155108Z"}
[speaker-94jzz] W0106 15:34:02.521299       1 warnings.go:70] discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice
[speaker-47s49] {"caller":"level.go:63","level":"info","msg":"triggering discovery","op":"memberDiscovery","ts":"2022-01-06T15:34:06.437035115Z"}
[speaker-94jzz] {"caller":"level.go:63","level":"info","msg":"triggering discovery","op":"memberDiscovery","ts":"2022-01-06T15:34:06.448021176Z"}
[speaker-qgjvg] {"caller":"level.go:63","level":"info","msg":"triggering discovery","op":"memberDiscovery","ts":"2022-01-06T15:34:16.940303128Z"}
[speaker-r9cm6] W0106 15:34:19.768841       1 warnings.go:70] discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice
[speaker-47s49] W0106 15:36:01.175087       1 warnings.go:70] discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice
[speaker-tn7kn] W0106 15:36:22.459597       1 warnings.go:70] discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice
[speaker-tn7kn] {"caller":"level.go:63","configmap":"metallb-system/config","event":"configLoaded","level":"info","msg":"config (re)loaded","ts":"2022-01-06T15:37:12.542619862Z"}
[speaker-r9cm6] {"caller":"level.go:63","configmap":"metallb-system/config","event":"configLoaded","level":"info","msg":"config (re)loaded","ts":"2022-01-06T15:37:12.54285656Z"}
[speaker-7jkf8] {"caller":"level.go:63","configmap":"metallb-system/config","event":"configLoaded","level":"info","msg":"config (re)loaded","ts":"2022-01-06T15:37:12.543297128Z"}
[speaker-qgjvg] {"caller":"level.go:63","configmap":"metallb-system/config","event":"configLoaded","level":"info","msg":"config (re)loaded","ts":"2022-01-06T15:37:12.546479654Z"}
[speaker-47s49] {"caller":"level.go:63","configmap":"metallb-system/config","event":"configLoaded","level":"info","msg":"config (re)loaded","ts":"2022-01-06T15:37:12.557708702Z"}
[speaker-94jzz] {"caller":"level.go:63","configmap":"metallb-system/config","event":"configLoaded","level":"info","msg":"config (re)loaded","ts":"2022-01-06T15:37:12.560457427Z"}
[speaker-7jkf8] W0106 15:37:54.786781       1 warnings.go:70] discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice
[speaker-qgjvg] W0106 15:38:03.211716       1 warnings.go:70] discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice
[speaker-r9cm6] {"caller":"level.go:63","level":"info","msg":"triggering discovery","op":"memberDiscovery","ts":"2022-01-06T15:38:48.058227044Z"}
[speaker-tn7kn] {"caller":"level.go:63","level":"info","msg":"triggering discovery","op":"memberDiscovery","ts":"2022-01-06T15:38:57.516808568Z"}
[speaker-7jkf8] {"caller":"level.go:63","level":"info","msg":"triggering discovery","op":"memberDiscovery","ts":"2022-01-06T15:38:57.536112642Z"}
[speaker-47s49] {"caller":"level.go:63","level":"info","msg":"triggering discovery","op":"memberDiscovery","ts":"2022-01-06T15:39:06.720076454Z"}
[speaker-94jzz] {"caller":"level.go:63","level":"info","msg":"triggering discovery","op":"memberDiscovery","ts":"2022-01-06T15:39:06.739423482Z"}
[speaker-qgjvg] {"caller":"level.go:63","level":"info","msg":"triggering discovery","op":"memberDiscovery","ts":"2022-01-06T15:39:17.539594986Z"}
...

I followed the Troubleshooting guide on the web page. The arping tool only shows timeouts on every node in the cluster.

sudo arping -I ens18 131.159.88.8
ARPING 131.159.88.8
Timeout

Any ideas what could cause this behavior?

Thanks in advance!

Best, Matthias

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 28 (16 by maintainers)

Most upvoted comments

I created l2advertisement kind and after that the speakers started announcing the ips

@Manvir-nokia Thanks for the answer here. For anyone looking for a simple bit to patch this, I put this right after my IPAddressPool, like so:

---
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: primary-pool
  namespace: metallb-system
spec:
  addresses:
  - 192.168.1.210-192.168.1.220
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: l2adv
  namespace: metallb-system
spec:
  ipAddressPools:
  - primary-pool

Afterward, my nmap port scan immediately was returning as not-down. My ports still show as closed unfortunately, but that’s a separate problem that’s likely unrelated to metallb.

+10
dustinmhorvath on Nov 6, 2022

I was able to resolve the issue. the IPs are getting announced. However, thanks for checking.

@Manvir-nokia I’m facing the same issue, how did you fix it?

I created l2advertisement kind and after that the speakers started announcing the ips. you can also check if l2advertisement object exists in metallb-system namespace.

Following is the CLI to check l2advertisement object:

kubectl get l2advertisement -n metallb-system

+4
Manvir-nokia on Sep 2, 2022

After some time of intense debugging wit the help of @fedepaol the issue is resolved.

For future reference:

  • The MetalLB Announced ip will not respond to ICMP requests! Always use curl to see if the service is working.
  • If you use arping - make sure to execute it on a node which will not arp for the address itself - The node that homes the announcing metallb-speaker will ignore the arp requests from arping

Thanks again for your help @fedepaol

+3
Mtze on Jan 12, 2022

no pressure! Feel free to jump on #metallb-devel on slack if you need assistance

+1
fedepaol on Feb 1, 2023

@Manvir-nokia would you mind filing a new issue and providing the logs of the speakers and the configuration (and the result of the arping test?)

+1
fedepaol on Aug 29, 2022
Read more comments on GitHub
← metallb: MetalLB cannot peer with BGP routers that Calico is already peering with
metallb: metallb sending wrong arp when using multiple ethernet interfaces →