cilium: LB-IPAM: Unable to set single addresses as a pool
Is there an existing issue for this?
- I have searched the existing issues
What happened?
I tried to create a pool with only single IPv4 (/32) and IPv6 (/128) addresses inside it. I expected it to see that the pool has two IP addresses in total, but it doesn’t seem to get that.
Instead it’s having very hard time allocating IPs from it.
Cilium Version
cilium-cli: v0.12.1 compiled with go1.18.5 on linux/amd64 cilium image (default): v1.12.0 cilium image (stable): v1.13.0 cilium image (running): v1.13.0
Kernel Version
5.15.0-1025-raspi
Kubernetes Version
{
"clientVersion": {
"major": "1",
"minor": "26",
"gitVersion": "v1.26.1",
"gitCommit": "8f94681cd294aa8cfd3407b8191f6c70214973a4",
"gitTreeState": "archive",
"buildDate": "2023-02-06T14:43:47Z",
"goVersion": "go1.19.4",
"compiler": "gc",
"platform": "linux/amd64"
},
"kustomizeVersion": "v4.5.7",
"serverVersion": {
"major": "1",
"minor": "25",
"gitVersion": "v1.25.6+k3s1",
"gitCommit": "9176e03c5788e467420376d10a1da2b6de6ff31f",
"gitTreeState": "clean",
"buildDate": "2023-01-26T00:30:33Z",
"goVersion": "go1.19.5",
"compiler": "gc",
"platform": "linux/arm64"
}
}
Sysdump
Too large; It’s here
Relevant log output
level=info msg="Notify ingress controller for service ingress" subsys=ingress-controller
level=error msg="Failed to process Ingress event, skipping: {oldIngress:0x4000171f00 newIngress:0x40015bc100}" subsys=ingress-controller
level=error msg="Unable to allocate IP" error="provided IP is not in the valid range. The range of valid IPs is 185.218.193.58/32" subsys=lbipam
level=error msg="Unable to allocate IP" error="provided IP is not in the valid range. The range of valid IPs is 2001:67c:1104:e03e::443/128" subsys=lbipam
level=info msg="Notify ingress controller for service ingress" subsys=ingress-controller
level=error msg="Unable to allocate IP" error="provided IP is not in the valid range. The range of valid IPs is 185.218.193.58/32" subsys=lbipam
level=error msg="Unable to allocate IP" error="provided IP is not in the valid range. The range of valid IPs is 2001:67c:1104:e03e::443/128" subsys=lbipam
Anything else?
This is valid way for MetalLB, and I think it should work with Cilium too.
The pool manifest can be found here
The cilium ingress svc looks like this:
apiVersion: v1
kind: Service
metadata:
annotations:
io.cilium/lb-ipam-ips: 185.218.193.58,2001:67c:1104:e03e::443
meta.helm.sh/release-name: cilium
meta.helm.sh/release-namespace: kube-system
creationTimestamp: "2023-03-13T16:52:43Z"
labels:
app.kubernetes.io/managed-by: Helm
cilium.io/ingress: "true"
helm.toolkit.fluxcd.io/name: cilium
helm.toolkit.fluxcd.io/namespace: kube-system
name: cilium-ingress
namespace: kube-system
resourceVersion: "2315905"
uid: 2b9c0a9c-b58a-45dd-bd47-1e00f21533cc
spec:
allocateLoadBalancerNodePorts: true
clusterIP: 10.46.159.107
clusterIPs:
- 10.46.159.107
- 2001:67c:1104:e03e::c94b
externalTrafficPolicy: Cluster
internalTrafficPolicy: Cluster
ipFamilies:
- IPv4
- IPv6
ipFamilyPolicy: RequireDualStack
ports:
- name: http
nodePort: 31019
port: 80
protocol: TCP
targetPort: 80
- name: https
nodePort: 31816
port: 443
protocol: TCP
targetPort: 443
sessionAffinity: None
type: LoadBalancer
status:
conditions:
- lastTransitionTime: "2023-03-13T16:52:43Z"
message: ""
reason: satisfied
status: "True"
type: io.cilium/lb-ipam-request-satisfied
loadBalancer: {}
Code of Conduct
- I agree to follow this project’s Code of Conduct
About this issue
- Original URL
- State: closed
- Created a year ago
- Reactions: 17
- Comments: 20 (8 by maintainers)
I think the issue is clear (see https://github.com/cilium/cilium/issues/24351#issuecomment-1467637464), so removing the “triage” label.
Since this is a limitation of the existing allocation logic, the next step would be to change that. This is not the only issue we have with the allocator, also #21776 and #22005. I have also seen requests to allow range notation
192.168.1.123-192.168.1.127which would also not be supported by the current allocator logic.I believe that we can combine this into a nice set of requirements for an alternative IP allocator:
Having the same issue, unable to allocate a cidr of /32. With the introduction of L2 announcements (https://docs.cilium.io/en/latest/network/l2-announcements/) in v1.14 this is going to be an even bigger issue. If the goal is to help support smaller on-perm deployments such as campus networks, then it is critical to be able to allocate single IP cidrs or arbitrary ranges as described here: https://github.com/cilium/cilium/issues/24351#issuecomment-1479660360. Public IP’s are very valuable and not handed out on a whim, they’re often allocated one at a time or in arbitrary ranges.
I would go so far as to say the L2 announcements feature is not very useful in real world practice without an improved LB-IPAM capability. To me it would be important to have some kind of fix or clearly documented safe workaround for this to be released with the L2 feature.
+1 with @j-gooding.
I am considering using Cilium on our clusters on which we get public subnets for ingresses but on these subnets, 3 more IPs are reserved and should not be used by the LB IPAM.
Either “allow specifying reserved IP within a pool” or a “The allocator should allow for arbitrary start and stop IPs as long as start < stop” would be very nice
The reason why this doesn’t work comes down to an implementation detail. LB-IPAM uses an existing package for the IP allocation which reserves the first and last IP since these typically can’t be used. https://github.com/cilium/ipam/blob/master/service/ipallocator/allocator.go#L56
But this also applies when making the range of /32 or /31, which does not make sense. We need to do some work on the IP allocation logic to fix this one.
For now, a temporary workaround would be to add a /30 to your pool, this does make 2 IPs available not 1 however, so not exactly what you are after.
Unfortunately, that doesn’t work when Cilium doesn’t understand the pool properly, with /32 prefix which is a single IPv4 and instead says there’s no IPv4 addresses.