ingress-nginx: `port 80 is already in use. Please check the flag --http-port` on GKE, ingress-nginx version 1.1.2 and 1.1.3

Leaving the above comment in case it helps someone else in troubleshooting, but in fact when i run the v1.2.0 image on a working node, i do see the cap_net_bind_service=ep on the /nginx-ingress-controller. So i would say the main issue here for this project is the potentially misleading error.

Furthermore, after removing all ingress-nginx controller pods on the node, running crictl rmi --prune (to clean up unused images), and re-running the pod on the node (image would be pulled fresh) it worked fine.

This led me to believe there must have been some corruption in the cached image layers, removing the capability on the underlying file in /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/… I was able to replicate that by stopping the pod, removing the capability from the underlying file that had it, and launching the pod again (now not working).

## find files named nginx-ingress-controller. 
$ find . | grep nginx-ingress-controller
./io.containerd.snapshotter.v1.overlayfs/snapshots/299742/fs/nginx-ingress-controller
./io.containerd.snapshotter.v1.overlayfs/snapshots/299739/fs/nginx-ingress-controller

## higher numbered one has the capability, ower-numbered one does not, presumably lower layer of image before that was set (note - image could probably be smaller if the file was added and setcap run in one layer)
$ getcap ./io.containerd.snapshotter.v1.overlayfs/snapshots/299739/fs/nginx-ingress-controller
$ getcap ./io.containerd.snapshotter.v1.overlayfs/snapshots/299742/fs/nginx-ingress-controller
./io.containerd.snapshotter.v1.overlayfs/snapshots/299742/fs/nginx-ingress-controller = cap_net_bind_service+ep

## remove it from higher-numbered one:
$ setcap -r ./io.containerd.snapshotter.v1.overlayfs/snapshots/299742/fs/nginx-ingress-controller
$ getcap ./io.containerd.snapshotter.v1.overlayfs/snapshots/299742/fs/nginx-ingress-controller
$

## running a new pod, it encounters the issue

version 1.2.1 appears to still have this issue, but it does not occur in all environments. we have 2 identical machines with identical docker + k8s + helm setup, but one has this issue and the other one does not. downgraded one that did not work to version 1.0.0 and it works, but the other machine chugs away with 1.2.1.

thinking very hard on how to explain this to the customer.

@saowu thanks! It helped, but I also needed to set this cap for nginx binary with the same ./io.containerd.snapshotter.v1.overlayfs/snapshots/<id>/fs as well

for me, it’s fixed with sysctl -w net.ipv4.ip_unprivileged_port_start=1 on host machine for a permanent fix, run the script below echo ‘net.ipv4.ip_unprivileged_port_start = 1’ > /etc/sysctl.d/ip_unprivileged_port_start.conf

the error message is misleading, it’s actually a permission issue, not a port binding issue

@strongjz can you have a look at https://github.com/jrhunger/ingress-nginx/blob/troubleshooting-docs-update-ports/docs/troubleshooting.md#unable-to-listen-on-port-80443 and let me know if that is the kind of thing you are looking for?

@longwuyuan correct, the v1.2.0 image has the proper cap_net_bind_service when freshly pulled.

I have the same issue with Kubernetes v1.22.8 and containerd://1.5.8 The initial installation was fine using Kubespray v2.18.1 After reinstallation the ingress nginx on one node can’t start NAME READY STATUS RESTARTS AGE ingress-nginx-controller-8rxh6 1/1 Running 0 24m ingress-nginx-controller-cct24 1/1 Running 0 11m ingress-nginx-controller-kr99w 1/1 Running 0 9m36s ingress-nginx-controller-kvl5f 1/1 Running 0 10m ingress-nginx-controller-svqsd 0/1 CrashLoopBackOff 5 (2m49s ago) 5m41s

The logs from the failed pod is k logs -f ingress-nginx-controller-svqsd

NGINX Ingress controller Release: v1.0.4 Build: 9b78b6c197b48116243922170875af4aa752ee59 Repository: https://github.com/kubernetes/ingress-nginx nginx version: nginx/1.19.9

---

F0424 07:43:51.932493 7 main.go:67] port 80 is already in use. Please check the flag --http-port

… I have also tried to change daemonset ports: - containerPort: 80 hostPort: 8888

it doesnt help Port 80 isnt used anyway on the server netstat -lnpt | grep 80 tcp 0 0 0.0.0.0:8081 0.0.0.0:* LISTEN 3109/nginx: master

@praveenperera I have the same problem since this morning. I already had version 1.1.1 installed. With version 1.1.0 it worked again.