ingress-nginx: nginx-ingress pod crashloop if it can not redownload GeoIP2 database
NGINX Ingress controller version : 1.1.0
Kubernetes version : v1.21.7
Environment:
- Cloud provider or hardware configuration: BareMetal
- OS (e.g. from /etc/os-release): Ubuntu
- Kernel (e.g.
uname -a): 5.4.0-91-generic - How was the ingress-nginx-controller installed: helm ingress-nginx-4.0.13 1.1.0
What happened: If for any reason during the pod start nginx-ingress-controller fails to download the geoip2 database from https://download.maxmind.com/app/geoip_download… the pod will crashloop even if there is a previous database already downloaded and present in /etc/nginx/geoip using a persistent volume.
What you expected to happen: It should fail to start only if it can not download the geoip2 database AND it can not find a previous downloaded database on /etc/nginx/geoip. In the current way, If there is any problem with maxmind website, the ingresses will refuse to start.
How to reproduce it:
- Deploy nginx-ingress to use a persistent volume mounted at /etc/nginx/geoip/ and also set the maxmindLicenseKey.
- After the first successfully run, the geoip2 databases will be on /etc/nginx/geoip/
- Cut access to download.maxmind.com using a firewall to simulate a failure on their website
- Delete the nginx ingress pod to force it to be recreated. It will never start until it can connect to download.maxmind.com.
/kind bug
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 1
- Comments: 24 (11 by maintainers)
@k8s-triage-robot: Closing this issue, marking it as “Not Planned”.
In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
The current behavior is to fatal when Maxmind license is defined and the database cannot be downloaded. There are various reasons this is desirable, such as compliance requirements that certain countries be blocked, etc.
One way to work around a Maxmind outage is to run a Maxmind mirror or caching proxy for your organization. This is recommended as a best practice by Maxmind to minimize load on their infrastructure. It works well for us.
Another solution could be some way for you to declare that download failures are allowed in your systems, or allowed only when a file is already present. Maybe a new config option could be added to allow you to define this policy.