crowdsec-bouncer-traefik-plugin: [BUG] v1.4.2 Bouncer returns 403 Forbidden without any active decision
Describe the bug š
The Traefik bouncer middleware enters a permanent āfail-closedā state where it blocks all requests with a 403 Forbidden error on any router where it is enabled. This happens even when there are no active decisions or alerts for the client IP in the CrowdSec LAPI.
The block persists even after coordinated restarts of both the Traefik and CrowdSec containers. The only way to get access to the service is to disable the CrowdSec middleware on the router.
Expected behavior š
The bouncer should only block a request and return a 403 Forbidden error if there is an active ban decision for the clientās IP address issued by the CrowdSec LAPI. If no decision exists, the request should be allowed to pass through to the backend service.
Context š
We have performed extensive debugging and can confirm the following:
- Traefik and CrowdSec are running correctly: Both containers start without fatal errors in their logs.
- Network connectivity is working: A test from inside the Traefik container to the CrowdSec LAPI was successful:
- Command:
docker exec -it traefik /bin/sh -c "apk add curl && curl -v http://crowdsec:8080/v1/heartbeat" - Result:
HTTP/1.1 401 Unauthorized. This proves that DNS resolution and TCP/HTTP communication between the containers are working perfectly.
- Command:
- LAPI state is clean:
docker exec crowdsec cscli decisions listanddocker exec crowdsec cscli alerts listboth show no entries for the blocked client IP. - Log Acquisition is working:
docker exec crowdsec cscli metricsshows that the Traefikaccess.logis being read and parsed. - Traefik sees the correct Client IP: The Traefik
access.logshows the correct external client IP making the request and receiving a403response code. The issue is not related to a misconfiguration offorwardedHeaders.
Plugin Configuration (in traefik/docker-compose.yml labels):
labels:
- "traefik.http.middlewares.crowdsec.plugin.crowdsec.bouncerUrl=http://crowdsec:8080"
- "traefik.http.middlewares.crowdsec.plugin.crowdsec.crowdsecLapiKey=${CROWDSEC_BOUNCER_API_KEY}" # Key is provided via .env file
Router Configuration (on the target serviceās labels):
labels:
- "traefik.http.routers.myservice.middlewares=crowdsec@docker"
Version (please complete the following information):
- OS: Docker on Linux
- Traefik version: 3.0.4
- Plugin version: 1.4.2
- CrowdSec version: 1.6.8
- Redis ? : No
To Reproduce
Steps to reproduce the behavior:
- Set up Traefik and CrowdSec using Docker Compose with the configurations described above.
- Enable the CrowdSec middleware on a specific router.
- Ensure no decisions exist in CrowdSec for your client IP (
cscli decisions list). - Attempt to access the URL of the protected router from a web browser.
- See the
403 Forbiddenerror page. - Disabling the middleware by commenting out the label on the service immediately resolves the issue.
Additional Context
Downgrading the plugin to version v1.4.1 completely resolves the issue. The bug seems to be specific to version v1.4.2.
Additional Context
Context: worked that out while setting up the system with ChatGPT & Gemini
About this issue
- Original URL
- State: closed
- Created a year ago
- Reactions: 2
- Comments: 15 (4 by maintainers)
Good to know, I assume thats not released yet. The above log entries are coming through with the latest v1.4.2 release.
Edit: I can see this fix in the commit at 4bb6e830dc989deee1c7d0a76141a111d24c186a on 29th April. Current release date for v1.4.2 is 1st April.
I will be using the
crowdsecLapiKeyFileoption when I sit down to lock in and commit this config. Thatās a tomorrow task though.I dont think itās possible to put environment variables in Traefikās static config. This post here says as much. There is a list of the environment variables that Traefik does make available for use in itās static config here.