crowdsec-bouncer-traefik-plugin: [BUG] v1.4.2 Bouncer returns 403 Forbidden without any active decision

Describe the bug šŸ›

The Traefik bouncer middleware enters a permanent ā€œfail-closedā€ state where it blocks all requests with a 403 Forbidden error on any router where it is enabled. This happens even when there are no active decisions or alerts for the client IP in the CrowdSec LAPI.

The block persists even after coordinated restarts of both the Traefik and CrowdSec containers. The only way to get access to the service is to disable the CrowdSec middleware on the router.

Expected behavior šŸ‘€

The bouncer should only block a request and return a 403 Forbidden error if there is an active ban decision for the client’s IP address issued by the CrowdSec LAPI. If no decision exists, the request should be allowed to pass through to the backend service.

Context šŸ”Ž

We have performed extensive debugging and can confirm the following:

  1. Traefik and CrowdSec are running correctly: Both containers start without fatal errors in their logs.
  2. Network connectivity is working: A test from inside the Traefik container to the CrowdSec LAPI was successful:
    • Command: docker exec -it traefik /bin/sh -c "apk add curl && curl -v http://crowdsec:8080/v1/heartbeat"
    • Result: HTTP/1.1 401 Unauthorized. This proves that DNS resolution and TCP/HTTP communication between the containers are working perfectly.
  3. LAPI state is clean: docker exec crowdsec cscli decisions list and docker exec crowdsec cscli alerts list both show no entries for the blocked client IP.
  4. Log Acquisition is working: docker exec crowdsec cscli metrics shows that the Traefik access.log is being read and parsed.
  5. Traefik sees the correct Client IP: The Traefik access.log shows the correct external client IP making the request and receiving a 403 response code. The issue is not related to a misconfiguration of forwardedHeaders.

Plugin Configuration (in traefik/docker-compose.yml labels):

labels:
  - "traefik.http.middlewares.crowdsec.plugin.crowdsec.bouncerUrl=http://crowdsec:8080"
  - "traefik.http.middlewares.crowdsec.plugin.crowdsec.crowdsecLapiKey=${CROWDSEC_BOUNCER_API_KEY}" # Key is provided via .env file

Router Configuration (on the target service’s labels):

labels:
  - "traefik.http.routers.myservice.middlewares=crowdsec@docker"

Version (please complete the following information):

  • OS: Docker on Linux
  • Traefik version: 3.0.4
  • Plugin version: 1.4.2
  • CrowdSec version: 1.6.8
  • Redis ? : No

To Reproduce

Steps to reproduce the behavior:

  1. Set up Traefik and CrowdSec using Docker Compose with the configurations described above.
  2. Enable the CrowdSec middleware on a specific router.
  3. Ensure no decisions exist in CrowdSec for your client IP (cscli decisions list).
  4. Attempt to access the URL of the protected router from a web browser.
  5. See the 403 Forbidden error page.
  6. Disabling the middleware by commenting out the label on the service immediately resolves the issue.

Additional Context

Downgrading the plugin to version v1.4.1 completely resolves the issue. The bug seems to be specific to version v1.4.2.

Additional Context

Context: worked that out while setting up the system with ChatGPT & Gemini

About this issue

  • Original URL
  • State: closed
  • Created a year ago
  • Reactions: 2
  • Comments: 15 (4 by maintainers)

Most upvoted comments

The banned=true has been removed in the last version of the plugin as useless query param.

Good to know, I assume thats not released yet. The above log entries are coming through with the latest v1.4.2 release.

Edit: I can see this fix in the commit at 4bb6e830dc989deee1c7d0a76141a111d24c186a on 29th April. Current release date for v1.4.2 is 1st April.

You can pass it via docker secret/docker volume and use the param crowdsecLapiKeyFile with a path to a file. It’s well described on the README.

I will be using the crowdsecLapiKeyFile option when I sit down to lock in and commit this config. That’s a tomorrow task though.

We don’t have access to Env Var in a Traefik plugin (or maybe but never try it) because we are not in a real Go context but it’s loaded in Yaegi that is a Go interpreter written in Go (that’s fun). Is that ok for you ? (if not, recreate a new feature request and we will see what is possible with Env Var)

I dont think it’s possible to put environment variables in Traefik’s static config. This post here says as much. There is a list of the environment variables that Traefik does make available for use in it’s static config here.