docker-mailserver: [BUG] fail2ban does not block ips behind proxy
Bug Report
Context
Banning IP addresses through fail2ban in the container actually has no affect. I still see a lot of login attempts by IP addresses that are banned in fail2ban.
What is affected by this bug?
- fail2ban
When does this occur?
After a certain amount of failed login attempts fail2ban banns ip addresses. But today i realized that (at least at my setup) the ban has no effect.
How do we replicate the issue?
- Watch the mailserver log after an IP got banned
Behavior
Actual Behavior
IP addresses banned in fail2ban can actually connect to postfix/dovecot and try to login
Expected Behavior
IP addresses banned in fail2ban aren’t allowed to connect to postfix/dovecot (according to the ban reason/jail) and connections get canceled immediately.
Your Environment
- version:
v7.2.0
- available RAM:
4GB
- Docker version:
v20.10.1
Environment Variables
- DMS_DEBUG=0
- ENABLE_CLAMAV=0
- ONE_DIR=1
- ENABLE_FAIL2BAN=1
- ENABLE_MANAGESIEVE=1
- REPORT_RECIPIENT=1
- REPORT_INTERVAL=daily
- SSL_TYPE=letsencrypt
- SPOOF_PROTECTION=1
- POSTFIX_MAILBOX_SIZE_LIMIT=3000000000
- POSTFIX_MESSAGE_SIZE_LIMIT=52428800
- ENABLE_SPAMASSASSIN=1
- SA_TAG=2.0
- SA_TAG2=6.31
- SA_KILL=6.31
- SA_SPAM_SUBJECT=****SPAM****
Relevant Stack Traces
fail2ban status:
Every 60.0s: ./setup.sh debug fail2ban
Banned in dovecot: 212.70.149.70
Banned in postfix: 212.70.149.70
Banned in postfix-sasl: 212.70.149.70, 178.239.168.169
fail2ban-jail.cf
[DEFAULT]
# "bantime" is the number of seconds that a host is banned.
bantime = 604800
# A host is banned if it has generated "maxretry" during the last "findtime"
# seconds.
findtime = 10800
# "maxretry" is the number of failures before a host get banned.
maxretry = 3
mailserver log with grep on a banned IP:
docker-mailserver | Jan 22 10:16:42 mail postfix/smtps/smtpd[1874]: connect from unknown[212.70.149.70]
docker-mailserver | Jan 22 10:16:52 mail postfix/smtps/smtpd[1874]: Anonymous TLS connection established from unknown[212.70.149.70]: TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)
docker-mailserver | Jan 22 10:17:15 mail dovecot: auth: passwd-file(black@mydomain.tld,212.70.149.70): unknown user (SHA1 of given password: 3e5a4c)
docker-mailserver | Jan 22 10:17:17 mail postfix/smtps/smtpd[1874]: warning: unknown[212.70.149.70]: SASL LOGIN authentication failed: UGFzc3dvcmQ6
docker-mailserver | Jan 22 10:17:21 mail postfix/smtps/smtpd[1874]: lost connection after AUTH from unknown[212.70.149.70]
docker-mailserver | Jan 22 10:17:21 mail postfix/smtps/smtpd[1874]: disconnect from unknown[212.70.149.70] ehlo=1 auth=0/1 rset=1 commands=2/3
docker-mailserver | Jan 22 10:18:40 mail postfix/smtps/smtpd[1874]: connect from unknown[212.70.149.70]
docker-mailserver | Jan 22 10:18:49 mail postfix/smtps/smtpd[1874]: Anonymous TLS connection established from unknown[212.70.149.70]: TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)
docker-mailserver | Jan 22 10:19:13 mail dovecot: auth: passwd-file(bomb@mydomain.tld,212.70.149.70): unknown user (SHA1 of given password: 377d0e)
docker-mailserver | Jan 22 10:19:15 mail postfix/smtps/smtpd[1874]: warning: unknown[212.70.149.70]: SASL LOGIN authentication failed: UGFzc3dvcmQ6
docker-mailserver | Jan 22 10:19:18 mail postfix/smtps/smtpd[1874]: lost connection after AUTH from unknown[212.70.149.70]
docker-mailserver | Jan 22 10:19:18 mail postfix/smtps/smtpd[1874]: disconnect from unknown[212.70.149.70] ehlo=1 auth=0/1 rset=1 commands=2/3
docker-mailserver | Jan 22 10:20:37 mail postfix/smtps/smtpd[1874]: connect from unknown[212.70.149.70]
docker-mailserver | Jan 22 10:20:47 mail postfix/smtps/smtpd[1874]: Anonymous TLS connection established from unknown[212.70.149.70]: TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)
docker-mailserver | Jan 22 10:21:11 mail dovecot: auth: passwd-file(booking@mydomain.tld,212.70.149.70): unknown user (SHA1 of given password: 31db12)
docker-mailserver | Jan 22 10:21:13 mail postfix/smtps/smtpd[1874]: warning: unknown[212.70.149.70]: SASL LOGIN authentication failed: UGFzc3dvcmQ6
docker-mailserver | Jan 22 10:21:17 mail postfix/smtps/smtpd[1874]: lost connection after AUTH from unknown[212.70.149.70]
docker-mailserver | Jan 22 10:21:17 mail postfix/smtps/smtpd[1874]: disconnect from unknown[212.70.149.70] ehlo=1 auth=0/1 rset=1 commands=2/3
docker-mailserver | Jan 22 10:22:35 mail postfix/smtps/smtpd[1874]: connect from unknown[212.70.149.70]
docker-mailserver | Jan 22 10:22:44 mail postfix/smtps/smtpd[1874]: Anonymous TLS connection established from unknown[212.70.149.70]: TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)
docker-mailserver | Jan 22 10:23:08 mail dovecot: auth: passwd-file(boom@mydomain.tld,212.70.149.70): unknown user (SHA1 of given password: 7010e6)
docker-mailserver | Jan 22 10:23:10 mail postfix/smtps/smtpd[1874]: warning: unknown[212.70.149.70]: SASL LOGIN authentication failed: UGFzc3dvcmQ6
docker-mailserver | Jan 22 10:23:14 mail postfix/smtps/smtpd[1874]: lost connection after AUTH from unknown[212.70.149.70]
docker-mailserver | Jan 22 10:23:14 mail postfix/smtps/smtpd[1874]: disconnect from unknown[212.70.149.70] ehlo=1 auth=0/1 rset=1 commands=2/3
docker-mailserver | Jan 22 10:24:34 mail postfix/smtps/smtpd[1874]: connect from unknown[212.70.149.70]
docker-mailserver | Jan 22 10:24:43 mail postfix/smtps/smtpd[1874]: Anonymous TLS connection established from unknown[212.70.149.70]: TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)
docker-mailserver | Jan 22 10:25:06 mail dovecot: auth: passwd-file(boot@mydomain.tld,212.70.149.70): unknown user (SHA1 of given password: 41cdec)
docker-mailserver | Jan 22 10:25:08 mail postfix/smtps/smtpd[1874]: warning: unknown[212.70.149.70]: SASL LOGIN authentication failed: UGFzc3dvcmQ6
docker-mailserver | Jan 22 10:25:12 mail postfix/smtps/smtpd[1874]: lost connection after AUTH from unknown[212.70.149.70]
docker-mailserver | Jan 22 10:25:12 mail postfix/smtps/smtpd[1874]: disconnect from unknown[212.70.149.70] ehlo=1 auth=0/1 rset=1 commands=2/3
docker-mailserver | Jan 22 10:26:33 mail postfix/smtps/smtpd[1874]: connect from unknown[212.70.149.70]
docker-mailserver | Jan 22 10:26:42 mail postfix/smtps/smtpd[1874]: Anonymous TLS connection established from unknown[212.70.149.70]: TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)
docker-mailserver | Jan 22 10:27:06 mail dovecot: auth: passwd-file(bot@mydomain.tld,212.70.149.70): unknown user (SHA1 of given password: 6d041d)
docker-mailserver | Jan 22 10:27:08 mail postfix/smtps/smtpd[1874]: warning: unknown[212.70.149.70]: SASL LOGIN authentication failed: UGFzc3dvcmQ6
docker-mailserver | Jan 22 10:27:12 mail postfix/smtps/smtpd[1874]: lost connection after AUTH from unknown[212.70.149.70]
docker-mailserver | Jan 22 10:27:12 mail postfix/smtps/smtpd[1874]: disconnect from unknown[212.70.149.70] ehlo=1 auth=0/1 rset=1 commands=2/3
docker-mailserver | Jan 22 10:28:32 mail postfix/smtps/smtpd[1874]: connect from unknown[212.70.149.70]
docker-mailserver | Jan 22 10:28:41 mail postfix/smtps/smtpd[1874]: Anonymous TLS connection established from unknown[212.70.149.70]: TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)
docker-mailserver | Jan 22 10:29:05 mail dovecot: auth: passwd-file(bug@mydomain.tld,212.70.149.70): unknown user (SHA1 of given password: 083d1c)
docker-mailserver | Jan 22 10:29:07 mail postfix/smtps/smtpd[1874]: warning: unknown[212.70.149.70]: SASL LOGIN authentication failed: UGFzc3dvcmQ6
docker-mailserver | Jan 22 10:29:11 mail postfix/smtps/smtpd[1874]: lost connection after AUTH from unknown[212.70.149.70]
docker-mailserver | Jan 22 10:29:11 mail postfix/smtps/smtpd[1874]: disconnect from unknown[212.70.149.70] ehlo=1 auth=0/1 rset=1 commands=2/3
docker-mailserver | Jan 22 10:30:32 mail postfix/smtps/smtpd[1874]: connect from unknown[212.70.149.70]
docker-mailserver | Jan 22 10:30:42 mail postfix/smtps/smtpd[1874]: Anonymous TLS connection established from unknown[212.70.149.70]: TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)
docker-mailserver | Jan 22 10:31:06 mail dovecot: auth: passwd-file(cisco@mydomain.tld,212.70.149.70): unknown user (SHA1 of given password: bf4153)
docker-mailserver | Jan 22 10:31:08 mail postfix/smtps/smtpd[1874]: warning: unknown[212.70.149.70]: SASL LOGIN authentication failed: UGFzc3dvcmQ6
docker-mailserver | Jan 22 10:31:12 mail postfix/smtps/smtpd[1874]: lost connection after AUTH from unknown[212.70.149.70]
docker-mailserver | Jan 22 10:31:12 mail postfix/smtps/smtpd[1874]: disconnect from unknown[212.70.149.70] ehlo=1 auth=0/1 rset=1 commands=2/3
So I don’t know why this happens. Maybe someone familiar with the functionality can assist me in debugging this behavior further.
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 17 (17 by maintainers)
I’d agree this is fairly implementation specific, but nonetheless may serve as a helpful template/guide of a working example for other K8s operators. In the meantime since this has been non-functional on K8s, I’ve taken to just manually updating an ACL on my primary router when I see repeated IP ranges in the weekly logs. That is hardly a sustainable or ideal process and I’d love to automate it. At the end of the day what would be most helpful is if there was a built in way to fire a webhook off to indicate an IP or range needed banning, and let the operator handle it however they want (e.g., modify Cilium like @georglauterbach, update a custom ACL, etc).
I’m reviving this thread because I stumbled upon this problem two weeks ago and finally had time to tackle it. It is somewhat overengineered, I guess, but I actually quite like it. Please tell me whether we should add this to the documentation.
Fail2Ban recognizes the correct IP addresses in the logs, and hence we only need to take them and insert them into a configuration of a component that can enforce the ban. The Container Network Interface is very much suited for this issue, as it already manages cluster-internal, ingress, and egress traffic. Because we are running behind a proxy, we need to drop ingress traffic before it reaches the proxy - otherwise, we lose the actual origin IP (covered by the PROXY protocol).
We can tell F2B to invoke a script as the ban and unban action in
/etc/fail2ban/jail.local
:and provide an appropriate configuration for this action in
/etc/fail2ban/action.d/bash.conf
:Now, here is the somewhat tricky part; the script
/tmp/docker-mailserver/fail2ban-proxy.sh
depends slightly on the container network interface you’re using. You can write a script that uses default Network Policies, but I opted for Cilium Network Policies becauseCiliumNetworkPolicies
are more concise.Now we create the network policy:
Make sure to not have another network policy that (accidentally) allows all traffic on the PROXY protocol ports
My ingress, Traefik, lives in the
ingress
namespace. We use thespec.ingress.0.fromCIDRSet.0.except[]
list for disallowed IP addresses. Now that we know all the parameters, we can write thefail2ban-proxy.sh
script:Contents of
fail2ban-proxy.sh
IMO, the script is quite readable; but maybe this is just me having read too much Bash already I guess 😆 🤣
And that’s almost it. When you read through the script, you have already noticed that
send_payload_to_api_server
usesSERVICE_ACCOUNT_TOKEN
and the like. This is because Role-Based Access Control (RBAC) is enabled in my cluster (and you should have it enabled too!). Hence, we need to create aServiceAccount
, aRole
, and aRoleBinding
to allow the DMS pod to access theCiliumNetworkPolicy
(which is in another namespace by the way!).The
ServiceAccount
must be applied in the same namespace that DMS is deployed in:The
Role
andRoleBinding
have to be applied in the namespace that your ingress resides in:This way, we do not need
ClusterRole
s andClusterRoleBinding
s, and DMS can only accessCiliumNetworkPolicies
in the ingress’ namespace.And that’s it.
Should I add this to the Kubernetes documentation page?
CC @radicand @wernerfred @polarathene
Given how your traffic flow is laid out, iptables is the enforcer of the blocking rules and would need to be the one understanding the proxy headers on the incoming requests to enforce the blocks. I haven’t done any research around this, but I’d suggest starting there, and then opening a request with upstream fail2ban to create rules compatible with blocking origin from the proxy headers (assuming this is feasible to do). Outside of this, what I personally do (as I run this setup under Kubernetes and the iptables solution just doesn’t work at all), is I monitor brute requests in the daily Postfix summary email, and then block ranges on my router. It’s some effort at the start, but I’ve found most come from the same 20-30 ranges and I haven’t had to add anything in the last few weeks.
PS:
jaq
does currently not supportindex()
, but it will with the upcoming version 1.4.0. Hence, until v1.4.0 ofjaq
has been released, you will actually need to installjq
.I forgot about this completely 👍🏼
The solution proposed is agnostic the cluster except for the JSON paths in the network policy, it’s not custom tailored I’d argue.
👍🏼
I thought about this too, but it’s better to let Traefik handle application routing and Cilium IP routing and firewalling. In the end, Traefik is not supposed to act as a firewall. Cilium is definitely the better option.
That’s not the issue at all, and I only have one node. The issue is that, inside the container, the traffic comes from Traefik, and only by inspecting the package’s content would you discover the original IP. Having nftables drop the connection inside the container is useless, because there is only traffic from the proxy (Traefik
I don’t think that’s a particularly good solution either, mainly because it involves polling and I am not sure you’d actually implement it.
As this is the main problem there is no issue with the mailserver/fail2ban but a general issue when running software behind a proxy. Will close the issue and post an update anyway if I found a solution.
@aendeavor here you go: https://github.com/docker-mailserver/docker-mailserver/wiki/Installation-Examples#using-docker-mailserver-behind-proxy I wrote down a first shot. Will add links to blogs/docs that helped me during research later.
Interesting. I’m running Traefik as well, but the only service I did not put behind it is the mail server. I’d be interested in seeing your Traefik configuration. Could you post an anonymized version?
However, I can confirm Fail2Ban is fully operational when not running behind a proxy - so no issues there.