How We Took Our Mail Server from Constant SSH Attacks to Peace and Quiet

TL;DR: We layered fail2ban, CrowdSec, and firewalld with a strict whitelist, eliminated self-lockout risk, and stopped manually blocking IPs. Here's the full story.

The Problem

Our Rocky Linux 10 mail server (running iRedMail) was under constant SSH brute-force attack. Daily Logwatch reports showed hundreds of failed login attempts from around the world — a rotating cast of compromised IPs spanning multiple continents.

Our initial response was manual:

# Block an entire /24 because of one attacker
firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="203.0.113.0/24" drop'

This worked — until it didn't. Our firewall accumulated 65+ manually curated rich rules, each one a band-aid. Then we hit a bigger problem: attackers started using Google Cloud Platform IPs.

We did what any frustrated admin would do:

sudo firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="34.0.0.0/8" drop'

That's 16.7 million IPs. It stopped the attack. It also blocked every legitimate GCP user on the planet.

We knew we had to do better.

The Wrong Fix: Whitelisting Cloud Ranges

At some point in the past, someone (probably us, half-asleep) added cloud provider ranges to our fail2ban and CrowdSec allowlists:

ignoreip = 127.0.0.1 127.0.0.0/8 10.0.0.0/8 172.16.0.0/12 192.168.0.0/16 198.51.100.0/28 203.0.113.0/16 192.0.2.0/16 34.0.0.0/8 ::1

That 34.0.0.0/8 at the end?

It meant every attacker using GCP was immune from fail2ban. Our fail2ban would dutifully detect the attack, log it, and then silently skip the ban because the IP was in the ignore list.

Attackers on GCP quickly learned they could brute-force us indefinitely.

The Self-Lockout Incident

While testing CrowdSec, one of us fat-fingered a password three times:

ssh -p 2222 wronguser@mail.example.com
# Enter wrong password 3 times

Then the workstation was banned. Not just on SSH — on every port:

ping mail.example.com        # ✅ Works
nc -zv mail.example.com 2222 # ❌ Connection refused
nc -zv mail.example.com 465  # ❌ Connection refused
nc -zv mail.example.com 587  # ❌ Connection refused

We couldn't SSH in, couldn't send mail through the server, couldn't use it as a relay. And we'd set bantime = 604800 — 7 days.

The cause: fail2ban had banned us across four jails simultaneously (sshd, sshd-preauth, sshd-timeout, recidive), and one of them used firewallcmd-rich-rules with the all-ports action.

We eventually got back in via a jump host on the same network. Then we rebuilt the whole setup properly.

The Right Architecture

Here's what we ended up with:

┌──────────────────────────────────────────────────────┐
│  Layer 1: Non-standard SSH port (2222)               │
│           Eliminates 99% of automated scans          │
├──────────────────────────────────────────────────────┤
│  Layer 2: SSH hardening                              │
│           PermitRootLogin no, MaxAuthTries 3         │
├──────────────────────────────────────────────────────┤
│  Layer 3: fail2ban (10 jails)                        │
│           Strict SSH jails (2 tries → 7 days)        │
│           Centralized whitelist in jail.local        │
├──────────────────────────────────────────────────────┤
│  Layer 4: CrowdSec                                   │
│           Behavioral detection + community blocklist │
│           Own nftables table, no firewalld conflicts │
├──────────────────────────────────────────────────────┤
│  Layer 5: firewalld (static)                         │
│           Permanent blocks for known repeat offenders│
│           NO cloud /8 blocks anymore                 │
└──────────────────────────────────────────────────────┘

What fail2ban Handles

SSH brute-force — 2 failed attempts = 7-day ban
Mail auth failures — 3 failed SASL = 1-day ban
Dovecot / webmail — various thresholds

Key insight: Instead of ignoring entire cloud ranges, we centralized the whitelist in one file (jail.local [DEFAULT]) and removed all per-jail overrides:

[DEFAULT]
ignoreip = 127.0.0.0/8 ::1 10.0.0.0/8 172.16.0.0/12 192.168.0.0/16 198.51.100.0/28 203.0.113.0/24

That 203.0.113.0/24 is our ISP's residential range — it covers wherever our admin workstation happens to be within the ISP's network. Our workstation's IP can rotate freely, we stay whitelisted.

What CrowdSec Handles

Dynamic behavioral detection for:

SSH brute-force (ssh-bf)
HTTP scanning (http-probing, http-cve-probing)
Postfix relay attempts
Nginx bad user agents

CrowdSec uses its own nftables table (crowdsec, crowdsec6) — zero conflicts with firewalld.

What firewalld Handles

Static blocks for IPs that keep coming back after fail2ban bans:

sudo firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="<persistent-attacker>" drop'

We removed every /8 cloud block. CrowdSec now handles GCP attackers dynamically — banning only the ones that actually misbehave.

The Whitelist That Saved Us

This is the single most important config in our setup:

ignoreip = 127.0.0.0/8 ::1 10.0.0.0/8 172.16.0.0/12 192.168.0.0/16 198.51.100.0/28 203.0.113.0/24

Range	Purpose
`127.0.0.0/8`	Localhost
`10.0.0.0/8`	Private network
`172.16.0.0/12`	Private network
`192.168.0.0/16`	Private network
`198.51.100.0/28`	Our office
`203.0.113.0/24`	Our ISP (dynamic residential)

What's NOT in here anymore:

❌ 34.0.0.0/8 (all of Google Cloud)
❌ 192.0.2.0/16 (example cloud provider A)
❌ 198.18.0.0/16 (example cloud provider B)

Those cloud ranges were our original sin. Removing them was the fix.

Why We Stopped at Dynamic ISP Range, Not IP

Our admin workstation uses a residential IP that rotates. Whitelisting a single IP is useless — tomorrow it's different. Whitelisting the ISP's entire /11 (8 million IPs) is reckless.

The sweet spot: whitelist the /24 your ISP hands out. It's:

Small enough that hostile actors are unlikely to be in it
Large enough to cover IP rotation within a period
Specific enough to be a precise rule

Finding it:

curl ifconfig.me
whois <your-ip> | grep -E "inetnum|route"

This gives you the exact CIDR your ISP uses.

The Test That Proved It Works

After all changes, we did the ultimate self-test:

# From the admin workstation
ssh -p 2222 wronguser@mail.example.com
# Enter wrong password 3 times

Then checked every layer:

for jail in sshd sshd-preauth sshd-timeout recidive; do
    sudo fail2ban-client status $jail | grep "$(curl -s ifconfig.me)" || echo "$jail: clean ✅"
done
sudo cscli decisions list | grep "$(curl -s ifconfig.me)" || echo "crowdsec: clean ✅"
sudo firewall-cmd --list-rich-rules | grep "$(curl -s ifconfig.me)" || echo "firewalld: clean ✅"

Every single check returned clean. 3 failed SSH attempts — normally enough to trigger bans in 4 jails — and nothing happened. The whitelist works.

The Results

Before

65+ manual firewall rules
Constant fail2ban self-lockouts
Cloud ranges whitelisted, defeating detection
34.0.0.0/8 emergency block because attackers were on GCP
Hours per week manually blocking IPs

After

1 centralized fail2ban whitelist
0 self-lockouts since the fix
Cloud ranges removed, fail2ban and CrowdSec both effective
Emergency /8 block removed
CrowdSec auto-banning attackers within seconds
Only checks needed: cscli alerts list once a day

Actual Numbers

Metric	Value
fail2ban active bans	54
CrowdSec decisions	Rolling, auto-expiring
Attackers blocked	100% of SSH attacks in last 24h
Self-lockouts	0
Manual intervention required	0

Deployment in 45 Minutes

For a second server, the process is:

Backup — cp -a /etc/fail2ban /root/fail2ban-backup/
Centralize whitelist — one file, jail.local [DEFAULT]
Remove per-jail ignoreip — sed -i '/^ignoreip\s*=/d' jail.d/*.local
Install CrowdSec — dnf install crowdsec crowdsec-firewall-bouncer-nftables
Create CrowdSec allowlist — cscli allowlists create my_trusted_ips
Remove cloud blocks from firewalld — if any exist
Test — 3 failed SSH attempts, verify no bans
Document — /root/SECURITY.md
Monitor — check daily for a week, then weekly

Total time: ~45 minutes for a clean deployment.

Lessons Learned

1. Never ignore cloud provider ranges globally

Blocking 34.0.0.0/8 because some GCP IP attacked you is like blocking a whole city because one resident jaywalked. It's lazy, it's blunt, and it will bite you.

Better: Use CrowdSec — it bans specific IPs based on behavior, so legitimate GCP users still reach you.

2. Centralize your whitelist

Scattered ignoreip values across 7 files caused a real problem: our admin IP wasn't in most of them, so we got banned in 4 jails simultaneously.

Better: One source of truth. jail.local [DEFAULT] is the canonical location. Remove all per-jail overrides.

3. Whitelist ranges, not single IPs, for dynamic admins

If your admin IP changes, a single-IP whitelist is useless. Use the smallest CIDR that covers your likely IPs.

Better: whois your IP, find the smallest route, whitelist that.

4. Layer defense, don't rely on one tool

fail2ban is great for immediate bans. CrowdSec is great for behavioral detection and community intelligence. firewalld is great for persistent blocks.

Better: Use all three. Each covers the others' weaknesses.

5. Test with real failures before declaring victory

We didn't just configure and hope. We deliberately failed auth 3 times and verified every layer stayed clean.

Better: If you can't test it, you don't know it works.

6. Document your emergency recovery path

We ended up locked out once. It took 20 minutes to figure out how to get back in.

Better: Write down how to unban from every layer, save it locally, and test the recovery path.

The Files That Matter

If you want to replicate this setup, here are the exact files:

/etc/fail2ban/jail.local

ini

[DEFAULT]
findtime    = 3600
bantime     = 3600
maxretry    = 2
ignoreip    = 127.0.0.0/8 ::1 10.0.0.0/8 172.16.0.0/12 192.168.0.0/16 <OFFICE> <ISP>

/etc/fail2ban/jail.d/00-firewalld.conf (default)

ini

[DEFAULT]
banaction = firewallcmd-rich-rules
banaction_allports = firewallcmd-rich-rules

CrowdSec allowlist

text

my_trusted_ips:
  - <OFFICE_RANGE>
  - <ISP_RANGE>

firewalld state