Replacing fail2ban with CrowdSec

It didn't start with a breach. It started with noise.

My nginx feeds netdata, and anything unusual goes to Discord. It looks like this, and it even pastes the current Top IP and Top Path list along with the alert:

The diagnostics are generous. The problem was that it fired hundreds to thousands of times a day, and almost none of it meant my services were in trouble. It was scanners hammering /wp-login.php, /.env and every .php endpoint they know, generating 4xx. My site has no PHP at all, but they don't know that and they don't care. They just keep knocking.

(That particular alert is one of the rare genuine ones: /api/spotify/audio-features was hit 9,968 times in five minutes, and that was my own code stuck in a loop. I redacted the IPs, since those are my own egress addresses.)

The machine already had fail2ban on it, and the numbers looked good:

614 IPs banned sounds useful. But staring at that stream of notifications, one thing struck me: they keep changing IP. Ban one and the next arrives on a different line. Does banning your way down this road have an end at all?

This is the full record of replacing it with CrowdSec: why it suits this situation better than fail2ban, the three pitfalls I hit during setup, what this machine actually blocked over six days, and the three times I locked myself out (the third is the dumbest and the most worth writing down). It ends with the Discord unblock bot I built, because when your own defences shut you out, SSH is exactly the thing you can't reach.

What this machine looks like#

The pitfalls and the self-inflicted bans both depend on the architecture, so let me set the scene first.

A few points that matter later:

• Everything runs on one physical machine, on a home fixed line with a static IP, with the router forwarding 80/443 in. No cloud load balancer, and no second box to take over when something breaks.
• Blocking happens at nftables, which is upstream of nginx. A blocked request never even reaches the access log, and from your side it looks like a silent timeout rather than a refused connection. That positional relationship underpins every diagnosis later on.
• The CrowdSec engine and the bouncer are separate pieces: the engine reads nginx's log and decides, the bouncer enforces at the firewall. The two dashed lines in the sketch are that loop.
• A dozen subdomains sit behind nginx: the blog, the NAS, monitoring, MQTT, an assortment of small tools. A rate limit or an error in any one of them feeds the same access log, and therefore feeds CrowdSec.

One more thing that matters here: my main domain runs Cloudflare grey-cloud (DNS only, traffic goes straight to my house), but another admin points one of his subdomains at me through the orange cloud, and on that path the source IP is a Cloudflare node. That fact bit me once in the cross-monitoring article, written up here:

Why not fail2ban#

fail2ban isn't broken. It does its job exactly as advertised. The issue is its model: read your own log, accumulate your own failure counts, ban the IPs you personally have seen.

Against an adversary that rotates IPs, that's whack-a-mole. Every new IP is brand new to you and has to climb from zero to the threshold before it gets banned, and by then it has finished its sweep.

CrowdSec changes the premise: bad IPs other people have already met, you get to use directly.

That third row is a real architectural difference: CrowdSec splits detection from enforcement. The engine only decides who should be blocked; a separate bouncer touches the firewall. That split becomes important later.

One more thing was decisive for me: it's self-hosted. The other option at the time was putting the site behind Cloudflare and letting it absorb the scanning, but that means handing my traffic to someone else's proxy. CrowdSec reads the log locally and enforces on the local firewall, leaving my existing architecture untouched.

Three pitfalls, all from one root cause#

Installing CrowdSec is two lines:

curl -s https://install.crowdsec.net | sudo sh
sudo apt install -y crowdsec
sudo apt install -y crowdsec-firewall-bouncer-nftables   # Ubuntu 24.04 defaults to nftables

Then I lost an entire evening, and all three pitfalls were one domino knocking over the next.

old  /var/lib/crowdsec/hub/...
new  /etc/crowdsec/hub/...

Source                          Lines read   Lines parsed
file:/var/log/nginx/access.log  41           3

The fix is to clear the broken links and reinstall the rule sets:

sudo find /etc/crowdsec -xtype l -delete     # deletes only symlinks whose target doesn't exist
sudo cscli hub update
sudo cscli collections install \
  crowdsecurity/nginx crowdsecurity/sshd crowdsecurity/linux \
  crowdsecurity/whitelist-good-actors --force
sudo systemctl reload crowdsec

Two diagnostics that are easy to misread#

cscli metrics counters are cumulative. I spent a while thinking the parse rate hadn't improved after a fix, when newly parsed lines were simply being diluted by the pile of earlier failures. reload does not clear the counters; only restart does. To see the true post-fix ratio, restart and let it accumulate again.

Use cscli explain for a definitive answer. It prints the parser chain a single log line actually travelled, so whichever stage failed is immediately obvious:

sudo cscli explain --file /var/log/nginx/access.log --type nginx | tail -25

Once fixed you should see crowdsecurity/nginx-logs pass in green and flow down into scenarios, instead of nothing but parser failures.

Keep only one SSH source#

One more I tuned myself. SSH events can be read from /var/log/auth.log or from journalctl, and the two are two sources for the same set of events. Collecting both counts every failed login twice, which fires scenarios earlier than the threshold you configured and inflates the alert count.

Measured, both yielded 261 events, but auth.log needed 10.13k lines read to get there while journalctl needed only 330. So my acquis.yaml keeps journalctl alone:

source: journalctl
journalctl_filter:
  - "_SYSTEMD_UNIT=ssh.service"
labels:
  type: syslog

Six days, 1,619 attacks#

These are this machine's real numbers over the last six days (full cscli alerts list statistics):

Broken down by attack type:

The long tail is a parade of CVE probes: CVE-2017-9841 (PHPUnit RCE), http-cve-2021-42013, CVE-2022-41082 (Exchange), thinkphp-cve-2018-20062, netgear_rce, fortinet-cve-2018-13379, jira_cve-2021-26086. I run none of those, and they come knocking every day regardless.

The attacks don't come from "a bad guy's computer"#

The distribution of source AS numbers is the most interesting column of all:

The top seven are almost entirely cloud datacentres: Azure, DigitalOcean, Alibaba Cloud, Hetzner. The scanners run on VMs billed by the hour. Ban one and spinning up another takes seconds, on a different IP.

Which answers my original question straight from the data: on the ban-the-IP road, your opponent's costs are far lower than yours. And that's precisely where the community blocklist earns its keep, because that freshly spun-up VM has usually already hit somebody else before it reaches you.

Origin                       active_decisions
CAPI (community blocklist)   25.21k
crowdsec (security engine)        8

Not everything blocked is hostile#

One detail is worth being clear about. A ban list at one point contained 167.94.146.48, stopped by crowdsecurity/http-bad-user-agent. Looking it up, it belongs to Censys, a research organisation that surveys the internet, as does Shodan. They aren't trying to hit you; they're cataloguing the world's public services.

Whether you block them is your call, but it's better to know what you're blocking. Incidentally, well-behaved crawlers are not a worry: the whitelist-good-actors collection is installed by default, Google, Bing and the various SEO crawlers are all whitelisted, and it uses forward-confirmed reverse DNS rather than trusting the User-Agent, so they can't be impersonated.

I locked myself out three times#

This is the part most worth remembering. A system that bans automatically will eventually ban you.

Ip:<my egress IP>  crowdsecurity/nginx-req-limit-exceeded  ban  6 events

Three signals let you tell immediately#

Being blocked by CrowdSec and having a service go down really do look different:

When all three hold, you can jump almost straight to cscli decisions list and look for yourself.

The fix has two layers#

Use a whitelist parser for the LAN. Drop a parser into s02-enrich so all of RFC1918 can never be banned:

# /etc/crowdsec/parsers/s02-enrich/lan-whitelist.yaml
name: koimsurai/lan-whitelist
description: "Never ban LAN / RFC1918"
whitelist:
  reason: "private LAN"
  cidr:
    - "192.168.0.0/16"
    - "10.0.0.0/8"
    - "172.16.0.0/12"

That rule has let 16,037 events through so far, which gives you a sense of how annoying life would be without it.

Use an allowlist for public egress IPs. The LAN whitelist can't save you when you're out of the house, since your source is then a public IP:

sudo cscli allowlists create home-dev -d "my dev machine IPs"
sudo cscli allowlists add home-dev <your egress IP>
sudo cscli allowlists inspect home-dev

Letting myself out via Discord#

The shared problem across all three incidents is this: to lift a ban, you first have to get in. And when I was blocked, SSH was precisely the thing I couldn't reach.

The fallback at the time was my phone. I connect with Termux, which I like a lot under normal circumstances; it has everything you need. But typing sudo cscli decisions delete --ip ... one-handed on a touch keyboard while out somewhere, after first picking your own IP out of an entire table, is not what I'd call a good experience.

One more thing makes it worse: CrowdSec doesn't notify you about who it banned. Those notifications lines in profiles.yaml are all commented out unless you wire up the slack / http / email plugin yourself. So being blocked by your own system never announces itself. You simply discover you can't connect, and start guessing.

# /etc/crowdsec/profiles.yaml — this is the default, notifications all commented
decisions:
 - type: ban
   duration: 4h
# notifications:
#   - slack_default
#   - http_default

So I added two commands to my existing Discord bot, solving "look it up" and "lift it" together. But there's an architectural problem: the bot runs in a container while CrowdSec is a host service, and a container can't reach the host's cscli.

Mounting the docker socket or granting host root into the container would solve it, but that hands an internet-facing bot control of the entire machine. So I used a file queue instead:

The only thing the container can do is drop a request file into the queue. What actually executes cscli is a watcher running as root on the host, which reads the request, validates it, runs it, and writes the result back. Even if the bot were compromised, an attacker could only drop request files, restricted to a fixed set of actions, and would never gain control of CrowdSec.

The key lines of the watcher:

ALLOWLIST = "parole"          # add to this allowlist right after unbanning
ALLOWLIST_TTL = "4h"          # only 4 hours, never a permanent pass

def valid_ip(s):
    try:
        ipaddress.ip_address(s)   # blocks command injection
        return True
    except ValueError:
        return False

if action == "unblock":
    ip = str(req.get("ip", ""))
    if not valid_ip(ip):
        return {"ok": False, "output": f"invalid ip: {ip}"}
    rc, out = run(["decisions", "delete", "--ip", ip])
    ensure_allowlist()
    run(["allowlists", "add", ALLOWLIST, ip, "-e", ALLOWLIST_TTL])

Three design details:

Once built, /crowdsec_list lays the current ban list out on your phone, each entry with its IP, the scenario that triggered it, and the time remaining:

Once you've spotted your own entry, /crowdsec_unblock with the IP lifts it, and the reply tells you how long the parole runs:

Those IPs incidentally illustrate something: http-cve-2021-41773, http-wordpress-scan, http-sensitive-files are all the same handful from the earlier statistics, and looking up their AS numbers turns up Google Cloud and Azure again. Even the batch I grabbed as an illustration is still scanners running in cloud datacentres.

The thing that finally makes all of this work is quite simple: the bot dials out to Discord. Your IP being DROPped at the firewall blocks the inbound direction, and has no effect at all on the bot's existing outbound connection. So while you're locked out, it still receives your commands.

A web admin panel hosted on your own machine cannot do this, because you have to reach it to use it, and reaching it is exactly what you can't do. SSH from a phone does route around the block (mobile data is a different IP), but you still have to type full commands on a phone.

Back to the beginning: did Discord go quiet?#

It did, but not for the reason I had planned.

Alongside installing CrowdSec, I had also prepared a netdata alert-silencing file, intending to route the 4xx group to silent and keep only 5xx and "getting slower". It was written and the commands were ready.

Then I went and looked just now:

$ ls /etc/netdata/health.d/
(empty)

$ ls ~/Server/netdata-web-noise-silence.conf
-rw-rw-r-- 1 timo9378 timo9378 2922  netdata-web-noise-silence.conf

That silencing file was never deployed. It's still sitting in my home directory. And I genuinely haven't been bothered by a 4xx alert in a long time.

The reason is that the 4xx went away by itself. Sampling 6,038 lines from the same access log today:

The scanners get dropped by the bouncer at the firewall before they ever reach nginx, so that 404 never appears in the log at all, the 4xx ratio naturally stays under the threshold, and the alert never fires.

This outcome is considerably better than what I had planned. Silencing covers the gauge; this is nothing actually happening. Had I deployed the silencing file first, netdata would now stay quiet even during a genuine scanning wave and I'd have lost a useful signal. As it stands, when it does fire there's usually something real, like that alert at the top of the article, which was my own Spotify API stuck in a loop.

What became of fail2ban#

Honestly, I never formally retired it. After CrowdSec went live I simply stopped paying attention, and only later discovered it had settled into an awkward state: the service isn't running, and the config files are still there.

$ systemctl is-active fail2ban
inactive
$ systemctl is-enabled fail2ban
disabled

How it stopped is a little funny: while adding a LAN whitelist I dropped a file into jail.d/, and that file made it fail to parse at boot, so it never came back up. I didn't notice, because CrowdSec was blocking throughout that period and there were no symptoms at all.

The lesson here isn't "fail2ban is bad". It's this: when two overlapping defences coexist, the real danger isn't a conflict, it's believing both are on watch. Had it been CrowdSec that stopped back then, I'd probably have taken just as long to find out. If you're switching, switch cleanly, and confirm the new one is genuinely running.

The one-page checklist#