Cross-Monitoring with Another Self-Hoster

Self-hosting long enough leads to an awkward question: the machine you're monitoring is the one running the monitor, so who tells you when it dies?

I run Uptime Kuma to watch all my services, but it lives on the very machine it watches. A power cut, a dead uplink, a hard lockup, and the monitoring dies along with everything else. Not a single alert goes out.

The fix is to put the monitoring somewhere else. There's more than one way:

I went with the third one purely because the option was there: another self-hoster, each of us watching the other. Their Kuma watches me, mine watches them. The bonus is that the two sites have completely independent hardware, uplinks and power, and the other end is a human being who will just message you when things break.

This is a record of the whole thing: the certificate misconception that stalled us at the start, the v1 to v2 database migration, and the pitfall I found at the end. Turning on Trust Proxy made my setup more dangerous than leaving it off.

(The status page's look can be swapped out with custom CSS; the pixel theme above is what I settled on. Kuma keeps the status page separate from the dashboard: the dashboard needs a login, the status page can be public on its own and expose only the services you choose.)

No certificate exchange needed#

The initial setup was this: the other admin created a subdomain, status.<their-domain>, pointed its A record at my static IP, and Kuma ran on my machine.

Then we got stuck. My nginx uses Let's Encrypt, but the domain is theirs. Both of us assumed the same thing: for HTTPS to work, they'd have to hand me the certificate. They had a *.<their-domain> wildcard certificate, and I took that to be the only road available.

That premise was wrong from top to bottom.

One line, certbot --nginx -d status.<their-domain>, and it was done. Renewal came free with it: certbot.timer handles it exactly the way it does for my own domains.

A wildcard only binds "the moment of issuance"#

My misconception was "a wildcard is tied to DNS, so it has to be theirs." That's half right. The "tied to DNS" part refers to the validation method at the moment of issuance, which for a wildcard must be DNS-01. But that's a separate question from where the certificate lives afterwards.

Once signed, a certificate is just a file, sitting on whichever machine terminates TLS. It isn't bound to a DNS provider or to a particular server. And several different certificates can exist for the same hostname at once without conflict: a CA won't refuse to issue me a single-name certificate just because someone else holds a wildcard. Issuance isn't exclusive. Prove you control the hostname and you get your own.

Looked at the other way round, "just use their wildcard" walks straight into two problems:

Keep the three roles straight and it stops being confusing. The issuer is always the CA (Let's Encrypt). The applicant is me, since I run certbot and prove control. The domain owner is them, and the only thing they do is point the A record over, which was done long before any of this. Nobody needs to "sign on someone else's behalf."

Reverse proxy: don't forget WebSocket#

Kuma's dashboard leans heavily on Socket.IO to push live heartbeats and charts. If the reverse proxy is just one bare proxy_pass line, the WebSocket upgrade never happens. The symptom is that the live charts sit completely still, or the browser calls the page insecure (a ws:// mixed content connection inside an HTTPS page).

The lines the proxy block actually needs:

location / {
    proxy_pass http://127.0.0.1:3011;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_read_timeout 86400s;          # 24h, so nginx doesn't cut long-lived connections
}

One thing that's easy to misread: sending a raw upgrade request straight at /socket.io/ returns 400. That's Socket.IO itself rejecting a request with no sid parameter, not nginx blocking anything. Don't rush off to change nginx over that 400.

What's worth monitoring#

Once the services were wired up, two practices turned out worth writing down.

While you're in there, tick certificate expiry notification. Cross-monitoring already hits the other side's HTTPS on a schedule, so you may as well have it shout when a certificate is down to its last few days. It's a free extra layer of insurance.

curl -fsS "https://status.<your-domain>/api/push/<TOKEN>?status=up&msg=OK"

With everything connected, the dashboard ends up looking like this, a full column of heartbeat bars down the left:

v1 to v2: treat the migration as irreversible#

Later I moved Kuma from 1.23.17 to 2.4.0. The reason was the advanced features in 2.x, and before that I'd already learned a lesson the hard way: Kuma's login page is exposed on the public internet, so 2FA belongs on the account the moment it exists.

This upgrade is not a matter of swapping a tag. A few things are worth knowing first:

So the order is: back up, change the image to 2.4.0, bring it up, watch the migration log, verify.

In practice it ran far faster than the estimate. My side had 8 monitors and 12,278 heartbeats, and the migration finished in 4 seconds:

Uptime Kuma Version: 2.4.0
Aggregate Table Migration Completed
Listening on: ... ✓

One number will startle you at first glance: heartbeats went from 12,278 to 10,190, two thousand fewer. That's normal. The 2.x migration aggregates and compresses the old per-record heartbeats, and not a single monitor was lost.

Trust Proxy: more dangerous switched on#

After the v2 upgrade I noticed Trust Proxy was off in the settings, while my nginx does send X-Forwarded-For. With it off, Kuma treats every visitor as 127.0.0.1, which makes the login rate limit global: anyone brute-forcing the login page locks me out along with themselves.

So it looks like it should be on. And turning it on "should" be safe, since Kuma binds to loopback only, is unreachable from outside, and everything must pass through nginx. So I turned it on.

Then the test slapped me down.

$ # Test 1: no X-Forwarded-For
  Kuma logged → my real IP              ✅

$ # Test 2: forge an X-Forwarded-For myself
$ curl -H 'X-Forwarded-For: 203.0.113.99' https://status.example.com/...
  Kuma logged → 203.0.113.99            ❌ swallowed whole

(203.0.113.99 is from the RFC 5737 documentation range, so it can't reach a real person.)

Kuma believed my forged header completely. The fault isn't in Kuma, it's in this line of my nginx config:

proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
#                                 ↑ this appends, it does not overwrite

$proxy_add_x_forwarded_for means "keep the XFF the client sent and append $remote_addr after it." And Kuma (Express's trust proxy) reads the leftmost entry, which is the one the client supplied. So anyone can claim to be any IP.

The real risk: CrowdSec bans Cloudflare#

My first read was "the login rate limit is worthless, passwords can be guessed forever." But I already had 2FA on, so getting the password right still doesn't get you in. That wasn't the main problem.

The real problem sat with a different service. My machine runs CrowdSec, which in short is a modern fail2ban: it reads nginx's access log, spots someone scanning or brute-forcing, and bans that IP automatically.

Which is exactly where this goes wrong. Digging in, the topology turned out not to match my mental model. My own domain doesn't use Cloudflare's orange cloud, but theirs does, and it points at my house:

In other words, a slice of the requests reaching my nginx has passed through Cloudflare first (measuring recent logs, around 10% comes from CF ranges). When those requests are written to the access log, $remote_addr is the Cloudflare node's IP, not the real visitor, and that's the field CrowdSec uses to decide who to ban.

The consequences run in two directions, neither good:

• Someone scans or brute-forces me over that path → CrowdSec decides the attack comes from a CF node → it bans Cloudflare's ranges. The result is the other admin's status page becoming unreachable worldwide, and it's brutally hard to diagnose, because what got banned is CF rather than the attacker.
• Conversely, a real attacker hiding behind CF is invisible to CrowdSec, which only ever sees CF.

The fix: make XFF overwrite#

The problem is in nginx, not Kuma, and two things go together:

# 1. let nginx learn the real client IP behind Cloudflare
set_real_ip_from 173.245.48.0/20;   # generate every range from CF's official list, refresh periodically
real_ip_header CF-Connecting-IP;

# 2. make XFF overwrite instead of append
proxy_set_header X-Forwarded-For $remote_addr;

The first turns $remote_addr into the genuine visitor IP. The second throws away whatever the client sent and replaces it with nginx's own view of the origin, which kills the forgery. After the change, run the same forged request again to confirm Kuma no longer swallows a fake IP.

The side effects are good ones: CrowdSec starts seeing real attacker IPs, so the protection genuinely improves, and Kuma's logs show real IPs rather than a 192.168.0.1 injected by some device on the other end.

The one-page checklist#