The week after I moved house, my server started shutting itself down every night.

Not a crash, not a reboot — death by power loss: every boot ran an ext4 recovery, uptime reset to zero, and whatever had been running a second earlier left no note behind. It also picked its moment. Almost always the small hours.

My first thought was a loose cable, so I reseated everything. It died again the next night. I wondered whether the air conditioning and water heater were sharing a circuit with it after the move, so I turned them off. It died again. I checked temperatures: 50 to 60°C, perfectly normal. Eventually I started suspecting the move had shaken the PSU loose internally — a Super Flower platinum unit less than a year old, which I didn't much want to believe.

The problem is that every one of those was a guess, and each one took a full day to test. After a few days of this I realised what I was missing wasn't a better hypothesis. It was evidence: I had no idea what the power going into that machine looked like at the instant it died.

So I bought a UPS. Half for protection, and half to use as a measuring instrument.

This is everything that followed: how to pick the UPS, setting up NUT from nothing through to automatic safe shutdown, the one BIOS option you cannot afford to get wrong, and finally the UPS logs closing the case — the culprit being a power cable the cat had chewed.

Why a UPS can settle this#

Let me be clear about why it counts as an instrument, because the logic of everything below depends on it.

A UPS measures the state of its own output, and it reports what it's doing using a few short codes. You'll see them throughout, so here they are up front:

StatusMeaning
OLOn Line — mains is fine, the UPS is just standing by
OBOn Battery — mains is gone, it's discharging
LBLow Battery — nearly empty, time to shut down

With those three, a UPS that keeps logging can cleanly split the next death into two possibilities:

  • It recorded OB or a voltage drop at the moment of death → the problem is upstream of the UPS, mains is at fault.
  • It stayed OL the whole time, voltage steady, battery full → the power leaving the UPS was clean, so the problem is downstream: the power cable, the PSU, the motherboard.

That line cuts the search space in half, and it does it without guessing.

Picking one: 1500VA, pure sine wave, with AVR#

The CyberPower CP1500PFCLCD had three properties that mattered specifically for this diagnosis:

PFC pure sine wave output

Modern PC power supplies are almost all active PFC. Those units can react badly to the that cheap UPSes produce, and can simply cut out at the moment of transfer to battery — which would have manufactured a brand new problem with symptoms identical to the one I was chasing. That is the last thing you want mid-diagnosis.

Built-in AVR

eliminates one hypothesis outright: if the mains voltage really was fluctuating in the small hours, AVR flattens it on the spot. If the machine still dies after that, mains is off the list.

Full NUT support

can read it through the usbhid-ups driver, with no vendor software required. Plug in the USB cable and lsusb shows:

console
Bus 001 Device 005: ID 0764:0601 Cyber Power System, Inc.

NOTE

You don't need to plug in the network port This UPS has both USB and serial, and NUT over USB is enough. The network port is for models with a management card, or for sharing one UPS across several machines. For monitoring a single box it does nothing.

NUT: from nothing to automatic safe shutdown#

NUT's configuration is scattered across several files, and the service won't start at all if the permissions are wrong. I wrote the whole thing as one script so I can re-run it after a reinstall or a hardware change.

Understand the three layers first#

The config files look messy because NUT isn't one program — it's three, each with a distinct job. Once you know which is which, the files sort themselves out:

  • driver (mine is usbhid-ups) — the one that actually talks to the UPS, translating the vendor's protocol into NUT's data model.
  • upsd — takes what the driver collected and serves it, so other things can ask.
  • upsmon — the client. It continuously asks upsd for data and runs the shutdown when the battery gets low.

Which makes the five config files fall out like this:

bash
/etc/nut/nut.conf      # what role this machine plays (standalone or client/server)
/etc/nut/ups.conf      # driver: which driver, which UPS
/etc/nut/upsd.conf     # upsd: which address the data service listens on
/etc/nut/upsd.users    # upsd: who can read, who can issue commands
/etc/nut/upsmon.conf   # upsmon: when to shut down, and how to notify

Single-machine monitoring uses standalone:

ini
# /etc/nut/nut.conf
MODE=standalone
ini
# /etc/nut/ups.conf
maxretry = 3
pollinterval = 5

[cyberpower]
    driver = usbhid-ups
    port = auto
    desc = "CyberPower CP1500PFCLCD"
ini
# /etc/nut/upsd.conf — localhost only, nothing exposed
LISTEN 127.0.0.1 3493

WARNING

Change the LISTEN default If NUT's upsd listens on 0.0.0.0, anyone on your LAN can query your UPS state and, if the permissions aren't right, issue commands to it. There is no reason at all to expose it for single-machine monitoring — bind 127.0.0.1. For the password I generate one with openssl rand -hex 16 and write it straight into the file, since nobody has to remember it.

Permissions matter too, or the driver can't read its own config:

bash
chown root:nut /etc/nut/*.conf /etc/nut/upsd.users
chmod 640 /etc/nut/*.conf /etc/nut/upsd.users

How the safe shutdown actually happens#

It comes down to these lines in upsmon.conf:

ini
MONITOR cyberpower@localhost 1 upsmon <password> primary
MINSUPPLIES 1
SHUTDOWNCMD "/sbin/shutdown -h +0"
POWERDOWNFLAG /etc/killpower
POLLFREQ 5
POLLFREQALERT 5
FINALDELAY 5

Line by line:

DirectiveWhat it does
MONITORWhich UPS to watch and with which credentials. The trailing primary means this machine is allowed to tell the UPS to cut power as part of shutting down
MINSUPPLIESHow many power supplies must still be alive to keep running. 1 for a single-PSU machine
SHUTDOWNCMDWhat to run when it's time to shut down
POWERDOWNFLAGThe flag file the shutdown sequence leaves behind — covered below
POLLFREQ / POLLFREQALERTHow often to ask upsd normally, and how often once something is wrong
FINALDELAYHow many seconds of grace before the shutdown command goes out

The actual sequence looks like this, and the last stage is the one most people miss:

POWERDOWNFLAG points at a flag file. When upsmon decides it's time to shut down it creates that file first, and at the very end of the shutdown sequence the system scripts check for it: if it's there, they also tell the UPS to cut its own output.

Skip this and the machine goes down while the UPS keeps discharging, draining the battery to zero — and every deep discharge takes life off a lead-acid battery.

And if you skip the BIOS auto power-on box, everything before it is wasted: the machine shuts down cleanly and then sits there waiting for someone to press the power button.

Choosing the shutdown threshold#

The trigger is whatever the UPS itself considers low battery. The factory values on mine:

console
battery.charge.low: 10        # 10% charge remaining
battery.runtime.low: 300      # or an estimated 300 seconds left

Starting with the factory values is deliberate, because they're the least likely to misfire. Once you have real numbers from a few weeks of running, you can decide whether to trigger earlier. Mine currently measures:

12%
UPS load
1000W rated
61min
Estimated runtime
113V
Input voltage

At 12% load with an hour of runtime, starting the shutdown with five minutes left is plenty of margin. If your machine pushes the UPS to 60% and the runtime is down to the low teens, five minutes gets tight — at which point upssched can add an earlier trigger, for example "90 seconds on battery, start shutting down".

The BIOS option you cannot get wrong#

This is the single easiest step in the whole chain to ruin.

CAUTION

Restore after AC Power Loss must be On, not Last State Last State means "return to whatever state you were in before power was lost". But after NUT's low-battery shutdown, that state is off — so when mains returns the BIOS concludes the machine was supposed to be off, and leaves it off. You build the entire automatic recovery chain and it dies on the last step. On is the one that means "power up whenever there's power", regardless of what came before.

My board is an MSI MAG X870E TOMAHAWK, so the path is:

text
Press Del at boot → F7 for Advanced Mode
  → SETTINGS → Advanced → Power Management Setup
    → Restore after AC Power Loss = On

The BIOS runs before the operating system, so SSH and remote desktop can't see any of this — you need a physical screen and keyboard. To skip the frantic Del-mashing:

bash
sudo systemctl reboot --firmware-setup

That reboots straight into the firmware setup screen. You still have to be standing in front of it.

Getting the UPS state out#

Once it's set up, upsc will tell you anything you want. But I didn't want to SSH in and type a command every time.

One log line per minute#

This script runs from cron once a minute and does two things at once: writes a human-readable log line, and a JSON file for programs to read.

bash
DATA=$(upsc cyberpower 2>/dev/null) || exit 0
g() { grep -m1 "^$1:" <<<"$DATA" | cut -d' ' -f2-; }

echo "$(date '+%F %T') status=$(g ups.status) in=$(g input.voltage)V \
out=$(g output.voltage)V load=$(g ups.load)% batt=$(g battery.charge)% \
runtime=$(g battery.runtime)s" >> "$OUT"

Those log lines look like this, and they are what eventually closed the case:

console
2026-07-05 00:12:01 status=OL in=113.0V out=113.0V load=12% batt=100% runtime=3675s
2026-07-05 00:13:01 status=OL in=113.0V out=113.0V load=12% batt=100% runtime=3675s
2026-07-05 00:14:01 status=OL in=113.0V out=113.0V load=12% batt=100% runtime=3675s
                    ↑ the machine hard-cut moments after this line

Pushing events to Discord#

upsmon.conf can hang an external command off every event type. Add a NOTIFYCMD line, then append +EXEC to the flags of the events you want:

ini
NOTIFYCMD /path/to/ups-discord-notify.sh
NOTIFYFLAG ONBATT   SYSLOG+WALL+EXEC
NOTIFYFLAG LOWBATT  SYSLOG+WALL+EXEC
NOTIFYFLAG ONLINE   SYSLOG+WALL+EXEC

I push six events: transferred to battery, battery low and shutting down soon, shutdown triggered, mains restored, battery aged and due for replacement, and communications lost. Each message carries the current charge, minutes remaining, load and mains voltage.

The webhook file has to live somewhere upsmon can read it, owned root:nut with mode 640. That's the thing that stalled me the first time round — I'd put it under my home directory, where upsmon couldn't reach it.

Not standing up another subdomain#

I considered Home Assistant's NUT integration, or running something like peaNUT. But I already have plenty of subdomains, and spinning up an entire website to look at one UPS is excessive.

So instead it goes into the NAS dashboard I already run. There's one constraint: the backend runs in a docker bridge network and cannot reach the host's upsd:3493. Rather than expose upsd, let the host write a file and the container read it:

The JSON is the same thing the script above produces on the side:

json
{
  "status": "OL",
  "online": true,
  "battery_charge": 100,
  "battery_runtime": 3675,
  "ups_load": 12,
  "input_voltage": 113.0,
  "model": "CP1500PFCLCDa TW",
  "updated_at": "2026-08-04T13:55:01+0800"
}

This matches how the backend already reads other mounted files, and it means not opening a network permission for one read-only value.

NOTE

Something I found on the way Halfway through I noticed I couldn't see the monitoring page on my phone, and assumed the PWA was at fault. It wasn't — there is no PWA, no manifest and no service worker. What the phone gets is a completely separate mobile layout, and the monitoring page only exists in the desktop dock. So it wasn't "the PWA didn't carry it over", it was "the mobile layout never had it". That pattern — assuming problem A when it's actually problem B — showed up more than once in this whole affair.

Case closed: a cable the cat had chewed#

A few days after the UPS came online, the machine died again.

I went through the logs. Before, during and after: OL the entire time, mains steady at 113 to 115V, battery at 100%, and not a single OB event. nut-monitor had nothing either. But the boot afterwards was another ext4 recovery.

Clean power in, hard cut anyway. The problem was definitively downstream of the UPS.

With the search narrowed to the power cable, the PSU and the motherboard, I opened the case, reseated every connector, and swapped the power cable while I was at it. The old one was rated 12A 125V — and it had a hole in it where the cat had chewed through.

That cable sits exactly between the UPS output and the PSU inlet, which is completely outside the UPS's field of view. A cable with half its copper severed makes intermittent contact: a little vibration or thermal expansion and it goes connected, disconnected, connected. Which is precisely what "random, instantaneous, no warning" power loss looks like.

It also explained everything that hadn't added up:

ObservationHow the chewed cable explains it
UPS logs clean throughoutThe break is downstream of where the UPS measures, so it can't see it
Dies even at idleNothing to do with load — it's a mechanical contact failure
Only started after moving houseThe vibration of the move made the half-severed strands easier to separate
Prefers the small hoursLowest temperature, so thermal contraction pulls the contact apart

The replacement is rated 10A 125V. That looks like a downgrade, but 10A × 125V is 1250W, and the whole machine at full load draws roughly 350 to 400W — under 3A. Enormous margin.

1day
Before the swap
average interval between deaths
10.8days
After the swap
continuous uptime

From "dies almost nightly" to "dies once in ten days". A tenfold reduction.

The relapse that nearly misled me#

Eleven days after the cable swap, it died again. And my gaming PC rebooted at the same time.

Two machines going down together points at one obvious conclusion: a real power cut. From there the reasoning unfolded neatly — if mains genuinely failed, the UPS should have carried the server; since it didn't, the power cable must be plugged into the "Surge Only" bank rather than "Battery Backup". The two banks on the back of a CyberPower look nearly identical, and getting that wrong is extremely common.

The whole chain was very persuasive, right up until I checked why the gaming PC had rebooted.

Windows Update.

The premise collapsed and the entire chain went with it. Going back through the logs, that death was identical to the previous four: OL throughout, mains steady, battery full, no OB. Not a power cut, and not the wrong outlet bank — the same downstream dropout as before, just at a tenth of the frequency.

TIP

Two independent events coinciding is the easiest way to reason yourself off a cliff "Both machines went down" looks like about the strongest common-cause evidence you could ask for. It was a coincidence. When a new piece of evidence suddenly makes your hypothesis extremely persuasive, that is precisely the moment to go back and verify the evidence itself. If I hadn't gone and checked the Windows Update history, I'd have spent an entire evening studying UPS outlet groupings.

So what was that one, actually? Honestly, I never found out.

All I could establish was what it wasn't: not mains (the UPS logs were clean throughout) and not heat (85% idle at the moment of death, load 1 to 3, zero thermal alarms, temperatures normal). The most likely remaining explanation is that one end of the new cable wasn't fully seated, or another internal connector had worked loose again — C13/C14 connectors that feel seated but aren't are extremely common. I pushed both ends home again, and then nothing more came of it.

Nothing more came of it not because I gave up, but because four days later it died again in a completely different way. My attention moved entirely, and this residual fault never recurred often enough to verify. So the honest conclusion is: the chewed cable cut the frequency tenfold, and the remainder is still unsolved.

I did do one more thing that day, though. Since "overheating" had been ruled out rather quickly, I added a temperature logger — CPU temperature and load, once a minute — as insurance for next time.

Four days later it earned its keep immediately:

console
15:12  cpu=90.1C  load=8.65
15:22  cpu=90.0C  load=32.53
15:31  cpu=90.0C  load=84.42
15:43  cpu=90.0C  load=156.54
15:50  cpu=90.0C  load=196.01   ← cron can no longer get scheduled

That one wasn't power loss. That was the system cooking itself: temperature pinned at 90°C for forty minutes, load climbing from 8 to 196, with the UPS normal throughout. Nothing to do with electricity, and a different story entirely.

NOTE

Every investigation leaves an instrument behind, and the next case gets cracked by the one the last case left I bought the UPS to investigate power loss, and its logs cracked the chewed cable. I added the temperature logger to rule out heat, and four days later it caught the thermal runaway red-handed. That isn't luck. It's what happens when the thing you add at every "I'm not sure" is a recorder rather than a guess — those recorders start accumulating evidence before you know you need them.

Where it stands as of writing#

journalctl --list-boots will tell you whether each reboot was a clean shutdown or a hard cut: the ones with systemd-shutdown at the end are clean, the ones that leave nothing behind are not.

console
07-14 → 07-15 20:21   no shutdown record   ← the residual power fault
07-16 → 07-19 15:56   no shutdown record   ← the thermal runaway
07-19 → 07-20 17:15   systemd-shutdown
07-20 → eight more    systemd-shutdown     ← all of them me rebooting it

The last unplanned death was 07-19, and there hasn't been a single hard cut in the sixteen days since.

But the residual power fault cannot be called closed. I've rebooted the machine eight times in that window for various reasons, so the longest single continuous run is only seven days — while the interval that produced the last power fault was 10.8 days. Which means no continuous run so far has exceeded the interval between the previous failures. It hasn't been proven fixed; it just hasn't had the opportunity.

To actually declare it closed, it needs to sit quietly for more than eleven consecutive days. That's a bar I set for myself, and I'm writing it down because "it hasn't happened since" isn't evidence at all unless you attach a timescale to it.

The one-page checklist#

UPS + NUT self-hosting checklist
  1. Three things matter when choosing a UPS: PFC pure sine wave (active-PFC supplies are picky about waveform), AVR (voltage excursions shouldn't touch the battery), and NUT support (no vendor software required).
  2. The UPS is also an instrument: log ups.status / input.voltage / battery.charge once a minute, and the next failure can be cleanly split into "mains problem" and "downstream of the UPS problem".
  3. Bind upsd to 127.0.0.1 — single-machine monitoring has no reason to be exposed.
  4. Config permissions root:nut 640, or the driver won't start.
  5. Don't omit POWERDOWNFLAG, or the battery drains to zero after the OS goes down.
  6. BIOS On, not Last State — get this wrong and the machine never comes back after NUT's clean shutdown.
  7. Start with the factory shutdown threshold, and only move to upssched once you have real runtime numbers.
  8. Keep the webhook file under /etc/nutupsmon can't read your home directory.
  9. Use a file to hand state to containers: host writes JSON, bind-mount it in. Cleaner than exposing upsd.
  10. Route cables through trunking or put them in a chew-proof sleeve. This one has nothing to do with servers. It's for the cat.
  11. Every time you eliminate a hypothesis, leave a recorder behind. Next time something breaks, you'll already have it.
參考連結