A DIY Furbo with Frigate + Home Assistant

I have two tabby cats, aki and kuro. Ever since I got them, ads for the Furbo pet camera have been following me around: remote treat tossing, an AI pet sitter, a cat diary edited for you every day, vomit detection. After seeing them enough times I was genuinely tempted, and for a while I almost placed an order. Then I actually looked into it and found that almost every feature I wanted was locked behind a subscription, and its recordings only keep the clips where the AI decides there's a pet.

As it happens, there was already a Tapo C210 running in my living room the whole time. It just has no SD card, so all it does is live view, no playback. So in the end I didn't buy a Furbo. Instead I took its feature list as my target and built it myself with Frigate and Home Assistant: recordings I can scroll back through, notifications when the cats are up to something, and an automatic daily rundown of "what the cats did today". It took about two weeks. The result: recording, notifications, a cat diary written by a local AI, and a daily highlight video all work. The one thing I couldn't pull off is telling which cat is which. I trained a classification model and validated it on frames where both cats are in shot: 38% accuracy, lower than the 50% you'd get by guessing.

This post goes in order: the Furbo downsides I found, a from-scratch setup guide (compose and config included; swap in your own credentials and they'll run), which local vision model to pick, how I proved the cat-ID model doesn't work, how the diary and highlight video are built, and finally how much VRAM and disk the whole thing eats.

I almost bought a Furbo, until I looked closer#

Here's what I found on Furbo's Taiwan site in October 2026:

• Camera: the 360° Cat Camera page says "from NT$1,699", but that price requires bundling a Furbo Nanny subscription (at least 3 months). Without a contract, the standalone camera is NT$5,399 on the official Shopee store. There's no power adapter in the box (NT$299 extra), and Wi-Fi is 2.4GHz only.
• What works without a subscription: live view, two-way audio, treat tossing, the cat teaser, and meow notifications. No recording of any kind can be played back.
• Furbo Nanny subscription: the Standard plan is NT$1,840 for the first year paid annually (regular monthly price NT$219 per month), with 3 days of recordings; the Premium plan starts at NT$188 per month and keeps 7 days. In Taiwan the two plans differ only in retention: the cat diary, vomit detection, and multi-pet ID are in both. Each extra camera adds another NT$40 per month.

What really killed it for me was how it records. Furbo's playback consists of clips saved when the AI detects a pet, not continuous recording. A lot of the negative reviews on the Taiwan App Store revolve around this, e.g. "the moments I want to see often weren't recorded" and "it sends me meow notifications all day, but it can't detect the vomiting", and disconnects are another common complaint. Furbo's own help docs also spell out the threshold for the "Pet Emergency" alert: two excessive vomiting detections within a day, or two seizure detections within 5 minutes; listed sources of false alarms include lowering the head on a patterned rug, changes in light and shadow, and putting a toy down. So it looks like a head-down posture check plus a repetition threshold. I already have a C210 in the living room, so buying a Furbo would mean one more 2.4GHz-only camera plus a yearly subscription fee, in exchange for 3 days of AI-picked clips.

There's a reason the ads push so hard, too. An October interview in Business Next (數位時代) mentioned that Furbo's subscription revenue overtook hardware after 2023 and now makes up more than half of total revenue. The strategy is to keep the camera's entry price low and tier the subscription by how long recordings are kept.

Still, digging through the reviews made it clear which feature people love most: the daily time-lapse "cat diary" video, 5 seconds to 1 minute long, set to music.

Compared with my existing HA dashboard, the difference was obvious: mine showed "is there a cat on screen right now", while Furbo sells "what the cat just did". So I set three goals: recording playback, event summaries, and a daily highlight reel.

Building it from scratch#

Here's what the whole setup is made of:

• Camera: Tapo C210, main stream 2304×1296, substream 1280×720, 2.4GHz only, no SD card, so Frigate is its only recording option.
• Host: a Linux server that's on 24/7, with an RTX 5060 Ti 16GB, and recordings on a 16TB hard drive.
• Frigate 0.17.2: recording, object detection, and event management, running in Docker.
• Ollama: runs a vision-language model on the host to write a summary for each event.
• MQTT broker: carries state between Frigate and Home Assistant.
• Home Assistant: notifications, dashboard, diary page.

Below are the seven steps in the order I actually set things up. Each one is collapsed; expand it if you're following along. Credentials and IPs have been stripped from the config files; fill in your own and they'll work.

rtsp://攝影機帳戶:密碼@鏡頭IP:554/stream1   # 主串流
rtsp://攝影機帳戶:密碼@鏡頭IP:554/stream2   # 子串流

sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
docker run --rm --gpus all ubuntu nvidia-smi

curl -fsSL https://ollama.com/install.sh | sh
ollama pull qwen3-vl:8b-instruct
sudo systemctl edit ollama

[Service]
Environment="OLLAMA_HOST=0.0.0.0:11434"
Environment="OLLAMA_KEEP_ALIVE=-1"

sudo ufw allow from 172.28.0.0/24 to any port 11434 proto tcp

# mosquitto/compose.yml
services:
  mosquitto:
    image: eclipse-mosquitto:2
    container_name: mosquitto
    restart: unless-stopped
    ports:
      - "1883:1883"
    volumes:
      - ./config:/mosquitto/config
      - ./data:/mosquitto/data

# mosquitto/config/mosquitto.conf
listener 1883
allow_anonymous false
password_file /mosquitto/config/passwd
persistence true
persistence_location /mosquitto/data/

docker run --rm -it -v ./config:/mosquitto/config eclipse-mosquitto:2 \
  mosquitto_passwd -c /mosquitto/config/passwd frigate
docker run --rm -it -v ./config:/mosquitto/config eclipse-mosquitto:2 \
  mosquitto_passwd /mosquitto/config/passwd hass

docker build . --build-arg MODEL_SIZE=Nano --rm --output . -f- <<'EOF'
FROM python:3.12 AS build
RUN apt-get update && apt-get install --no-install-recommends -y libgl1 && rm -rf /var/lib/apt/lists/*
COPY --from=ghcr.io/astral-sh/uv:0.10.4 /uv /bin/
WORKDIR /rfdetr
# 先裝 CPU 版 torch,下載量小很多
RUN uv pip install --system torch==2.8.0 --index-url https://download.pytorch.org/whl/cpu
RUN uv pip install --system rfdetr[onnxexport] onnx==1.19.1 transformers==4.57.6 onnxscript
ARG MODEL_SIZE
RUN python3 -c "from rfdetr import RFDETR${MODEL_SIZE}; x = RFDETR${MODEL_SIZE}(resolution=320); x.export(simplify=True)"
FROM scratch
ARG MODEL_SIZE
COPY --from=build /rfdetr/output/inference_model.onnx /rfdetr-${MODEL_SIZE}.onnx
EOF

services:
  frigate:
    image: ghcr.io/blakeblackshear/frigate:0.17.2-tensorrt
    container_name: frigate
    restart: unless-stopped
    shm_size: 1g                 # 偵測用的影格放這裡,太小會直接 crash
    ports:
      - "127.0.0.1:5000:5000"    # 沒有認證的 API,只給同一台的 HA 用
      - "8971:8971"              # 有登入頁的網頁介面
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - ./config:/config
      - /mnt/hdd/frigate/media:/media/frigate
    tmpfs:
      - /tmp/cache:size=1g       # 錄影暫存放記憶體,不要一直寫 SSD
    env_file: .env
    environment:
      - TZ=Asia/Taipei
    mem_limit: 10g
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
    devices:                     # 明列裝置檔,見下方說明
      - /dev/nvidia0:/dev/nvidia0
      - /dev/nvidiactl:/dev/nvidiactl
      - /dev/nvidia-uvm:/dev/nvidia-uvm
      - /dev/nvidia-uvm-tools:/dev/nvidia-uvm-tools
      - /dev/nvidia-modeset:/dev/nvidia-modeset

networks:
  default:
    ipam:
      config:
        - subnet: 172.28.0.0/24  # 釘死網段,ufw 規則才不會失效

FRIGATE_RTSP_USER=攝影機帳戶
FRIGATE_RTSP_PASSWORD=攝影機帳戶密碼
FRIGATE_MQTT_USER=frigate
FRIGATE_MQTT_PASSWORD=剛剛設的密碼

mqtt:
  enabled: true
  host: 172.28.0.1               # 從容器連主機上的 broker(Docker 網段的 gateway)
  port: 1883
  user: '{FRIGATE_MQTT_USER}'
  password: '{FRIGATE_MQTT_PASSWORD}'

detectors:
  onnx:
    type: onnx

model:
  model_type: rfdetr
  width: 320
  height: 320
  input_tensor: nchw
  input_dtype: float
  path: /config/model_cache/rfdetr.onnx
  # rfdetr 用內建的 COCO 91 類標籤,不要設 labelmap_path

ffmpeg:
  hwaccel_args: preset-nvidia    # 用顯卡解碼

go2rtc:                          # Frigate 內建的串流轉發,鏡頭只被拉一次
  streams:
    keting:
      - rtsp://{FRIGATE_RTSP_USER}:{FRIGATE_RTSP_PASSWORD}@鏡頭IP:554/stream1
    keting_sub:
      - rtsp://{FRIGATE_RTSP_USER}:{FRIGATE_RTSP_PASSWORD}@鏡頭IP:554/stream2

genai:
  provider: ollama
  base_url: http://172.28.0.1:11434
  model: qwen3-vl:8b-instruct

semantic_search:                 # 讓 Explore 頁可以用文字搜事件,只索引開啟後的新事件
  enabled: true
  model: jinav1
  model_size: large

review:
  alerts:
    labels: [person]             # 只有人算 alert,貓留在 detections
  genai:
    enabled: true
    preferred_language: Traditional Chinese
    detections: true
    image_source: recordings
    activity_context_prompt: |
      這是台灣一般住家的客廳監視器,不是防盜攝影機。
      (完整內容見下面「選本機視覺模型」那節)

audio:
  enabled: true
  listen: [meow, hiss, caterwaul, speech, scream, yell, fire_alarm]

cameras:
  keting:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/keting
          input_args: preset-rtsp-restream
          roles: [record, detect]
        - path: rtsp://127.0.0.1:8554/keting_sub
          input_args: preset-rtsp-restream
          roles: [audio]         # 音訊另開一條連線,掛子串流省頻寬
    detect:
      width: 2304                # 0.17 要寫死;跑主串流的原因見「分辨兩隻貓」
      height: 1296
      fps: 5
    motion:
      mask:
        - 0,0,0.38,0,0.38,0.075,0,0.075
      contour_area: 25
    objects:
      track: [cat, person]       # 沒養狗,dog 拿掉(偵測器會把貓認成狗)
      genai:                     # 物件描述用英文,給語意搜尋
        enabled: true
        objects: [cat, person]
        object_prompts:
          cat: >-
            Frames from an indoor living-room camera.
            In ONE short sentence, state what the cat is doing.
            If two cats are present, say so.
            Do NOT describe coat colour, pattern or markings.
            Start directly with "A cat" or "Two cats". No preamble.
            If no cat is visible, reply exactly: nothing notable.
    record:
      enabled: true
      continuous:
        days: 3
      motion:
        days: 14
      alerts:
        retain:
          days: 30
      detections:
        retain:
          days: 14
    snapshots:
      enabled: true
      bounding_box: true
      retain:
        default: 14

docker logs frigate 2>&1 | grep -i password

# /config/diary/recent_summary.py:印出最近一則有摘要的片段,沒有就印空字串
import json, sys, time, urllib.request

after = int(time.time()) - int(sys.argv[1]) * 60
revs = json.load(urllib.request.urlopen(
    f"http://127.0.0.1:5000/api/review?after={after}&cameras=keting&limit=100"))
done = [r for r in revs if (r["data"].get("metadata") or {}).get("title")]
if done:
    r = max(done, key=lambda r: r["start_time"])
    m = r["data"]["metadata"]
    t = time.strftime("%H:%M", time.localtime(r["start_time"]))
    print(f"{t} {m.get('shortSummary') or m['title']}"[:90])

# configuration.yaml
shell_command:
  frigate_recent_summary: "python3 /config/diary/recent_summary.py 10"

- id: cat_hiss_alert
  alias: 客廳偵測到貓哈氣
  mode: single
  triggers:
    - trigger: state
      entity_id: binary_sensor.keting_hiss_sound
      from: "off"
      to: "on"
  actions:
    - action: shell_command.frigate_recent_summary
      response_variable: ctx
    - variables:
        scene: "{{ (ctx.stdout | default('', true)) | trim }}"
    - action: notify.mobile_app_你的手機
      data:
        title: 客廳偵測到哈氣
        message: >-
          {{ now().strftime('%H:%M') }},畫面裡有
          {{ states('sensor.keting_cat_count') }} 隻貓。
          {%- if scene %} 剛才的畫面:{{ scene }}{%- endif %}
        data:
          image: /api/camera_proxy/camera.keting   # 附上當下的畫面
          url: /lovelace/living                    # 點通知打開客廳頁
          push:
            interruption-level: time-sensitive     # 穿透 iOS 專注模式

Two things to tune once it's running#

The config above already includes both fixes; this section explains why they're needed.

The first is that motion was triggering almost constantly. I counted: of 2412 ten-second recording segments, 2348 were flagged as having motion (97.3%), but only 6.3% actually had an object in them. There were two causes: the timestamp burned into the top-left corner changes every second and accounts for over 80% of the changed pixels in an otherwise static frame, and every 2 to 4 seconds there's a burst of full-frame compression noise. This doesn't just waste GPU: record.motion.days: 14 effectively became 14 days of continuous recording.

motion:
  mask:
    - 0,0,0.38,0,0.38,0.075,0,0.075   # 遮掉左上角的時間戳
  contour_area: 25                    # 濾掉壓縮雜訊,貓的面積大約 500

After the change, the motion ratio dropped to 31%, and the detector runs at 0 fps when nothing is happening.

The second is that GenAI summaries don't cover cats by default. Frigate's reviews are split into alerts and detections; cats fall under detections, and review.genai.detections defaults to false:

review:
  genai:
    enabled: true
    detections: true          # 預設 false,貓的事件不會有摘要
    image_source: recordings  # 預設用 180p 的預覽圖,會漏掉人和貓

I also switched image_source to recordings (480p). With the default preview images, there was a segment where a person was clearly detected (confidence 0.96), yet the summary said "the living room is completely still".

Picking a local vision model#

I compared three models on the same set of frames. What was actually in them was two tabby cats, plus a few empty frames as a hallucination control group:

I went with qwen3-vl:8b. Testing also turned up a few problems with Frigate's defaults:

• The default object prompt is "analyze these images containing {label}", which tells the model up front that the thing is definitely there, so even an empty frame gets described as having a cat.
• Setting only preferred_language: Traditional Chinese doesn't keep Simplified Chinese out. The default review summary produced Simplified-character sentences like "花盆从沙发搬运到凳子上" ("the flowerpot was moved from the sofa onto the stool"), and also made up a person named "Joe". At first I thought gemma3:4b just couldn't write Traditional Chinese; later I found that the language instruction in Frigate's built-in prompt is too weak, and the model itself can write Traditional just fine. You need to state "always use Traditional Chinese" and "never make up names" explicitly in your own prompt.
• The default prompt is written for outdoor security: it talks about driveways, packages, and strangers late at night, none of which fits a living room.
• Object descriptions default to English, and every one opens with filler like "Here's an analysis of the cat's behavior based on the images...", which is why the prompt says No preamble.

I rewrote the prompt to fit my home: this is the living room of a home in Taiwan, not security footage; the only fixed objects are the sofa, coffee table, cat tree, litter box, cardboard box, electric fan, and air purifier; the cat tree has an enclosed round cat bed on it; there's no dog in the house; don't describe coat color, don't invent names, don't say how many seconds something lasted.

Those last few rules were all forced on me by hallucinations, and I caught them myself while reading through the diary: there is no cage anywhere in my home. After the first version had run for a few days, the summaries said "the cat is in a cage" 14 times in one day (it was actually the round cat bed on the cat tree), mentioned a "dog", and had one entry saying "the cat is retching with its head down, lasting about 136 seconds". The 136 seconds was actually the total length of that recording; the model only looked at a few frames of it. The "dog" wasn't entirely the vision model's fault: Frigate's detector itself mistakes a cat for a dog about 8 times a day (confidence 0.77 to 0.95), so I just removed dog from objects.track.

review:
  genai:
    activity_context_prompt: |
      這是台灣一般住家的客廳監視器,不是防盜攝影機。
      - 住戶(成人)與兩隻貓在此活動,兩者進出都是完全正常的事。
      - 沒有車道、門廊、包裹等室外元素,不要提到這些。
      - 畫面裡固定會出現的東西只有這些:沙發、茶几、矮凳、貓塔、
        貓砂盆、紙箱、立式電風扇、空氣清淨機。
      - 貓塔上有一個封閉式的圓形貓窩。那是貓窩,不是籠子,家裡沒有任何籠子。
      - 畫面裡只拍得到其中一個貓砂盆(家裡有兩個)。
        「進入貓砂盆內、蹲踞或撥砂」才寫成使用貓砂盆;
        只是走過旁邊、在旁邊休息,就寫「在貓砂盆旁」。
      - 家裡沒有養狗。偵測結果若標成 dog 那是誤判,不要在描述裡提到狗。
      - 一律使用繁體中文(台灣用語)。
      - 禁止臆測或編造人名。未知人物一律寫「一名成人」或「住戶」。
      - 只描述畫面中確實看得到的東西,看不清楚就說看不清楚。
      - 不要描述貓的毛色或花紋,統一寫「一隻貓」。
      - 禁止聲稱任何行為「持續了幾秒/幾分鐘」。你看到的是抽樣出來的
        少數幾張影格,不是連續影片。
      - 不要用「全程」「整段時間」「持續」這類字眼。
      - Level 0(正常):住戶活動、貓的任何活動、日常家事。這是絕大多數情況的正確答案。
      - Level 1(可疑):不認識的人在屋內;貓明顯異常(持續嘔吐姿勢、抽搐、
        長時間倒地不動);物品翻倒。
      - Level 2(危險):破壞、打鬥、火煙、有人跌倒後無法起身。

Frigate's GenAI actually uses the model in two places, and I have them write in different languages:

• Review summaries (review.genai): turn a stretch of events into a title and summary, in Traditional Chinese, for notifications and the diary.
• Object descriptions (objects.genai): one sentence for each tracked object, stored in the semantic search index. Jina CLIP v1, which semantic search uses, only supports English (v2 is multilingual), so this prompt is written in English: one sentence, no describing coat color, and reply nothing notable if no cat is visible.

That way each side works in its own language, and nothing needs translating.

After changing the prompt there was one more problem: the old summaries were already in the database. I first ran the old and new prompts on the same frames to confirm the new version really stopped writing about cages and made-up durations, and only then rewrote the old ones. For example, for the 21:27 segment the old version said "the cat seems unwell, retching with its head down continuously", and the new one says "a cat walks around by the cardboard box next to the standing fan" (both originally in Chinese). Frigate only has an API for regenerating "object descriptions" (PUT /api/events/<id>/description/regenerate), not review summaries, and Reindex in the web UI just recomputes vectors from the existing text without rewriting anything. So I wrote a separate script that pulls frames from the recordings, applies the new prompt, and rewrites with the same model. The results go into a file on the side, which the diary prefers when reading, and Frigate's database is left untouched. I only rewrote entries showing signs of hallucination: 72 out of 721, about 3.6 seconds each. When I spot-checked a few days later, hallucinations had dropped from about 10% to 2.5% (31 out of 1238), and most of what's left is wording like "the whole time" and "throughout".

Telling the two cats apart#

This is the one part that didn't work. aki and kuro are littermates, a brother and sister, both tabbies, one leaning warm brown and the other dark gray. aki is a bit darker, and even I often mix them up on the camera feed.

First, how have other people done it#

Before getting my hands dirty, I did a round of research to find out whether telling identical-looking cats apart is even doable.

The simplest approach is collars in different colors, which turns the problem from "recognize which cat it is" into "find the red blob in the frame". Unfortunately they hate wearing collars, so that was out right away. Our pet air purifier (the one the cat is sitting on in the video) weighs the cat, and I looked into using that too: the two cats weigh differently, so whoever sits on it tells you who it is. It turns out its API doesn't return the weight, so that was a dead end too.

Academically this is called individual re-identification, and there are actually quite a few papers:

• One study used footage from "Hello Street Cat" feeding stations in China and trained a Siamese network on 2,796 photos of 69 cats; the best combination (VGG16 with contrastive loss) reached 97% test accuracy.
• Another compared ResNet, DenseNet, EfficientNet, ConvNeXt, and Siamese networks at recognizing individual cats, and concluded that conventional CNNs trained with transfer learning performed best.
• The studies I read had one thing in common: coat color is the least reliable feature, and what works is the shape of the face and the stripes along the flanks. Nose prints are unique like fingerprints, but you need a macro shot to capture them; off-the-shelf tools like Double Take and CompreFace only recognize human faces.

Commercial products aren't any better. Furbo's multi-pet ID (Pet ID) needs a subscription and works by uploading a photo of each pet showing its face and part of its body. The official docs give no accuracy figures at all and don't mention what happens with look-alike pets. The App Store reviews are pretty blunt, though: one person said they'd made a profile for each pet, and they "don't look alike at all", yet it still can't tell them apart; another said it keeps calling the male cat by the female's name and mixes him up with another cat or the dog, wrong about half the time. If it gets pets that look nothing alike wrong, I don't see it doing better with two littermates with the same coat.

The accuracy numbers in these studies look great, but they assume close, clear footage, and mostly involve picking one cat out of many cats with different coats. My situation is the opposite: only two cats, littermates, same coat, and the camera 3 meters away. So I decided to try Frigate's built-in features first instead of building my own training pipeline from the start.

Ruled out first: having the vision model read coat color#

qwen's summaries sometimes say "black cat" and sometimes "tabby", which looked like something I could use directly. But within the same event, one second it says "both are tabbies" and the next "one tabby and one black cat". I tallied 33 events by box position and brightness: the 6 "black cat" mentions all appeared on the right side of the frame by the doorway, which is the backlit spot, and the brightness there wasn't actually any darker. The model was reading position and lighting, not the cat itself, so I took coat color out of the prompt.

Custom classification in Frigate 0.17#

Frigate 0.17 has built-in object classification that can run a second classification on detected cats and write the result into the sub label. The example in the official docs happens to be telling two cats apart.

classification:
  custom:
    cats:
      enabled: true
      name: cats
      threshold: 0.8
      object_config:
        objects: [cat]
        classification_type: sub_label

You set it up on the Classification page in the Frigate web UI: give it a name, pick the object cat, type sub_label, enter aki and kuro as the classes, then pick training images class by class; anything you don't select is automatically filed under none. Classes have to be added one at a time. The first time, I typed "aki / kuro" as a single class, and the title of the image picker became "Select all images with: aki / kuro".

While picking images I found that in a lot of them even I couldn't tell which cat it was. Training images are fixed at 224×224, and detection was originally running on the 1280×720 substream, so the cropped cats had a short side of only about 187px and were being upscaled to 224. After moving detection to the 2304×1296 main stream, the crops became about 241px, and you could see the outline of the face and the stripes along the flanks:

The cost of the main stream is that shm_size has to go up (that's where the 1g in step 5 came from), and detection now processes 3.24 times as many pixels, though the motion ratio didn't climb back up (31% to 34.5%).

A few rules I only learned by using it:

• Training images can only be picked from Frigate's own crops; you can't upload your own photos. The maintainers' reasoning is that the crops have to match how detection actually crops.
• Training runs on the CPU and usually takes 1 to 3 minutes. After you hit train, the spinner in the web UI sometimes keeps spinning forever; that's the status not updating, not a hang.
• The rule for assigning a name: each classification has to clear the threshold (I set 0.8), and at least 60% of at least 3 attempts have to agree on the same name before it's written into the sub label.

I trained it several times. The first version had only 24 images (aki 11, kuro 3, none 10). Once it was live, the model labeled aki and kuro at a ratio of 3.7 to 1, which is exactly the 11 to 3 of the training set: it learned the ratio, not the cats. The final training set was aki 188, kuro 204, none 41.

How do you know if the model is right#

The model labels every cat with a name, but I can't judge by eye whether those labels are correct either. The method I ended up with is the same-frame check: when both cats are on screen at the same time, the model has to give them different names. That's a ground truth that needs no human judgment.

There's a catch when actually computing it: Frigate sometimes loses a cat and picks it back up as a second object, which looks like both cats are in the frame. So pairs whose boxes overlap a lot (IoU above 0.3) have to be excluded; only pairs that barely overlap count as genuinely two cats.

At first, after removing the lost-and-reacquired ones, only 2 real same-frame pairs were left, nowhere near enough to judge. By September 5 there were 34 real same-frame pairs, and only 13 of them got different names: 38%. The two cats were each assigned about half the time, so random guessing would be expected to hit 50%. The model did worse than guessing.

Then I ran the model on its own training set: 96.2% accuracy. 96% on the training set and 38% on data it hasn't seen means it memorized the training images. The cause is the training data itself: the 392 images actually came from only about 30 events. The way I picked images was to click a tracked object in the web UI and add all its crops in one go, so each event contributed 5 to 20 nearly identical images in a row, and the effective sample size was only around 30.

The training set had another bias: at night the C210 switches to infrared black-and-white, and color cues disappear entirely. Of aki's 188 images, 138 were black-and-white (73%), versus only 26% for kuro. aki is darker to begin with, so while labeling, I'd filed all the dim images under aki. That's also what I saw in the Recent tab: most got labeled aki, and kuro was only recognized in bright, clear shots. Still, the model got 92.5% of kuro's black-and-white images right, so the main cause is still memorization, not just "see black-and-white, guess aki". Frigate's official advice is blunt about this too: diversity matters more than quantity, and selecting dozens of nearly identical images at once is one of the fastest ways to make a model worse.

I also tried an approach that needs no labeling: for semantic search, Frigate already stores a CLIP vector for every detected cat. Checking whether those vectors could separate the two across 171 same-frame pairs gave 56.1%, only slightly better than guessing.

Verdict: the camera is the bottleneck#

The living-room camera is about 3 meters from the cats, and a cat's face takes up only about 6% of the frame width. Two cats with the same coat just don't have enough distinguishing features at that distance, and switching models or labeling a few hundred more images probably won't help much. A more workable direction is a close-range camera, like one at the feeding area, 50 cm from the cat.

For now I've taken the aki and kuro counts out of the diary and the notifications, and put a warning card on the page. Once the close-up camera on the feeder has collected enough data, I'll revalidate with the same same-frame method.

Cat diary#

The diary is the feature I use most out of this whole project. Frigate generates a summary for each event, and I built a dedicated page in HA that refreshes every 30 minutes and pushes that day's version at 10 PM:

The architecture is simple: every 30 minutes HA runs a Python script via shell_command. The script pulls the day's events and summaries from Frigate's /api/review and writes them out as a JSON file in HA's config directory; HA then reads that JSON with a sensor, and the page lays it out with Markdown cards. The 10 PM push reads the same file, so the numbers on the page and in the notification always match.

I ran into a few problems along the way:

• At first the whole diary was computed with templates inside an HA script, and once there was enough data it hit HA's 256KB template output limit (Template output exceeded maximum size of 262144 characters). I switched to a Python script that generates the JSON, with HA only reading it.
• Frigate 0.17 labels objects that pass the custom classification as cat-verified. I'd been filtering with "cat" in objects, which silently missed 66% of cat events.
• The day's highlight video is embedded at the top of the page, but after a new video was made the browser kept playing the old one. The reason: HA's /local/ responses carry a 31-day cache header, and even a hard refresh doesn't help. What finally fixed it was appending the file's modification time to the video URL as a version number (?v= plus the mtime). The video is embedded with a video tag directly inside a Markdown card; the width has to be written as a width="100%" attribute, because style gets stripped by HA's allowlist. The highlight video is only generated at 9:50 PM, so for 22 hours a day "today's video" doesn't exist; when there isn't one for today, it falls back to the most recent one.
• Files under /local/ can be read without logging in, so the folder holding the videos is named with a random string, which effectively makes the URL the password.
• For push notifications, my family's iPhones go through the HA app (see step 7 for why). The diary is set as a passive notification so it doesn't keep popping up; unusual sounds are set as time-sensitive.

Besides the diary, I added a few small metrics Furbo doesn't have: activity compared with the same time slot over the past 7 days, how long someone was in the living room keeping the cats company, and an alert when the cats haven't been seen for too long. That last one started out as a fixed 4 hours, and then a question occurred to me: what if they've gone off to sleep in a bedroom? I only have the one camera in the living room. So the threshold became twice the longest past gap, with a minimum of 4 hours, checked every 30 minutes, and the notification text says outright "only the living-room camera". As for the litter box, only one of the two is in view, and the model can't tell "walking past" from "using it", so I only show a count of "near the litter box" as a rough trend.

Daily highlight video#

This one is modeled on Furbo's most popular feature: every night at 9:50 PM, it picks clips from the day's events and cuts them into a video sped up 1.6×, which goes at the top of the diary page. Furbo's diary only covers 7 AM to 7 PM. I didn't copy that, because my cats are most active in the middle of the night, with thirty-something events each at 2 AM and 3 AM.

When I watched the first version, something felt off: the clips of the two cats sleeping together on the sofa at night, so deeply they rolled belly-up, didn't make the cut. How was it picking? Digging in, I found two problems. One, the loop went hour by hour in increasing order and stopped once it had 10 clips, so it only ever picked from midnight to 9 AM. Two, it sorted by detection confidence, and nearly every cat detection is between 0.98 and 0.99, so the sort was effectively random.

After fixing that, I first tried a composite score, weighting for both cats being in frame and for clip length. But I didn't think both cats in frame was the point. Highlights should be the cuter clips where something is happening, and the only place that knows "what's going on in this clip" is the one-line GenAI summary. So I switched to having the same qwen3-vl:8b score every event from 1 to 5 based on its own summary: playing, chasing, grooming, stretching, the two cats interacting, or interacting with a person is a 5, and sitting still is a 1. There were a few gotchas here too:

• Ollama's format: "json" only guarantees valid JSON, not the shape. Once the model spat out JSON formatted like a security report. Switching to structured outputs with a JSON Schema made it reliable.
• Hand an 8B model 95 items to rank at once and it only returns the top 10. I switched to batches of 20, scoring each item.

The whole script is Python, run by cron at 21:50 every day, and the core of it is three API calls. Here's a simplified version:

import json, urllib.request

FRIGATE = "http://127.0.0.1:5000/api"
OLLAMA = "http://127.0.0.1:11434/api/generate"

# 1. 拿當天的 review 和 GenAI 摘要。參數是複數 cameras,
#    寫成單數 camera 會被靜默忽略,兩台鏡頭的片段混在一起
revs = json.load(urllib.request.urlopen(
    f"{FRIGATE}/review?after={start}&before={end}&cameras=keting&limit=2000"))

# 0.17 通過自訂分類的物件會變成 cat-verified,要把後綴拿掉再比
segs = [r for r in revs
        if any(o.removesuffix("-verified") == "cat" for o in r["data"]["objects"])
        and r["data"].get("metadata")]

# 2. 讓同一顆模型依摘要打分,format 一定要給 JSON Schema
SCHEMA = {"type": "object", "properties": {"scores": {"type": "array", "items": {
    "type": "object",
    "properties": {"i": {"type": "integer"}, "s": {"type": "integer"}},
    "required": ["i", "s"]}}}, "required": ["scores"]}

lines = [f"{i} {r['data']['metadata']['title']}" for i, r in enumerate(segs[:20])]
body = json.dumps({
    "model": "qwen3-vl:8b-instruct",
    "prompt": SCORE_PROMPT.format(items="\n".join(lines)),  # 1 到 5 分的評分標準
    "stream": False, "format": SCHEMA, "options": {"temperature": 0.1},
}).encode()
req = urllib.request.Request(OLLAMA, data=body, headers={"Content-Type": "application/json"})
scores = json.loads(json.load(urllib.request.urlopen(req))["response"])["scores"]

# 3. 依時間範圍直接切片段,不要抓整個 event 的 clip
#    (最長的事件有 2513 秒,伺服器要現場轉檔 42 分鐘)
clip = f"{FRIGATE}/{seg['camera']}/start/{t0:.3f}/end/{t1:.3f}/clip.mp4"

Once I have the clips, ffmpeg speeds them up 1.6× (setpts for the video, atempo for the audio), and concat joins them into one file. For the join, every clip's encoding parameters have to match, so I only take the living-room camera, and whether there's an audio track has to be the same across all of them.

After a week of this I still felt the highlights were often a bit off. This time it wasn't picking the wrong segments; the clips themselves were cut wrong. Every clip was a fixed 9.6 seconds, but plenty of events are shorter than that: 9 of 30 clips were shorter than the capture length, and one was even 0 seconds. When I traced back why it was fixed at 9.6 seconds, the answer was that there was no technical reason at all; I'd just picked it offhand. Now it varies with event length (2 to 10 seconds of output), with the total kept to 180 seconds.

The last problem was repetition. One of the cats sleeps in the round cat bed on the cat tree a lot, so several clips in a day's highlights were all the same cat bed:

My first idea was to detect duplicates from the summary text, but I figured matching titles weren't the problem; what matters is what's on screen, so I actually pulled out the thumbnails and looked. Text similarity turned out to be an anti-signal: the same description can go with completely different frames, while different descriptions can be the same cat bed. Comparing whole-frame similarity didn't work either, since the static background dominates. What I ended up using is the detection box's center position plus its area: a center distance under 0.1 and an area within a factor of two count as the same spot, with at most 3 clips per spot. Selection has two stages: the first 70% of the time goes purely by score, and the last 30% fills in time slots that normally get picked less. I don't mind longer videos, I'm only worried about disk space, so the total is capped at 180 seconds: measured at about 59KB per second, that's around 10MB a video, and under 100MB for 7 days.

These days it produces about 18 to 26 clips a day, a video a little over two minutes long and around 12MB, kept for 7 days. Below is 30 seconds from the October 12 highlight (it had audio originally; I removed it for posting here): in the small hours a cat walks in from the doorway, jumps onto the cat tree at 5 AM, and lets out a big yawn up there.

By the way, while writing this post I took the yawning frame from that video and asked qwen3-vl about it directly (without my prompt). Its answer, originally in Chinese, was "the black cat in the frame is standing on the top level of the cat tree... mouth open, seemingly hissing". The tabby became a black cat and the yawn became a hiss, which is exactly why the prompt earlier has to be so detailed.

Resource usage#

This whole thing runs on my home server. Below are the numbers I actually measured on October 13 (by then I'd added a camera on a feeder, so there are two decode streams).

For VRAM, here's what nvidia-smi shows:

The big one is the vision model. I have Ollama keep the model in VRAM permanently (OLLAMA_KEEP_ALIVE=-1), because every reload means waiting (back when I used gemma3:4b, the first summary took 36 seconds). Going by these numbers, an 8GB card can't fit this combo, and 12GB or more gives you some breathing room. The highlight scoring uses the same 8B model; switching to the 27B version would push the resident 8B out of VRAM.

Other resources:

• GPU utilization: normally 2% to 3%, with the whole card idling at about 31W. Frigate's detection only runs when there's motion, about 6ms per run, and detection fps is 0 when nothing's happening. What actually maxes out the GPU is writing summaries: I measured about 3 seconds per summary, during which power draw jumps to 160 to 170W. The living room sees about 160 to 240 events a day, which adds up to roughly 10 minutes a day of the GPU at full load.
• CPU and memory: docker stats shows the whole Frigate container at around 17% (100% equals one core), with about 5GB of memory; I set the container limit to 10GB.
• Disk: continuous recording from the living-room camera is about 10GB a day. My original retention plan was 14 days of continuous recording, 30 days of motion, and 60 days of events. I didn't want to take up too much disk or have the drive writing like crazy all day, so I cut that to 3 days continuous, 14 days motion, and 30 days events. The living room's recordings currently total about 69GB, plus about 6.7GB of snapshots and event clips. The highlight video is 12MB a day, kept for 7 days, so it's negligible. In other words, how far back you can watch depends entirely on how big your disk is: at 10GB a day for the living room, 1TB gets you about three months of continuous recording. That's a completely different way of thinking from Furbo tiering its subscription by 3 or 7 days.

Versus Furbo#

Actually, even before I got the 38% result, I was already shopping for an automatic feeder with a camera. The original idea was to cover the floor angle the living-room camera can't see, with automatic feeding as a bonus. After the cat ID failed, it picked up another job: when a cat eats, it's only a few dozen centimeters from the camera, which is exactly the close-up that telling the cats apart needs. The one I ended up buying is now hooked into Frigate too; how I set it up, and whether cat ID got any more accurate, will go in another post.