Deploying a PostHog Data Pipeline Locally

A while back I ran into a data-sync requirement at work. It started with the front end failing to reliably get a user's department (dept) name during login. So the DBA proposed a fix: go straight to the source — build a data pipeline that pulls the existing member records out of the database and syncs them, filling in that missing dept field along the way. To pull this off, we decided to stand up a PostHog Data Pipeline locally.

The catch: this thing runs in a slightly bizarre environment. All our base services live inside Docker in Docker (DinD), with a Caddy wrapped around the very outside for traffic routing. Don't ask me why we stack this many layers — I'm not entirely sure either. Probably because the company's public-facing address is a single fixed IP from the ISP, and to serve different services off that one physical IP via different DNS names, someone slapped this Caddy layer on top.

Anyway, running a full microservice stack — Temporal and MinIO included — inside this DinD-plus-Caddy setup, I figured I'd just copy the official Docker Compose, run it, wire up the DB, and be done. That, it turned out, was where the debugging nightmare began.

The ODBC Driver and Certificate Problem#

It all began with PostHog's UI throwing a cold, blank "connection error" on the DB sync task. No detailed log, no stack trace.

I curl'd and used Python's pymssql from inside the container: TCP and the direct connection were perfectly fine. But the moment I switched to pyodbc — which PostHog depends on underneath — it threw a file not found.

A quick look at the container's config (/etc/odbcinst.ini) cracked it. PostHog's image ships ODBC Driver 18 by default, and Microsoft made an aggressive change there: it forces encrypted connections by default (Encrypt=yes). In a local dev environment like ours, with no trusted TLS certificate configured, the connection gets rejected outright.

Lost in a Maze of Microservices#

The database was connected, but after I hit "Reload," the task just sat there, dead.

I suspected the link between the Django API (the Web container) and the Temporal Server had broken. A quick ping from inside: ping: temporal: Name or service not known. The Web container had no idea who temporal was. In the Docker Compose world that's fatal — without the right environment variables injected, the app just dumbly looks for localhost.

Cracked open .env and the config, and sure enough, something was missing. Added these two lines and restarted:

TEMPORAL_HOST=temporal
TEMPORAL_PORT=7233

curl temporal:7233 again — through. The task in the logs submitted cleanly, and the status finally turned to Running.

The Missing Worker, and Some Hardcore Debugging#

Status was Running, but the task fell into a black hole — zero progress. In the UI, a whole row of Member / Dept syncs was either stuck or flat-out Failed, and clicking in only surfaced an inscrutable TypeError — no clue where it died.

Since the vague error told me nothing, I went straight to Temporal's Admin Tools to find out exactly which link was stuck. First, list the workflows currently in Running state:

docker exec -it deploy-temporal-admin-tools-1 temporal workflow list \
  --address temporal:7233 --query "ExecutionStatus='Running'"

In the printed table I found the external-data-job task, copied its WorkflowId, and pulled up its detailed errors and execution history:

temporal workflow show --address temporal:7233 -w external-data-job(...)

  ID          Time                    Type
   1  2026-02-23T09:56:10Z  WorkflowExecutionStarted
   2  2026-02-23T09:56:10Z  WorkflowTaskScheduled

Reading these two lines was the key to the whole case:

• Started: The Temporal server successfully received the request to launch external-data-job (i.e. the DB pipeline).
• Scheduled: Temporal placed the task into the "Task Queue" and is waiting for a Worker to pick it up.

And the problem is right there: nothing after that. Normally the next line should be WorkflowTaskStarted (a Worker has begun processing). Which tells you one thing — there is no Worker to take this task.

To confirm exactly which queue it was waiting on, I added -o json and filtered for taskqueue:

temporal workflow show ... -o json | grep -i "taskqueue"

# Output:
# TaskQueue:{Name:data-warehouse-task-queue, Kind:Normal}

Draw out this "queue with no listener" deadlock and it's obvious at a glance:

Now that I knew what was missing, I could add it myself. I stepped into the existing Worker container and checked the startup script's help:

docker exec -it deploy-temporal-django-worker-1 \
  python manage.py start_temporal_worker --help

#  --task-queue TASK_QUEUE
#                        Task queue to service

Just add --task-queue data-warehouse-task-queue to the startup command and you're set.

An Aside: The Frustration of Editing Someone Else's Config#

Now the fix was just editing docker-compose.yml to add this dedicated Worker. But I have to vent for a second.

Anyway, I dropped this brand-new service definition into docker-compose.yml:

temporal-data-warehouse-worker:
    extends:
        file: docker-compose.base.yml
        service: temporal-django-worker
    command: python manage.py start_temporal_worker --task-queue data-warehouse-task-queue
    environment:
        SITE_URL: https://$DOMAIN
        # These few were the big traps added later:
        ENCRYPTION_SALT_KEYS: ${ENCRYPTION_SALT_KEYS}
        CDP_REDIS_HOST: redis
        REDIS_URL: redis://posthog-redis:6379
    depends_on:
        - db
        - redis
        - clickhouse
        - kafka
        - temporal

Once this dedicated Worker was up, the Pipeline finally kicked off for real.

The Art of Disguise: MinIO and S3 Credentials#

After all that wrangling, I could finally pull the member and dept data out. But at the very last step — writing into the MinIO data lake — it threw credential provider was not enabled and NoSuchBucket.

PostHog uses the deltalake library underneath to write data, and this thing insists on AWS credentials. We're on open-source MinIO, so I had to play a little "disguise" in the Worker's environment variables — feed it a set of fake AWS credentials whose names line up but which actually point at the local MinIO:

environment:
  - AWS_ACCESS_KEY_ID=minioadmin
  - AWS_SECRET_ACCESS_KEY=minioadmin
  - AWS_REGION=us-east-1
  - AWS_S3_ENDPOINT=http://minio:9000

Then a simple Boto3 script, into the container, to create the missing data-warehouse Bucket by hand:

import boto3

s3 = boto3.client(
    's3',
    endpoint_url='http://minio:9000',
    aws_access_key_id='minioadmin',
    aws_secret_access_key='minioadmin'
)

s3.create_bucket(Bucket="data-warehouse")

Watching MinIO's console finally sprout row after row of efficient Parquet files, the data had at last settled into the lake.

An Aside: The Hour I Spent Staring at Logs, Questioning My Life#

With every config filled in and the Bucket created, I hit "Sync" in the UI, full of confidence.

The task launched, and the Temporal Worker started spewing logs like mad. At first, watching the data get pulled row by row was almost therapeutic — but ten minutes passed, then twenty, then half an hour... the logs kept scrolling endlessly, and the status was still Running.

Because there were DinD and Caddy wrapped around the outside, the network was relatively complex, and I started to wonder whether some TCP timeout had fired somewhere, or whether the Worker had quietly OOM'd and jammed the task in some loop. I'd even opened another terminal, ready to restart the entire Docker daemon.

But the truth was on docker stats the whole time:

So after nearly an hour of nervously staring at the terminal, a single line finally popped out: Sync completed successfully. And it hit me: right — this is a Full Sync. The company's years of accumulated member and dept history is simply enormous, and the Worker was just honestly chunking that data, converting formats, and writing it into MinIO. I'd scared myself, start to finish.

Core Analysis: How Does the Data Actually Flow in This Architecture?#

The data flow, wrapped up in DinD and Caddy, is genuinely a bit convoluted. PostHog's Data Warehouse design isn't about simply "copying" external data into its own relational database — it follows the modern Data Lake architecture. When we trigger the member and dept sync, the data's actual path looks like this:

1. Extract & Transform: Once the Temporal DW Worker receives its schedule, it connects into the external database over ODBC, pulls out the massive member and dept datasets, and in memory converts them into the columnar Parquet format.
2. State Tracking: To make sure an hour-long job like this can resume after an interruption, the Worker continuously writes CDC progress and cursors into the Redis pointed to by CDP_REDIS_HOST.
3. Load to Data Lake: The transformed data doesn't go into Postgres — it's packed into a pile of .parquet files and slammed straight into MinIO's data-warehouse Bucket via the S3 API we just disguised.
4. Query Engine: Here's the clever part — PostHog's high-performance analytics engine, ClickHouse, maps those Parquet files in MinIO directly as External Tables.
5. UI & Dashboard: When a user pulls up charts in the PostHog UI, or wants to filter user events by dept, ClickHouse — at blazing speed — JOINs the local event data (Events) with the member/dept data just synced into MinIO, and hands the result back to the Web API.

Draw the outermost Caddy, that love-hate DinD layer, and the data-lake pipeline inside it all together, and it looks roughly like this:

After a whole weekend of wrangling, my biggest takeaway is this: microservice "decoupling" is genuinely a double-edged sword. Layer on the company's kind of complex infrastructure — Caddy plus DinD — and debugging cost shoots straight up. A single missing environment variable, one silent default bump in an underlying driver, one queue with no listener — any one loose screw can bring the whole Pipeline to a quiet halt, without even throwing a decent error.