Wiring NLLB Translation into Rust and Breaking Through Every Layer of the Windows Toolchain

I wanted to move a fully offline NLLB translation feature from Python into Rust. The model is an int8-quantized NLLB-200 600M, reached through ct2rs (ctranslate2-rs), the binding onto CTranslate2. On Linux everything was clean and tidy.

But the moment it moved to Windows, I hit three walls in a row on the same road: it wouldn't link, it wouldn't shut down, and it crashed on machines with no GPU.

This post takes the three walls apart one by one. The first two are reefs in the Windows/MSVC toolchain; the third is a shutdown deadlock that had been hiding for two years, plus an NVIDIA driver bug that turned out to be none of my doing.

Two copies of protobuf, colliding#

The first wall stopped me before it could even run — it simply wouldn't link.

Besides translation, this desktop program statically links in a few other offline ML components: one is sherpa-onnx (which carries its own protobuf), with ONNX Runtime underneath it; and on the translation side, the NLLB tokenizer goes through sentencepiece, which also embeds a protobuf (the lite version). Two copies of protobuf, and under MSVC static linking they can't coexist. A full app link spat out:

libsherpa_onnx_sys.rlib(parse_context.obj) : error LNK2005:
  "google::protobuf::internal::ReadSizeFallback / InlineGreedyStringParser / ..."
  already defined in libsentencepiece_sys.rlib(parse_context.obj)
fatal error LNK1169: one or more multiply defined symbols found

204 LNK2005 + one LNK1169, all google::protobuf::internal::*. The cause is simple: sentencepiece's built-in protobuf-lite uses an un-renamed namespace google::protobuf, the same namespace as the full protobuf inside sherpa / ONNX Runtime; two identically-named google::protobuf::* symbols met in the same link, and the linker didn't know which copy to keep — a straight hit on ODR.

I laid the options out and weighed three roads:

• (a) /FORCE:MULTIPLE — tell the linker to stop caring and link anyway. It'll silently pick one protobuf — but pick wrong and you're done: ORT needs the full protobuf's descriptors, sentencepiece's lite version doesn't have them, and if lite wins, ORT dies quietly at runtime. That's ODR roulette.
• (b) Make both sides share one protobuf — patch the -sys crate's C++ build to force everyone onto a single system protobuf. Clean, but heavy; and ORT wants a new version while sentencepiece is stuck on an ancient built-in lite version, so the versions themselves are incompatible.
• (c) Split NLLB into a separate process/sidecar — each protobuf lives in its own binary, the link conflict vanishes (and it dodges the next CRT wall for free), at the cost of losing in-process and having to carry IPC serialization and lifecycle myself.

In the end I took none of the three. The real fix was a fourth, and the cleanest: don't have that protobuf at all. sentencepiece is only there to tokenize, so I turned off ct2rs's sentencepiece feature entirely and switched to the pure-Rust tokenizers crate reading tokenizer.json. With that one flip, sentencepiece-sys, its embedded protobuf-lite, and prost along with it all disappear from the dependency graph — the conflict isn't suppressed, it no longer exists. And the model directory already ships tokenizer.json, so it's zero deployment change. After removing it, LNK2005 dropped to zero and the exe linked cleanly.

/MT or /MD — the whole chain has to pick one side#

With protobuf gone, the second wall shot straight up — this time the CRT (C runtime).

ct2rs builds CTranslate2 as /MT (static CRT), while sentencepiece-sys was originally /MD (dynamic CRT); the MSVC linker refuses to mix /MT and /MD objects and throws LNK2038 (RuntimeLibrary mismatch, a hard reject). Even after removing sentencepiece, Rust std defaults to /MD, which still doesn't match CT2's /MT, earning a batch of LNK4098 + LNK2005 (malloc / free / __CxxFrameHandler duplicates).

The fix is a .cargo/config.toml that forces the whole chain to /MT:

[target.'cfg(target_os = "windows")']
rustflags = ["-C", "target-feature=+crt-static"]

(Scoped by target, so Linux/macOS CI is completely untouched.)

At first this scared me — a workspace-wide +crt-static, wouldn't that disturb the linking of all the C deps like sherpa-onnx and ORT? Digging in, I found it's actually the only correct answer, not a risky compromise:

• The sherpa static libs I use are a prebuilt static-MT release — the directive baked into them is RuntimeLibrary=MT_StaticRelease, and that CRT is unchangeable (downloaded and unpacked, not compiled locally). So /MT is the only value that can align with it; conversely, "change ct2rs to /MD" is physically impossible in this project — it would only shatter sherpa's /MT.
• ONNX Runtime uses load-dynamic (runtime LoadLibrary) — it doesn't participate in the link at all, so the CRT has zero effect on it.

In other words, +crt-static isn't dragging everyone down; it's standing on the same side as sherpa, which was /MT all along.

So what actually differs between /MT and /MD? Not the few hundred KB of duplicated code — the state. Each /MT module carries its own private CRT state, and the heap is the deadliest. The classic death: malloc on module A's heap, hand it to module B to free → heap corruption that detonates minutes later somewhere completely unrelated. /MD avoids it because everyone shares the one heap in ucrtbase.dll.

While I'm at it, let me puncture a misconception I had: /MT does not mean "you can't have a sidecar." The CRT only needs to align at module boundaries — same binary must match; DLL↔EXE in the same process may differ, only CRT resources (malloc'd pointers, FILE*) mustn't cross; and process↔process (a sidecar) shares nothing at all, so the CRT is irrelevant. A sidecar is precisely the layer where the CRT stops mattering: two address spaces, you physically can't malloc in A and free in B.

This also explains why sherpa had no CRT pain the whole way while CT2 was a minefield: sherpa exposes a C API (POD, opaque pointers, paired Create* / Destroy*, a C ABI stable for decades), so it's safe as a DLL with any CRT; CT2 exposes a C++ API (cxx — std::string / std::vector / std::future cross the boundary directly, and C++ has no stable ABI), so it can only be linked statically and must have its CRT aligned. FFI is "how you call"; the CRT is "who owns that state after the call" — the two are orthogonal, and you can have perfect FFI and still die at free().

With walls one and two down, it finally compiled and ran on Windows, translating fast and accurately (今天天氣很好 → The weather is nice today.).

The problem was the moment I closed it.

Translation done, I drop the Translator — and the whole program freezes right there. Not slow — dead still: CPU at zero, the screen no longer updating, Ctrl + C won't kill it, and in the end I have to open Task Manager and force it. And it's selective — it strikes only on Windows, only on CPU; the same code on CUDA exits perfectly clean.

I checked the issues and found I wasn't the first to hit it. This pit had been lying in a corner of this binding for a full two years, with a "working" workaround the whole time — except what that workaround does is leak the entire model.

And so the second half of this post exists. Follow it down and you find it's actually a two-year-long detective report hiding two deadlocks — and the final culprit has the same name both times: a thread_local destructor that, as a thread is ending, joins a thread pool under Windows' loader lock.

How many thread pools are stacked here#

To follow the fight ahead, you first need to know how many thread pools hide under "closing a translator." From the outside in:

• CTranslate2's own ThreadPool: each Translator has a ReplicaPool underneath managing a set of workers. When the model is destroyed, ~ThreadPool closes the queue and then calls worker->join() on each worker. This is the layer that ends up deadlocked — the victim.
• The thread pool for CPU parallelism: under OPENMP_RUNTIME=NONE (this binding's default), CT2's intra-op parallelism uses a static thread_local BS::thread_pool.
• Ruy's internal thread pool: int8 GEMM goes through the Ruy backend, and the first time each worker runs a GEMM it lazily builds a thread_local ruy::Context, which in turn keeps its own set of threads.

On the Rust side, cxx wraps the C++ object as a UniquePtr, and on drop cxx calls the C++ destructor for you.

So "closing a translator," a seemingly harmless act, is really a chain of destruction: Rust drop → C++ destructor → ~ReplicaPool → ~ThreadPool → join() on each worker. And before a worker truly ends, it still has to run the destructors of the thread_local objects hanging on it.

The deadlock hides right in that crack — "the worker is trying to end while still running a thread_local destructor."

Two years ago, this pit was traded away by "leaking a whole model"#

Rewind two years. The earliest report was simple: someone ran the official nllb.rs example with facebook/nllb-200-distilled-600M, translation was fine, but afterward the process hung for several minutes and Ctrl + C did nothing. Narrowing it down repeatedly, the culprit pinned to one line — stuck at drop(t), where t is the translator.

The maintainer reproduced it and quickly pointed to the scene: worker->join() never returns. But there's one sentence here that kept the case unsolved for two years — that worker thread looks like it ended perfectly normally, so what is the join even waiting for? A thread that "looks already finished," yet keeps whoever joins it waiting until the end of time. It doesn't add up.

The handling at the time was to route around it. Since it hangs at destruction, just don't let the destructor run: on the Rust side, add a #[cfg(windows)] impl Drop that bypasses the UniquePtr's drop, the C++ destructor doesn't execute, and naturally it can't hang at join.

The hang was gone. But the price is written honestly in the maintainer's own closing note: even when the Translator is dropped, RAM/VRAM isn't released — it waits for the whole process to end and the system to reclaim it. In other words — this isn't a fix; it's trading "the whole model never releases" for "it doesn't hang." For a short-lived program that runs and quits, you'll never get that bill; for a resident desktop program, it leaks once per load/unload round.

Then the case just hung there. The only new clue in between was one line the maintainer added later: CUDA doesn't hang, only CPU does, and "why the join gets stuck in some situations is still a mystery."

Remember that CPU/CUDA asymmetry. Later it points straight at the culprit.

OpenMP dodges one deadlock, not the other#

My first instinct on picking this up was to route the whole road around: this pattern where a thread_local destructor hangs — can I avoid touching it entirely?

Yes. As noted, CT2's CPU parallelism under the default OPENMP_RUNTIME=NONE uses that static thread_local BS::thread_pool, whose destructor joins on thread exit and then dies on Windows. But switch to OpenMP for parallelism and this thread_local thread-pool path is removed wholesale at compile time — no thread_local, no hanging destructor. So in theory, build CT2 as an OpenMP runtime (in the binding that's the openmp-runtime-comp feature) and the first deadlock vanishes.

Sounds smooth. Then I turned the feature on and MSVC threw a link error straight at me:

LINK : fatal error LNK1181: cannot open input file 'gomp.lib'

openmp-runtime-comp in the build script unconditionally does cargo:rustc-link-lib=gomp, and gomp is GNU's OpenMP runtime, which simply doesn't exist in the MSVC toolchain. MSVC has its own: its /openmp stuffs a /DEFAULTLIB:VCOMP directive into each object, the runtime is linked in automatically, and you specify nothing. So on MSVC, that gomp line isn't just redundant — it makes the link fail outright.

The fix: skip that link on MSVC, leave other toolchains as-is:

// build.rs: gomp is GNU's OpenMP runtime, absent on MSVC (LNK1181).
// MSVC's /openmp auto-emits /DEFAULTLIB:VCOMP, linking the runtime automatically.
if env::var("CARGO_CFG_TARGET_ENV").as_deref() != Ok("msvc") {
    println!("cargo:rustc-link-lib=gomp");
}

The feature linked, the OpenMP path connected, and that BS::thread_pool thread_local destructor was removed at compile time. Full of confidence, I reran load → translate → drop.

Still hangs.

Same spot, same worker->join(), same never-returns. OpenMP had clearly flattened the first path — so why is it still stuck? — and that's when it hit me: I'd only torn out one of the deadlocks. There's a second on this road, and it has nothing to do with which runtime CPU parallelism uses; OpenMP can't touch it. And it is the true lead of this whole case.

CUDA doesn't hang, CPU does#

Back to the clue I'd set aside.

That sentence already wears the answer on its face. Where do the CPU and CUDA paths differ? In the GEMM. int8 GEMM on CPU goes through the Ruy backend, CUDA goes through cuBLAS — the two touch entirely different things. So what does the CPU GEMM touch that CUDA can't? This, in src/cpu/backend.cc:

ruy::Context *get_ruy_context() {
  static thread_local ruy::Context context;
  return &context;
}

The first time each worker runs a Ruy GEMM, it lazily builds a thread_local ruy::Context. And ruy::Context's destructor does one thing: join Ruy's internal set of threads. Because it's thread_local, the moment this destructor runs is — exactly as the worker is ending.

The truth starts to converge here. On CUDA, the GEMM never calls get_ruy_context() from start to finish; the worker has no such thread_local at all, nothing to join, and the thread exits perfectly clean. On CPU it's the opposite: run int8 GEMM even once and this thread_local ruy::Context is hanging on the worker, waiting to be destroyed at the worst possible moment. The two-year "CPU/CUDA mystery" was never a mystery — it isn't that the join itself is flaky; it's that the CPU path has one extra thread_local destructor the CUDA path doesn't.

So why does "join as the worker ends" deadlock? Lay out the destruction order:

1. ~ReplicaPool → ~ThreadPool closes the queue and calls worker->join() on the worker
2. the worker's run() loop returns, the thread begins terminating
3. as the thread terminates it runs the thread_local destructors hanging on it, so ~ruy::Context goes to join Ruy's set of threads
4. this join happens on a thread that's already ending — deadlock

The key is the execution context in step 3. On Windows, thread exit triggers the thread_local/TLS destructor callbacks, and this whole flow runs holding the loader lock. Going to join another batch of threads right at that moment — threads that may also need the loader lock to finish — gives you a textbook loader-lock deadlock: the one holding the lock is waiting on others, and those being waited on are waiting on the lock. So ~ThreadPool hangs forever at worker->join().

This also answers the two-year-old paradox of "the worker looks like it ended normally": run() did return, the thread looks like it's wrapping up — but it's deadlocked in the join inside a thread_local destructor, never completing the last mile, so whoever joins it from outside never gets it back.

Once you see the root cause, there's only one direction for the fix. Move ruy::Context's destruction from "auto-triggered on thread exit" to "done by hand in a normal execution context."

CT2 happens to have a ready-made hook — ReplicaWorker::finalize() is called inside the worker's run(), before the thread actually exits, and it exists precisely for this kind of per-thread cleanup. Step one: change ruy::Context from "an auto-destructed thread_local object" to "a heap pointer you can clear by hand," and add a clear_ruy_context():

// backend.cc: context becomes heap-allocated, its lifetime no longer tied to thread-local destruction.
static thread_local ruy::Context* ruy_context = nullptr;

ruy::Context *get_ruy_context() {
  if (!ruy_context)
    ruy_context = new ruy::Context();
  return ruy_context;
}

void clear_ruy_context() {
  delete ruy_context;
  ruy_context = nullptr;
}

Step two: call it inside destroy_context() for CPU devices — that function is exactly the cleanup point finalize() reaches:

// devices.cc: release this worker's ruy::Context in a normal execution context (not at thread exit),
// to avoid hanging ThreadPool shutdown when joining Ruy threads on Windows.
#ifdef CT2_WITH_RUY
    if (device == Device::CPU) {
        cpu::clear_ruy_context();
    }
#endif

The join now happens in a normal context — no loader lock, the thread hasn't started terminating — and completes fine. And because the destructor now runs to completion, that Rust-side Drop bypass that leaked the whole model to dodge the hang can be removed too. Two deadlocks — one routed around with OpenMP (and its inability to link on MSVC fixed along the way), one solved at the root.

Autopsy: CPU leaks RAM, CUDA leaks VRAM#

Fixing the deadlock is one thing; I also wanted to confirm "no more leak" was real. So on Windows/MSVC 14.44/x64, int8 NLLB-200 600M, Ruy backend, I ran 5 rounds of load → translate → unload and measured RSS after each:

Three rows, which happen to be the three stages of this case. Row one is the "working" workaround: ~870 MB leaked per round (the whole model unreleased), 4.3 GB gone in 5 rounds, the slope climbing all the way — a resident program can't hold out. Row two is an intermediate: even if I leak only that ruy::Context and not the model, it still leaks ~250 MB per round — that's Ruy's prepacked cache held down by the context; it shows "leak a little less" isn't the answer — as long as the context isn't destroyed correctly, the cache behind it keeps sitting there. Row three is the fix: 16, 17, 18, 18, 19, basically flat, clearing the context at finalize and releasing the prepacked cache with it. And rebuilding the context after destruction is no problem at all — translation output is byte-identical across rounds; clear it, use it again, same result.

Stack every round's RSS for the three approaches on one chart, and who's bleeding round by round versus who's a flat line hugging the bottom splits at a glance:

One more thing I didn't measure at first and only reasoned from code: this leak doesn't discriminate by device. That workaround that skips the destructor to dodge the deadlock is #[cfg(target_os = "windows")] — it looks at "is this Windows," not which device — so whether CPU or CUDA, as long as it's Windows, drop is skipped and the whole model still isn't released. On CUDA it leaks VRAM: the unfixed version, run 5 rounds, climbs VRAM from 5516 to 8911 MB (~840 MB per round, 4.65 GB over 5 rounds), while the fork releases fully on both paths. CUDA is even worse — a laptop with 8 GB of VRAM overflows around round 4, whereas CPU RAM at least has virtual memory to fall back on.

The gap between 885 and 19 isn't measurement error. It's the distance between "keep it from hanging" and "actually fix it" — both are equally green in a smoke test, both exit normally; only by running five full rounds and watching memory do you see one bleeding and the other not.

A GPU build that crashes when there's no GPU#

Deadlock fixed, leak sealed, I thought I was through with ct2rs. Then, after building the CUDA version, something even stranger showed up: on a machine with no usable GPU, it randomly SIGSEGVs (exit 139) after translating. And the conditions are absurdly picky — I ran the combinations into a table:

Only one combination blows up: built with CUDA, but run on CPU, and actually translated — 4–5 crashes in 6. Everything else is 0.

To catch this kind of ghost — "crashes tens of seconds later, with a stack that has nothing to do with the culprit" — I hooked up a VEH exception handler + backtrace symbolization + DLL load/unload logging, and caught the scene:

ACCESS_VIOLATION at 0x7ffec97a8190: execute (DEP) — that address state=FREE (unmapped)
RIP = 0x7ffec97a8190; stack has only ntdll thread-pool frames: TpSetWaitEx …
[dll] load   nvdxgdmal64.dll  0x7ffec97a0000..0x7ffec9834000
[dll] unload nvdxgdmal64.dll  0x7ffec97a0000..0x7ffec9834000   ← after unload, crash at +0x8190

A DEP execution violation — the CPU jumped to execute an address that's already unloaded and not even mapped. Both crashes landed at the same RVA +0x8190 (different base due to ASLR). The stack had not a single CTranslate2/ruy frame, all ntdll thread-pool.

The culprit is the NVIDIA driver. nvcuda.dll's init pulls in nvdxgdmal64.dll (the driver's DXG DMA allocator); when cudaGetDeviceCount() finds no usable device (err=100 cudaErrorNoDevice), that DLL is FreeLibrary'd — but a wait callback it registered on the Windows thread pool is never deregistered. That orphan callback fires ~18–29s later, jumping into the now-unloaded module → DEP violation. This one chain explains every observation: only WITH_CUDA loads nvcuda; only the CPU device reaches "no usable GPU"; only translating keeps it alive long enough for the callback to fire (the load→drop-only version exits in 2.9s, faster than the callback, hence 0/12); and it's random because that's a scheduling race.

Whose bug is this? I once took it for granted and said "this is a known driver bug" — and got slapped by my own hand. I wrote a 12-line, cudaGetDeviceCount()-and-a-wait-loop-only pure-C program: no CT2, no ruy, no Rust, and it still crashes 3/4–4/6. So this is definitively the NVIDIA driver's own bug, nothing to do with CT2, ct2rs, or my fork.

The driver side is no help (that PyTorch reporter's rollback didn't even save it), and I can't control the client's driver version, so I still have to block it on my end. The fix is surgical: use LdrRegisterDllNotification to watch for DLL loads, and the instant nvdxgdmal64.dll is loaded, immediately pin it with GetModuleHandleExA's GET_MODULE_HANDLE_EX_FLAG_PIN flag — NVIDIA's later FreeLibrary then can't unmap it, the module stays put, and when the orphan callback fires it hits code that's still mapped — harmless. Baseline 4–5/6 → 0/8 after pinning, with not a single unload event — the causal chain closes.

Triggering it takes four conditions all at once: an NVIDIA driver installed, the program actually initializing CUDA, CUDA finding no usable device, and the process then staying alive tens of seconds. Real-world "driver present, no usable GPU" cases are more common than you'd think: CUDA_VISIBLE_DEVICES set wrong, hybrid-GPU laptops with the discrete GPU disabled (Optimus/MUX), a driver too old for the runtime, remote-desktop sessions with no GPU, a GPU held exclusively by another session… on the client's end you can't guess which one it'll be.

A few closing thoughts#

Draw the whole destruction chain out and the deadlock point and where the fix intervenes become obvious:

With the three walls down, looking back, they're really the same shape of pit: each piece is fine on its own — it only goes wrong assembled into this specific combination. Two protobufs each compile fine; /MT is happy alone in one binary; thread_local destructs obediently on other platforms; cudaGetDeviceCount() is a perfectly legal call. All the pain happens at the boundaries and in the combinations — protobuf colliding in the same link, CRT colliding in the same binary, join colliding under the loader lock, the callback colliding with an already-unloaded module.

And the most interesting thing on this whole road is why the two-year deadlock was never truly solved: not because it was too hard for anyone to understand, but because that workaround was "good enough" — it made the hang disappear, and the leak it traded for happened to hide in a corner most people won't notice: the process ends and the memory comes back anyway. Short-lived programs never get that bill, so the problem stayed "solved" for two years. Routing around a problem and solving it look a lot alike — especially when the cost of routing around is deferred to the future, pushed to someone else, pushed to "the OS will clean it up."

thread_local is convenient — it manages a lifetime for you automatically; /FORCE:MULTIPLE, skipping drop, "the driver will fix it eventually" are all convenient too. But "convenient" often means it moves the cost to a moment you can't see right now, an environment you can't see, someone you can't see. And this whole trip, I was mostly just carrying those moved-away costs back in front of me, one by one.

References#

• CTranslate2 #2076 — Fix Windows shutdown deadlock and leak with the Ruy backend
• ctranslate2-rs #144 — Don't link gomp on MSVC
• ctranslate2-rs #74 — the leak workaround that skips Drop on Windows
• ctranslate2-rs #64 — Process doesn't exit / hangs at the end on Windows (the original issue)
• discuss.python.org — a public report of nvdxgdmal64.dll_unloaded