I'm building a strictly-offline desktop notes app, using BlockNote as the editor core. BlockNote has an official AI package, @blocknote/xl-ai, that does selection-based AI (improve writing / simplify / fix spelling / translate / continue), complete with a gorgeous (strikethrough + highlight, compared right in place) and Accept/Reject. The official demo looks incredibly slick.

My need was simple: wire this AI up to the Gemma 3 27B model we run locally (offline, no cloud API). The official package does it all, so hooking up an endpoint should take ten minutes, right?

It turned out to be the start of yet another debugging nightmare.

First, Understand How xl-ai Talks to a Backend#

Step one, obviously, is reading the docs. Digging into xl-ai's docs, it turns out the whole AI flow is built on top of Vercel AI SDK. Its outward-facing hook is a — you can plug in your own transport to decide "how the request gets sent to the LLM backend."

The standard approach in the docs is to build a createOpenAICompatible adapter with the Vercel AI SDK transport, point it at your own OpenAI-compatible endpoint, and hand it to AIExtension:

ts
import { createOpenAICompatible } from "@ai-sdk/openai-compatible"
import { AIExtension, createBlockNoteAIClient } from "@blocknote/xl-ai"

// point the model at our local OpenAI-compatible chat completions
const client = createBlockNoteAIClient({ baseURL: "http://localhost:7501/v1", apiKey: "x" })
const model  = createOpenAICompatible({ name: "local", fetch: client.fetch })
  .chatModel("google/gemma-3-27b-it")

// attach AIExtension; transport uses its default (internally fires tool-calling requests via model)
useCreateBlockNote({ extensions: [AIExtension({ model })] })

Looks reasonable. Our backend happens to have an OpenAI-compatible chat completions route, so just hook it up, right?

And that's when I walked straight into the first wall.

xl-ai Assumes Your LLM Does Tool-Calling#

After wiring up the transport, the AI did nothing at all, or threw an internal error outright. Digging in revealed the essence of xl-ai's pipeline:

text
transport → chat.sendMessage → tool-input-delta → parsePartialJson → operation

What xl-ai gives the LLM is not "please rewrite this text," but a called applyDocumentOperations, whose input is an operations array where each op looks like { type: "update"|"add"|"delete", id, block: <html> }. It expects the LLM to use the tool-calling format, streaming out this tool's input JSON delta as it generates (one character at a time), and xl-ai uses to parse and apply that unfinished JSON as it arrives.

In other words, xl-ai assumes what's behind it is a cloud LLM that dutifully does tool-calling (GPT-4, Claude, that kind).

But our local Gemma is nowhere near stable at tool-calling; it only spits out plain text. Forcing the transport onto it means making xl-ai run parsePartialJson on plain text, which slams straight into its internal around "the same op yielded multiple times, partial → complete transitions." I tried two paths:

  • buffer mode (wait for the whole segment, then send at once)
  • append-only streaming (append as it receives)

Both died, with the classic error No matching function for add.

The essence of the problem is a format mismatch — xl-ai receives the tool's input JSON one character at a time via the AI SDK's tool-input-delta, assembling something like this:

jsonc
// what xl-ai expects: applyDocumentOperations' input, partial → filled in char by char
// (in real SSE, one tool-input-delta per line; inputTextDelta accumulates this JSON string)
{ "operations": [{ "type": "update", "id": "blk-1$", "block": "<p>rewriting…</p>" }] }

NOTE

The $ suffix on each id is xl-ai's convention, telling the model "this is an id, not text."

But Gemma returns this:

text
The rewritten content.

Plain text. Feeding that to parsePartialJson only parses out a pile of undefined, plus the partial→complete race, hence No matching function for add.

The conclusion is clear: xl-ai wants a structured tool call, Gemma only gives plain text, and the transport path is dead on a local model.

What About Writing a ChatTransport That Fakes Tool-Calling? Also Dead#

Since Gemma can't tool-call, could I write my own ChatTransport on the front end and wrap the plain text into the tool call format xl-ai wants? I made a lib/blocknote-ai-transport.ts to try:

ts
// gist: implement the Vercel AI SDK ChatTransport, internally call our existing /chat/stream (plain text),
// buffer the whole segment, then emit it to xl-ai in the shape of an applyDocumentOperations tool call.
class BlocknoteAITransport implements ChatTransport<UIMessage> {
  async *sendMessages({ messages, body }) {
    // 1. call the plain-text endpoint, buffer the whole output
    let buf = ""
    for await (const delta of chatStream({ userPrompt: build(messages) })) {
      buf += delta
      // (meanwhile you can push a fake "typewriter" animation in the toolbar for visual feedback)
    }
    // 2. once buffered, assemble the applyDocumentOperations tool call xl-ai expects and emit once
    const toolCallId = crypto.randomUUID()
    yield { type: "tool-input-start", toolCallId, toolName: "applyDocumentOperations" }
    yield { type: "tool-input-delta", toolCallId, inputTextDelta: JSON.stringify({
      operations: parseTextIntoOps(buf),   // split the plain text into an operations array yourself
    })}
    yield { type: "tool-input-available", toolCallId }
    yield { type: "finish" }
  }
}

This one is clean — parsePartialJson receives well-formed JSON, hits no race, and xl-ai handles it as a normal tool call. But by design it's fake streaming: I have to buffer Gemma's whole output before I can assemble the tool call and emit; a long response means ten-to-twenty seconds of blank while the user stares at a dead editor. The toolbar typewriter effect is just for show — the editor itself is frozen.

Conclusion: clean but slow, not real streaming. Abandon this, and keep looking for a path that "skips the buffer AND bypasses tool-call."

Why Not Just Force Gemma to Emit JSON on the Backend?#

At this point you might think: don't Ollama, vLLM, and llama.cpp already support (format: "json" / guided_json / GBNF grammar)? Just constrain Gemma on the backend to output the tool-call JSON xl-ai wants — back to path one. I seriously considered it, but several practical reasons blocked it:

  1. HTML inside a JSON string is fragile: each op's block is an HTML string, and the model has to correctly escape ", balance <p>...</p>, handle Unicode escaping. A grammar can only guarantee the JSON is valid JSON — it can't govern the correctness of the HTML inside the string. Gemma 3 27B is mediocre at HTML anyway; common issues are broken tags or eaten attributes, and grammar can't save that layer.
  2. Token cost and generation speed: xl-ai's schema wraps a layer of applyDocumentOperations.operations[].{type,id,block}; the scaffolding tokens alone are a lot, and with the extra JSON-escape characters, local inference time clearly stretches — a net negative for a UX where "the user hits the button and wants to see characters appear immediately."
  3. Even if it passes, you hit the same wall: see the "bonus round" section later — even a cloud model using the right tool-call format still produces an ugly inline diff on long text. Forcing JSON only solves "the parser doesn't blow up"; it doesn't remove the ceiling the diff logic itself has on long text whose structure got mangled.
  4. An op-forwarder is actually cleaner and more controllable: the backend takes plain text and deterministically assembles ops itself — no JSON parse / escape risk, easy to test and debug (a lot of bugs this time were found with backend print + replaying HAR).

In short: forcing JSON dumps the race/format responsibility onto the model, and Gemma can't necessarily catch it; and even if it does, the UX still hits the same ceiling. Better to let the backend act as a "translator" — cheap and controllable.

The Official Package Hid a manual-execution Escape Hatch#

Not wanting to give up that diff effect, I went back to xl-ai's examples. Web-fetching around, I found the source of this official example:

raw.githubusercontent.com
https://raw.githubusercontent.com/TypeCellOS/BlockNote/main/examples/09-ai/05-manual-execution/src/App.tsx

05-manual-execution — the name alone was right. Fetching the whole App.tsx to read, it demonstrates two ways that completely bypass transport / chat / parsePartialJson:

  1. executeOne(chunk): apply a single block change (apply synchronously, then delay the accept).
  2. Streaming with objects: use , feeding operation objects one by one with writer.write(operationObject).

The two modes look roughly like this:

ts
import { aiDocumentFormats, StreamToolExecutor } from "@blocknote/xl-ai"

const provider = aiDocumentFormats.html.getStreamToolsProvider({ withDelays: true })
const tools    = provider.getStreamTools(editor, selectionInfo)
const executor = new StreamToolExecutor(tools)

// mode 1: executeOne — apply one op at a time (change synchronously, then accept)
await executor.executeOne({ type: "update", id: "blk-1", block: "<p>the fixed one</p>" })

// mode 2: streaming with objects — write operation objects yourself
const writer = executor.writable.getWriter()
writer.write({ operation, isUpdateToPreviousOperation: false, isPossiblyPartial: true, metadata: {} })
// …write multiple partials as you receive…
await writer.close()

The example also touches on how getStreamToolsProvider({ withDelays, defaultStreamTools, selectionInfo }) produces streamTools.

WARNING

There's a detail I didn't notice at the time that hurt me badly later: the official example's write calls for partial / complete are all direct calls, not awaited. This fire-and-forget is the key to real streaming — remember it (the pit is coming later).

The key realization: StreamToolExecutor is the real execution end that applies operations onto + the suggest-changes plugin, and it's fully decoupled from chat / transport. I don't need Gemma to speak tool-calling at all — I just need to assemble the operation object myself and feed the executor. The diff and Accept/Reject are all done by executor + suggest-changes, so I still get them.

The operation object looks like this:

jsonc
{
  "operation": { "type": "update", "id": "<block-id>", "block": "<p>the rewritten content</p>" },
  "isUpdateToPreviousOperation": true,   // whether this op continues/replaces the previous one
  "isPossiblyPartial": true,             // content not fully generated yet (streaming)
  "metadata": {}
}

The First Version Worked, but It Was Fake Streaming#

Following the manual-execution path, the first version (also the first commit) did this:

  • The front end overrides xl-ai's invokeAI and calls the backend's plain-text stream /chat/stream itself.
  • On receiving Gemma's plain text, the front end splits the text into chunks by \n\n paragraph boundaries and assembles update / add operation objects.
  • Then writer.write feeds StreamToolExecutor, which applies live and produces the diff.

I also had to add one thing by hand. Some background first: BlockNote is built on ProseMirror, and all state changes go through an object called a , applied to the document only via view.dispatchTransaction(tr). To make the AI's changes be treated as "suggestions" (the strikethrough + highlight diff marks, with Accept/Reject) rather than written straight into the document, you have to intercept at this layer and rewrite the transaction's insert/delete/replace steps into the corresponding suggestion-mark steps.

xl-ai registers the plugin, but natively it inserts the diff marks manually itself; we bypassed that path, so we have to wrap the view's dispatchTransaction in a withSuggestChanges layer ourselves, so the AI's changes get treated as "suggestions" with the strikethrough highlight and Accept/Reject:

ts
import { withSuggestChanges } from "@handlewithcare/prosemirror-suggest-changes"
import { AIExtension } from "@blocknote/xl-ai"

export function installManualAiRunner({ editor }) {
  // 1. permanently wrap the view's dispatch into a suggest-changes version (AI change → suggestion mark)
  const view = editor.prosemirrorView
  view.setProps({ dispatchTransaction: withSuggestChanges(view.props.dispatchTransaction) })

  // 2. override AIExtension's invokeAI / abort to route through our manual flow
  const aiExt = editor.getExtension(AIExtension)
  aiExt.invokeAI = async (opts) => { await runManualAi({ editor, aiExt, opts }) }
  aiExt.abort    = () => { cancelRef.current?.(); aiExt.closeAIMenu(); return Promise.resolve() }
}

Note: this version has no tool-calling format anywhere. It's plain text coming in, ops assembled by hand on the front end.

It "worked," but the effect was fake streaming — the editor didn't change live; every time it waited for the event done (the whole segment finished) instant before applying the text all at once, a very different feel from the official demo's character-by-character reveal. On a long response the screen just froze, then everything popped in at the end. I looked at it and decided this was far from "working" and had to be fixed.

I only later caught the root cause of the fake streaming: I made a mistake the official example didn't — I used await writer.write(...).

ts
// ❌ fake streaming: await serializes every write, stuck on the same microtask chain, all crammed to the end
for await (const op of ops) {
  await writer.write(op)
}

// ✅ real streaming: fire-and-forget, each partial op is applied by the executor immediately (the official example's way)
onOp: (op) => {
  void writer.write(op).catch((e) => console.warn("write op failed:", e))
}

The official example's writes for partial / complete are all direct calls, not awaited (fire-and-forget), so each partial op is applied by the executor immediately; my extra await stuck all the writes on the same chain, so they all crammed to the end → fake streaming. Only after switching to void writer.write(op) did it start to feel like step-by-step application.

But even without awaiting the write, the layer of "the front end assembling operations from plain text itself" is still fragile: you have to catch the partial / complete timing yourself, handle block boundaries yourself, decide which segment maps to which block yourself… tons of edge cases. Rather than force it on the front end, move this layer to the backend.

Move the Op-Assembly Layer to the Backend, Impersonating a Tool Call#

Since the front end assembling ops from plain text is too fragile, change the division of labor: let the backend assemble the operations. And the backend is right next to Gemma, able to receive tokens and emit ops at the same time — real streaming far more natural than the front end buffering-then-assembling.

I asked the backend to open a new endpoint /chat/stream/op: it takes Gemma's plain text and, on the backend, receives the delta while assembling clean BlockNote operation objects, pushing them back to the front end over . The front end degrades into a thin op-forwarder: receive an op, writer.write it to the executor, no text parsing needed at all.

ts
chatStreamOp(req, {
  onOp: (op) => { void writer.write(op) },          // the backend emits an op, just forward it
  onDone: async () => { await writer.close(); await executor.finish() },
})

SSE events: status (model loading state) / op (one operation) / done / error. The core of the backend assembling ops is wrapping Gemma's delta into that xl-ai object format:

python
def _op_event(block_id: str, html: str, is_update: bool, partial: bool) -> str:
    data = {
        "operation": {"type": "update", "id": block_id, "block": html},
        "isUpdateToPreviousOperation": is_update,   # first op=False, subsequent=True (continuing the same block)
        "isPossiblyPartial": partial,               # streaming=True, finalized=False
        "metadata": {},
    }
    return f"event: op\ndata: {json.dumps(data, ensure_ascii=False)}\n\n"

# main loop gist: receive Gemma's delta, and emit the text accumulated so far as a "growing block"
async for line in resp.aiter_lines():
    delta = parse_openai_delta(line)
    cur_text += delta
    yield _op_event(cur_id, f"<p>{cur_text}</p>", not is_first_op, partial=True)
    is_first_op = False

This is the truth behind the "tool call" in the title — it's not that Gemma actually does tool-calling; it's the backend impersonating that structured format (translating plain text into an operation stream) and feeding xl-ai's native executor. The ops are backend-constructed so they're clean, the backend generates-and-emits so it's real streaming, and it goes through the executor so diff / Accept-Reject are all natively supported.

Beautiful in theory. In practice, every edge case after this had to be stepped on one by one. First an overview, then each dismantled:

  • Pit 1 — No tool can handle update: xl-ai's default provider binds the tool set to the command (continue only opens add, improve only opens update…), mismatched with our "backend always emits update."
  • Pit 2 — update vs add+delete: diff-strategy choice — character-level in-place diff or whole-block replacement, each with landmines.
  • Pit 3 — \n\n vs \n: the backend's segment separator disagrees with the LLM's actual output habit → everything crams into the first block.
  • Pit 4 — paragraph counts don't line up: Simplify merges paragraphs, Improve expands them → LLM output paragraph count ≠ selected block count.
  • Pit 5 — streaming isn't smooth: xl-ai's tools only apply once every 50 characters + I turned off the withDelays animation.

Pit 1: The Executor Doesn't Recognize the Tool the Backend Emits#

The first test after connecting the backend op-stream, /ai (no selection, pure continuation), threw this.

The crack: xl-ai's provider is "command-restricted" — the continue command only opens the add tool. But our backend emits update ops for all commands, and with no update tool on hand the executor throws "no tool can handle update."

Fix: don't use xl-ai's restricted provider, build one with all tools open:

ts
getStreamToolsProvider({ defaultStreamTools: { add: true, update: true, delete: true } })

Pit 2: update or add + delete?#

Selection-based rewriting has two approaches, each with landmines:

  • update in place: xl-ai does a character-level inline diff (original struck through, new highlighted, compared in place) — exactly that demo effect. But mid-stream (isPossiblyPartial=true) it's conservative — unsure whether later tokens will re-match the original, so it daren't delete the original's tail early, and only commits the deletion once the op finalizes (isPossiblyPartial=false). It shows up as "the first few sentences diff normally, then it stalls mid-stream, and the rest of the segment only appears all at once at done" — real streaming feeding fake streaming.
  • add + delete: insert the new content as a new block, then delete the original block. Stable, smooth streaming, but it's block-level (new content appears as a whole block, the old one deleted whole), not character-level in-place comparison.

I went back and forth and finally chose update (I wanted that in-place UX). And /ai continuation, being pure add insertion with no "should I delete the original" hesitation, was smooth the whole time, so it's fixed on add inserted after the cursor.

Pit 3: One Character Cost Me an Afternoon (\n\n vs \n)#

The "stuck + a bunch of original text untouched" symptom I stared at screenshots guessing for a long time; in the end I cracked it by replaying the real payload from the against the backend and seeing exactly what ops it emitted.

Test: select 7 paragraphs (7 blocks) for Improve writing. Pull that request body out of the browser's saved HAR, replay it verbatim against the backend, and count how many ops each block received:

python
import json, httpx, collections

body = json.loads(open("payload.txt").readline())          # request body pulled from the HAR
per  = collections.Counter()
with httpx.stream("POST", "http://127.0.0.1:7502/chat/stream/op", json=body) as r:
    ev = None
    for line in r.iter_lines():
        if line.startswith("event:"): ev = line[6:].strip()
        elif line.startswith("data:") and ev == "op":
            op = json.loads(line[5:])["operation"]
            per[op["id"][:8]] += 1
print(per)

Output (before the fix):

text
block update order: [('32671a2a', 425)]   ← all 425 ops hit the 1st block
finalized block count: 1                   ← the other 6 blocks got zero ops

The whole output crammed into the first block, the other 6 blocks kept their original text. The reason was embarrassingly dumb: the backend split segments and switched blocks on the blank line \n\n, but Gemma's output paragraphs are separated by a single \n (matching the input format) — there's never a \n\n → it never switches block → everything crams into block[0]:

python
# before: split on \n\n → Gemma output has no \n\n → the while never enters → never switch block
while "\n\n" in pending:
    head, pending = pending.split("\n\n", 1)
    ...

# after: split on single \n, each segment maps to the next selection_id; skip blank lines, merge into the last block
while "\n" in pending:
    head, rest = pending.split("\n", 1)
    head = head.strip()
    if not head:                                   # blank line → don't switch block
        pending = rest; continue
    if block_idx >= len(req.selection_ids) - 1:    # last block: merge the rest in
        pending = head + " " + rest; break
    yield _op_event(cur_id, f"<p>{head}</p>", not is_first_op, partial=False)  # finalize this segment
    written_ids.add(cur_id)
    block_idx += 1
    cur_id = req.selection_ids[block_idx]           # switch to the next selected block
    is_first_op = True
    pending = rest

Changing the split from \n\n to a single \n, replaying the same payload:

text
block appearance order: [all 7 ids hit]
finalized block count: 7 / 7

A one-character difference cost me an afternoon.

Pit 4: Paragraph Counts Don't Line Up, Simplify Quietly Merges Paragraphs#

After fixing the split, short text was fine, but long-text Simplify still "changed halfway, with the rest deleted paragraph by paragraph."

Replaying a 21-paragraph Simplify:

text
input 21 paragraphs, 21 selection_ids
→ only 7 blocks got updated, the other 14 unmatched

This time it's not the split — the paragraph count itself doesn't line up: Simplify condenses 21 paragraphs into 7 output paragraphs, so 1-to-1 naturally only matches the first 7 blocks, and the other 14 keep their original text.

Fix: at the end, delete every block that was never updated from start to finish. There's also an off-by-one here — you can't use "the last block index written" to compute the tail (Gemma's tail often has an extra \n that advances the cursor one too far and misses a deletion), so switch to actually recording which block ids were written (a set), and at the end delete those not in the set:

python
written_ids = set()
# ...every time you yield an update op, written_ids.add(cur_id)...
for sid in selection_ids:
    if sid not in written_ids:
        yield delete_event(sid)   # delete anything never written, to avoid leftover original text

Verification: 21 selections, output 3 paragraphs → update 3 + delete 18 = covers 21, zero leftovers.

Pit 5: The "Smoothness" Is Actually an Animation, Not Real Per-Token#

The diff was right, but in my testing the characters "popped out block by block," lacking that official-demo smoothness of characters appearing one by one.

The reason: xl-ai's update/add tools have internal throttling — they only apply once the block content grows by about 50 characters:

ts
// inside xl-ai's update tool's execute (throttling gist)
let r = 50
return { execute: async (op) => {
  if (op.isPossiblyPartial) {
    const len = JSON.stringify(op.block).length
    if (len < r) return                  // not grown past the threshold → skip, don't apply
    r = len + 50                         //  bump the setpoint up by +50
  } else { r = 50 }                      // non-partial (finalized) resets the threshold
  // …compute diff, attach suggestion mark, optionally await animation delay
  for (const step of diffSteps) {
    if (opts.withDelays) await sleep()   // ★ withDelays:false lacks this line → 50 chars applied at once
    applyStep(step)
  }
}}

And I had set the provider to (figuring the backend was already streaming, no need for more animation). The two together mean 50 chars per jump — very choppy.

Setting withDelays back to true, xl-ai inserts a tiny delay between each diff step, and that 50-char chunk gets revealed character by character — visually just like the demo, characters appearing one at a time:

ts
const provider = aiDocumentFormats.html.getStreamToolsProvider({
  withDelays: true,                                    // ★ animation on; the demo default is also true
  defaultStreamTools: { add: true, update: true, delete: true },
})

Turns out that "smoothness" is an animation, not real per-token.

Wired All Together, It Looks Like This#

After a big detour, it ended up like this:

The final division of labor:

  • Selection commands (fix-spelling / improve / translate) → backend emits update ops, a character-level in-place diff on each selected block; when output paragraphs are fewer than the selection, the extra tail blocks are deleted at the end.
  • /ai continuation (no selection) → backend emits add ops inserted after the cursor, streamed character by character.
  • Front-end provider with all three tools open, withDelays: true, dispatchTransaction wrapped in withSuggestChanges.

Against the initial dead end (forcing the transport to make Gemma tool-call), the core mental shift is: don't force the local model to speak xl-ai's language; add an adapter layer on the backend that translates plain text into operations and feeds the executor directly.

Blocked by Corporate MAC, Stuck Repackaging Endlessly#

There was a painful stretch in the middle: front-end changes couldn't be tested locally with pnpm dev — the company's blocked it. Every front-end change required a full repackage of a desktop installer (~2 minutes) + reinstall to verify — iteration slow enough to question my life choices.

Then I thought of a workaround: the backend Python service is actually a standalone local (running on localhost:7501), ultimately compiled into a packed into the installer — I can't recompile the .pyd, but I can use a lightweight venv (only fastapi/httpx/opencc/tiktoken installed, no torch at all) to run just that op-stream route separately on 7502, and bake the front-end build to point at 7502. That way the backend logic can be edited live and restarted for testing, no .pyd recompile each time. Replaying HAR to find the split bug was only possible thanks to this setup.

The Real Ceiling Is Actually the Model Not Being Strong Enough#

After fixing all the code problems, one remained that's not a bug — it's model capability:

Gemma 3 27B gets lazy on long input. Replaying an "entire document crammed into a single block (1647 chars)" Improve writing, comparing input and output:

text
input 1647 chars → output 1472 chars
input/output "identical ending" = 1044 chars

In other words, Gemma only genuinely rewrote the first ~600 characters, and echoed the last 1044 back verbatim, unchanged. The diff faithfully displayed this as "the front part changed, the back part completely untouched" — I too thought it was a bug at first; only after digging out this data was I sure it's not a program issue, it's the model slacking off.

Two derived situations:

  1. The whole document crammed into a single block: only one block, the backend can't split segments, so it can only repeatedly diff the giant block — slow and messy diff.
  2. Paragraph structure scrambled by the model: the prompt clearly says "keep the same paragraph count," yet Gemma still merges paragraphs, and the 1-to-1 mapping collapses (patched by Pit 4's tail deletion, but the shape is still ugly).

The root fix is per-block invocation: N selected blocks means N LLM calls, each block getting only its own segment — each is small, the model rewrites it fully, always 1-to-1 with no misalignment. The cost is N calls, higher latency, and it needs both front end and backend changed. Left as the next-phase enhancement.

Not Just Gemma — the Official Demo Blows Up in Similar Situations Too#

To confirm whether this is really our local model's problem, I went to BlockNote's official demo page, selected the same passage (including the repeated fragments a prior AI had left) and hit Simplify — it blew up the same way, the diff a chaotic mess, no telling what it was changing.

Pulling the request body the official demo sends, that tool definition confirms the single-tool + operations-array design:

jsonc
"toolDefinitions": {
  "applyDocumentOperations": {
    "inputSchema": {
      "type": "object",
      "properties": {
        "operations": {
          "type": "array",
          "items": { "anyOf": [
            { "type": "object", "description": "Update a block", "properties": {
                "type": { "enum": ["update"] },
                "id":   { "type": "string", "description": "id of block to update" },
                "block":{ "type": "string", "description": "html of block (MUST be a single HTML element)" }
            }, "required": ["type", "id", "block"] }
            // ...add / delete share the same schema
          ]}
        }
      },
      "required": ["operations"]
    }
  }
}

The partial streaming of the input looks like this (one tool-input-delta per line, inputTextDelta filled in one character at a time):

jsonc
{"type":"tool-input-delta","toolCallId":"call_xxx","inputTextDelta":"目"}
{"type":"tool-input-delta","toolCallId":"call_xxx","inputTextDelta":"的"}
{"type":"tool-input-delta","toolCallId":"call_xxx","inputTextDelta":"與"}
// ...accumulating the whole operations JSON character by character
{"type":"tool-input-available","toolCallId":"call_xxx","toolName":"applyDocumentOperations"}
{"type":"finish-step"} / {"type":"finish","finishReason":"tool-calls"}

And the selectedBlocks I submitted already contained the repetition a prior AI had left ("and according to their and according to their feedback improve the translation system their and according to their…"); a cloud model doing Simplify on that mess naturally won't produce a pretty diff — this is no longer about model strength, but that an inline diff on "structurally-mangled long text" is just inherently ugly.

IMPORTANT

In other words: the mapping-misalignment and ugly-diff problems we hit are partly a ceiling of BlockNote's in-place diff approach, not a bug unique to the local Gemma. Swap in OpenAI/Claude and you'd only win on "it won't echo half-way through" — the diff still struggles on long text.

Final Thoughts#

The biggest takeaway from this trip: behind an official package's "beautiful demo" there's usually the assumption of a very strong backend (here, a tool-calling cloud model). Once your environment doesn't fit that assumption (a local small model that only emits plain text), forcing its transport onto you only slams into a pile of internal race conditions; the right posture is to find its escape hatch (manual-execution / StreamToolExecutor), add an adapter layer yourself to bridge the two formats — and then step on the edge cases it didn't handle for you, one by one: tool-set restriction, partial-diff timing, split separators, paragraph-count mapping, streaming throttle animation…

In the end the feature works, and short-to-medium text feels great. The remaining long-text problem, honestly, isn't something the front end or backend can squeeze out more of — that's the model's own ceiling. Knowing where the ceiling is counts as a gain too.