Limits

2026-08-31

Things worth knowing before you design around them.

No embeddings

/v1/embeddings is not served and /api/embed answers 501 with a message saying so. If your pipeline needs embeddings, get them elsewhere.

One model per process

/v1/models and /api/tags report the single loaded model. To serve a second model, run a second tempo9 on another port.

Vision needs a Core ML tower

Image input works when the server is started with --tower <dir>, where dir is the directory containing tower_meta.json — the coreml/ subdirectory of an exported tower, not its parent:

--tower ~/models/qwen3vl_tower/coreml     # correct
--tower ~/models/qwen3vl_tower            # missingAsset: .../tower_meta.json
--tower ~/models/…/model-mmproj.gguf      # same error — a GGUF projector is not a tower

A GGUF mmproj is what a tower is exported from, offline; it is not a drop-in. A GGUF vision projector inside an Ollama model is detected but not loaded — the server prints a line saying so rather than leaving you to wonder why images do nothing.

No authentication

Nothing checks a key, and there is no flag to make it. This is defensible only because the listener binds 127.0.0.1 explicitly and never the wildcard, so the server is unreachable from other machines.

Do not put it behind a tunnel, a reverse proxy, or a port forward and leave it that way. That removes the only thing protecting it.

KV cache spills to disk, by default

Tempo9 keeps reusable KV prefixes in RAM and, when they overflow the RAM watermark, writes them to disk so a later session restores from SSD instead of re-computing the prompt. This is on by default.

What it does and does not buy you. Measured on a restarted engine with the same prompt, against the engine's own restored= counter and with the cache directory wiped as a control: with the cache preserved the first request after a restart reports restored=47 spans and the second request takes 0.61 s; with the directory wiped the same request reports restored=0 and takes 5.20 s. The restore is real and it is worth 8.5x — but the first request after a restart pays full prefill either way. An earlier version of this page implied that first request came back fast. It does not.

What that means for your disk:

• Cache files live in ~/Library/Caches/tempo9/kv (AS_KV_OFFLOAD_PATH moves it), capped at 16 GB (AS_KV_OFFLOAD_GB changes the cap; oldest entries are evicted first).
• The location is deliberate: macOS may purge it under disk pressure, and Time Machine does not back it up. Losing the cache costs a re-prefill, never data.
• The cache survives restarts — the on-disk index reloads at startup.
• A cache from a different model, dtype or KV layout is detected by a config hash and wiped rather than served back wrong.

To keep the disk untouched:

AS_KV_OFFLOAD=0 tempo9 --gguf model.gguf

An unwritable or full cache directory is not an error — the engine logs it and runs without offload.

Ollama's store is read, never written

Using --ollama opens Ollama's blobs in place. Nothing is copied, converted, moved, or written into their store. The one file Tempo9 creates is a symlink in its own cache — see Ollama models.