Tempo9
Server-style inference.
On a Mac you own.
English · 中文版
Apple Silicon is ready for serious local AI workloads. Tempo9 is the missing runtime layer: continuous batching, shared KV, paged KV that grows with use, SSD KV offload, multi-request scheduling, and GPU + Neural Engine execution on one Mac.
Read the manual See the measurements Install status
104.9 tok/s uses --speculation-k 4 on 200+ token
tool-call JSON; default settings measure 91.8 tok/s. The complete method is
published with the results; the raw repository will open with the first release.
Many sessions, one local model
Tempo9 currently serves multiple agents through one shared runtime. Continuous batching adds new requests to the batch already in flight. Paged KV allocates cache pages only for tokens that exist. Prefix caching computes and stores a shared system prompt or tool history once. Under memory pressure, cold KV pages can move directly between GPU-accessible unified memory and SSD.
Together, these mechanisms reduce both waiting time and duplicated memory. In a four-agent coding soak, all 80 turns completed. With either a 1.5K or 10K shared system prompt, every Tempo9 first-token latency range stayed below the comparison engine. The full ranges are shown below.
Four agents · lower is better
Every Tempo9 range ends before the matching llama.cpp range
Ranges across two to three runs per cell; twelve requests per run. Axis: seconds to first token.
Qwen3.5-35B-A3B q3km · 16-inch MacBook Pro, M5 Pro, 24 GB, macOS Tahoe 26.5, stock settings. Both engines read the same GGUF and use their default prefix caches. Each cell contains two to three runs of twelve requests. This section compares first-token latency only; concurrent decode throughput has not yet met the reproducibility bar for publication. Method and data →
Long context without preallocation
A declared context length should set a limit, not reserve the entire limit. Tempo9 uses Paged KV to allocate cache pages on demand, prefix caching to share common pages across sessions, and SSD offload to free unified memory when pages go cold.
In the measurements below, increasing the declared context from 8K to 128K leaves Tempo9's startup footprint at roughly 0.55 GB. Memory grows only as tokens become resident.
KV + working memory · lower is better
From 8K to 128K, startup memory stays nearly flat
Declared capacity at startup
Scale: 0–4 GB
Memory after tokens arrive
Scale: 0–2.2 GB
The panels use different scales. The left measures declared capacity before use; the right measures memory after tokens become resident.
Prefix caching removes repeated compute as well. In a 15K-token shared-prefix test, a later request reused 14,976 cached tokens and reduced prefill from 10.3 seconds to 0.48 seconds.
Qwen3.5-35B-A3B q3km · M5 Pro · 24 GB · stock settings. The chart reports KV plus working memory and excludes file-backed model weights; its two panels use different scales. Method and data →
A tool-call service path tested for accuracy
Tempo9 tests more than raw model output. The measured path starts at the compatible API, parses tool definitions, generates arguments, and returns the tool call to the client. All 3,641 BFCL v4 single-turn cases went through the same OpenAI-compatible HTTP endpoint an agent uses.
That endpoint has also been exercised with Codex, Claude Code, OpenClaw, and other agent frameworks that accept a local model endpoint. The agent stays; its inference backend changes.
Category breakdown
| BFCL v4 category | Tempo9 |
|---|---|
| simple (Python) | 95.25% |
| multiple | 94.50% |
| parallel | 89.00% |
| parallel multiple | 83.50% |
| Non-Live irrelevance | 84.17% |
Qwen3.5-35B-A3B q3km · M5 Pro · 24 GB. Official BFCL v4
evaluator. This page reports the completed single_turn groups; it
does not fold untested multi-turn, web-search, or memory cases into an overall
score.
Structured generation with XGrammar
Tempo9 can constrain decoding with a JSON Schema through XGrammar, keeping JSON and tool arguments structurally valid as tokens are generated. The grammar enforces form; BFCL separately tests whether the tool and its arguments are correct.
Structured output also benefits from speculation
On JSON and tool-call workloads, draft acceptance from the model's own MTP
head reaches 0.724. With --speculation-k 4, single-stream decode
rises from 91.8 to 104.9 tok/s. A paired set of 2,501 BFCL cases showed no
significant accuracy shift. Speculation remains off by default because byte-for-
byte output is not yet invariant across speculation depths.
Qwen3.5-35B-A3B q3km · M5 Pro · 24 GB. One stream, 200+ token tool-call JSON, decode throughput excluding TTFT. Method and data →
From LLM to VLM
VLM: vision and language in parallel
Moving from a text-only LLM to a vision-language model adds an image-encoding path. Tempo9 places the vision transformer (ViT) on the Neural Engine and the language model on the GPU. They run concurrently, so image encoding does not take the GPU away from text generation.
Vision and language compete for the same processor.
NPU encodes N+1 while the GPU keeps decoding N.
Text-only reference, without an image: 73.6 tok/s. The 2.8× comparison uses the same image and language-model workload; only ViT placement changes.
How it stays concurrent under load
The placement rows above are Tempo9 internal controls; only the ViT execution device changes. The same paging model also lets a larger vision-language model fit:
On a vision model both engines can run, answering the same question about the same image:
| Gemma 4 12B, one image | Tempo9 | llama.cpp |
|---|---|---|
| Encoding the image | 28 ms | 131 ms |
| Decoding the answer | 28.2 tok/s | 33.9 tok/s |
Qwen3-VL-30B-A3B and Gemma 4 12B · M5 Pro · 24 GB.
The 2.8× compares NPU and GPU ViT placement under the same image workload, not
the processors' isolated peak performance. The Gemma 4 rows are three runs
each, with both engines reading the same GGUF and mmproj.
Method and data →
Two ways to use Tempo9
Command-line server
Start a local service with a GGUF model, then point the client at Tempo9.
tempo9 --gguf <model.gguf> --port 11435
# OpenAI client base_url = http://127.0.0.1:11435/v1
# Anthropic client base_url = http://127.0.0.1:11435 (/v1/messages)
Endpoints: /v1/chat/completions ·
/v1/messages · /v1/responses · /v1/models.
It speaks OpenAI, Anthropic and Ollama-style APIs. Port 11435 sits next to
Ollama's 11434 so both can run side by side.
macOS SDK
The native macOS SDK embeds the runtime and model loader in an application. There is no separate local service to install or manage, and dependencies can be signed with the app for integrations designed around Mac App Store sandbox and review requirements.
A complete runtime for local agents and VLMs
Continuous batching, Paged KV, prefix caching, and SSD offload address the scheduling and memory demands of several long-context agents. Running the Neural Engine and GPU in parallel keeps a local VLM generating while it handles images. Together, these capabilities bring workloads that normally expect a server runtime onto one Mac.
Local operation follows from that architecture: model weights, context, and tool-call data are handled by the process on the machine, with no cloud account or telemetry service required.
Model architectures tested end to end
Nine architectures and sixteen GGUF model files are currently verified.
Architecture names follow the standard Transformers classes in each model's
config.json. The list is generated by loading and running models,
not by inferring support from code paths.
Text-only language models
| Architecture | Models | Tested examples |
|---|---|---|
Qwen3ForCausalLM | Qwen3 | Qwen3 0.6B |
LlamaForCausalLM | Llama | Llama 3.1 8B, Llama 3.2 1B, TinyLlama 1.1B |
MistralForCausalLM | Mistral, Ministral | Mistral 7B v0.3, Ministral 3B |
GptOssForCausalLM | GPT-OSS | GPT-OSS 20B (MXFP4) |
Multimodal language models
| Architecture | Models | Tested examples |
|---|---|---|
Qwen3_5ForConditionalGeneration | Qwen3.5, Qwen3.8 | Qwen3.5 0.8B / 4B / 9B, Qwen3.8 27B |
Qwen3_5MoeForConditionalGeneration | Qwen3.5 MoE | Qwen3.5 35B-A3B |
Qwen3VLForConditionalGeneration | Qwen3-VL | Qwen3-VL 4B |
Qwen3VLMoeForConditionalGeneration | Qwen3-VL MoE | Qwen3-VL 30B-A3B |
Gemma4UnifiedForConditionalGeneration | Gemma 4 Unified | Gemma 4 12B |
Every model listed ran on the same 24 GB M5 Pro. Each completed load and ordinary question answering, plus a 10K-token test where the model's context limit allows it. Representative models from the multimodal architectures also completed real-image inputs. The full matrix keeps untested and failed cells visible.
Want another model?
Open an issue. Model support is demand-driven: an architecture with someone asking for it gets built before one that merely exists. Tell us the model and the quantisation you want and it goes on the list.
Examples and developer documentation
Start with a working example, then use the reference for the integration you are building.
Examples
Developer documentation
The first public build is almost ready
The manual and measurements are public now. Signed binaries and the Homebrew tap are the remaining release work.