On a 256GB unified-memory M3 Ultra running a resident omlx inference service, TTFT, decode speed, long-context prefill, concurrency, and memory usage for two models — all measured numbers.
Running a large model on your own desk, the three questions you hear most are: How long until the first token? How fast is each token? How many people can it serve at once?
We set up a resident inference service with omlx on a Mac Studio (M3 Ultra / 256GB) and ran each of the two models separately. Everything below is measured data under the same methodology, not estimates from a benchmark utility.
Revision note (2026-09-29 17:10): The initial Qwen3.8 Flash Next data in this article came from a single probe; at the time DeepSeek was still resident in memory and did not have exclusive resources, so the numbers were distorted. We have now re-run the full test with "only one model in memory," and all data in this article has been replaced. The initial data is void.
0.0.0.0:8000, OpenAI-compatible endpoint /v1/chat/completions, model directory ~/.omlx/modelsA few server-side settings that directly affect the results:
max_concurrent_requests: 4max_context_window: 409600max_tokens: 20000chunked_prefill: disabledThe two models under test:
Deepseek-V4-Flash-oQ3e-mtp — oQ3e quantization, 24 shards, 123 GB on disk, measured resident memory 119.10 GBQwen3.8-Flash-Next-4bit — measured resident memory 104.07 GBFour metrics — first, let's be clear about what each one measures:
The server itself reports fields such as time_to_first_token, generation_tokens_per_second, and prompt_eval_duration; we take these values directly and verify them a second time on the client using timestamps.
Short prompt (about 30 characters), three rounds:
The slightly higher first-round TTFT (0.34 s) is a normal warm-up effect; it then settles at just over 0.1 s. At 0.1 seconds to first token, the user doesn't feel any "waiting" — this number is critical for voice interaction, because the voice chain still has recognition and synthesis stages, leaving little room for the large model.
Long-context prefill (8472-token input):
4-way concurrency (128 tokens per stream):
Under 4-way concurrency, per-stream speed drops from 30.5 tok/s to about 15.2 tok/s (roughly halved), but total throughput rises from 30.5 to 55.5 tok/s, nearly doubling.
Same platform, same methodology, with only this model in memory:
Short prompt (about 30 characters), three rounds:
The three decode-speed readings vary by only 0.2 tok/s — extremely stable.
Long-context prefill (8700-token input):
4-way concurrency (128 tokens per stream):
This one is key: under concurrency, per-stream speed drops only from 33.35 to about 25 tok/s (a decline of about 25%), while aggregate throughput actually rises to 89.2 tok/s, more than double a single stream. Concurrency efficiency is clearly better than DeepSeek's tier.
On the same machine with the same 4-way concurrency, a head-to-head comparison:
The conclusion is a bit counterintuitive: Qwen3.8 Flash Next, which feels "lighter" in parameter count, is better on every metric on this machine. Its 4bit quantization is friendlier to unified memory, and its prefill performance is especially strong.
If you only run one model, choose Qwen3.8 Flash Next first. DeepSeek V4 Flash's strengths lie in its context window (the server can open it up to the 400K-token level) and its positioning for long-text processing, suiting scenarios like "long documents, long sessions" rather than competing on throughput.
Let's be clear about one premise first: this 256GB machine can hold only one model at a time.
We tested this: with DeepSeek resident (the server process RSS once fluctuated between 88 and 126 GB), requesting Qwen3.8 Flash made the service return directly:
HTTP 507 Insufficient Storage
It's not a full disk — it's that unified memory can't be allocated. According to the server logs, the two models' resident usage is 119.10 GB and 104.07 GB respectively — the weights alone add up to 223 GB, and once you add the KV cache and the system's own overhead, 256GB leaves no room.
So: to switch models, the previous one must first leave memory. If your business requires "both models on standby at all times, using whichever the request needs," the 256GB tier isn't enough — you have to go higher.
The service has kept its own ledger from launch to now (as of 2026-09-29 17:10):
By model:
An 89.6% cache hit rate is the most valuable optimization on this machine. The same system prompt and the same conversation history don't need to be prefilled again the second time — the 17.8 s long-context time-to-first-token above shortens dramatically once the cache hits. For local inference, keeping the prompt prefix stable pays off more than swapping hardware.
The capability boundaries of this M3 Ultra in a sentence:
The implication for our voice box work is direct: with wake word, recognition, and synthesis all handled on-device and locally, it's feasible to keep one model locally for fallback conversation. There are two prerequisites: keep the context short and keep the prompt prefix stable. The cloud model handles complex semantics, while the local model handles "chatting even offline" — this division of labor holds up on the measured data.
Supplementary test-environment notes: The above are single-run results, without repeated trials to take extremes; the client and server are on the same LAN, and network latency was not separately isolated. Machine load, thermal state, and background processes all affect the numbers, so re-testing on another machine or another day may vary by 10% up or down.
All data cited in this article comes from omlx's self-reported timing fields (time_to_first_token, generation_tokens_per_second, prompt_eval_duration), the server logs, and cumulative statistics (stats.json), with no extrapolation or conversion.