What actually runs on a 32 GB Mac mini — June 2026

• Mac mini M4 — 10-core CPU, 10-core GPU
• 32 GB unified memory — single pool, shared CPU/GPU
• ~120 GB/s bandwidth — the binding constraint at decode time

Three constraints, in order of how often they decide things:

1. Memory is the hard wall. Subtract ~6 GB for macOS, ~3–5 GB for the iPad sim and dev apps, and the largest LLM that comfortably co-resides is ~22 GB on disk (≈ 26 GB hot at 64 K context). Anything larger swaps to disk and throughput collapses.
2. Decode is bandwidth-bound. Once weights are in memory, tokens-per-second is set by how fast active parameters stream through the GPU. MoEs win because only the active experts move per token.
3. Prefill is compute-bound and quadratic in context. A 46 K-token document costs 9–11 minutes of prefill on every cold cache hit. Not tunable — that's physics on this rig.

Reasoning — 5-step word problem, ~180 input tokens

* cold-load TTFTs — model being swapped in. pure prefill on short prompts is 1–3 s.

Vision — LickBank app screenshot

Long-doc summarization — ~46 K-token Frankenstein excerpt

The MoE 26B-A4B at 46 K context decodes 2.3× faster than the dense 12 B and 4.6× faster than the dense 14 B. Its prefill (170 s) is also 3.2× faster than the 12 B's (546 s) — because Apple Silicon prefill scales with active params, not total.

Tool-call fidelity — 7-case suite (Hermes use case)

Fidelity doesn't separate the fitting models. Every MoE that fits in 32 GB hits 7/7. The differentiator is decode speed, and decode speed is bandwidth-bound. Qwen3-Coder-Next (MoE 80 B, 42 GB at 4-bit) doesn't fit at all — Metal OOMs on first prefill.

Quick reference — our seven installed presets

~37 GB on disk total (three Qwen presets share NVFP4 blobs at 21 GB; three Gemma presets share theirs at 16 GB):

The key insight

Klein 9B's pre-quantized MLX-Q4 build is smaller on disk than Klein 4B's FP16 (8.9 GB vs 22 GB) but produces visibly sharper output. The size flip is because Klein 4B ships at full precision and quantizes at runtime via -q 4, while Klein 9B was pre-quantized by the community uploader. Do not pass -q 4 to Klein 9B — re-quantizing already-quantized weights degrades them measurably.

Prompting note

Klein is trained on natural-language descriptive prose, not tag soup. Push toward photo: camera + lens ("shot on Sony A7R V, 85mm f/1.4"), light ("golden hour", "overcast diffused"), texture ("skin pores visible", "fabric weave", "film grain"), genre ("candid photograph", "documentary"). Avoid CGI tells: "8k", "masterpiece", "highly detailed", "cinematic", "trending on artstation". FLUX.2 has no negative-prompt support — express exclusions in the positive prompt.

What we tested

The architecture lesson

This was the surprise of the whole project. We assumed an audio LLM would win — they're the new shiny — but a 2.97 GB specialist transformer trained on the actual task crushed both audio-native LLMs and image-vision LLMs operating on spectrograms. SongFormer's deterministic output (identical labels on CPU and MPS) is also operationally easier than any LLM call.

1 · Decode is bandwidth-bound, not compute-bound

On a 3 B-active MoE at 4-bit, generating one token requires reading roughly 1.5 GB of weights through memory. Apple's M4 mini has ~120 GB/s bandwidth. The arithmetic ceiling is 120 / 1.5 ≈ 80 tok/s; runtime overhead and KV-cache reads halve that to ~40 tok/s. Every 3 B-active MoE we tested clusters at 34–42 tok/s — Qwen, Nemotron, Gemma, GLM, all land there.

2 · MoE pays off twice — at decode and at long-context prefill

Decode reads only the active experts (~3–4 B params) regardless of total size. Prefill on Apple Silicon also scales with active-param count, not total. That's why the 26B-A4B QAT does long-doc summarization 3× faster than the dense 12 B Gemma — same prefill cost as a 4 B model, judged quality of a 26 B.

3 · Speculative decoding doesn't help compute-light models

Speculative decoding gives 1.4–1.7× on dense models — we measured 1.71× on Qwen3.6-27B-dense via MTPLX on this exact box. But its payoff scales with per-token compute, and a 3 B-active MoE is already compute-light. Grafted MTP on Qwen3.6-35B-A3B at depth 1 / 2 / 3 gave 0.94× / 0.73× / 0.58× — speculative makes the MoE slower. Confirmed by mlx-lm PR #990's author independently.

4 · Lower-bit quants are not free throughput

Tried Unsloth's UD-3-bit Qwen3.6-35B-A3B expecting a decode win. Got 34.7 tok/s vs uniform 4-bit's 41.5. The "3-bit" UD quant is mixed-precision: 270 layers at 8-bit, only 117 at 3-bit. In byte terms it's heavier than uniform 4-bit, and mixed-precision dequant kernels add overhead. MLX's 4-bit kernel is also far more optimized than its 3-bit path.

5 · The think toggle is the only 5× knob

Of all the sampling/runtime tweaks tested, only one moves wall-clock by a factor of five. Disabling thinking on Qwen3.6 cuts a 293 s response to 55 s with no decode-rate change — the model just writes 1,900 tokens instead of 9,500. Everything else (num_ctx, num_predict, temperature, sampling) is in the 1.0–1.5× range.

6 · The 32 GB wall

Even with NVFP4 quantization, the largest model that comfortably fits with a usable context window and KV cache is ~26 GB resident. Qwen3-Coder-Next at 42 GB is a hard out. Mixing two large models hot (e.g. gemma-hermes + Klein 9B image gen) requires explicit eviction. This isn't tunable — it's the hardware ceiling. A 64 GB box would unlock 80 B-class models; a 32 GB box won't.

• Sample size: N=1 per axis on a single prompt for most benches. Decode tok/s numbers are physics — they're reliable. Quality ratings (5/5/5 etc.) are directional, scored by Claude as judge against a rubric.
• TTFT contamination: "Cold prefill" numbers when the model is being swapped in include load time, not just prefill. Pure prefill cost on short prompts is 1–3 s for any of these models.
• Quant comparison isn't apples-to-apples: Gemma's NVFP4 vs Qwen's uniform 4-bit are different formats. Some Gemma 4 wins come from NVFP4 being newer and better-tuned for Apple Silicon than the older 4-bit MLX kernel.
• Same-family judge bias: Claude judging models has a possible bias toward Claude-flavored output. The reasoning axis has a ground-truth answer (10:22 AM, 137 mi on the train problem) so quality scores there are grounded; doc summary and vision scores rely on Claude's read.
• Tooling churn: rapid-mlx, mlx-vlm, mlx-metal, and Ollama-MLX all moved meaningfully between Apr and Jun 2026. Numbers tagged to specific versions. Re-run before relying on a six-month-old measurement.

Bench scripts (for reproducibility)

Every script that produced a number on this page is checked into ./bench/. Walk-through and run instructions in the bench/README.

Raw data + reports

Raw run JSON and the markdown reports the post draws from are in bench/results/:

• REPORT.md — original 3-axis run, Qwen3-14B vs Qwen3-VL-8B (2026-06-03)
• REPORT_v2_gemma4_vs_qwen3.md — Gemma 4 12 B vs Qwen3 head-to-head (2026-06-04)
• REPORT_v3_throughput_hermes.md — qwen-hermes / gemma-hermes throughput (2026-06-04)
• REPORT_v4_gemma4_26b_a4b_qat.md — Gemma 4 26B-A4B QAT NVFP4 across all axes (2026-06-06)

Each *.json in the same directory has prompt, comp_tok, ttft_s, total_s, decode_tps, and the full streamed content — so any score can be re-judged with a different rubric or model.