Mostly because Apple Silicon is built for weight-streaming (memory bandwidth), not massive block-parallel computation.
What is Parallel Diffusion in LLMs?
To generate text, autoregressive models project one token at a time, append it to the key-value cache, and run a forward pass of query length 1. It does this times.
Parallel Diffusion LLMs (like Google’s DiffusionGemma) start with a 256-token canvas of random noise and iteratively refine it in parallel over e.g., 48 steps.
What is the bottleneck swap?
In standard autoregressive (AR) generation:
- The bottleneck is memory bandwidth. Weight files (2GB for a 1B model, 16GB for a 8B model) have to be streamed from the system memory to the GPU cores for every single token. The GPU compute units spend most of their time idle, waiting on memory.
In parallel diffusion generation:
- The bottleneck is raw compute. Because we refine all 256 canvas tokens simultaneously at every single step, we do more FLOPs in the attention and feed-forward networks (FFN) per step. The bottleneck shifts from memory bandwidth to how fast your GPU can multiply matrices.
Why is it slow on a Mac?
I recently profiled this using a converted MiniCPM5-1B checkpoint on an M5 Pro chip:
- AR Step Latency (length 1):
6.5 msper token. - Diffusion Step Latency (length 256):
50.9 ms.
Apple Silicon is designed around massive unified memory bandwidth, which hides the memory-bound limits of autoregressive generation extremely well. However, when we swap to a compute-bound diffusion workload, the Mac’s relatively modest GPU core count is exposed
Even though we did fewer steps, the slower step speed means diffusion is actually slower than autoregressive generation on Mac. On server GPUs (like an NVIDIA H100), the latency difference between length 1 and length 256 is almost zero, so the step reduction translates into 3x to 4x speedup.
What to do then?
If you are running locally on a Mac: stick to standard autoregressive models for now