Compute vs memory: one MoE layer
64 experts, 8 fire per token. Routing pattern is random Illustrative
Pipeline of the study
Setup as reported: 12 prompts × 6 categories + 3 length prompts, 3 repeats = 45 runs per cell
Bandwidth ceiling: tokens/s ≤ BW ÷ bytes per token
Peaks are public specs (~68 GB/s Jetson, ~200 GB/s M2 Pro). Curves are arithmetic, not measurements.
Implied bandwidth use ÷ peak
Bytes × reported tok/s ÷ peak BW. Above 1.0 is impossible. Derived arithmetic. Hover a cell.
Hover a cell.
Reported results
Choose metric and device. Lighter bars are derived or bounded, not exact table values.
Prompt-length slowdown (short → long)
OLMoE only; dense models slowed by comparable amounts
Where does the router time go?
cb_eval times every graph node and forces syncs; M2 values are an upper bound.
MoE block: what the routing metric sees
Three explanations, none directly measured
Energy per token does not reconcile
Jetson only. Reported J/token vs mean power ÷ throughput (episode arithmetic).
Overlooked on-device MoE work
Hover or tap a node
Each targets the memory-versus-compute tension the paper measures.
What would settle it
References
- Does MoE Actually Help Inference on Consumer and Edge Hardware? Alfarizy et al., 2026 (arXiv 2606.21428)
- HOBBIT: mixed-precision expert offloading, Tang et al., 2024
- EdgeMoE: sparse LLMs on mobile devices, Yi et al., 2023
- SwapMoE: tunable memory budget MoE serving, Kong et al., 2023
- Pre-gated MoE, Hwang et al., 2024
- Mixtral, DeepSeek-V2-Lite, Qwen1.5-MoE, Qwen3-30B-A3B technical reports
- Not All Experts are Equal: expert pruning and skipping, Lu et al., 2024
- Roofline model, Williams, Waterman, Patterson, 2009
- Fiddler / MoE-Lightning / Klotski-style offloading systems, 2024-2025