Does MoE Really Speed Up Inference on Edge Hardware?

Alfarizy, Nguyen, Richard, Razavi-Far, Cao · Univ. of New Brunswick · 2026
OLMoE-1B-7B vs three dense models · Apple M2 Pro & Jetson Orin Nano 8GB · llama.cpp Q4_K_M
arXiv 2606.21428 Interactive page

Compute vs memory: one MoE layer

64 experts, 8 fire per token. Routing pattern is random Illustrative

Pipeline of the study

Setup as reported: 12 prompts × 6 categories + 3 length prompts, 3 repeats = 45 runs per cell

Bandwidth ceiling: tokens/s ≤ BW ÷ bytes per token

Peaks are public specs (~68 GB/s Jetson, ~200 GB/s M2 Pro). Curves are arithmetic, not measurements.

Implied bandwidth use ÷ peak

Bytes × reported tok/s ÷ peak BW. Above 1.0 is impossible. Derived arithmetic. Hover a cell.
Hover a cell.

Reported results

Choose metric and device. Lighter bars are derived or bounded, not exact table values.

Prompt-length slowdown (short → long)

OLMoE only; dense models slowed by comparable amounts

Where does the router time go?

cb_eval times every graph node and forces syncs; M2 values are an upper bound.

MoE block: what the routing metric sees

Three explanations, none directly measured

Energy per token does not reconcile

Jetson only. Reported J/token vs mean power ÷ throughput (episode arithmetic).

Overlooked on-device MoE work

Hover or tap a node
Each targets the memory-versus-compute tension the paper measures.

What would settle it

References

  1. Does MoE Actually Help Inference on Consumer and Edge Hardware? Alfarizy et al., 2026 (arXiv 2606.21428)
  2. HOBBIT: mixed-precision expert offloading, Tang et al., 2024
  3. EdgeMoE: sparse LLMs on mobile devices, Yi et al., 2023
  4. SwapMoE: tunable memory budget MoE serving, Kong et al., 2023
  5. Pre-gated MoE, Hwang et al., 2024
  6. Mixtral, DeepSeek-V2-Lite, Qwen1.5-MoE, Qwen3-30B-A3B technical reports
  7. Not All Experts are Equal: expert pruning and skipping, Lu et al., 2024
  8. Roofline model, Williams, Waterman, Patterson, 2009
  9. Fiddler / MoE-Lightning / Klotski-style offloading systems, 2024-2025