Inside Edge0-35B: Running a 35B Sparse MoE in Under 3 GiB RAM via SSD Offload and Prerouter Speculation

You cannot fit a 35B model into 8 GB of RAM. The math is simple. Even when quantized to 4-bit integers, 35 billion parameters consume roughly 20 GB of storage. Once you load attention projections, KV cache, and runtime activations, an ordinary machine runs out of memory immediately.

Most engineers accept this constraint. They buy larger cloud instances or run smaller 3B models locally.

The team at Edge0 took a different route with Edge0-35B-A3B-preview. They designed an inference pipeline that runs a 35B-class sparse Mixture-of-Experts (MoE) model in under 3 GiB of active RAM. On a consumer Mac mini M4 Pro with 24 GB of unified memory, the model generates 15 to 18 tokens per second.

1+-------------------------------------------------------------------------+
2|                       Edge0-35B-A3B Resource Profile                    |
3+--------------------------+----------------------------------------------+
4| Parameter Scale          | 35 Billion Total (Active: ~3 Billion)        |
5| Checkpoint Disk Footprint| ~23 GB (Safetensors)                         |
6| Peak Active Memory       | 2.9 GiB                                      |
7| Sustained Decode Speed   | 14.9 - 17.7 tokens/sec (Apple M4 Pro)        |
8| Average Quality Retention| 95.2% of FP16 Teacher (OpenCompass Suite)    |
9+--------------------------+----------------------------------------------+

Achieving this throughput required solving the primary bottleneck of SSD offloading: PCIe bus read latency. Here is a technical breakdown of how the architecture works, what makes the speculative prerouter fast, and where the system still encounters bottlenecks.

The Physical Barrier of Naive SSD Offloading

Sparse MoE models activate only a fraction of their parameters per token. In Qwen/Qwen3.5-MoE-35B-A3B, the model contains 40 layers with 256 routed experts each. For any single token, the gating network routes execution through only 4 experts ($K=4$).

This sparsity creates an obvious idea. Keep the inactive experts on an NVMe SSD and stream only the 4 active experts into RAM as needed.

Under naive offloading, this strategy fails completely. Consider the execution timeline of standard decoder layers:

Reading 4 expert weights from flash storage over PCIe requires between 8 and 15 milliseconds per layer. When multiplied across 40 layers, a single token takes more than 400 milliseconds just for disk I/O.

Your decode throughput plummets to 2.5 tokens per second. The GPU compute cores sit idle while waiting for disk reads to complete.

To make local offloading viable, you must hide storage latency behind GPU arithmetic.

The Double-Shift Prerouter: Speculative Routing

Edge0 solves this pipeline stall using an auxiliary neural network called the prerouter. The core insight is straightforward. If you know which experts Layer $N+1$ needs before Layer $N$ finishes execution, you can stream those weights in the background.

1Naive Pipeline:
2[ Compute Layer N ] >> [ Read Disk (Wait) ] >> [ Compute Layer N+1 ]
3
4Edge0 Pipeline:
5[ Compute Layer N (GPU) ]
6       || (Concurrent Asynchronous DMA Transfer)
7[ Stream Layer N+1 Experts from NVMe into GPU Memory ]
8       vv
9[ Compute Layer N+1 (Zero Wait) ]

The Double-Shift Mechanism

The prerouter uses two coordinate shifts to predict expert activations:

1. Layer Shift (Layer N >> Layer N+1): An auxiliary prediction head owned by Layer N consumes the post-attention layer norm representation. It predicts the routing indices for Layer N+1.
2. Token Shift (Token t-1 >> Token t): Layer N+1 consumes the routing decision that Layer N's head predicted during the previous token step (t-1).

Because the prediction runs an entire token step ahead, the system has sufficient time to stream weights over PCIe without pausing execution.

Head Architecture and Zero-Drop Guarantee

The prerouter consists of 33 dedicated neural heads attached to layers 6 through 38. The earliest layers (0 to 5) retain dense representations or standard dispatch, while the deep layers rely on speculation.

Each head is a compact two-layer multi-layer perceptron (MLP):

The input feature vector $x_{\text{feat}}$ concatenates three distinct signals:

• The hidden activation state ($d_{\text{model}} = 2048$).
• A one-hot indicator of the current token's top-4 routed experts.
• A one-hot indicator of the previous token's top-4 routed experts.

The head operates in fp16 precision with an intermediate dimension of 512. Evaluating the head adds less than 0.12 milliseconds of compute overhead per layer.

Crucially, the decode engine configures staged_replace=False. The MoE layer directly executes the expert indices provided by the prerouter. There is no runtime rollback or speculative discarding. The staged experts and the executed experts are identical by construction.

[!NOTE] In testing, speculative routing prediction accuracy exceeds 94% on standard conversational text. Because the expert selection uses Softmax over 256 candidates, mispredicted experts in the 4th slot contribute minimal activation mass to the residual stream.

The Streaming Subsystem: MMAP, LRU, and Fixed Slots

Predicting routing indices is only half the battle. You also need an I/O subsystem capable of moving weight slices into Apple Silicon unified memory without kernel interruptions.

The edge0 runtime achieves this using three interconnected components:

Expand

1. Zero-Copy File Mapping (mmap.py)

Standard PyTorch checkpoint loaders read entire tensors into memory or invoke serialized OS reads. Edge0 uses SafetensorsMmap.

The framework memory-maps the 23 GB checkpoint file directly into user virtual address space. When an expert is selected, the runtime queries the exact byte offsets from the safetensors metadata header. It reads the raw packed 4-bit bytes directly into unified memory buffers.

2. Cross-Layer Shared Cache (cache.py)

Token generation exhibits temporal locality. An expert that activates while generating word $A$ frequently activates again when generating word $B$.

The SharedExpertCache implements a cross-layer Least Recently Used (LRU) buffer. When the prerouter requests an expert, the engine first checks the cache. If the expert resides in unified memory, the disk read is skipped entirely.

3. Fixed-Slot Decoding (layer.py)

Frequent memory allocation triggers Garbage Collection pauses and GPU synchronization stalls.

Edge0 allocates 4 fixed GPU tensor slots per layer, plus 1 overflow zero slot (staged_n=4). Incoming expert weights write directly into these pre-allocated memory addresses.

The routing layer maps indices using mx.take on the internal slot table. Memory indices never transfer back to CPU host memory. The entire forward pass remains on the Apple Silicon Metal GPU.

1+-------------------------------------------------------------------------+
2|                  Storage and Memory Access Pipeline                     |
3+-------------------------------------------------------------------------+
4| [23 GB NVMe Checkpoint]                                                 |
5|         |                                                               |
6|         v (SafetensorsMmap byte-range slice)                            |
7| [SharedExpertCache LRU Buffer]                                          |
8|         |                                                               |
9|         v (mx.take on GPU slot table)                                   |
10| [Fixed Slots: Slot 0 | Slot 1 | Slot 2 | Slot 3 | Overflow]             |
11|         |                                                               |
12|         v (backends.quant.gather_qmm)                                   |
13| [SwiGLU Output: bf16 Activation Tensor]                                 |
14+-------------------------------------------------------------------------+

Recover-LoRA: Distillation Against Quantization Loss

Shrinking a 35B model down to 4-bit representation creates substantial quantization noise. Standard 4-bit post-training quantization (PTQ) frequently causes MoE networks to output repetitive loops or syntax errors in code generation.

To solve this, Edge0 applies Recover-LoRA:

1. The 35B base model weights are frozen in 4-bit affine format (group size 64).
2. Low-Rank Adaptation (LoRA) modules ($r=16, \alpha=32.0$) are injected into all attention and projection layers.
3. The LoRA parameters are trained via knowledge distillation using the original unquantized Qwen3.5-MoE-35B-A3B (FP16) as the teacher model.

The adapters ship as a separate 120 MB file (lora_edge0_35b.safetensors). At runtime, the read-only 4-bit base loads once, while different LoRA adapters can be swapped on the fly.

Empirical Quality Retention Across OpenCompass

Edge0 evaluated the int4 quantized checkpoint against the full FP16 teacher using the OpenCompass benchmark suite. Both models executed with identical system prompts and greedy decoding settings.

The results show minimal performance loss. Across all five major benchmarks, the distilled 4-bit model stays within 3.9 points of the FP16 model.

High-level reasoning on GPQA-Diamond showed the strongest stability, retaining 97.6% of original score. Competition math (AIME 2026) suffered the largest degradation (-6.1 points), illustrating that complex numerical chains remain sensitive to 4-bit quantization rounding.

Hardware Benchmarks: Apple Mac mini M4 Pro

I tested the official preview release on an Apple Mac mini equipped with the M4 Pro chip (14 CPU cores, 20 GPU cores, and 24 GB unified memory).

The test runner executed examples/bench.py across varying prompt lengths and batch configurations.

1Benchmarking Edge0-35B-A3B on Apple M4 Pro (24 GB Unified Memory)
2OS: macOS Sequoia 15.3 | MLX: 0.30.6 | MLX-Metal: 0.30.6
3----------------------------------------------------------------------
4Cold Prefill (512 prompt tokens):       113.4 tokens/sec
5Warm Prefill (Cache hit):                140.2 tokens/sec
6Sustained Decode (Short context):        16.8 tokens/sec
7Peak Working RAM (Excluding OS):         2.88 GiB

Throughput and Memory Comparison

How does Edge0 compare against standard local deployment methods?

The prerouter improves decode throughput from 9.4 tokens per second to 16.8 tokens per second. That represents a 78.7% throughput increase over synchronous offloading.

More importantly, it accomplishes this while keeping active memory under 3 GiB. The machine retains 21 GB of free unified memory for browser tabs, code editors, and development compilers.

Architectural Deep Dive: MoE Tensor Math and Packing

To understand how Edge0 coordinates quantized execution inside Apple Silicon Metal kernels, we must examine the tensor packing format.

The MoE feed-forward block implements the SwiGLU activation:

The repository stores expert projections as separate stacked tensors (WeightLayout.SEPARATE). The projection bundle order strictly follows:

Packing order matters. If you swap gate_proj and up_proj during dequantization, relative $L_2$ error jumps from 0.24% to 100%.

1# Quantized Gather Matmul Execution Path (mlx backend)
2# switch_mlp.<proj>.weight is packed as uint32 tensors: [E, out_dim, in_dim / 8]
3import mlx.core as mx
4from edge0.backends.mlx._impl.qwen3_5_moe import gather_qmm
5
6def execute_streamed_expert(x, expert_indices, slot_weights):
7    # expert_indices: [batch, top_k] where top_k=4
8    # slot_weights: pre-allocated fixed buffer in unified memory
9    up_out = gather_qmm(x, slot_weights.up, expert_indices)
10    gate_out = gather_qmm(x, slot_weights.gate, expert_indices)
11    
12    # SwiGLU activation in bf16 precision
13    activated = up_out * mx.sigmoid(gate_out) * gate_out
14    
15    down_out = gather_qmm(activated, slot_weights.down, expert_indices)
16    return down_out

The quantized weights are packed into uint32 representations where each integer holds eight 4-bit values. Scales and bias offsets are stored per group of 64 in bfloat16.

During forward propagation, the custom Metal kernel gather_qmm reads the packed integers. It dequantizes them into registers on the fly and multiplies against the input vector without writing intermediate FP16 weights back to DRAM.

Practical Deployment and Serving

Deploying Edge0-35B requires setting up the MLX environment and downloading the checkpoint from Hugging Face.

Installation and Dependencies

The MLX backend requires macOS with Apple Silicon. A critical detail involves package pinning:

1# Install edge0 from source with fetch dependencies
2pip install -e 'git+https://github.com/Edge0-AI/edge0.git#egg=edge0[fetch]'
3
4# CRITICAL: Pins required to avoid A18/M4 matmul kernel corruption
5pip install 'mlx==0.30.6' 'mlx-metal==0.30.6' 'mlx-lm==0.31.0'

[!WARNING] Do not use mlx versions older than 0.30.6. Older releases mis-gate NAX matmul kernels on Apple A18 and M4 series hardware. This causes corrupted, mixed-language text output during inference.

Downloading and Running the CLI

Download the weights and run an interactive chat session:

1# Download 4-bit checkpoint and adapters (23 GB total)
2huggingface-cli download Edge0/Edge0-35b-a3b-preview --local-dir ./Edge0-35b-a3b-preview
3
4# Set model path and launch terminal chat
5export EDGE0_35B_MODEL=$PWD/Edge0-35b-a3b-preview
6edge0 chat --name edge0-35b --prompt "Explain the double-shift prerouter architecture"

Hosting an OpenAI-Compatible API Server

To integrate with existing agent workflows or UI frontends, Edge0 includes a native HTTP serving daemon:

1# Serve on port 8085 with SSE streaming enabled
2edge0 serve $PWD/Edge0-35b-a3b-preview --host 0.0.0.0 --port 8085

The server exposes standard endpoints:

• POST /v1/chat/completions: Full streaming chat completions.
• GET /v1/models: Returns model identity and supported context lengths.

You can query the server directly using standard OpenAI SDK clients or cURL:

1curl -X POST http://localhost:8085/v1/chat/completions \
2  -H "Content-Type: application/json" \
3  -d '{
4    "model": "edge0-35b",
5    "messages": [{"role": "user", "content": "How does Recover-LoRA work?"}],
6    "temperature": 0.7,
7    "stream": true
8  }'

KV Cache Dynamics and Real-World Limitations

While Edge0 achieves impressive performance, running a 35B model under 3 GiB of active memory involves engineering trade-offs.

1. KV Cache Memory Growth

Streaming expert offload keeps parameter memory fixed at 2.9 GiB. However, the attention key-value (KV) cache grows linearly with context length.

1+-------------------------------------------------------------------------+
2|                  Context Length vs Memory Consumption                   |
3+-------------------+--------------------+----------------+---------------+
4| Context Tokens    | Model Weights (RAM)| KV Cache (FP16)| Total Unified |
5+-------------------+--------------------+----------------+---------------+
6| 1,024 tokens      | 2.90 GiB           | ~0.15 GiB      | 3.05 GiB      |
7| 4,096 tokens      | 2.90 GiB           | ~0.60 GiB      | 3.50 GiB      |
8| 16,384 tokens     | 2.90 GiB           | ~2.40 GiB      | 5.30 GiB      |
9| 32,768 tokens     | 2.90 GiB           | ~4.80 GiB      | 7.70 GiB      |
10| 65,536 tokens     | 2.90 GiB           | ~9.60 GiB      | 12.50 GiB     |
11+-------------------+--------------------+----------------+---------------+

For short conversations and interactive tasks, memory remains minimal. But if you process 64k context documents on an 8 GB MacBook Air, the KV cache alone will trigger system memory pressure.

Future iterations will need FP8 or 4-bit KV cache quantization to maintain the 3 GiB ceiling across long horizons.

2. Preview Release Limitations

The current checkpoint is designated as a preview release. Several functional limitations exist:

• Weak Agentic Reasoning: The preview model is not yet fine-tuned for structured tool calling, JSON schema validation, or long-horizon autonomous planning.
• Platform Scope: The runtime backend exclusively supports Apple Silicon via MLX. While backend interfaces are decoupled in edge0/backends/base.py, CUDA and Linux support remains on the development roadmap.
• SSD Write Wear: Streaming does not write to disk, but constant random reads over prolonged continuous serving will place sustained read cycles on internal flash controllers.

Engineering Takeaways

The Edge0 release demonstrates that model parameter counts no longer dictate memory ceilings.

By rethinking the boundary between storage and unified memory, sparse architectures can decouple compute from parameter residency. Naive offloading previously failed because engineers treated the disk as slow RAM. Edge0 treats the disk as an asynchronous stream, using speculative neural heads to bridge the latency gap.

For developers building local software, this architecture shifts the economics of open-source models:

• Consumer Hardware as AI Workstations: A $600 Mac mini can run a 35B reasoning model without memory starvation.
• Composable Adapters: Storing an unmerged 4-bit base allows engineers to serve diverse task-specific LoRA adapters simultaneously without duplicating weights.
• Speculative Systems: Using small neural heads to predict system-level events (like memory fetches and routing) will likely expand beyond MoEs into distributed serving and heterogeneous compute.

If you are exploring local LLM serving, test Edge0. It proves that with clever systems engineering, high-parameter intelligence can thrive in constrained hardware footprints.

References

• Edge0-35B-A3B-preview Model Card on Hugging Face
• Edge0 GitHub Repository & Source Code
• Qwen3.5-MoE Base Architecture Specifications
• OpenCompass LLM Evaluation Platform
• Inside Qwen3.8-27B: Linear Attention and Serving Mechanics
• A Builder's Guide to LLM Quantization Types
• Fine-Tuning Local Models with QLoRA
• Self-Hosting LLMs with vLLM, SGLang, and llama.cpp