Artificial intelligence has reached a milestone breakthrough in on-device deployment. The open-source project Edge0 has recently officially released and open-sourced the Edge0-145B and Edge0-120B models, along with a unified inference framework and deployment tools. As the world's first open-source 145-billion-parameter model claimed to run smoothly locally on personal computers, Edge0-145B successfully brings programming, reasoning, and agent capabilities comparable to cutting-edge cloud models directly to users' local laptops.
Breaking Hardware Limits: How Does It Run a 145B Model on a Laptop?
For a long time, on-device AI has struggled to balance model capability and hardware feasibility: large models have many parameters and extremely high memory and bandwidth requirements; small models, although compact, are hard-pressed to handle complex professional tasks. Offloading heavy work to the cloud inevitably brings continuous API costs, data privacy restrictions, and a strong dependence on network connectivity.
To address this industry pain point, the Edge0 team carried out comprehensive innovation across four levels: underlying architecture, training, loading, and scheduling:
Sparse Mixture-of-Experts (MoE) architecture: The model cleverly shares large matrices among a massive number of sparse experts, greatly reducing the weights that need to be stored and moved while keeping the total number of experts unchanged. Out of the 145B total parameters, only about 10B participate in each inference computation.
On-device collaborative training from scratch: The model incorporates the hard constraints of on-device deployment into its design from the early stages of training, avoiding the performance loss caused by post-hoc compression or pruning of traditional cloud models.
Three-tier storage intelligent pipeline: Through a clever design, GPU memory, system memory, and solid-state drive (SSD) access form an efficient pipeline. High-frequency "hot experts" reside permanently in GPU memory, an approximately 40GB expert library is placed in system memory and loaded on demand, and approximately 48GB of vocabulary data is stored on the SSD for real-time reading, avoiding serial waiting.
Efficient CPU-GPU collaboration: The GPU is responsible for computing cached experts, while the CPU processes the remaining work in parallel, and speculative decoding is used to spread the cost of weight reading across more outputs.
Through the above innovations, Edge0-145B achieves impressive real-world performance on compatible personal computers (such as MacBook Pro with Apple M-series chips, or Windows platforms equipped with RTX 4090/5090, etc.): peak memory usage is only 29GB, prompt preprocessing speed exceeds 1,100 tokens/s, decoding generation speed exceeds 80 tokens/s, typical power consumption is controlled at around 120W, and it supports ultra-long contexts from 128K to 256K.
Programming and Reasoning Capabilities Rival the Cloud, Effortlessly Handling Complex Local Workflows
