On November 5, Baidu officially open-sourced its new base model, ERNIE 5.5 Flash Base. It set new SOTA results among models of the same size on multiple benchmarks spanning knowledge, reasoning, math, code, multilingual, and long-context tasks, outperforming comparable models such as DeepSeek-V4-Flash-Base and Hy3-preview-Base.
The model uses an MoE architecture, with 280 billion total parameters and 14 billion activated parameters. It was pretrained on 33.5 trillion tokens, with a pretraining context length of up to 128K, and supports a context length of up to 1 million tokens during fine-tuning and inference.
With efficiency as the main focus, ERNIE 5.5 Flash Base introduces several algorithmic innovations at the system level. In its attention mechanism, it introduces Latent-Gated Sparse Attention, combining HCA and MLA to compress historical information and provide fine-grained access to historical positions. It uses low-rank gated attention to reduce the number of gating projection weight parameters by 75%, while enhancing expressiveness through virtual head attention.
For expert routing, it adopts Multi-Router Latent MoE, using multiple independent routers to improve expert selection effectiveness. For the training objective, it introduces ECHO, which reuses the last layer of the backbone to learn to predict subsequent tokens and removes the need for an extra MTP module. On the large-model infrastructure side, it optimizes the distributed Muon optimizer and introduces the FlashMask-4 attention operator, further improving communication and memory utilization efficiency in large-scale training.
Competition among large models is now evolving toward coordinated progress in model capabilities, training efficiency, and use-case innovation. The open-sourcing of ERNIE 5.5 Flash Base lowers the barrier to specialized model development while providing developers with an efficient technical foundation for building customized specialized models.
