The long-standing approach of word-by-word prediction in large model era has been challenged by the Shanghai Artificial Intelligence Lab and the School of Artificial Intelligence, Shanghai Jiao Tong University, with the LUMIA Lab team. They introduced a novel pre-training task called Next Concept Prediction (NCP) and launched the world's first base model with a discrete latent space of 8.9B scale, NCP-ArchPreview, which redefines the calculation of training efficiency.
These numbers are striking. The model was pre-trained on 5.73T of Dolma-3 data but only used 51.3% of the Token budget to match the final pre-training Loss of OLMo-3-7B, effectively improving the convergence speed by 1.95 times and systematically increasing computational Pareto efficiency by 1.74 times; in downstream tasks, it outperformed by 2.45 points, with a near 6-point improvement in GSM8K mathematical reasoning. In other words, while others need a full box of computing power to reach that level, this model gets there with just half a box of fuel.
The skeleton of NCP-ArchPreview is a three-stage latent space concept processing pipeline. It first builds an extremely large discrete concept representation space using product quantization, then explicitly models future concepts through differentiable next concept prediction, causal leakage prevention feedback mechanisms, and hierarchical residual routing—instead of guessing the next word blindly, it first clarifies what the next concept should look like. This shift to latent space concept prediction fundamentally breaks the traditional word-by-word prediction paradigm.
The surprises don't stop at pre-training. The team found that fine-tuning just 17M latent space parameters can surpass LoRA, without the problem of forgetting; the training throughput increased while memory usage decreased. After injecting this concept representation into a draft model, the average acceptance length during speculative decoding increased by 4.17%, and reached 7.59% in code tasks, opening a new path for inference acceleration. In their technical report, the team also generously shared their experiences and solutions, built a trillion-scale proxy metric screening mechanism, and open-sourced all assets including full-stage checkpoints and evaluation frameworks. For researchers looking for new breakthroughs in model fine-tuning and inference acceleration, this latent space idea is undoubtedly a freshly made roadmap.
