Shanghai AI Lab and Shanghai Jiao Tong University Propose a New Pre-training Paradigm for NCP, 8.9B Hidden Space Model Achieves Equal Performance with Half the Tokens

Shanghai AI Lab and SJTU LUMIA Lab propose Next Concept Prediction (NCP) and release NCP-ArchPreview, the first 8.9B discrete latent space base model. Using only 51.3% token budget on 5.73T corpus, it matches OLMo-3-7B final pretraining loss, greatly boosting efficiency.....