Large models are "set in stone" the moment they leave the factory. Feed them new knowledge, and it either disappears as soon as the conversation is over, or, if forced in, makes them forget everything they used to know. Google tried to resolve this dilemma on October 27 by unveiling a "Sleep Framework." The related paper is titled "Language Models Need Sleep," and it bluntly states: models should also consolidate memory through sleep, just like humans. This approach, released publicly by Google Research, targets the industry's chronic "knowledge cutoff" problem.
Work while awake, organize while asleep
The framework divides a model's operation into two phases. In the awake phase, it receives input as usual and first places the information into short-term memory; once it shifts into sleep, it pauses taking on tasks and focuses on memory consolidation and self-improvement. The key is "memory consolidation": with the help of parameter expansion, mixture-of-experts memory blocks, and knowledge distillation, fragile short-term context is written into long-term parameters without overwriting existing knowledge. There is also a "dreaming" phase, where the model generates and filters synthetic training samples on its own, practicing repeatedly through reinforcement learning to make reasoning and generalization smoother. Compared with traditional repeated fine-tuning, which causes "catastrophic forgetting," sleep-style consolidation is like installing an anti-forgetting gate in the model.
The ambition of zero forgetting
The experimental data is enticing: the framework beats most baselines on continual translation, long-context understanding, fact absorption, and mathematical reasoning; it can digest contexts up to about 10 million tokens, and in multi-stage continual learning on unseen languages, performance drops almost not at all; it also achieves higher accuracy on math benchmarks such as AIME 24/25. If it can really be put into practice, it means the "knowledge cutoff" may be broken—models would no longer become outdated the moment they are released, but would grow their abilities while being used, without forgetting their old strengths. For healthcare, finance, and legal scenarios that need to continuously absorb new information, this mechanism is clearly more cost-effective than periodic retraining.
