As AI chatbots become deeply involved in human emotional lives, how models respond to users' psychological crises has become the most urgent ethical defense in the industry. Recently, the AI field has seen a major personnel change: Andrea Vallone, the former head of OpenAI's "Model Policy" research, has officially left and followed her former boss Jan Leike to competitor Anthropic.
Core Challenge: An Unprecedent "Emotional Quagmire"
During her time at OpenAI, Vallone led the team responsible for the deployment of GPT-4 and the next-generation reasoning model GPT-5. She faced a topic almost completely "vacant" in the global AI industry: When a model detects that a user shows excessive emotional dependence or even sends signals of suicide or self-harm, should the AI remain coldly indifferent or intervene?
Vallone once admitted that this research had almost no existing precedents. She not only participated in designing mainstream safety training methods such as "rule-based reward," but also tried to balance "usefulness" and "emotional safety boundaries" in the model's responses.
Industry Pain: The Shattered Safety Defense and Legal Storm
The flow of talent behind this is a collective anxiety about the safety of large models. In the past year, the AI field has witnessed multiple extreme negative events:
Extreme Tragedies: There have been multiple cases worldwide where teenagers and adults, after long-term "confiding" with AI, committed suicide or carried out violent crimes due to emotional manipulation or the collapse of the safety defenses in long conversations.
Legal Litigation: Several victims' families have filed lawsuits against relevant AI companies for negligence; the U.S. Senate held a special hearing to question the role and legal responsibilities of AI systems.
Surprising Data: According to a previous survey by OpenAI, tens of thousands of ChatGPT users show signs of mental health emergencies such as mania, psychotic symptoms, or suicidal tendencies every week.
Talent Gathering: Anthropic Strengthens Its "Safety Culture" Identity
After joining Anthropic's Alignment team, Vallone will directly report to Jan Leike. Leike was the lead of OpenAI's super alignment, and when he left in May 2024, he publicly criticized OpenAI's "safety culture has been replaced by shiny products."
