Today, as large AI models are rapidly evolving, the public generally believes that technological advancement is entirely driven by algorithm upgrades and massive automated training data. However, a newly exposed internal document has completely shattered this public perception. According to 404Media, an overseas media outlet, OpenAI has long had an internal project called "Project Lily," which involves human reviewers assessing real user chat records of ChatGPT.
In this work, employees known as "prompt reviewers" need to view anonymized samples of real user conversations to evaluate the quality of the model's responses. Their review focuses on whether the answers are relevant and checking for overly "AI-like" or condescending tones, emojis, or fawning language. Notably, the project's guidelines strictly prohibit AI from using any anthropomorphic expressions or fabricating "personal experiences." For example, it is allowed to say "I found some relevant information," but not to write "As a chef, I like..." or "I understand this feeling."
Although the work is relatively mechanical, the hourly wage is reportedly over $50 for tasks of low difficulty, attracting many people to participate.
However, the privacy risks hidden behind this mechanism have raised widespread concerns. Many users see ChatGPT as a temporary confidant or a source of emotional support, sharing their deepest secrets with it. Although OpenAI claims that chat records are anonymized and usernames are hidden, the company also admitted to the media that the filtering process may still miss some personal data, especially in short conversations, which are more likely to be at risk. More surprisingly, the conversation versions handed to reviewers often include a "user memory summary," which details the user's questions, interests, context, and even location information.
It should be clarified that "Project Lily" does not specifically check whether the model's answers are factually accurate, but only marks obvious errors. This means that effectiveness evaluation and safety reviews are usually handled by other independent teams. In addition, the majority of ordinary chatbots default to enabling the option "Allow use of your conversation records to improve the product." Although this setting is disabled by default for enterprise, business, and educational version customers, switching the toggle does not have retroactive effect. Previously stored records will continue to be retained, and even if users actively delete the conversation, the content may have already been collected and anonymized and stored in some dataset.
When asked by the media, OpenAI initially did not directly respond whether users were clearly informed that their chat records might be reviewed by humans, but provided a FAQ page mentioning that human reviews are conducted for model optimization purposes. After this in-depth report was published, OpenAI quietly updated the help page and added methods for users to opt out of data collection, but the revised document still avoided mentioning the fact that staff members directly read conversation content.
In fact, relying on human review to optimize models is not an exception in the tech industry. Google's Gemini explicitly states in its privacy center that some saved chat records may be reviewed by humans, and Anthropic holds a similar position and has a dedicated webpage to explain it. Perplexity's stance, however, is relatively vague, as its privacy policy neither confirms nor denies that staff can access chat logs. This also indicates that, on the path toward extreme intelligence in AI, human involvement remains an indispensable underlying support.
