Two ChatGPT users in California have lodged a proposed class-action lawsuit against OpenAI in the Northern District of California, accusing the company of routing actual user conversations to human contractors without clear notification.
Background of the lawsuit
The filing, served on September 2, alleges that OpenAI’s practice breaches the state’s Unfair Competition Law, the California Consumer Privacy Act, and common-law privacy protections. The plaintiffs contend that the company’s privacy policy does not explicitly list data-annotation or evaluation vendors as recipients of user content, leaving users unaware that a person—not just an algorithm—might see their messages. The complaint lists eight legal claims and seeks damages, restitution, punitive relief, and injunctive orders.
The role of Project Lily
According to the complaint, the disputed practice is part of an internal program dubbed “Project Lily.” Contractors hired through third-party staffing firms act as AI data reviewers or chatbot evaluators. Their tasks involve reading full user prompts, summarising the intent, and rating four candidate model responses on a scale of one to seven. This human feedback loop, known in the industry as reinforcement learning from human feedback (RLHF), is intended to improve model behavior, particularly reducing overly agreeable or overly human-like outputs.
The reviewers operate from a dashboard that includes a “user memories summary,” which aggregates prior interactions and can expose details such as location, occupation, or personal circumstances, even though usernames are removed. The plaintiffs argue that OpenAI’s automated filtering does not always prevent sensitive information from reaching reviewers, potentially exposing private data.
OpenAI’s response and potential remedies
OpenAI has been given until October 13 to file a formal answer. The plaintiffs request several specific changes: the default "Improve the model for everyone" toggle should be turned off, an opt-in mechanism should be required before any conversation is sent to a human reviewer, a clear warning should appear inside the chat interface, and any data derived from reviewed chats should be deleted and excluded from future model training. If the court grants the injunction, OpenAI could be compelled to adjust its privacy disclosures and data-handling practices.
Why it matters
The case highlights a growing tension between AI advancement and user privacy. As large language models rely on human-in-the-loop feedback to refine performance, the extent to which personal conversations are exposed to contractors raises legal and ethical questions. A ruling could set precedent for how AI companies disclose human-review processes and obtain consent, potentially influencing industry standards worldwide. Moreover, the outcome may affect user trust in conversational AI tools that many treat as personal assistants, therapists, or private search engines.
