OpenAI's training opt-out has a reasoning-token-shaped hole
Two policy stories in one afternoon: a researcher finds the wording of OpenAI's consumer opt-out may not cover what the model thinks, and California keeps building its AI oversight apparatus — this time the people who will do the auditing.
The guarantee ChatGPT's paid users opt into may stop short of the hidden reasoning that contains everything they typed. Shashwat Goel, a researcher who read OpenAI's consumer terms line by line, found that the opt-out protects "Input" and "Output" — and since customers never receive a model's hidden chain-of-thought tokens, it's unclear those tokens count as "Output" at all. The catch is that hidden reasoning is a lightly processed copy of the useful part of an interaction: your prompt, the model's work, the shape of the answer. If the opt-out only covers what comes back over the wire, OpenAI could post-process that reasoning automatically — no human ever reading it — and feed the result into a training pipeline without breaking the letter of the guarantee. The contrast he draws is sharper than the loophole: OpenAI's own enterprise agreement and Anthropic's consumer terms never require that you receive the Output for the opt-out to apply.
Goel says he emailed OpenAI's data policy contact two days ago and got no clarification, and he keeps the claim careful — this is an ambiguity in the terms, not proof that training on hidden reasoning happens. But a paid guarantee you have to reverse-engineer out of a definitions section isn't much of a guarantee; it's a negotiation. For enterprises whose lawyers read terms the way Goel does, the difference between "we don't train on your data" and "we don't train on the exact bytes you sent" is the whole contract.
California's next round of AI laws is about who checks the checkers. Governor Gavin Newsom signed two bills this week creating the nation's first framework for independent AI audits: SB 813 establishes independent verification organizations that can assess AI systems and models, and AB 1405 sets up a state registry of auditors with standards for their independence, transparency, and integrity. It's the difference between a lab grading its own homework and an auditor it doesn't pick — we covered the state's first mandate that chatbots alert parents when teens signal self-harm just yesterday, and the auditor framework extends the same logic from products to the models behind them. The open question is the one every audit regime eventually faces: whether the registry produces referees or a licensed class of rubber stamps.
What to watch: whether OpenAI clarifies the opt-out language — silence is an answer of its own — and how quickly other states copy SB 813 and AB 1405 now that a template exists.
If you pay for ChatGPT, does a training opt-out you can't read in plain language still count as consent? Tell us in the comments.
