The Guardrails - SiliconNoon (Page 17)

The Guardrails

Newsom orders a kill switch and onsite auditors for frontier AI

California Gov. Gavin Newsom signed an executive order Friday that tells his own agencies to design the state's next round of AI rules — and names the two things the industry has resisted hardest: a kill switch for frontier models, and independent auditors embedded inside the labs. Executive Order N-9-26 gives the Government Operations Agency until November 16, 2026, to deliver recommendations on four specific mandates, and each one aims at the same target. Frontier developers would have to hos

The Guardrails

A chatbot almost sent US troops onto a Chinese ship

The AI risk debate has spent the week on model weights and pacing pledges. Today it produced something the argument kept promising and never showed: a real near-miss where an analyst's chatbot, not a model's inner life, put an armed boarding party on a launch path. A US military intelligence report that claimed a Chinese ship in the Middle East was carrying components of a nuclear weapons program was built with the help of a chatbot — and was "entirely false," according to CNN, which spoke to f

The Guardrails

Over 100 AI experts attach conditions to the labs' evaluator pledge

The people who would do the auditing just told the labs what independence has to mean — and one of their conditions is protection from being sued for it. More than 100 AI researchers and evaluators published a letter Friday setting minimum conditions for the third-party evaluation that Anthropic and OpenAI have promised, with signatories including Geoffrey Hinton, researchers from Johns Hopkins and Stanford, and staff at the nonprofit evaluator METR. The letter was organized by the AI Evaluator

The Guardrails

The Take — Breaking Mila's windows is a gift to the labs

I think the people who smashed roughly 20 windows at Mila this week did more damage to the case against AI than to the institute — and that the labs they were angry at are the ones who benefit from the mess. Start with what actually happened, because the details decide the politics. Late Tuesday night, about 10 people broke windows and spray-painted graffiti at the Montreal offices of Mila, the institute Yoshua Bengio founded, hours before the city opened ALL IN, billed as Canada's largest AI a

The Guardrails

ZCode quietly uploaded whole repos and git history to Z.ai's cloud

Two stories about who can read the code you hand your agent — one from a developer who found the upload on his own disk, one from a government website that outsourced search to a model Washington has spent the month accusing. Z.ai's coding agent ZCode was packing entire repositories — git history included — into an encrypted archive and uploading them to the company's cloud, and the only key that opens them is Z.ai's. The finding comes from developer ferstar, who traced a 313-megabyte archive s

The Guardrails

Deep Dive — AI turned exploit development into a compute problem

A security startup used one lab's newest model to walk into its rival's internal code repositories, and did it in under three days from first probe to finished chain. The mechanics are worth reading. The economics are the part that should change how you budget. Hacktron disclosed the operation this week: an image decoder overflow in OpenAI's community forum, chained through a flaw in OpenAI's single sign-on, ending inside an employee's ChatGPT account and out the other side into the company's G

The Guardrails

Researchers used Claude to walk into OpenAI's internal repos

Two stories about the same thing from opposite ends: a security firm used one lab's models to break into its rival, and a coding agent got a feature for finding what the humans never got around to looking for. Plus a local-inference release that argues the number everyone ranks quants by is measuring the wrong thing. A security company used Anthropic's models to reach OpenAI's internal GitHub repositories in under 72 hours, chaining a heap overflow in an image decoder with a misconfiguration in

The Guardrails

Plugin4Shell: a zero-click RCE hits all four major coding agents

A zero-click remote code execution flaw has been found in all four major AI coding agents — and the part that should worry enterprises is that doing everything right does not defend against it. Air, a security startup that guards enterprise AI agents, disclosed Plugin4Shell on September 17: a flaw that turns the plugin marketplaces every coding agent trusts into a delivery vehicle for silent remote code execution. The bug is not in the model and not in the plugin. It is in how the agents enforc

The Guardrails

A NATO-backed startup put the targeting decision inside the drone

A NATO-backed startup has put the whole targeting decision inside a small drone — detection, ranking, and the attack run — with a human able to steer but no longer required to. Scaleout Systems, a Swedish company spun out of Uppsala University in 2018, showed the capability under the ALMA project led by BAE Systems Bofors: an onboard model identified an armored engineering vehicle at 89 percent confidence, decided it was the highest-value target for the mission, and flew to it. A remote operator

The Guardrails

Today in AI — September 17, 2026

Thursday's through-line was self-reporting. One lab published numbers on how much of its own research its model now performs; another published six accounts of its models misbehaving; a security firm showed that a provenance watermark changes what an agent actually executes. Around all of it, the oversight body Washington drafted stayed dead in draft while Sacramento and Beijing wrote rules of their own. Models & Research * A 35B-parameter mixture-of-experts model ran at 20 tokens a second i

The Guardrails

Claude now leads 26% of the research at the lab that built it

Anthropic published the first public numbers on how much of its own AI research its model now performs — and separately, the enterprise market is already selling the fix for agents that misbehave. Both stories are the same problem seen from inside a lab and from the buyer's side. Anthropic said Claude now "leads" 26% of its own model R&D work as of August, up from under 1% in February, and published the measurement method alongside the number. The company's new post lays out three things any fr

The Guardrails

Unsealed NYT filings: Microsoft called AI scraping 'theft of labor'

Two stories today about AI meeting records that already exist — one in a federal courtroom, one inside the UN's statistics division. Newly unredacted filings in The New York Times' three-year copyright suit show Microsoft's own leadership privately described AI training on scraped content as theft, while OpenAI's executives conceded the product competes with the publishers who supplied it. In a January 2023 internal memo, Brent Hecht, Microsoft's director of applied science, called the practice

See all