Amazon Web Services AI Data Center in New Carlisle, Indiana, October 2, 2025.
Noah Berger | Getty Images
The most important new jobs in artificial intelligence are likely to come with office badges, company laptops, and access to some of the most heavily guarded systems in tech: the massive language models of AI leaders including Anthropic and OpenAI.
What you may not have is the power to stop them.
Anthropic CEO Dario Amodei has proposed continuing to embed third-party safety evaluators within the Frontier AI company as part of a plan to slow the progress of increasingly sophisticated models, after former Anthropic researcher Jacob Coxon resigned and warned that Frontier Laboratories was racing ahead with systems it may not be able to control.
In an essay published over the weekend that led to a regulatory divide between the AI community and factions within the government, including President Trump, Amodei pledged to provide third-party evaluators with the same access as internal risk teams and the right to publish their findings without the company’s editorial control, subject to limited editing. He explicitly cited embedded banking supervisors as a precedent, writing that his proposal “has precedent in the banking industry, where regulatory ‘supervisors’ are sometimes embedded along with employees.”
But Julie Andersen Hill, dean of the University of Wyoming School of Law and an expert on banking regulation, said comparisons with banking industry regulation are inaccurate without the power to flip an entire operation’s “kill switch.” “If you don’t give them that power, I don’t know what they’re doing,” Hill said.
At the largest banks, government inspectors have offices within the institution, access to internal systems and employees, and a continuous presence. Hill said he could order banks to cease operations, limit growth, force changes in management or, in extreme cases, shut down.
Anthropic’s proposed evaluators would be able to investigate and report, but Amodei’s plan would not give evaluators the same enforcement powers or legal authority to prevent models from being trained or released. “That’s fundamentally different, because banking regulators have much more power than that,” Hill said.
Amodei wrote that the evaluator needs to provide a “neutral third party who can actually see the details,” but acknowledged that Anthropic still decides what to include and what to omit from existing public information.
Details released so far suggest that evaluators will have special access to Frontier AI systems, but that their formal authority over the companies developing them will be limited. Neither Anthropic’s proposal nor OpenAI’s existing third-party evaluation framework gives external evaluators independent authority to halt model development or deployment.
Anthropic and Open AI has similar safeguards in place, but has not yet released details about how its new embedded evaluator effort will work and did not respond to requests for comment.
What the AI evaluator sees in the model
Albert Ziegler, head of AI at cybersecurity firm XBOW, who is leading a team evaluating the models’ capabilities, said his company has received early access to unreleased models from Anthropic, OpenAI and other major developers, and Ziegler is personally involved in evaluating them. He said XBOW conducts its work in its own environment, not as a commissioned test, and typically shares its results with model providers.
Ziegler says that everyday reality, at least for now, is less cinematic than the existential framework.
Amodei wrote that new regulations are needed because within six to 12 months, a swarm of misplaced agents could take over large swaths of the internet, causing hundreds of billions of dollars in damage. Ziegler says his team may find that the model produces meaningless results under unusual formatting demands or that external safety checkers need to intervene more frequently. “But the kind of insidious and catastrophic consequences caused by subterfuge combined with unprecedented capabilities that people fear is not something we ourselves have seen,” he said.
Black box testing can reveal whether a model is capable of performing a dangerous task, but determining whether a larger system is dangerous may require access to instructions surrounding the model, tools and permissions, safety controls, and logs of attempted actions, Ziegler said. Still, he added, serious risks could only emerge from a combination of circumstances that the evaluator would never have triggered.
“It’s true that we don’t have a veto,” he said.
He said evaluators can discover and document risks that developers have overlooked and “force informed decisions before release.”
However, the final decision lies with the company.
Despite having broader powers, the banking model is incomplete, Hill said.
Supervisors fail to prevent large-scale collapses and are regularly accused of getting too close to the institutions they oversee after a crisis. Continued oversight is also costly for regulated companies, potentially strengthening large incumbents that can absorb costs while making it harder for smaller competitors to enter.
Links to top AI labs remain a concern
The proposal has also drawn criticism from those who argue that Anthropic and OpenAI are using safety concerns to push for a regulatory approach that could insulate them from competition and accountability. Anthropic also faces allegations of conflicts of interest involving evaluators it proposed to use.
If an AI company could choose its evaluators, control what they see, and be free to ignore their conclusions, “it’s a lot like an in-house compliance department,” Hill said. “If Anthropic wants to have an in-house compliance department, there’s nothing stopping them from doing that at this point,” she said. “There’s no need for the government to do that. It seems like what the existing AI people just want is someone to monitor them and tell the public, ‘Look, I looked behind the curtain and there’s nothing bad going on there,'” she said. “This is somewhat unusual from a regulatory standpoint.”
Amodei cited the independent nonprofit Model Evaluation Threat Research (METR) as an example of a potential embedded evaluator. Anthropic has previously worked with METR, most recently asking them to investigate a cybersecurity assessment incident involving Claude. Joe Benton, also a former anthropologist, recently left the company and joined METR to work on built-in assessments of AI risks. In addition to providing direct expertise to METR, his move demonstrates how small and interconnected the frontier AI safety field remains.
This placement shows the structural tension surrounding the word “independent.” Developers choose who receives access, define its boundaries, and maintain control over what happens after discovery.
Access alone does not mean evaluators are independent, said Deborah Raji, a researcher at the University of California, Berkeley who specializes in algorithmic auditing and AI accountability. In an established audit system, auditors must meet both competency requirements and rules governing conflicts of interest and conduct, he said.
“If you don’t meet the behavioral standards of independence, you are effectively not qualified to be an auditor,” Raji said. “It discredits the whole process.”
Raji said an authority independent of companies should decide who is qualified to carry out assessments, what assessors can inspect and where their results should be reported.
“You can’t wake up one day and decide you’re qualified to be a bank examiner, and the company being audited can’t randomly assign you to a qualified bank examiner,” she said. “Otherwise, it would be the same as companies asking random friends to check their homework.”
METR said it does not accept cash payments or donations from AI companies or their executives. However, the organization acknowledged in its own Frontier Risk report that some employees have strong social ties to employees at AI companies and share research centers with some lab employees. These relationships do not prove that the work is at risk, but they do highlight how small and interconnected the emerging assessment field remains.
“We allow all sorts of weird things.”
Raji said it’s hard to ignore all the baggage that comes with a relationship with METR. “They have financial and ideological entanglements, suspected personal conflicts of interest (i.e. marriages of researchers to OpenAI board members, hires who are former Frontier Institute employees, etc.), and are locked into Anthropic and OpenAI with little obligation to replace them, especially in recent years,” she said.
She added that the current structure of these relationships is “really abnormal and allows for all sorts of strange things.”
To its credit, Raji says METR has been very open about the details of its contract with Open AI for the Hug Face study. “But you will see how OpenAI sets the parameters of the arrangement and controls the ‘scope’ of the audit, what was accessed, what was not accessed, what findings were made public, and what findings were not made public.”
Ziegler said he doesn’t feel the early access relationship diminishes XBOW’s independence. Developers want to know when something goes wrong, rather than having teams verify predetermined conclusions.
The conflict is not unique to AI, said Christina Ho, chief assurance officer at accounting firm Oath and former director of the Public Company Accounting Oversight Board, a Congressional nonprofit that oversees audits of public companies and SEC-registered broker-dealers. She said there is a persistent tension between independence and self-preservation because auditors are paid by clients and auditors must challenge their work. However, since the financial crisis, the law has been amended to allow auditors to be held accountable under the Sarbanes-Oxley Act.
AI adds a second problem: expertise. Traditional audits often focus on whether a company followed proper processes and controls. “You have to be able to verify not only the model development process, but also the actual system and its outputs,” Ho says. “Right now, there are very few people who can do that.”
Even if these potential disputes could be addressed, Hill said, the AI proposal still does not provide a detailed legal framework, which she said is important for bank examiners. “Leaving supervisors on their own without any standards doesn’t really work,” she says.
Frontier AI currently does not have a comparable set of operating rules that define what is prohibited, what discretion evaluators have, or what consequences follow significant findings. Access and publication rights may help with external oversight, but they cannot provide regulatory credibility as long as the company retains control. “It’s not both ways,” Hill said. “You can’t control everything and still expect reliability as if you had given up control.”
The ultimate difference, Raji said, is whether an adverse finding yields a result.
“The purpose of an audit is to force the person being audited to make some consequential judgment,” he said. “If you do an audit and nothing happens, that’s audit laundering.”
In Hill’s view, if the risks are as great as AI leaders claim, the ultimate test is straightforward. “If we truly believe that AI has the power to disrupt society, we need an independent regulator with the ability to stop it,” she said.
