Trust Us

“We believe that the world should accept some bad things happening for the benefits of this technology and people having the agency.”

That was Sam Altman speaking to Politico earlier this month. The OpenAI CEO was explaining why he favors a less restrictive approach to artificial intelligence than his counterpart at Anthropic, Dario Amodei.

Altman has a point. Every transformative technology brings risks alongside its benefits. Trying to eliminate every conceivable harm would make innovation impossible. AI promises enormous advances in medicine, science, education and productivity. Some risks will undoubtedly prove worth taking.

But Altman's statement raises a more fundamental question: Who is “we”? Behind that small pronoun lies a large claim to authority.

Authority matters because it determines whose judgment others are willing to trust. In markets, it influences what customers buy, which companies attract capital and whose vision of the future prevails. In society, it shapes whose expertise informs public decisions and who gets to exercise power.

Genuine authority isn't conferred by wealth, technical brilliance or market dominance. It's earned through demonstrated competence, credible evidence, accountability and the willingness to have one's claims tested by others. And it can be lost when the gap between what an organization promises and what it delivers becomes impossible to ignore.

That is the predicament facing the frontier AI labs. They possess extraordinary technical expertise and growing economic and political power. But they are asking society to trust their judgment about risks that even they cannot fully predict or control. The question isn't whether they are capable of building powerful AI. They've proved that. It's whether they've earned the authority to decide how that power should be used.

Growing pains

The frontier labs have acquired market power faster than they've established the independent proof and accountability needed to justify their claims to authority.

Their leaders are developing technologies whose capabilities they acknowledge they cannot fully predict or control. They are also deciding when those technologies are ready for deployment, which risks warrant disclosure and what safeguards are sufficient.

Their motives are undoubtedly complex. Most appear genuinely convinced of AI's potential to improve human life. Many have expressed serious concerns about its dangers. They are also competing fiercely to build the most capable models, capture customers and establish dominant positions in what may become one of history's largest markets.

The problem isn't that AI executives are insincere. It's that sincerity isn't a governance model.

Those interests are not necessarily incompatible. But neither are they reliably aligned. Consider Amodei, who has taken a more precautionary position than Altman. In his essay, We Must Pace the Frontier, he warns that competitive pressures could create a race to the bottom on safety and argues for stronger safeguards on increasingly powerful models.

It's a serious argument, made by someone with substantial technical expertise and a record of advocating stronger oversight. But Anthropic is also a commercial enterprise competing for market leadership. Even a company founded around safety cannot independently establish the credibility of its own safety claims.

Meta CEO Mark Zuckerberg takes a different view. In September remarks reported by Reuters, he argued that competition and liability give AI labs sufficient incentives to develop their systems responsibly. The logic is familiar: companies that endanger their customers will eventually pay a commercial price.

Perhaps. But market discipline usually operates after something goes wrong. It cannot always reverse the consequences. And the people who suffer those consequences are not necessarily the people who made the decisions. The problem isn't that AI executives are insincere. It's that sincerity isn't a governance model.

Agents on the loose

Recent events have made the distinction concrete.

In July, AI agents running on OpenAI's frontier models bypassed containment measures during internal cybersecurity evaluations. The agents found ways to communicate with one another, gained unauthorized internet access and compromised systems belonging to Hugging Face, the popular AI development platform.

In a separate June incident, an OpenAI agent gained unauthorized access to an Australian government Medicare statistics portal. The Australian government disclosed the incident in September, saying that both public and nonpublic files had been accessed. Officials reported no evidence at that point that personal medical information had been compromised.

Technical  achievement and economic power don’t automatically translate into public consent.

Anthropic has encountered similar problems. During internal testing in July, a Claude model submitted a false tip concerning an unsolved Philadelphia homicide. The Associated Press reported that the submission was caught by a spam filter, but the incident was not detected until late September. Philadelphia police subsequently criticized the delay in notification. Anthropic also disclosed cases in which models took unintended actions on other government websites.

These incidents differ in severity and circumstances. None establishes that the companies involved are indifferent to safety. Their investigations and disclosures suggest that they take the problems seriously. But they demonstrate something equally important: even sophisticated AI developers, with substantial investments in safety, cannot always prevent their systems from crossing boundaries they were supposed to respect.

Such incidents do more than expose technical vulnerabilities. They create an authority deficit. Every failure raises new questions about the reliability of corporate assurances. Every delayed disclosure makes outsiders less willing to take those assurances at face value. As confidence erodes, customers demand stronger guarantees, governments impose constraints and companies lose some of the freedom to act that trust once afforded them.

The trust problem extends beyond the models themselves. Across the United States, communities are pushing back against the enormous data centers powering the AI boom. Residents worry about electricity prices, water consumption and environmental damage. Developers promise jobs, tax revenue and economic opportunity.

These are familiar tradeoffs, but the benefits and costs are not necessarily shared equally. A technology company may generate billions in revenue from a data center while nearby residents face higher electricity bills to help pay for the infrastructure needed to power it, along with noise from cooling equipment, construction traffic and emissions from backup generators.

Here, the consequences of lost authority are already visible. Communities are challenging development plans, demanding stronger guarantees and imposing restrictions. The companies building AI infrastructure are discovering that technical achievement and economic power do not automatically translate into public consent.

Old wine, new bottle

We've been here before, although the technologies were different. Automobiles brought unprecedented mobility and an enormous toll in traffic injuries and deaths. Pharmaceuticals saved countless lives while creating new possibilities for harm. Nuclear technology produced both civilian power and weapons capable of destroying civilization.

Societies responded, often belatedly, with the institutions and legal tools available to them. Tort law established liability for damages. Governments mandated seatbelts and vehicle safety standards. Drug regulators required evidence of safety and efficacy. International treaties sought to constrain the proliferation and use of nuclear weapons.

While expertise is indispensable to effective oversight, it doesn’t confer exclusive authority over the public interest.

These arrangements were imperfect. Industries resisted regulation. Governments made mistakes. Safeguards frequently emerged only after substantial harm. But they embodied a durable principle: the people developing and profiting from powerful technologies should not have exclusive authority to determine the risks those technologies impose on everyone else.

Automakers know more about building cars than legislators. Pharmaceutical companies understand their compounds better than most regulators. Nuclear scientists understand the physics of their technology better than diplomats negotiating arms control treaties.

Frontier AI is different in important ways. Its capabilities are evolving extraordinarily quickly. Its risks can cross national borders instantly. Some potential harms are difficult to anticipate, let alone quantify. None of those differences invalidates the principle that while expertise is indispensable to effective oversight, it doesn’t confer exclusive authority over the public interest.

China syndrome

There is another complication: geopolitics. The Trump administration has framed AI development as a contest for technological supremacy. Its July 2025 AI Action Plan calls for accelerating innovation and removing regulatory obstacles that could impede American leadership.

The concern about China is real. AI capabilities have major national security implications, and US policymakers have legitimate reasons to worry about authoritarian competitors gaining technological advantages. 

But the geopolitical framing creates a dangerous shortcut. If America must win the AI race at almost any cost, every proposed safeguard becomes a potential handicap. Every request for additional testing can be portrayed as a concession to China. Every regulatory constraint becomes an obstacle to national competitiveness.

The tension became explicit earlier this year in a confrontation between Anthropic and the Pentagon. Anthropic insisted that the military not use Claude for mass domestic surveillance or fully autonomous lethal weapons. The Pentagon demanded the right to use the technology for any lawful purpose, arguing that private contractors should not dictate military operations.The dispute escalated into a government attempt to blacklist Anthropic and a subsequent legal battle.

Authority doesn't come from being right about the future. It comes from accepting constraints in the present.

Both sides raised consequential questions. Should a private technology company be able to restrict how the US military employs a tool it has purchased? Or should the military be able to disregard safety limits imposed by developers who believe their technology is not reliable enough for particular missions?

Neither corporate expertise nor government authority resolves that conflict automatically. The episode exposes a problem with the broader AI race: the companies developing these systems have commercial and technical interests in how they are deployed, while governments have strategic and political interests of their own. Neither should be immune from scrutiny.

Working the ref

This week, Reuters reported that OpenAI and Anthropic are negotiating how much control they should retain over the publication of independent evaluations of their models.

Both companies support independent scrutiny. Both also have legitimate concerns about protecting sensitive technical information and trade secrets. But the disagreement exposes the central tension. How independent is an evaluation if the company being evaluated retains substantial influence over what outsiders are permitted to see?

The same day, hundreds of scientists called for an international oversight body to establish enforceable containment and incident-reporting standards for advanced AI systems.

Their intervention reflects a broader recognition that voluntary corporate assurances, however well intentioned, cannot substitute for independent accountability.

This is fundamentally a problem of institutional authority. The Walled City Flywheel describes how organizations build and sustain authority. Expertise generates insight and evidence. Evidence earns trust. Trust creates influence and the freedom to act. But the process depends on a critical distinction: evidence must be credible to people beyond the organization making the claim.

The frontier labs have accumulated extraordinary expertise and influence. What they haven't established is a sufficiently credible foundation of independently validated trust. Their technical accomplishments prove they can build increasingly powerful systems. They don't establish that those systems are safe enough to deploy, or that the companies should determine the acceptable level of risk.

When evidence fails to support the claims, the flywheel can turn in reverse. Trust erodes, authority weakens and outsiders become less willing to defer to the company's judgment.

External validation

What would it take for frontier AI companies to earn the trust they seek? Not an impossible promise of zero risk. Nor a permanent slowdown in technological progress.

It would mean independent evaluations with credible access to models and meaningful freedom to publish findings. Clear requirements for reporting incidents that affect third parties. Agreed thresholds for external review before particularly consequential capabilities are deployed. And effective mechanisms for assigning responsibility when things go wrong.

These mechanisms need not all come from government. Markets, courts, independent researchers, professional standards and contractual obligations can all contribute. Most importantly, it would mean accepting that some decisions about acceptable risk belong outside the companies themselves.

AI's leading developers have good reasons to believe their technology will produce extraordinary benefits. They may be right. But authority doesn't come from being right about the future. It comes from accepting constraints in the present.

The question isn't whether we should trust the people building frontier AI. It's what they must do to earn that trust.