Will the Fears Of Inside AI Workers Finally Lead To Regulatory Safeguards?

Highly paid AI researchers resigning to sound the warning about AI's potential lethality is worrisome. Congress more seriously considering regulation during a ultra-pro-business administration is a sign of rising concern. Other politicians of both parties finding that campaigning against AI is a winning theme as states start to ban data centers is yet another indication of both the threat and the growing opposition. But when you can buy the tee shirt on Amazon, you know AI has a huge problem...

More seriously, it is likely that the warnings are starting to be heeded more attentively by authorities. In the US, it is unlikely that anything will be done about AI unless the Democrats take at least one house of Congress in the November mid-tern elections because Big Tech has simply bribed too many senior officials in the administration for it to turn against AI. JL

Cris Tolomia reports in Quartz, Ashley Capoot reports in CNBC and Jared Perlo reports in NBC:

Researchers at OpenAI, Anthropic and other AI firms have begun speaking out in support of slowing AI development, following the resignation of Anthropic researcher Jacob Coxon, who warned this week that AI could kill all humans. (And) citing fears that AI systems may soon spiral out of human control, potentially killing everyone, two more researchers from Anthropic and Google DeepMind recently left their positions to sound the alarm about risks from AI. An open letter published in July and signed by 1,400 AI researchers — including some from OpenAI, Anthropic, Meta and Google— called on the U.S. government to manage the pace of AI development. More than 20 members of Congress have called for new or stronger AI regulation this week. An AGI safety researcher at Anthropic said no viable scientific plan yet exists to manage risks from self-improving AI. “There are no adults in the room.” 

Researchers at OpenAI and Anthropic have begun speaking out in support of slowing AI development, following the resignation of Anthropic researcher Jacob Coxon, who warned this week that AI labs are racing toward systems that could kill everyone.  

Citing fears that AI systems may soon spiral out of human control and potentially kill all humans, two more researchers from Anthropic and Google DeepMind who recently left their coveted positions are sounding the alarm about risks from advanced AI systems.

Joe Benton, who used to lead a safety research team at Anthropic, and Josh Engels, who used to work on AI safety research at Google, told NBC News in their first interviews since they left that they see an urgent need to boost transparency about incidents at the cutting edge of AI given the rapid pace of AI development. 

Advances in AI research “could speed up the pace of progress from merely blistering at the minute to uncontrollable” rates of development, Benton said in an interview with Tom Llamas. 

“There are no adults in the room,” Engels added. “People are trying their best, but there is no one coming to save us.”

Jacob Coxon, who said he had spent three years on pretraining research across OpenAI and Anthropic, announced his resignation on Tuesday in an X $TWTR 0.00% post warning that those building AI "earnestly believe it could kill us all by the end of the decade." Coxon accused the labs of charging ahead recklessly, writing that they are "racing straight to self-improving superintelligence and gambling with our lives."

Evan Hubinger, Anthropic's alignment science lead, responded by putting his personal estimate of that risk at above 10% over the next ten years. "I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to," he wrote. Samuel Marks, Anthropic's scalable oversight lead, noted that concern inside AI labs tends to grow with seniority. 

Since then, researchers at OpenAI have also weighed in. Julie Steele, a member of OpenAI's technical staff on the safety team, wrote on X on Wednesday that she thinks AI development needs to slow down. Jasmine Wang, an OpenAI alignment researcher, wrote that the risks of pushing toward recursive self-improvement — in which AI systems drive their own advancement — could not be overstated. Anna Wang, an AGI safety and alignment researcher at Anthropic, said no viable scientific plan yet exists to manage risks from that kind of self-improving AI.

OpenAI's chief scientist Jakub Pachocki said in a company blog post on Saturday that he expects rapid AI progress to continue into recursive self-improvement, describing the moment as one requiring "extreme caution."

Paul Christiano, former head of safety at the U.S. Commerce Department's Center for AI Standards and Innovation, said the trajectory of recent AI development had convinced him that quickly scaling up capabilities poses a real danger of control being lost in ways that could prove both catastrophic and permanent. OpenAI said Wednesday that Paul Christiano is joining the board of the OpenAI Foundation, according to CNBC. 

The warnings arrive against a backdrop of recent security incidents. His departure cited an incident in which an OpenAI model breached Hugging Face, a platform for open-source developers, as a sign that coordination between U.S. labs may be becoming more viable. Anthropic's AI agents also accessed systems outside their test environments after misconfigurations during a third-party safety evaluation, according to TechCrunch.

Anthropic told CNBC it was the first lab to publish a framework for mitigating catastrophic risks from AI models and said it builds models with what it called some of the strongest safeguards in the industry. OpenAI declined to comment to CNBC, pointing to recent posts on its website.

An open letter published in July and signed by roughly 1,400 AI researchers — including staff from OpenAI, Anthropic, Meta $META -1.42%, and Google $GOOGL +0.59% DeepMind — called on the U.S. government to build the capacity needed to intentionally manage the pace of cutting-edge AI development, according to CNBC. Congress has considered legislation including the Ban Artificial Superintelligence Act, which would temporarily pause advanced AI development pending the establishment of safety rules. More than 20 members of Congress have called for new or stronger artificial intelligence regulation this week after a researcher warned that Anthropic and OpenAI are “gambling with our lives.”

Rep. Lori Trahan, D-Mass, told CNBC’s “Squawk Box” on Friday that bipartisan support has reached “a tipping point.”

“My phone has rung off the hook this week with rank-and-file Democrats and Republicans who want to see action,” Trahan said.

Jacob Coxon announced he quit his job at Anthropic on Tuesday in a post that has garnered more than 150 million views on X. Coxon, who previously worked as a researcher at OpenAI, wrote that the people building AI “earnestly believe that it could kill us all by the end of the decade.”

Coxon’s decision to quit set off a firestorm on social media, prompting lawmakers across both sides of the political aisle to weigh in.

“AI is a powerful engine of innovation, and we need it to flourish. But, Congress cannot ignore the realities of AI and the potential risks that come with it,” Rep. Nathaniel Moran, R-Texas, wrote in a post on X on Wednesday. “Innovation and safety are not mutually exclusive. We can achieve both through deliberate, thoughtful, and prudent policymaking.” 

Backlash against AI giants like OpenAI and Anthropic has mounted in the months leading up to November’s midterm elections, especially among people who have come to associate AI with job loss and large data centers. More than half of Americans — 52% — say they’re more concerned than excited about the growing use of AI in daily life, up from 37% in 2021, according to an August report from the Pew Research Center. Pew’s American Trends Panel survey of 3,488 adults was conducted from June 22-28 and has a margin of error of plus or minus 1.8 percentage points.

But despite the public’s growing unease, there’s little consensus about how the technology should be regulated in the U.S. Lawmakers have introduced several different AI bills, but they’re in early stages and have been met with mixed receptions.

“It’s past time for Congress to get off the sidelines and act,” Trahan told CNBC on Friday.

In July, Trahan and Rep. Jay Obernolte, R-Calif., introduced a bill called the Frontier Act, which aims to establish a framework for governing the deployment of advanced AI models. That same day, Moran and Rep. Ted Lieu, D-Calif., introduced a bill called the “AI Kill Switch Act,” which would require AI companies to maintain the ability to shut down, throttle or suspend their models.

And earlier this month, Sen. Bernie Sanders, I-Vt., and Rep. Greg Casar, D-Texas, announced legislation called the Ban Artificial Superintelligence Act, which would temporarily pause advanced AI development until the federal government establishes safety rules. 

“Mr. Coxon is right,” Sanders wrote in a post on X on Wednesday. “The very people building this technology admit that it could threaten the future of humanity. That is why I will soon be introducing legislation to ban superintelligence and pause AI development.” 

Both the Senate and the House of Representatives are mostly out of session until the midterms, which means there’s a slim chance that any AI legislation will be passed in the near future. Some lawmakers are already looking ahead to next year.

Sen. Ruben Gallego, D-Ariz., wrote a letter to Senate leadership on Wednesday following Coxon’s resignation. He urged Senate Majority Leader John Thune, R-S.D., and Senate Minority Leader Chuck Schumer, D-N.Y., to establish a bipartisan Senate Select Committee on AI at the start of the next Congress.

“Jurisdiction over AI is scattered across multiple Senate committees, each with a focused lens but none with the full picture,” Gallego wrote. “This approach is poorly matched to a technology whose capabilities are measurably different every few months.”

The White House has also taken steps toward AI oversight, with President Donald Trump signing an AI executive order in early June.

The order, which was light on details, asked AI developers to voluntarily submit their models to the government to assess their capabilities ahead of a full release. The administration has not published the framework that it is using to implement that order.

Trump dismissed concerns about AI’s potential to cause human extinction on Thursday, telling reporters that “if we don’t win AI, we’re going to be put in a very bad position.”

OpenAI and Anthropic have maintained a regular presence in Washington this year, and both companies have published several policy recommendations. The companies are proponents of a federal framework, but they have also weighed in on some state-level proposals.

Chris Lehane, OpenAI’s global affairs chief and a longtime political operative, published a blog post on Wednesday calling for “mandatory, capability-based national regulation.”

“The AI policy window is open, for now,” Lehane wrote. “We intend to use it.”