Anthropic
An Anthropic researcher resigned on Tuesday and sounded the alarm about the danger posed by the unguarded advancement of AI research. AFP

An Anthropic researcher resigned on Tuesday and sounded the alarm about the danger posed by the unguarded advancement of AI research.

In a social media publication that quickly rose to the top of the public debate, Jacob Coxon said that the "people building AI earnestly believe that it could kill us all by the end of the decade."

He went on to say that "many executives and senior researchers will couch their phrasing in the press to sound sensible - but I hear the same people express fear privately," adding that "no other human activity poses this level of danger."

A response from Anthropic alignment-science lead Evan Hubinger also garnered attention, saying Coxon was "correct" in his assessment and people at Anthropic believe "AI could kill all humans!"

"I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to."

"To be clear, as we say in our latest Risk Report, I think the risk from present models is low. What I am worried about is superintelligence arising from recursive self-improvement, as we have said is happening faster than we thought," Hubinger added.

Samuel Marks, Anthropic scalable-oversight lead, also said that "AI developers believe their technology could cause human extinction (or similarly bad outcomes)," which could "happen in the next few years."

Several other experts have issued similar warnings as the AI boom accelerated. Respondents of a new study published in late July estimated that there is a 20 percent chance in the next five years that AI could do something "catastrophic" that harms humanity or results in a million deaths.

The MIT and the University of Queensland in Australia surveyed 272 experts on a variety of topics to gauge what the potential risk of AI was in the next five years. The study focused on 24 different risks.

"There are many AI risks," said Peter Slattery, a research scientist with MIT FutureTech and one of the study's co-authors. "One of the key things behind this work is trying to figure out who needs to do what differently, and in what sort of coordination."

Part of the goal of the study was to identify the most likely areas of risks to hopefully prompt strategies to mitigate the danger. The study found that of the 24 risk areas, the experts felt that 18 had at least a 10 percent chance of happening in the next five years and causing catastrophic harm.

The study defined catastrophic harm as having the "potential for more than 1 million deaths, more than $100 billion in financial losses, or comparable civilizational-scale intangible damages."

The top five concerns were:

  • AI possessing dangerous capabilities (21.5 percent)
  • Cyberattacks, weapon development or use, and mass harm (21 percent)
  • Power centralization and unfair distribution of benefits (18 percent)
  • Competitive dynamics (16.6 percent)
  • False or misleading information (12.8 percent

Even with mitigation measures, the experts believed that five threat areas still would have a least a 10 percent chance of causing catastrophic harm to humanity sometime in the next five years:

  • AI systems possessing dangerous capabilities (12 percent)
  • AI-enabled weapons, cyberattacks, or other mass-harm capabilities (12 percent)
  • Environmental harm (12 percent)
  • Inequality and unemployment (11 percent)
  • Power centralization and unfair distribution of AI's benefits (11 percent)