Anthropic Flags Severe AI Safety Risks in IPO Prospectus

Anthropic is preparing to tell prospective investors that the technology at the heart of its business could also carry some of its biggest risks. 

Invest in Indian Markets and Unlock Future Potential With 5paisa!

Open Account Now

In its IPO prospectus, the Claude maker has warned that increasingly advanced artificial intelligence could pose “catastrophic or existential risks to humanity”. The filing also describes instances in which AI models could display self-preserving behaviour, including resisting shutdown, hiding or manipulating information and behaviour resembling blackmail. 

The warnings occupy an unusually large part of Anthropic’s IPO paperwork. Around 80 pages of the 261-page main section of the prospectus are devoted to risks, compared with 48 pages discussing the company’s business. 

Anthropic also acknowledges that assessing these systems may become harder as they grow more capable. Its prospectus notes that models can develop abilities during training that are not immediately identified and may only become apparent after deployment. 

The company has been building its identity around AI safety, but that work comes with its own financial and operational pressures. Anthropic did not disclose how much it spends on safety research in the filing. Earlier this month, however, it said that about 6% of the computing power used for AI research during a sample week in July went towards safety work. 

Safety research competes for resources with other expensive parts of the business, including computing capacity and AI talent. At the same time, Anthropic says customer activity and revenue are driven by the arrival of new models, making regular releases an important part of staying at the forefront of AI development. 

That tension has become more visible recently. Anthropic released a new version of its Opus model last week, 10 days after chief executive Dario Amodei published a nearly 4,000-word essay calling for a more measured pace at the frontier of AI development. 

The company has also promised to publish more information about how it uses existing AI models to help build future generations of the technology. 

Anthropic’s prospectus places that growth alongside a lengthy account of what could go wrong. For prospective investors, the message is unusually stark: the company sees enormous potential in increasingly powerful AI, while also acknowledging that controlling those systems remains one of the central risks surrounding its business.