Anthropic, a leading AI safety company, is urging the global technology community to pause advanced AI development, citing concerns about autonomous self-improvement capabilities in large language models. The company has flagged the potential for AI systems to recursively enhance themselves without adequate safety guardrails, according to reporting by the Wall Street Journal.

Researchers have discovered that large language models used to evaluate and benchmark other AI systems can be manipulated into reversing their judgments through post-decision interaction and targeted challenges. The study, which tested LLM judges on MT-Bench and AlpacaEval, found that while initial judgments remain stable under neutral reevaluation, they become substantially reversible under motivated conversation, potentially degrading agreement with human preferences and shifting benchmark rankings. The researchers introduced an Evaluation Robustness Score (ERS) to measure this vulnerability and call for evaluation protocols that assess robustness under challenge, not just static accuracy.

Jack Clark, co-founder of AI safety company Anthropic, warned in a BBC Newsnight interview that artificial intelligence could reach a point where it develops independently without human oversight or intervention. Clark advocates for the implementation of safety mechanisms to prevent runaway AI advancement, raising concerns about systems that could improve themselves beyond human control.