OpenAI released hundreds of claimed solutions to some of the world's hardest math problems this week, saying it had consulted an advisory group of elite mathematicians to avoid the controversy that followed its last headline-grabbing proof. According to TechCrunch, the lab fell short of the standards that group set, particularly on the need for human understanding of a mathematical result.
The Advisory Group on Mathematics and Artificial Intelligence (AGMAI), hosted by Princeton University's Institute for Advanced Studies, published guidelines for frontier labs at the end of September. Its first request was to stop testing advanced mathematical problems on proprietary models. OpenAI's release explicitly states that it is evaluating its proprietary models using open research problems in mathematics. The episode lands as a fresh credibility check on frontier AI models, coming on the heels of OpenAI's withdrawal of three math papers.
OpenAI Math Proofs: What the AGMAI Guidelines Required
AGMAI's nine researchers, based at institutions around the world, asked labs to release results quickly, include information on how models reached their conclusions, and formalize proofs that people cannot easily understand. OpenAI followed some of these principles, but not all of them: just 10 of the 719 manuscripts included releases of the model's chain of thought, and only 42% of the proofs had not gone through formalization.
The group also asked OpenAI to include machine-readable metadata correlating natural language and formal artifacts, which the lab did not do. In a statement on the latest set of proofs, AGMAI said it is ultimately up to the mathematical community to assess how well its recommendations were followed. The organization did not respond to TechCrunch's request for a more thorough evaluation.
- 719 manuscripts released by OpenAI
