Despite the extensive promotional campaign that accompanied the OPNAI AI model, recently, independent tests showed a performance less than expectations, which opened the door to a wave of questions about the extent of technical companies ’transparency in announcing the capabilities of their advanced models, and returned to the interface the discussion about the gap between marketing data and practical reality.
Excessive results from Openai
Last December, Openai announced that its O3 model was able to solve more than 25% of Frontiermath tests, which is one of the most difficult mathematics tests directed to smart models, and this performance was presented during a live broadcast with the participation of researcher Mark Chen, who confirmed that the results came after internal tests using an advanced computer structure.
The truth is behind the declared numbers
What seemed to be an exceptional achievement that was soon exposed that it does not reflect the actual reality. The EPOCH AI, responsible for developing a FRONTIERMAT, published in the United States, published the results of new tests conducted on the public version of O3, which showed that the model was unable to exceed 10% of the correct answers.
Different copies and settings
According to the institute, this disparity may be due to the use of a stronger version in Openai tests compared to those that were presented to users.
The ARC Prize Foundation, which was officially tested before its launch, indicated that the version it received was completely different in terms of capabilities, which was also confirmed by “Winda Chu” from the Openai team, explaining that the current model is designed to be more rapid and efficient in daily use, which may explain the decline in performance in high skills tests.
Transparency under a microscope
This incident is not the first in the world of artificial intelligence, it has already been criticized for major companies such as “Mita” and “Xai” affiliated with Elon Musk regarding the publication of test data that does not actually apply to the public versions.
The future of the model
With Openai’s willingness to launch a new version called “O3-PRO”, the importance of verifying the credibility of the declared numbers increases, especially since these models have become an essential part of the tools of work, education and development, and are supposed to be based on accurate standards that can be trusted.
