OpenAI Criticizes AI Evaluation Benchmark: 731 Questions, Nearly a Third Have Flaws. 8-Month Passing Rate Rises from 23% to 80%, Now Ineffective
OpenAI publicly questioned the SWE-Bench Pro benchmark, pointing out that about 30% of its 731 test tasks have evaluation flaws. The benchmark, launched by Scale AI, is an industry authority for measuring large model programming capabilities. However, OpenAI warned that the passing rate of cutting-edge models has surged from 23.3% to 80.3% within 8 months, which is unusually fast, indicating doubts about the reliability of the evaluation.