Search AI Products and News

AWS AI Labs recently introduced SWE-PolyBench, a multilingual open-source benchmark designed to provide a more comprehensive framework for evaluating AI programming assistants. With advancements in large language models (LLMs), AI programming assistants capable of generating, modifying, and understanding software code have shown significant progress. However, current evaluation methods remain limited, with many benchmarks focusing solely on single languages like Python, failing to offer a complete picture.

At a recent Goldman Sachs conference, OpenAI CFO Sarah Friar revealed the company is developing an AI agent called "A-SWE" designed to completely replace software engineers. Friar stated that this new AI will not only augment the productivity of existing engineers but also independently handle tasks ranging from application development to quality assurance, troubleshooting, and documentation. She indicated that A-SWE will "double" the size of companies' development teams. Image source omitted.

Bug fixing is a persistent challenge in software development. ByteDance's Doubao large model team has exciting news: they've launched Multi-SWE-bench, the first multi-lingual software engineering (SWE) dataset. This new dataset aims to evaluate and improve the ability of large models to automatically fix code errors. Multi-SWE-bench significantly expands the scope compared to previous single-language datasets. It covers not only Python, but also...

Recently, OpenAI researchers admitted in a newly published paper that despite the advanced nature of current AI technologies, these models still cannot compete with human programmers. OpenAI CEO Sam Altman previously stated that by the end of this year, AI is expected to surpass 'junior' software engineers, but research findings show that these AI models still face significant challenges. Image credit: Image generated by AI, image licensed from service provider Midjourney in the study.

Recently, OpenAI released a significant report on AI programming capabilities, highlighting the current state of AI in software development through a $1 million real-world development project. The benchmark test, named SWE-Lancer, covered 1,400 real projects from Upwork, comprehensively assessing AI performance in both direct development and project management areas. The results indicated that the best-performing AI model, Claude 3.5 Sonnet, achieved a success rate of 26.2% in coding tasks and reported performance in project management.

OpenAI has released SWE-bench Verified, aiming to more accurately assess AI performance in software engineering tasks and address the limitations of the original SWE-bench, such as overly strict unit tests, ambiguous problem descriptions, and challenging development environment setups. The new benchmark improves assessment consistency and reliability by introducing a containerized Docker environment, significantly enhancing the performance scoring of AI models. GPT-4o solved 33.2% of the samples under the new benchmark, while the best open-source agent framework has...

AI startup Cognition has launched the AI programmer 'Genie', scoring 30.08% on the authoritative testing platform SWE-Bench, far surpassing Devin and GPT-4, and becoming the world's strongest AI programming assistant. Co-founder Alistair Pullen showcased the project in 2022, aiming to create an AI that can automatically code, debug, and optimize like a human. After over a year of development, Genie utilizes special data that incorporates the reasoning processes of human programmers.