Search AI Products and News

Bug fixing is a persistent challenge in software development. ByteDance's Doubao large model team has exciting news: they've launched Multi-SWE-bench, the first multi-lingual software engineering (SWE) dataset. This new dataset aims to evaluate and improve the ability of large models to automatically fix code errors. Multi-SWE-bench significantly expands the scope compared to previous single-language datasets. It covers not only Python, but also...

In geoscience, geological maps are crucial tools for understanding the Earth's surface and subsurface structures. However, interpreting these complex diagrams requires specialized knowledge and extensive experience. To enhance intelligence in this field, Microsoft Research Asia recently introduced GeoMap-Bench, a new benchmark dataset for evaluating the performance of multimodal large language models (MLLMs) in understanding geological maps. The launch of GeoMap-Bench marks a significant step forward in AI applications for geological map interpretation. Microsoft researchers, in collaboration with...

In today's rapidly advancing AI landscape, effectively evaluating and comparing the capabilities of different generative AI models is a significant challenge. Traditional AI benchmarking methods are increasingly showing their limitations, prompting AI developers to explore more innovative evaluation approaches. Recently, a website called "Minecraft Benchmark" (MC-Bench) has emerged, uniquely leveraging Microsoft's sandbox game Minecraft to facilitate model assessment.

On December 5th, the Byte Bean Bag model team launched the latest code model evaluation benchmark - FullStack Bench, covering over 11 real-world scenarios, supporting 16 programming languages, and including 3374 questions. Compared to previous evaluation standards, this benchmark can more accurately assess the code development capabilities of large models across a broader programming domain, facilitating the optimization of models in real-world programming tasks. Current mainstream code evaluation benchmarks, such as HumanEval and MBPP, typically focus on basic and advanced.

In a recent study, the OpenAI research team launched a new benchmark called MLE-bench, aimed at assessing the performance of AI agents in machine learning engineering. This research particularly focuses on 75 machine learning engineering-related competitions from Kaggle, intending to test agents' abilities in various skills required in the real world, including model training, dataset preparation, and experiment execution. To facilitate better evaluation, the research team utilized foundational data from Kaggle's public leaderboard to establish performance metrics for each competition.

DeepSeek V2.5 demonstrates exceptional performance in the field of artificial intelligence, particularly in code generation and chat models. Through comparative testing with GPT-4, it has achieved significant improvements across multiple metrics, including win rates, MT-Bench, and AlignBench scores. In terms of code generation capabilities, DeepSeek V2.5 achieved a HumanEval score of 89% and a LiveCodeBench score of 41%, showcasing its ability to generate high-quality, executable code.

OpenAI has released SWE-bench Verified, aiming to more accurately assess AI performance in software engineering tasks and address the limitations of the original SWE-bench, such as overly strict unit tests, ambiguous problem descriptions, and challenging development environment setups. The new benchmark improves assessment consistency and reliability by introducing a containerized Docker environment, significantly enhancing the performance scoring of AI models. GPT-4o solved 33.2% of the samples under the new benchmark, while the best open-source agent framework has...

AI startup Cognition has launched the AI programmer 'Genie', scoring 30.08% on the authoritative testing platform SWE-Bench, far surpassing Devin and GPT-4, and becoming the world's strongest AI programming assistant. Co-founder Alistair Pullen showcased the project in 2022, aiming to create an AI that can automatically code, debug, and optimize like a human. After over a year of development, Genie utilizes special data that incorporates the reasoning processes of human programmers.

Arthur has launched the open source tool ArthurBench for evaluating and comparing the performance of large language models. ArthurBench helps companies test the performance of different language models on specific use cases and provides metrics such as accuracy, readability, and risk mitigation for comparison. Financial services firms, automotive manufacturers, and media platforms have already begun using ArthurBench, accelerating analysis and providing more accurate answers.