Popular GitHub Repositories for Benchmarking

An adversarial example library for constructing attacks, building defenses, and benchmarking both

Python SDK for AI agent monitoring, LLM cost tracking, benchmarking, and more. Integrates with most LLMs and agent frameworks including OpenAI Agents SDK, CrewAI, Langchain, Autogen, AG2, and CamelAI

Merlion: A Machine Learning Framework for Time Series Intelligence

AutoRAG: An Open-Source Framework for Retrieval-Augmented Generation (RAG) Evaluation & Optimization with AutoML-Style Automation

Repository for benchmarking graph neural networks (JMLR 2023)

Collection of popular and reproducible image denoising works.

[NeurIPS 2024] OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments

Unifying Variational Autoencoder (VAE) implementations in Pytorch (NeurIPS 2022)

[ACL 2024 ?] Video-ChatGPT is a video conversation model capable of generating meaningful conversation about videos. It combines the capabilities of LLMs with a pretrained visual encoder adapted for spatiotemporal video representation. We also introduce a rigorous 'Quantitative Evaluation Benchmarking' for video-based conversational models.

A streamlined and customizable framework for efficient large model evaluation and performance benchmarking

A Toolbox for Adversarial Robustness Research

Unsupervised time series anomaly detection library

T3Bench: Benchmarking Current Progress in Text-to-3D Generation

Benchmarking Generalized Out-of-Distribution Detection

[PVLDB 2024 Best Paper Nomination] TFB: Towards Comprehensive and Fair Benchmarking of Time Series Forecasting Methods

PytorchAutoDrive: Segmentation models (ERFNet, ENet, DeepLab, FCN...) and Lane detection models (SCNN, RESA, LSTR, LaneATT, BézierLaneNet...) based on PyTorch with fast training, visualization, benchmarking & deployment help

[NeurIPS'21 Outstanding Paper] Library for reliable evaluation on RL and ML benchmarks, even with only a handful of seeds.

Long Range Arena for Benchmarking Efficient Transformers

Benchmarking Knowledge Transfer in Lifelong Robot Learning

Windows Agent Arena (WAA) ? is a scalable OS platform for testing and benchmarking of multi-modal AI agents.