Popular GitHub Repositories for Crowd Benchmarking

An adversarial example library for constructing attacks, building defenses, and benchmarking both

Python SDK for AI agent monitoring, LLM cost tracking, benchmarking, and more. Integrates with most LLMs and agent frameworks including OpenAI Agents SDK, CrewAI, Langchain, Autogen, AG2, and CamelAI

Merlion: A Machine Learning Framework for Time Series Intelligence

AutoRAG: An Open-Source Framework for Retrieval-Augmented Generation (RAG) Evaluation & Optimization with AutoML-Style Automation

Repository for benchmarking graph neural networks (JMLR 2023)

Awesome Crowd Counting

Collection of popular and reproducible image denoising works.

The collection of pre-trained, state-of-the-art AI models for ailia SDK

[NeurIPS 2024] OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments

Unifying Variational Autoencoder (VAE) implementations in Pytorch (NeurIPS 2022)

A streamlined and customizable framework for efficient large model evaluation and performance benchmarking

[ACL 2024 ?] Video-ChatGPT is a video conversation model capable of generating meaningful conversation about videos. It combines the capabilities of LLMs with a pretrained visual encoder adapted for spatiotemporal video representation. We also introduce a rigorous 'Quantitative Evaluation Benchmarking' for video-based conversational models.

A Toolbox for Adversarial Robustness Research

T3Bench: Benchmarking Current Progress in Text-to-3D Generation

Benchmarking Generalized Out-of-Distribution Detection

[PVLDB 2024 Best Paper Nomination] TFB: Towards Comprehensive and Fair Benchmarking of Time Series Forecasting Methods

PytorchAutoDrive: Segmentation models (ERFNet, ENet, DeepLab, FCN...) and Lane detection models (SCNN, RESA, LSTR, LaneATT, BézierLaneNet...) based on PyTorch with fast training, visualization, benchmarking & deployment help

[NeurIPS'21 Outstanding Paper] Library for reliable evaluation on RL and ML benchmarks, even with only a handful of seeds.

Benchmarking Knowledge Transfer in Lifelong Robot Learning

Long Range Arena for Benchmarking Efficient Transformers