Popular GitHub Repositories for Evaluation

Supercharge Your LLM Application Evaluations ?

OpenCompass is an LLM evaluation platform, supporting a wide range of models (Llama3, Mistral, InternLM2,GPT-4,LLaMa2, Qwen,GLM, Claude, etc) over 100+ datasets.

AutoRAG: An Open-Source Framework for Retrieval-Augmented Generation (RAG) Evaluation & Optimization with AutoML-Style Automation

The easiest tool for fine-tuning LLM models, synthetic data generation, and collaborating on datasets.

SuperCLUE: 中文通用大模型综合性基准 | A Benchmark for Foundation Models in Chinese

A unified evaluation framework for large language models

An open-source visual programming environment for battle-testing prompts to LLMs.

Accelerating the development of large multimodal models (LMMs) with one-click evaluation module - lmms-eval.

Open-source evaluation toolkit of large multi-modality models (LMMs), support 220+ LMMs, 80+ benchmarks

UpTrain is an open-source unified platform to evaluate and improve Generative AI applications. We provide grades for 20+ preconfigured checks (covering language, code, embedding use-cases), perform root cause analysis on failure cases and give insights on how to resolve them.

Avalanche: an End-to-End Library for Continual Learning based on PyTorch.

:cloud: :rocket: :bar_chart: :chart_with_upwards_trend: Evaluating state of the art in AI

An automatic evaluator for instruction-following language models. Human-validated, high-quality, cheap, and fast.

The official GitHub page for the survey paper "A Survey on Evaluation of Large Language Models".

? Must-read papers and blogs on LLM based Long Context Modeling ?

The ultimate LLM Ops platform - Monitoring, Analytics, Evaluations, Datasets and Prompt Optimization ?

The production toolkit for LLMs. Observability, prompt management and evaluations.

Sharing both practical insights and theoretical knowledge about LLM evaluation that we gathered while managing the Open LLM Leaderboard and designing lighteval!

High-fidelity performance metrics for generative models in PyTorch

Evaluate your LLM's response with Prometheus and GPT4 ?