Popular GitHub Repositories for Llm As A Judge

The open-source LLMOps platform: prompt playground, prompt management, LLM evaluation, and LLM Observability all in one place.

Evaluate your LLM's response with Prometheus and GPT4 ?

? Agent-as-a-Judge and DevAI dataset

xVerify: Efficient Answer Verifier for Large Language Model Evaluations

CodeUltraFeedback: aligning large language models to coding preferences

This is the repo for the survey of Bias and Fairness in IR with LLMs.

Code and data for "Timo: Towards Better Temporal Reasoning for Language Models" (COLM 2024)

Code and data for Koo et al's ACL 2024 paper "Benchmarking Cognitive Biases in Large Language Models as Evaluators"

A set of examples demonstrating how to evaluate Generative AI augmented systems using traditional information retrieval and LLM-As-A-Judge validation techniques

LLM-as-judge evals as Semantic Kernel Plugins

根信号评估平台的 MCP（主控处理器）

automated evaluation of llm generated responses on aws