deepeval

A framework for few-shot evaluation of language models.

Debug, evaluate, and monitor your LLM applications, RAG systems, and agentic workflows with comprehensive tracing, automated evaluations, and production-ready dashboards.

AI Observability & Evaluation

? Open-Source Evaluation & Testing for AI & LLM systems

the LLM vulnerability scanner

Python SDK for AI agent monitoring, LLM cost tracking, benchmarking, and more. Integrates with most LLMs and agent frameworks including OpenAI Agents SDK, CrewAI, Langchain, Autogen, AG2, and CamelAI

AutoRAG: An Open-Source Framework for Retrieval-Augmented Generation (RAG) Evaluation & Optimization with AutoML-Style Automation

The LLM's practical guide: From the fundamentals to deploying advanced LLM and RAG apps to AWS using LLMOps best practices

《大模型白盒子构建指南》：一个全手搓的Tiny-Universe

The open-source LLMOps platform: prompt playground, prompt management, LLM evaluation, and LLM Observability all in one place.