Popular GitHub Repositories for Infra Monitoring

? Open source LLM engineering platform: LLM Observability, metrics, evals, prompt management, playground, datasets. Integrates with OpenTelemetry, Langchain, OpenAI SDK, LiteLLM, and more. ?YC W23

Python SDK for Agent AI Observability, Monitoring and Evaluation Framework. Includes features like agent, llm and tools tracing, debugging multi-agentic system, self-hosted dashboard and advanced analytics with timeline and execution graph view

Free MLOps course from DataTalks.Club

CLI tool for configuring and monitoring Claude Code

Build high-quality LLM apps - from prototyping, testing to production deployment and monitoring.

SkyPilot: Run AI and batch jobs on any infra (Kubernetes or 15+ clouds). Get unified execution, cost savings, and high GPU availability via a simple interface.

AI Observability & Evaluation

Evidently is ??an open-source ML and LLM observability framework. Evaluate, test, and monitor any AI-powered system or data pipeline. From tabular data to Gen AI. 100+ metrics.

Real-time Claude Code usage monitor with predictions and warnings

Next-generation AI Agent Optimization Platform: Cozeloop addresses challenges in AI agent development by providing full-lifecycle management capabilities from development, debugging, and evaluation to monitoring.

Python SDK for AI agent monitoring, LLM cost tracking, benchmarking, and more. Integrates with most LLMs and agent frameworks including OpenAI Agents SDK, CrewAI, Langchain, Autogen, AG2, and CamelAI

? Open source LLM observability platform. One line of code to monitor, evaluate, and experiment. YC W23 ?

Stunning status pages, batteries included!

Deepchecks: Tests for Continuous Validation of ML Models & Data. Deepchecks is a holistic open-source solution for all of your AI & ML validation needs, enabling to thoroughly test your data and models from research to production.

The open-source AIOps and alert management platform

The open-source LLMOps platform: prompt playground, prompt management, LLM evaluation, and LLM Observability all in one place.

Debugging, monitoring and visualization for Python Machine Learning and Data Science

? Awesome System for Machine Learning ⚡️ AI System Papers and Industry Practice. ⚡️ System for Machine Learning, LLM (Large Language Model), GenAI (Generative AI). ? OSDI, NSDI, SIGCOMM, SoCC, MLSys, etc. ?️ Llama3, Mistral, etc. ?‍? Video Tutorials.

?️ Open-source, developer-first LLMOps platform designed to streamline prompt design, version management, instant delivery, collaboration, troubleshooting, observability and more.

A AI-Driven, Distributed and high-performance monitoring system, for comprehensive monitoring and management of kafka cluster.