Popular GitHub Repositories for Manuscript Metrics

? Open source LLM engineering platform: LLM Observability, metrics, evals, prompt management, playground, datasets. Integrates with OpenTelemetry, Langchain, OpenAI SDK, LiteLLM, and more. ?YC W23

The LLM Evaluation Framework

Evidently is ??an open-source ML and LLM observability framework. Evaluate, test, and monitor any AI-powered system or data pipeline. From tabular data to Gen AI. 100+ metrics.

Python SDK for AI agent monitoring, LLM cost tracking, benchmarking, and more. Integrates with most LLMs and agent frameworks including OpenAI Agents SDK, CrewAI, Langchain, Autogen, AG2, and CamelAI

High-level library to help with training and evaluating neural networks in PyTorch flexibly and transparently.

《大模型白盒子构建指南》：一个全手搓的Tiny-Universe

mean Average Precision - This code evaluates the performance of your neural net for object recognition.

Quantized Attention that achieves speedups of 2.1-3.1x and 2.7-5.1x compared to FlashAttention2 and xformers, respectively, without lossing end-to-end metrics across various models.

A comprehensive set of fairness metrics for datasets and machine learning models, explanations for these metrics, and algorithms to mitigate bias in datasets and models.

Machine learning metrics for distributed, scalable PyTorch applications.

Basic Utilities for PyTorch Natural Language Processing (NLP)

Avalanche: an End-to-End Library for Continual Learning based on PyTorch.

Sharing both practical insights and theoretical knowledge about LLM evaluation that we gathered while managing the Open LLM Leaderboard and designing lighteval!

Neural network graphs and training metrics for PyTorch, Tensorflow, and Keras.

Powerful Analytics Solution. Setup in 30 seconds. Display all your data on a Simple, AI-powered dashboard. Fully self-hostable and GDPR compliant. Alternative to Google Analytics, MixPanel, Plausible, Umami & Matomo.

一款多语言的API，满足您所有可观测性需求。轻量、快速，可替代LGTM :rocket: 独立的OpenTelemetry、Loki、Prometheus、Tempo、Pyroscope兼容 :star:

Measures and metrics for image2image tasks. PyTorch.

Evaluation code for various unsupervised automated metrics for Natural Language Generation.

A collaboratively written review paper on deep learning, genomics, and precision medicine

High-fidelity performance metrics for generative models in PyTorch