Popular GitHub Repositories for Pyspark Validation

structured outputs for llms

The standard data-centric AI package for data quality and machine learning with messy, real-world data and labels.

Evidently is ??an open-source ML and LLM observability framework. Evaluate, test, and monitor any AI-powered system or data pipeline. From tabular data to Gen AI. 100+ metrics.

Simple and Distributed Machine Learning

Core ML tools contain supporting tools for Core ML model conversion, editing, and validation.

? Open-Source Evaluation & Testing for AI & LLM systems

State of the Art Natural Language Processing

Deepchecks: Tests for Continuous Validation of ML Models & Data. Deepchecks is a holistic open-source solution for all of your AI & ML validation needs, enabling to thoroughly test your data and models from research to production.

A high-level machine learning and deep learning library for the PHP language.

Petastorm library enables single machine or distributed training and evaluation of deep learning models from datasets in Apache Parquet format. It supports ML frameworks such as Tensorflow, Pytorch, and PySpark and can be used from pure Python code.

Apache Spark & Python (pySpark) tutorials for Big Data Analysis and Machine Learning as IPython / Jupyter notebooks

Hopsworks - Data-Intensive AI platform with a Feature Store

Starter code to solve real world text data problems. Includes: Gensim Word2Vec, phrase embeddings, Text Classification with Logistic Regression, word count with pyspark, simple text preprocessing, pre-trained embeddings and more.

PySpark + Scikit-learn = Sparkit-learn

Automatically find issues in image datasets and practice data-centric computer vision.

Perception toolkit for sim2real training and validation in Unity

The toolkit to test, validate, and evaluate your models and surface, curate, and prioritize the most valuable data for labeling.

A full pipeline AutoML tool for tabular data

A Comprehensive Framework for Building End-to-End Recommendation Systems with State-of-the-Art Models

Distributed scikit-learn meta-estimators in PySpark