Popular GitHub Repositories for Pyspark

Simple and Distributed Machine Learning

State of the Art Natural Language Processing

Petastorm library enables single machine or distributed training and evaluation of deep learning models from datasets in Apache Parquet format. It supports ML frameworks such as Tensorflow, Pytorch, and PySpark and can be used from pure Python code.

Apache Spark & Python (pySpark) tutorials for Big Data Analysis and Machine Learning as IPython / Jupyter notebooks

Hopsworks - Data-Intensive AI platform with a Feature Store

Starter code to solve real world text data problems. Includes: Gensim Word2Vec, phrase embeddings, Text Classification with Logistic Regression, word count with pyspark, simple text preprocessing, pre-trained embeddings and more.

PySpark + Scikit-learn = Sparkit-learn

A Comprehensive Framework for Building End-to-End Recommendation Systems with State-of-the-Art Models

Distributed scikit-learn meta-estimators in PySpark

MorphL Community Edition uses big data and machine learning to predict user behaviors in digital products and services with the end goal of increasing KPIs (click-through rates, conversion rates, etc.) through personalization

Isolation Forest on Spark

Toolkit for Apache Spark ML for Feature clean-up, feature Importance calculation suite, Information Gain selection, Distributed SMOTE, Model selection and training, Hyper parameter optimization and selection, Model interprability.

Build reliable AI and agentic applications with DataFrames

BERT, AWS RDS, AWS Forecast, EMR Spark Cluster, Hive, Serverless, Google Assistant + Raspberry Pi, Infrared, Google Cloud Platform Natural Language, Anomaly detection, Tensorflow, Mathematics

Relation Extraction using Deep learning(CNN)

Spark 2.0 Python Machine Learning examples

This project aims to build an enterprise-grade offline data warehouse solution based on e-commerce platform order data.

This repository provides a comprehensive implementation of a deep neural network-based recommendation system similar to YouTube's. The repo is organized to include the core Python implementation of the model and a Spark-based Scala solution for data generation and model serving.

Inference in Bayesian Belief Networks using Probability Propagation in Trees of Clusters (PPTC) and Gibbs sampling