Machine-Learning-Pipeline-LR-Pyspark

Open source platform for the machine learning lifecycle

the portable Python dataframe library

Simple and Distributed Machine Learning

酷玩 Spark: Spark 源代码解析、Spark 类库等

Apache Linkis builds a computation middleware layer to facilitate connection, governance and orchestration between the upper applications and the underlying data engines.

Interactive and Reactive Data Science using Scala and Spark.

Kubernetes operator for managing the lifecycle of Apache Spark applications on Kubernetes.

BigDL: Distributed TensorFlow, Keras and PyTorch on Apache Spark/Flink & Ray

Implementing best practices for PySpark ETL jobs and applications.

Petastorm library enables single machine or distributed training and evaluation of deep learning models from datasets in Apache Parquet format. It supports ML frameworks such as Tensorflow, Pytorch, and PySpark and can be used from pure Python code.