Popular GitHub Repositories for Airflow

PipelineAI

A Data Engineering & Machine Learning Knowledge Hub

An end-to-end LLM reference implementation providing a Q&A interface for Airflow and Astronomer

Construct a modern data stack and orchestration the workflows to create high quality data for analytics and ML applications.

Apache Liminals goal is to operationalise the machine learning process, allowing data scientists to quickly transition from a successful experiment to an automated pipeline of model training, validation, deployment and inference in production. Liminal provides a Domain Specific Language to build ML workflows on top of Apache Airflow.

A lightweight tool to get an AI Infrastructure Stack up in minutes not days. K3ai will take care of setup K8s for You, deploy the AI tool of your choice and even run your code on it.

Repo that relates to the Medium blog 'Keeping your ML model in shape with Kafka, Airflow' and?MLFlow'

Full-stack Highly Scalable Cloud-native Machine Learning system for demand forecasting with realtime data streaming, inference, retraining loop, and more

Arquitetura CRM de Baixo Custo com Gen AI, projetada para startups que precisam processar e analisar dados de vendas de forma eficiente.

Airflow Pipeline for Machine Learning

Integrating Airbyte, Kafka, Airflow and MLflow on Azure Linux VMs within private network to continuously retrain LSTM Attention model with 1-minute stock prices and redeploy it on Azure ML AKS real-time inference endpoint behind serverless Azure API

? A scalable, production-ready data pipeline for real-time streaming & batch processing, integrating Kafka, Spark, Airflow, AWS, Kubernetes, and MLflow. Supports end-to-end data ingestion, transformation, storage, monitoring, and AI/ML serving with CI/CD automation using Terraform & GitHub Actions.

Kafka variant of the MLOps Level 1 stack

An airflow provider for anomaly detection.

Real Time Twitter Sentiment Analysis Product

A scalable, declarative, low-code framework for real-time and batch feature calculation/management (quant finance, anomaly/fraud detection, etc.), predictive ML training/inference and simulation. Built on top of Ray

This is an overview of a MLOps architecture that includes both Airflow and MLflow running on separate Docker containers.

An example of setting up local audio ML training pipeline on Airflow with MLFlow experiment tracking and custom python library.

:shark: Airflow plugin to scale machine learning tasks with Valohai and get automatic version control

End-to-End deployment of E-commerce customers segmentation using Clustering Machine learning algorithms in Google Cloud Platform and MLOps Tools