End-to-End-Data-Pipeline

Public

? A scalable, production-ready data pipeline for real-time streaming & batch processing, integrating Kafka, Spark, Airflow, AWS, Kubernetes, and MLflow. Supports end-to-end data ingestion, transformation, storage, monitoring, and AI/ML serving with CI/CD automation using Terraform & GitHub Actions.

airflow apache docker elasticsearch flink grafana great-expectations hadoop influxdb kafka

Creat：2025-02-15T13:26:35

Update：2025-03-23T10:56:25

https://hoangsonww.github.io/End-to-End-Data-Pipeline/

Stars

Stars Increase

Related projects

N8n

Hot

Fair-code workflow automation platform with native AI capabilities. Combine visual building with custom code, self-host or cloud, 400+ integrations.

125288

4年前

+718today

Comfyui

Hot

ComfyUI docker images for use in GPU cloud and local environments. Includes AI-Dock base for authentication and improved user experience.

83946

4个月前

+196today

Netdata

alerting

X-Ray Vision for your infrastructure!

75265

4个月前

+32today

Uptime Kuma

Hot

docker

A fancy self-hosted monitoring tool

72524

4个月前

+93today

Superset

analytics

Apache Superset is a Data Visualization and Data Exploration Platform

67334

4个月前

+25today

Echarts

apache

Apache ECharts is a powerful, interactive charting and data visualization library for browser

64090

4个月前

+16today

Traefik

consul

The Cloud Native Application Proxy

55800

5个月前

+23today

Appwrite

android

Build like a team of hundreds_

52035

4个月前

+6today

Gitea

bitbucket

Git with a cup of tea! Painless self-hosted all-in-one software development service, including Git hosting, code review, team collaboration, package registry and CI/CD

49797

4个月前

+26today