Popular GitHub Repositories for Dags

Build applications that make decisions (chatbots, agents, simulations, etc...). Monitor, trace, persist, and execute on your own infrastructure.

Python package for the creation, manipulation, and learning of Causal DAGs

? LyricWave - Your AI Music Composer ? Compose Unique MP4 Songs Effortlessly! LyricWave uses AI to create personalized music by harmonizing lyrics with captivating melodies and synthetic vocals. Unleash your musical creativity today! ??

WebApp to bring together Text Summarization and Sentiment Analysis of the stock related news to better understand the stock price trends.

DAGs on DAGs! Smart PySpark custom transformation runner

LangDAG is a specialized orchestration framework for building LLM (Large Language Model) agent workflows using DAGs(Directed Acyclic Graphs), written in Python.

Run DAGs of MLflow Projects in Kubernetes using Concurrent for MLflow

Airflow DAGs repository synced for the MLOps plattform of the repository mlops-eks-mlplatform

This repository contains the codebase for the BuildItAll Big Data Processing Platform, a case study project designed to manage large daily data for a hypothetical Belgian client.

A comprehensive ETL pipeline for analyzing passenger satisfaction data. Features a modern data architecture with Apache Airflow for extraction, dbt/Snowflake for transformation, Python/Pandas for cleaning, and interactive dashboards for visualization with NextJS.

EPL Fotmob Data Visualizations

A simple demonstration of an Airflow-Kafka-Spark (AKS) stack for online time series forecasting.

In this project, you will create an end-to-end Airflow pipeline, integrated with MLflow, for CodePro, an EdTech startup, to perform lead scoring and maximize profitability while minimizing the Customer Acquisition Cost (CAC). The assignment involves data collection, preprocessing, and periodic model retraining within Airflow, alongside development

Preparing for Airlfow Certification - Testing Knowledge -> New Connection of Neurons in Brain with Training++

Airflow Pipeline for Lead Scoring to Maximize Profit with retraining pipeline and Development experimentation using mlflow

This is an end-to-end AWS Cloud ETL project. This data pipeline orchestration uses Apache Airflow on AWS EC2 as well as AWS Lambda. It demonstrates how to build ETL data pipeline that would perform data transformation using Lambda function as well as loading into a Redshift cluster table. The data would then be visualized using Amazon QuickSight.

This is an end-to-end AWS Cloud ETL project. This orchestration uses Apache Airflow on AWS EC2 as well as AWS Glue. It demonstrates how to build ETL pipeline that would perform data transform using Glue job/crawler as well as loading into a Redshift table. It also shows how to connect Amazon Athena to Glue Data Catalog, and Power BI to Redshift.

This is an end-to-end AWS Cloud ETL project. This data pipeline orchestration uses Apache Airflow on AWS EC2 as well as Snowpipe. It demonstrates how to build ETL data pipeline that would perform data transformation using Python on Apache Airflow as well as automatic ingestion into Snowflake data warehouse via Snowpipe. Also features Power BI.

Automation of Iris flower classes Mlflow experimental logging and prediction

Personal project aimed at developing a ML service which resembles a production environment system