insect-observation-data-pipeline

Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.

ClickHouse? is a real-time analytics database management system

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

DuckDB is an analytical in-process SQL database management system

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.

QuestDB is a high performance, open-source, time-series database

Apache Doris is an easy-to-use, high performance and unified analytics database.

An orchestration platform for the development, production, and observation of data assets.

The world's fastest open query engine for sub-second analytics both on and off the data lakehouse. With the flexibility to support nearly any scenario, StarRocks provides best-in-class performance for multi-dimensional analytics, real-time analytics, and ad-hoc queries. A Linux Foundation project.

OceanBase is an enterprise distributed relational database with high availability, high performance, horizontal scalability, and compatibility with SQL standards.