Popular GitHub Repositories for Multimodal Synthesis

The all-in-one Desktop & Docker AI application with built-in RAG, AI agents, No-code agent builder, and more.

?? - a deep learning toolkit for Text-to-Speech, battle-tested in research and production

Agno is a lightweight library for building Multimodal Agents. It exposes LLMs as a unified API and gives them superpowers like memory, knowledge, tools and reasoning.

[NeurIPS'23 Oral] Visual Instruction Tuning (LLaVA) built towards GPT-4V level capabilities and beyond.

?? Build multimodal AI applications with cloud-native stack

Large-scale Self-supervised Pre-training Across Tasks, Languages, and Modalities

MiniCPM-o 2.6: A GPT-4o Level MLLM for Vision, Speech and Multimodal Live Streaming on Your Phone

? Leon is your open-source personal assistant.

:sparkles::sparkles:Latest Advances on Multimodal Large Language Models

A scalable generative AI framework built for researchers and developers working on Large Language Models, Multimodal, and Speech AI (Automatic Speech Recognition and Text-to-Speech)

State-of-the-Art Deep Learning scripts organized by models - easy to train and deploy with reproducible accuracy and performance on enterprise-grade infrastructure.

MNN is a blazing fast, lightweight deep learning framework, battle-tested by business-critical use cases in Alibaba. Full multimodal LLM Android App:[MNN-LLM-Android](./apps/Android/MnnLlmChat/README.md)

Easy-to-use Speech Toolkit including Self-Supervised Learning model, SOTA/Streaming ASR with punctuation, Streaming TTS with text frontend, Speaker Verification System, End-to-End Speech Translation and Keyword Spotting. Won NAACL2022 Best Demo Award.

Implementation of DALL-E 2, OpenAI's updated text-to-image synthesis neural network, in Pytorch

A framework to enable multimodal models to operate a computer.

A fast, local neural text to speech system

End-to-End Speech Processing Toolkit

Amphion (/?m?fa??n/) is a toolkit for Audio, Music, and Speech Generation. Its purpose is to support reproducible research and help junior researchers and engineers get started in the field of audio, music, and speech generation research and development.

Use PEFT or Full-parameter to finetune 450+ LLMs (Qwen2.5, InternLM3, GLM4, Llama3.3, Mistral, Yi1.5, Baichuan2, DeepSeek-R1, ...) and 150+ MLLMs (Qwen2.5-VL, Qwen2-Audio, Llama3.2-Vision, Llava, InternVL2.5, MiniCPM-V-2.6, GLM4v, Xcomposer2.5, Yi-VL, DeepSeek-VL2, Phi3.5-Vision, GOT-OCR2, ...).

SeaTunnel is a multimodal, high-performance, distributed, massive data integration tool.