Popular GitHub Repositories for Mulit Modality

[NeurIPS'23 Oral] Visual Instruction Tuning (LLaVA) built towards GPT-4V level capabilities and beyond.

:sparkles::sparkles:Latest Advances on Multimodal Large Language Models

? Scalable embedding, reasoning, ranking for images and sentences with CLIP

a state-of-the-art-level open visual language model | 多模态预训练模型

The Enterprise-Grade Production-Ready Multi-Agent Orchestration Framework. Website: https://swarms.ai

Align Anything: Training All-modality Model with Feedback

Simple command line tool for text to image generation using OpenAI's CLIP and Siren (Implicit neural representation network). Technique was originally created by https://twitter.com/advadnoun

Official repo for "Mini-Gemini: Mining the Potential of Multi-modality Vision Language Models"

? Otter, a multi-modal model based on OpenFlamingo (open-sourced version of DeepMind's Flamingo), trained on MIMIC-IT and showcasing improved instruction-following and in-context learning ability.

InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactions

Open-source evaluation toolkit of large multi-modality models (LMMs), support 220+ LMMs, 80+ benchmarks

Lumina-T2X is a unified framework for Text to Any Modality Generation

[ICLR 2024] Official implementation of " ? Time-LLM: Time Series Forecasting by Reprogramming Large Language Models"

[ACL 2024 ?] Video-ChatGPT is a video conversation model capable of generating meaningful conversation about videos. It combines the capabilities of LLMs with a pretrained visual encoder adapted for spatiotemporal video representation. We also introduce a rigorous 'Quantitative Evaluation Benchmarking' for video-based conversational models.

Awesome Person Re-identification

【ICLR 2024?】 Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment

Parsing-free RAG supported by VLMs

Long-RL: Scaling RL to Long Sequences

Chatbot Arena meets multi-modality! Multi-Modality Arena allows you to benchmark vision-language models side-by-side while providing images as inputs. Supports MiniGPT-4, LLaMA-Adapter V2, LLaVA, BLIP-2, and many more!

Multi-modality pre-training