Popular GitHub Repositories for Vlms

Unified Efficient Fine-Tuning of 100+ LLMs & VLMs (ACL 2024)

Solve Visual Understanding with Reinforced VLMs

An on-premises unstructured data extraction tool powered by vision language models.

MLX-VLM is a package for inference and fine-tuning of Vision Language Models (VLMs) on your Mac using MLX.

Open Source Generative Process Automation (i.e. Generative RPA). AI-First Process Automation with Large ([Language (LLMs) / Action (LAMs) / Multimodal (LMMs)] / Visual Language (VLMs)) Models

AI Inference Operator for Kubernetes. The easiest way to serve ML models in production. Supports VLMs, LLMs, embeddings, and speech-to-text.

Awesome-Jailbreak-on-LLMs is a collection of state-of-the-art, novel, exciting jailbreak methods on LLMs. It contains papers, codes, datasets, evaluations, and analyses.

Parsing-free RAG supported by VLMs

A curated list of 3D Vision papers relating to Robotics domain in the era of large models i.e. LLMs/VLMs, inspired by awesome-computer-vision, including papers, codes, and related websites

Advanced Quantization Algorithm for LLMs/VLMs.

A hub for various industry-specific schemas to be used with VLMs.

Official repository for VisionZip (CVPR 2025)

[CVPR'24] HallusionBench: You See What You Think? Or You Think What You See? An Image-Context Reasoning Benchmark Challenging for GPT-4V(ision), LLaVA-1.5, and Other Multi-modality Models

Official repo of VLABench, a large scale benchmark designed for fairly evaluating VLA, Embodied Agent, and VLMs.

Unleashing the Power of VLMs in Autonomous Driving via Reinforcement Learning and Reasoning

llama.cpp (GGUF LLMs) and llava.cpp (GGUF VLMs) for ROS 2

Official Repository of OmniCaptioner

Custom nodes for using fal API. Video generation with Kling, Runway, Luma. Image generation with Flux. LLMs and VLMs OpenAI, Claude, Llama and Gemini.

This repository collects research papers of large Foundation Models for Scenario Generation and Analysis in Autonomous Driving. The repository will be continuously updated to track the latest update.

An implementation of GRPO for Unsloth's VLMs training