Popular GitHub Repositories for Chunks

Video-based AI memory library. Store millions of text chunks in MP4 files with lightning-fast semantic search. No database needed.

ChatGPT PROMPTs Splitter. Tool for safely process chunks of up to 15,000 characters per request

A fast, lightweight and easy-to-use Python library for splitting text into semantically meaningful chunks.

? GUI for editing LLM vector embeddings. No more blind chunking. Upload content in any file extension, join and split chunks, edit metadata and embedding tokens + remove stop-words and punctuation with one click, add images, and download in .veml to share it with your team.

BERT classification model for processing texts longer than 512 tokens. Text is first divided into smaller chunks and after feeding them to BERT, intermediate results are pooled. The implementation allows fine-tuning.

? semantic-chunking ? semantically create chunks from large document for passing to LLM workflows

smart-llm-loader is a lightweight yet powerful Python package that transforms any document into LLM-ready chunks. Spend less time on preprocessing headaches and more time building what matters. From RAG systems to chatbots to document Q&A, SmartLLMLoader handles the heavy lifting so you can focus on creating exceptional AI applications.

A powerful MCP server that enables Claude to discover and call any API endpoint through semantic search. Intelligently chunks OpenAPI specifications to handle large API documentation, with built-in request execution capabilities. Perfect for integrating private APIs with Claude Desktop.

A Python tool for splitting large Markdown files into smaller sections based on a specified token limit. This is particularly useful for processing large Markdown files with GPT models, as it allows the models to handle the data in manageable chunks.

Python scripts that converts PDF files to text, splits them into chunks, and stores their vector representations using GPT4All embeddings in a Chroma DB. It also provides a script to query the Chroma DB for similarity search based on user input.

Translate Markdown files from one language to another using OpenAI's API while retaining original formatting. This Jupyter notebook tokenizes input text, splits into chunks, translates with OpenAI, and reconstructs output to preserve Markdown structure. Useful for localizing documentation, articles, books, and other long-form Markdown content.

Culturally-compliant video storage. Embeds searchable text chunks into pixelated media for lightning-fast semantic search. Zero-database, maximum compliance.

CLI for semantic search on your computer. Searches text files and identifies the most relevant chunks to your query.

The project is a Streamlit-based app that allows users to upload PDF files, processes the text into chunks, stores it in a FAISS vector store using Google Generative AI embeddings, and enables conversational Q&A with the documents using Google Generative AI’s chat model for answers.

Semantic Chunking is a Python library for segmenting text into meaningful chunks using embeddings from Sentence Transformers.

Weaviate Lexplorer is an open-source app for discovering Lex Fridman's podcast insights using Next.js for the GUI, Ollama for running LLMs locally and the Weaviate vector database local instance via Docker for storing transcription chunks

?‍♂️ MagicXML is a FastAPI-based service designed to fetch, process, and convert XML data into structured CSV files. It is optimized for handling large XML files by processing them in chunks asynchronously, making it suitable for heavy data processing tasks.

Analyzing the performance of Mamba-based vs. Transformer-based Retrieval Augmented Language Models over increasing k number of chunks retrieved.

A Small project for Telegram BOT, providing concise, digestible chunks of news updates from various sources as per desired news topics

Python-based system designed to transcribe audio files, split the transcripts into manageable chunks, create text embeddings using HuggingFace models, and employ advanced question-answering models for retrieval-based QA.