Popular GitHub Repositories for Common Crawl

? Turn entire websites into LLM-ready markdown or structured data. Scrape, crawl and extract with a single API.

Repository to track the progress in Natural Language Processing (NLP), including the datasets and the current state-of-the-art for the most common NLP tasks.

Crawl a site to generate knowledge files to create your own custom GPT from a URL

common data analysis and machine learning tasks using python

Intelligent (edge and LLM) proxy for agents. Move faster by letting Arch handle the pesky heavy lifting in building agentic apps. Fast ?? task clarification, agent routing, seamless integration of prompts with tools for common tasks, and unified access and observability of LLMs.

ManimML is a project focused on providing animations and visualizations of common machine learning concepts with the Manim Community Library.

An AI-powered data science team of agents to help you perform common data science tasks 10X faster.

A collection of production-ready Generative AI Agent templates built for Google Cloud. It accelerates development by providing a holistic, production-ready solution, addressing common challenges (Deployment & Operations, Evaluation, Customization, Observability) in building and deploying GenAI agents.

A Unity library for common movement AI

Flexible Node.js AI-assisted crawler library

Common interface for interacting with AI agents. The protocol is tech stack agnostic - you can use it with any framework for building agents.

A Solution Accelerator for the RAG pattern running in Azure, using Azure AI Search for retrieval and Azure OpenAI large language models to power ChatGPT-style and Q&A experiences. This includes most common requirements and best practices.

This tool will daily crawl https://arxiv.org and use LLMs to summarize them.

ChatWeb can crawl web pages, read PDF, DOCX, TXT, and extract the main content, then answer your questions based on the content, or summarize the key points.

Evaluating and reproducing real-world robot manipulation policies (e.g., RT-1, RT-1-X, Octo) in simulation under common setups (e.g., Google Robot, WidowX+Bridge) (CoRL 2024)

(deprecated) A system for making generative systems

Collective Knowledge (CK), Collective Mind (CM) and Common Metadata eXchange (CMX): community-driven projects to facilitate collaborative and reproducible research and to learn how to run AI, ML, and other emerging workloads more efficiently and cost-effectively across diverse models, datasets, software, and hardware using MLPerf.

Python API client for AI providers that intends to replace LangChain and LangGraph for most common use cases.

AWS Generative AI CDK Constructs are sample implementations of AWS CDK for common generative AI patterns.

PyNLPl, pronounced as 'pineapple', is a Python library for Natural Language Processing. It contains various modules useful for common, and less common, NLP tasks. PyNLPl can be used for basic tasks such as the extraction of n-grams and frequency lists, and to build simple language model. There are also more complex data types and algorithms. Moreover, there are parsers for file formats common in NLP (e.g. FoLiA/Giza/Moses/ARPA/Timbl/CQL). There are also clients to interface with various NLP specific servers. PyNLPl most notably features a very extensive library for working with FoLiA XML (Format for Linguistic Annotation).