Popular GitHub Repositories for Md Crawler

Your AI second brain. Self-hostable. Get answers from the web or your docs. Build custom agents, schedule automations, do deep research. Turn any online or local LLM into your personal, autonomous AI (gpt, claude, gemini, llama, qwen, mistral). Get started - free.

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Use LLMs to dig out what you care about from massive amounts of information and a variety of sources daily.

A community-driven way to read and chat with AI bots - powered by chatGPT.

Leaked GPTs Prompts Bypass the 25 message limit or to try out GPTs without a Plus subscription.

README file generator, powered by AI.

Flexible Node.js AI-assisted crawler library

Basic Memory is a knowledge management system that allows you to build a persistent semantic graph from conversations with AI assistants. All knowledge is stored in standard Markdown files on your computer, giving you full control and ownership of your data. Integrates directly with Obsidan.md

AnyCrawl ?: A Node.js/TypeScript crawler that turns websites into LLM-ready data and extracts structured SERP results from Google/Bing/Baidu/etc. Native multi-threading for bulk processing.

A scalable, mature and versatile web crawler based on Apache Storm

An Obsidian plugin to interact with your privacy focused AI-Assistant making your second brain even smarter!

Automate webpages at scale, scrape web data completely and accurately with high performance, distributed AI-RPA.

ChatWeb can crawl web pages, read PDF, DOCX, TXT, and extract the main content, then answer your questions based on the content, or summarize the key points.

Social media (Weibo) comments analyzing toolbox in Chinese 微博评论分析工具, 实现功能: 1.微博评论数据爬取; 2.分词与关键词提取; 3.词云与词频统计; 4.情感分析; 5.主题聚类

A script to effortlessly extract your entire ChatGPT data export from JSON files to nicely-formatted markdown files.

:newspaper: Let ChatGPT Summarize Hacker News for You

End-To-End Molecular Dynamics (MD) Engine using PyTorch

Official repository for "Craw4LLM: Efficient Web Crawling for LLM Pretraining"

Medical Graph for Neo4j

Opensource Korean chatbot framework