Best Open Source webscraping Libraries

A curated list of the most popular GitHub repositories tagged with webscraping. Select any project to visualize its architecture and dive into the codebase using RepoMind's AI engine.

#1firecrawl/firecrawl

🔥 The Web Data API for AI - Turn entire websites into LLM-ready markdown or structured data

84,470TypeScript

Analyze Code

#2huginn/huginn

Create agents that monitor and act on your behalf. Your agents are standing by!

48,730Ruby

Analyze Code

#3assafelovic/gpt-researcher

An autonomous agent that conducts deep research on any data using any LLM providers.

25,371Python

Analyze Code

#4ScrapeGraphAI/Scrapegraph-ai

Python scraper based on AI

22,734Python

Analyze Code

#5D4Vinci/Scrapling

🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl!

9,101Python

Analyze Code