back to home

Best Open Source spider Libraries

A curated list of the most popular GitHub repositories tagged with spider. Select any project to visualize its architecture and dive into the codebase using RepoMind's AI engine.

#1NaiboWang/EasySpider

A visual no-code/code-free web crawler/spider易采集：一个可视化浏览器自动化测试/数据采集/爬虫软件，可以无代码图形化的设计和执行爬虫任务。别名：ServiceWrapper面向Web应用的智能化服务封装系统。

44,099JavaScript
Analyze Code

#2gocolly/colly

Elegant Scraper and Crawler Framework for Golang

25,102Go
Analyze Code

#3jhao104/proxy_pool

Python ProxyPool for web spider

23,156Python
Analyze Code

#4shengqiangzhang/examples-of-web-crawlers

一些非常有趣的python爬虫例子,对新手比较友好,主要爬取淘宝、天猫、微信、微信读书、豆瓣、QQ等网站。(Some interesting examples of python crawlers that are friendly to beginners. )

14,574HTML
Analyze Code

#5s0md3v/Photon

Incredibly fast crawler designed for OSINT.

12,687Python
Analyze Code

#6crawlab-team/crawlab

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架

12,159Go
Analyze Code

#7guyueyingmu/avbook

AV 电影管理系统， avmoo , javbus , javlibrary 爬虫，线上 AV 影片图书馆，AV 磁力链接数据库，Japanese Adult Video Library,Adult Video Magnet Links - Japanese Adult Video Database

9,909PHP
Analyze Code

#8ihmily/DouyinLiveRecorder

可循环值守和多人录制的直播录制软件，支持抖音、TikTok、Youtube、快手、虎牙、斗鱼、B站、小红书、pandatv、sooplive、flextv、popkontv、twitcasting、winktv、百度、微博、酷狗、17Live、Twitch、Acfun、CHZZK、shopee等40+平台直播录制

9,375Python
Analyze Code

#9bda-research/node-crawler

Web Crawler/Spider for NodeJS + server-side jQuery ;-)

6,785TypeScript
Analyze Code