Moorcheh

A distributed, high-concurrency crawler and task scheduler written in Go for large-scale data extraction.

Moorcheh operates as a distributed engine for managing massive crawling workloads across multiple nodes. It leverages Redis for task queuing and state persistence, enabling horizontal scaling as data demands increase. The system includes native support for proxy rotation and custom middleware: these features bypass rate limits and maintain high throughput for complex scraping (indexing over 5,000 pages per minute).

https://github.com/moorcheh/moorcheh