Discord Join Discord
☰

I crawled 133K pages
on a home PC.

One developer. Rust. A tiered anti-bot crawler that escalates from HTTP to headless Chrome when it gets blocked. 37 GB on spinning HDDs. ₹0 infrastructure spend. Every dataset free and open.

Browse Datasets See How It's Built ↓

0

Pages Crawled

0

Public Datasets

0

On Spinning HDDs

₹0

Cloud Spend

How It's Built

No Lambda functions. No managed services. Just a Rust binary, spinning HDDs, and stubbornness.

TIER 1 — HTTP

rquest with Chrome TLS/JA4 fingerprint impersonation. Looks like a real browser at the TCP level. Handles 85% of pages.

TIER 3 — BROWSER

Headless Chromium via raw CDP. Auto-escalates when Tier 1 gets blocked by JS SPAs. Browser auto-relaunches if Chrome dies.

STORAGE

LMDB-backed URL queue. Parquet persistence. Disk-space watchdog kills the crawl before your drive fills up. Graceful SIGTERM shutdown.

queen.log — qween_crawler
// Live from qween_crawler — real log output
[WARN] tier-1 blocked → escalating to tier-3 url="https://cloud.google.com/ai/generative-ai"
reason=phase-b:heavy-js-spa-script-count budget_left_ms=19253
[WARN] site redesign detected — content hash changed domain="google.com"
[ERROR] fetch failed worker_id=29 url=https://cloud.google.com/apigee/docs/getstarted
error=fetch timeout after 15s
[INFO] crawled=133,812 | errored=3,313 | frontier=634,632 | inflight=47
bytes_total=37,501,408,394 | tier1=114,342 | tier3=672

Yes, those are real errors. The crawler handles them and keeps going.

The Machine

Everything runs on one desktop.

Intel i7-12700F (20 threads) · GTX 1650 · 16 GB RAM · Kali Linux · 360 GB HDD + 930 GB external. The hostname is WannaCry. No cloud. No GPUs rented. Just patience and spinning rust.

┌──(adhyansh㉿WannaCry)-[~]
└─$ fastfetch --logo none
CPU: i7-12700F (16+4) @ 4.90 GHz
GPU: NVIDIA GTX 1650
Memory: 9.42 GiB / 15.46 GiB (61%)
Disk (/home): 105 GiB / 358 GiB — ext4 HDD
Disk (ext): 123 GiB / 931 GiB — exfat HDD
OS: Kali GNU/Linux Rolling
Uptime: 5 hours, 33 mins

Let's Connect

Want to request a custom crawl? Need a specific dataset processed? Or just want to talk about Rust and web scraping?

Email Me Discord Join Discord

Built by

Adhyansh Verma, 16.

I started crawling the web because I noticed that most open datasets for AI are either ancient Common Crawl dumps or gated behind corporate APIs. The first version of my crawler created so many files the OS filesystem choked. So I rewrote everything in Rust around Parquet — and kept going.

Free for research & open source. Commercial use? Custom crawls? Let's talk.

Known limitation: forum boilerplate isn't fully stripped from some web crawls yet. Working on it.