Breadth-first web crawler & offline source asset reconstructor
WebSource Harvester (Web Crawler)
WebSource Harvester is an educational web crawling and source reconstruction framework engineered in Python. It executes depth-controlled Breadth-First Search (BFS) traversals across websites to capture client-visible static assets including HTML documents, linked stylesheets, scripts, media, fonts, XML sitemaps, and downloadable PDFs. To ensure complete offline fidelity, the engine dynamically recalculates and rewrites resource URI links relative to each document's local directory hierarchy, ensuring nested routes (such as /blog/*) render offline without broken dependencies. The tool strictly enforces same-origin boundaries and respects server trust models without attempting intrusive backend access.
Problem
Offline web archiving tools frequently break on nested URL paths by using root-relative path links, or scrape uncontrollably without precise link-hop depth limits.
Solution
A Python crawler with file-relative path rewriting, configurable link-depth limits (--depth 0, 1, 2, 1-2), BFS queue traversal, same-origin enforcement, URL decoding, and URL export capabilities.
Outcome
A dependable client-side web analysis and asset harvesting engine demonstrating Python web scraping, BeautifulSoup DOM manipulation, path normalization, and clean architecture.
Standout Features
- Depth-controlled BFS crawling (depth 0, 1, 2, or custom ranges) with same-origin gates
- Directory-aware relative path rewriting ensuring nested HTML routes load offline
- Comprehensive asset harvesting: HTML, CSS, JS, media, fonts, srcset variants, and PDFs
- Robust parsing with URL decoding, query collision hashing, and --export-urls list generation
Stack & Signals
- Python 3.10+
- BeautifulSoup4
- lxml
- urllib / requests
- Regex
- GitHub Actions
Canonical destination