A simple Python-based website crawler that lets you input a URL, crawl all internal links on that domain, display results in a GUI, and export them to a CSV file.
This project uses Tkinter for the interface and BeautifulSoup + Requests for web crawling.
- 🖥️ Clean GUI built with Tkinter
- 🌐 Crawls all internal links within the same domain
- 📊 Progress bar and live status updates while crawling
- 💾 Saves all discovered URLs to
links.csvin the project folder - ⚙️ Can be packaged into a standalone Windows
.exewith PyInstaller
- Install dependencies:
pip install requests beautifulsoup4
Run the app:
bash Copy code python site_crawler_gui.py Enter any website URL (e.g., https://example.com) and click Start Crawling.
All discovered links will appear in the results box and be saved to links.csv.
🧱 Build a Windows App (Optional) To build an .exe version (no console window):
bash Copy code pyinstaller --noconsole --onefile "site_crawler_gui.py" You’ll find the executable in the dist/ folder.
📂 Project Structure bash Copy code WebCrawler/ │ ├── site_crawler_gui.py # Main application file ├── links.csv # Generated after crawling ├── README.md # Project overview (this file) └── CHANGELOG.md # Version history and updates 🧑💻 Requirements Python 3.9+
Packages: requests, beautifulsoup4