mirror of
https://github.com/NohamR/phpBB-forum-scraper.git
synced 2026-02-21 18:15:43 +00:00
5615658452
Fork and refactor project for scraping macserialjunkie.com: add a new phpBB spider (uses python-dotenv for credentials, form login enabled, multiple start_urls, robust ID/time/text extraction and pagination) and an SQLitePipeline that saves posts to posts.db with a tqdm progress bar. Update settings to use the SQLite pipeline, increase concurrency, reduce download delay, disable robots.txt, set JOBDIR for resume and silence logs; add .env.example and .python-version, update README and requirements (add tqdm), tidy .gitignore, and add pyproject.toml. Also reorganize package layout (rename/move phpBB_scraper modules), remove legacy pipeline and old spider implementations, and add a dependency lock file (uv.lock).
phpBB Forum Scraper
This is a fork of phpBB-forum-scraper customized for scraping macserialjunkie.com.
Python-based web scraper for phpBB forums. Project can be used as a template for building your own custom Scrapy spiders or for one-off crawls on designated forums. Please keep in mind that aggressive crawls can produce significant strain on web servers, so please throttle your request rates.
Requirements:
- Python web scraping library, Scrapy.
- Python HTML/XML parsing library, BeautifulSoup.
Scraper Output
The phpBB.py spider scrapes the following information from forum posts:
- Username
- User Post Count
- Post Date & Time
- Post Text
- Quoted Text
If you need additional data scraped, you will have to create additional spiders or edit the existing spider.
Edit phpBB.py and Specify:
allowed_domainsstart_urlsusername&passwordforum_login=Falseorforum_login=True
Running the Scraper:
cd phpBB_scraper/
scrapy crawl phpBB
uv run scrapy crawl phpBB -s JOBDIR=crawls/resume-002
NOTE: Please adjust settings.py to throttle your requests.