mirror of
https://github.com/NohamR/phpBB-forum-scraper.git
synced 2026-02-21 18:15:43 +00:00
Fork and refactor project for scraping macserialjunkie.com: add a new phpBB spider (uses python-dotenv for credentials, form login enabled, multiple start_urls, robust ID/time/text extraction and pagination) and an SQLitePipeline that saves posts to posts.db with a tqdm progress bar. Update settings to use the SQLite pipeline, increase concurrency, reduce download delay, disable robots.txt, set JOBDIR for resume and silence logs; add .env.example and .python-version, update README and requirements (add tqdm), tidy .gitignore, and add pyproject.toml. Also reorganize package layout (rename/move phpBB_scraper modules), remove legacy pipeline and old spider implementations, and add a dependency lock file (uv.lock).
20 lines
178 B
Plaintext
20 lines
178 B
Plaintext
# Exclude Files
|
|
*.DS_Store
|
|
*.json
|
|
*.txt
|
|
*.tsv
|
|
*.xls
|
|
*.xlsx
|
|
*.pyc
|
|
!requirements.txt
|
|
|
|
# Exclude Directories
|
|
.idea/
|
|
.vscode/
|
|
__pycache__/
|
|
.ipynb_checkpoints/
|
|
.env
|
|
|
|
/crawls
|
|
posts.db
|