Commit Graph

7 Commits

Author SHA1 Message Date
√(noham)²
5615658452 Add macserialjunkie spider and SQLite pipeline
Fork and refactor project for scraping macserialjunkie.com: add a new phpBB spider (uses python-dotenv for credentials, form login enabled, multiple start_urls, robust ID/time/text extraction and pagination) and an SQLitePipeline that saves posts to posts.db with a tqdm progress bar. Update settings to use the SQLite pipeline, increase concurrency, reduce download delay, disable robots.txt, set JOBDIR for resume and silence logs; add .env.example and .python-version, update README and requirements (add tqdm), tidy .gitignore, and add pyproject.toml. Also reorganize package layout (rename/move phpBB_scraper modules), remove legacy pipeline and old spider implementations, and add a dependency lock file (uv.lock).
2026-01-31 13:30:41 +01:00
David Ascienzo
29cb84e271 Debugged FormRequest for Authentication 2020-04-21 18:49:20 -04:00
Dascienz
e1172ff8cd Refactoring and README updates 2020-02-05 21:12:01 -05:00
Dascienz
28185eb7be gitignore amendments, indentation and spacing 2019-08-18 13:49:08 -04:00
David Ascienzo
76df7378f3 Updated README.md 2018-08-19 15:53:45 -04:00
David Ascienzo
bf2fe60993 Updated README.md 2018-08-19 15:51:11 -04:00
David Ascienzo
01fcfb586b Uploading revised phpBB forum scraping code. 2018-08-19 15:47:44 -04:00