√(noham)²
5615658452
Add macserialjunkie spider and SQLite pipeline
...
Fork and refactor project for scraping macserialjunkie.com: add a new phpBB spider (uses python-dotenv for credentials, form login enabled, multiple start_urls, robust ID/time/text extraction and pagination) and an SQLitePipeline that saves posts to posts.db with a tqdm progress bar. Update settings to use the SQLite pipeline, increase concurrency, reduce download delay, disable robots.txt, set JOBDIR for resume and silence logs; add .env.example and .python-version, update README and requirements (add tqdm), tidy .gitignore, and add pyproject.toml. Also reorganize package layout (rename/move phpBB_scraper modules), remove legacy pipeline and old spider implementations, and add a dependency lock file (uv.lock).
2026-01-31 13:30:41 +01:00
Dave Ascienzo
d0178052c9
Merge pull request #11 from Dascienz/developer
...
Developer
2020-05-28 21:28:47 -04:00
David Ascienzo
12ad5b5f11
Additional support on xpaths for poor phpBB formatting
2020-05-28 21:29:52 -04:00
David Ascienzo
7b850d872d
Additional support on xpaths for poor phpBB formatting
2020-05-28 21:29:06 -04:00
David Ascienzo
3f6be6c3b6
Removed unused imports
2020-04-21 18:59:53 -04:00
Dave Ascienzo
41b21bc0d1
Merge pull request #9 from Dascienz/developer
...
Removed unused imports
2020-04-21 18:58:56 -04:00
Dave Ascienzo
97b4cc8dcb
Merge pull request #8 from Dascienz/dev
...
Removed .idea/ from repo
2020-04-21 18:52:14 -04:00
David Ascienzo
4b4a3f6f09
Removed .idea/ from repo
2020-04-21 18:53:17 -04:00
David Ascienzo
86e6f6a62a
.gitignore updates
2020-04-21 18:51:35 -04:00
Dave Ascienzo
65c8eaf738
Merge pull request #7 from Dascienz/dev
...
.gitignore updates
2020-04-21 18:50:47 -04:00
David Ascienzo
29cb84e271
Debugged FormRequest for Authentication
2020-04-21 18:49:20 -04:00
Dave Ascienzo
96548fff9c
Merge pull request #6 from Dascienz/dev
...
Debugged FormRequest for Authentication
2020-04-21 18:48:36 -04:00
Dave Ascienzo
29dd97420f
Merge pull request #4 from Dascienz/dev
...
Refactoring and README updates
2020-02-05 21:12:19 -05:00
Dascienz
e1172ff8cd
Refactoring and README updates
2020-02-05 21:12:01 -05:00
Dave Ascienzo
584d362856
Merge pull request #2 from ScottMnemonic/master
...
https://github.com/Dascienz/phpBB-forum-scraper/issues/1
2019-09-24 11:26:46 -04:00
Mnemonic
c579babe4f
Added clarifiers and fixed the return
2019-09-24 02:23:01 -07:00
Dascienz
e387766d63
gitignore updates
2019-08-19 20:32:13 -04:00
Dascienz
2bd8a19ca0
Fixed QuoteText tags and added pyc files to gitignore
2019-08-19 20:02:33 -04:00
Dascienz
28185eb7be
gitignore amendments, indentation and spacing
2019-08-18 13:49:08 -04:00
David Ascienzo
76df7378f3
Updated README.md
2018-08-19 15:53:45 -04:00
David Ascienzo
bf2fe60993
Updated README.md
2018-08-19 15:51:11 -04:00
David Ascienzo
01fcfb586b
Uploading revised phpBB forum scraping code.
2018-08-19 15:47:44 -04:00
