mirror of
https://github.com/NohamR/phpBB-forum-scraper.git
synced 2026-02-22 02:25:43 +00:00
5615658452
Fork and refactor project for scraping macserialjunkie.com: add a new phpBB spider (uses python-dotenv for credentials, form login enabled, multiple start_urls, robust ID/time/text extraction and pagination) and an SQLitePipeline that saves posts to posts.db with a tqdm progress bar. Update settings to use the SQLite pipeline, increase concurrency, reduce download delay, disable robots.txt, set JOBDIR for resume and silence logs; add .env.example and .python-version, update README and requirements (add tqdm), tidy .gitignore, and add pyproject.toml. Also reorganize package layout (rename/move phpBB_scraper modules), remove legacy pipeline and old spider implementations, and add a dependency lock file (uv.lock).
15 lines
291 B
Python
15 lines
291 B
Python
# -*- coding: utf-8 -*-
|
|
|
|
# Define here the models for your scraped items
|
|
#
|
|
# See documentation in:
|
|
# http://doc.scrapy.org/en/latest/topics/items.html
|
|
|
|
import scrapy
|
|
|
|
|
|
class PhpbbScraperItem(scrapy.Item):
|
|
# define the fields for your item here like:
|
|
# name = scrapy.Field()
|
|
pass
|