noham/papeer
1
0
Fork 0
mirror of https://github.com/NohamR/papeer.git synced 2026-05-25 04:17:19 +00:00
Code Issues Packages Projects Releases Wiki Activity

chain selctors, depth & quiet options, split main commands

Browse Source
This commit is contained in:
lapwat
2022-02-04 19:42:27 +01:00
parent 26b144fb73
commit 4b760c9562
10 changed files with 528 additions and 279 deletions
Download Patch File Download Diff File Expand all files Collapse all files

200
README.md
View File

@@ -1,101 +1,126 @@
# Papeer
Papeer is a tool that lets you scrape content from the internet. It can scrape any web page, keeping only relevant content (formatted text and images) and removing ads and menus. You can export the content to Markdown, EPUB or MOBI files.
Papeer is a powerful an **ereader internet vacuum**. It can scrape any website, removing ads and keeping only the relevant content (formatted text and images). You can export the content to Markdown, EPUB or MOBI files.
# Table of contents
- [Usage](#usage)
- [Examples](#examples)
* [Grab a single page](#grab-a-single-page)
* [Grab several pages](#grab-several-pages)
+ [`selector` option](#-selector--option)
+ [`depth` option](#-depth--option)
+ [Display table of contents](#display-table-of-contents)
+ [Scrape time](#scrape-time)
- [Installation](#installation)
* [From source](#from-source)
* [From binary](#from-binary)
+ [Linux / MacOS](#linux---macos)
+ [Windows](#windows)
* [MOBI support (optional)](#mobi-support--optional-)
- [Autocompletion](#autocompletion)
- [Dependencies](#dependencies)
# Usage
## Scrape a web page
The `get` command lets you retrieve the content of any web page.
```
Browse the web in the eink era
Scrape URL content
Usage:
papeer [flags]
papeer [command]
papeer get URL [flags]
Available Commands:
completion generate the autocompletion script for the specified shell
get Scrape URL content
help Help about any command
ls Print table of content
version Print the version number of papeer
Examples:
papeer get https://www.eff.org/cyberspace-independence
Flags:
-a, --author string book author
-d, --delay int time to wait before downloading next chapter, in milliseconds (default -1)
-f, --format string file format [stdout, md, epub, mobi] (default "stdout")
-h, --help help for papeer
--images retrieve images only
-i, --include include URL as first chapter, in resursive mode
-l, --limit int limit number of chapters, in recursive mode (default -1)
-n, --name string book name (default: page title)
-o, --offset int skip first chapters, in recursive mode
--output string file name (default: book name)
-q, --quiet hide progress bar
-r, --recursive create one chapter per natigation item
-s, --selector string table of content CSS selector, in resursive mode
-t, --threads int download concurrency, in recursive mode (default -1)
Use "papeer [command] --help" for more information about a command.
-a, --author string book author
--delay int time in milliseconds to wait before downloading next chapter, use with depth/selector (default -1)
-d, --depth int scraping depth
-f, --format string file format [stdout, md, epub, mobi] (default "md")
-h, --help help for get
--images retrieve images only
-i, --include include URL as first chapter, use with depth/selector
-l, --limit int limit number of chapters, use with depth/selector (default -1)
-n, --name string book name (default: page title)
-o, --offset int skip first chapters, use with depth/selector
--output string file name (default: book name)
-q, --quiet hide progress bar
-s, --selector strings table of contents CSS selector
-t, --threads int download concurrency, use with depth/selector (default -1)
--use-link-name use link name for chapter title
```
# Examples
## Scrape a whole website
## Grab a single page
If a navigation menu is present on a website, you can scrape the content of each page.
The `get` command lets you retrieve the content of a web page.
You can activate this mode by using the `depth` or `selector` options.
### `depth` option
This option defaults to 0, `papeer` will grab only the main page.
If you specify a value greater than 0, `papeer` will grab only the pages as deep as the value you specify.
> Using `include` option will include all intermediary levels.
### `selector` option
If this option is not specified, `papeer` will grab only the one page.
If this option is specified, `papeer` will select the links (a HTML tag) present on the main page, then grab each one of them.
You can chain this option to grab several level of pages with diferent selectors for each level.
## Display the table of contents
Before actually scraping a whole website, it is a good idea to use the `list` command. This command is like a **dry run**, which lets you vizualize the content before actually retrieving it. You can use several options to customize the table of contents extraction, such as `selector`, `limit`, `offset` and `include`. Type `papeer list --help` for more information about those options.
```sh
papeer get https://www.eff.org/cyberspace-independence
# A Declaration of the Independence of Cyberspace
# ===============================================
# Governments of the Industrial World, you weary giants of flesh and steel, I come from Cyberspace, the new home of Mind. On behalf of the future, I ask you of the past to leave us alone. You are not welcome among us. You have no sovereignty where we gather...
papeer list https://12factor.net/ -s 'section.concrete>article>h2>a'
```
## Grab several pages (recursive mode)
The `recursive` option lets you extract the table of content of a website, then scrape the content of each one of its pages.
### Display table of content
Before trying the `recursive` option, it is a good idea to use the `ls` option, which lets you vizualize the content that will be retrieved. You can use several options to customize the table of content extraction, such as `selector`, `limit`, `offset` and `include`. Type `papeer help` for more information about those options.
```sh
papeer ls https://12factor.net/ -s 'section.concrete > article > h2 > a'
# # NAME URL
# 1 I. Codebase https://12factor.net/codebase
# 2 II. Dependencies https://12factor.net/dependencies
# 3 III. Config https://12factor.net/config
# 4 IV. Backing services https://12factor.net/backing-services
# 5 V. Build, release, run https://12factor.net/build-release-run
# 6 VI. Processes https://12factor.net/processes
# 7 VII. Port binding https://12factor.net/port-binding
# 8 VIII. Concurrency https://12factor.net/concurrency
# 9 IX. Disposability https://12factor.net/disposability
# 10 X. Dev/prod parity https://12factor.net/dev-prod-parity
# 11 XI. Logs https://12factor.net/logs
# 12 XII. Admin processes https://12factor.net/admin-processes
```
# NAME URL
1 I. Codebase https://12factor.net/codebase
2 II. Dependencies https://12factor.net/dependencies
3 III. Config https://12factor.net/config
4 IV. Backing services https://12factor.net/backing-services
5 V. Build, release, run https://12factor.net/build-release-run
6 VI. Processes https://12factor.net/processes
7 VII. Port binding https://12factor.net/port-binding
8 VIII. Concurrency https://12factor.net/concurrency
9 IX. Disposability https://12factor.net/disposability
10 X. Dev/prod parity https://12factor.net/dev-prod-parity
11 XI. Logs https://12factor.net/logs
12 XII. Admin processes https://12factor.net/admin-processes
```
### Scrape time
Once you are satisfied with the table of content listed by the `ls` command, you can actually scrape the content of those pages. You can use the same options that you specified for the `ls` command. In recursive mode, you also have the possibility to use `delay` and `threads` options.
Once you are satisfied with the table of contents listed by the `ls` command, you can actually scrape the content of those pages. You can use the same options that you specified for the `ls` command. You can specify `delay` and `threads` options when using `selector` or `depth` options.
```sh
papeer get https://12factor.net/ --recursive -s 'section.concrete > article > h2 > a' --format=md
# [======================================>-----------------------------] Chapters 7 / 12
# [====================================================================] 1. I. Codebase
# [====================================================================] 2. II. Dependencies
# [====================================================================] 3. III. Config
# [====================================================================] 4. IV. Backing services
# [====================================================================] 5. V. Build, release, run
# [====================================================================] 6. VI. Processes
# [====================================================================] 7. VII. Port binding
# [--------------------------------------------------------------------] 8. VIII. Concurrency
# [--------------------------------------------------------------------] 9. IX. Disposability
# [--------------------------------------------------------------------] 10. X. Dev/prod parity
# [--------------------------------------------------------------------] 11. XI. Logs
# [--------------------------------------------------------------------] 12. XII. Admin processes
# Markdown saved to "The_Twelve-Factor_App.md"
papeer get https://12factor.net/ --selector='section.concrete>article>h2>a'
```
```
[======================================>-----------------------------] Chapters 7 / 12
[====================================================================] 1. I. Codebase
[====================================================================] 2. II. Dependencies
[====================================================================] 3. III. Config
[====================================================================] 4. IV. Backing services
[====================================================================] 5. V. Build, release, run
[====================================================================] 6. VI. Processes
[====================================================================] 7. VII. Port binding
[--------------------------------------------------------------------] 8. VIII. Concurrency
[--------------------------------------------------------------------] 9. IX. Disposability
[--------------------------------------------------------------------] 10. X. Dev/prod parity
[--------------------------------------------------------------------] 11. XI. Logs
[--------------------------------------------------------------------] 12. XII. Admin processes
Markdown saved to "The_Twelve-Factor_App.md"
```
# Installation
@@ -108,21 +133,30 @@ go get -u github.com/lapwat/papeer
## From binary
### On Linux / MacOS
### Linux / MacOS
```sh
platform=linux # use platform=darwin for MacOS
release=0.3.3
curl -L https://github.com/lapwat/papeer/releases/download/v$release/papeer-v$release-$platform-amd64 > papeer
# use platform=darwin for MacOS
platform=linux
release=0.4.0
# download and extract
curl -L https://github.com/lapwat/papeer/releases/download/v$release/papeer-v$release-$platform-amd64.tar.gz > papeer.tar.gz
tar xzvf papeer.tar.gz
rm papeer.tar.gz
# move to user binaries
chmod +x papeer
sudo mv papeer /usr/local/bin
```
### On Windows
### Windows
Download [latest release](https://github.com/lapwat/papeer/releases/download/v0.3.3/papeer-v0.3.3-windows-amd64.exe).
Download [latest release](https://github.com/lapwat/papeer/releases/download/v0.4.0/papeer-v0.4.0-windows-amd64.exe).
## Install kindlegen to export websites to MOBI (optional)
## (optional) MOBI support
Install kindlegen to convert websites, Linux only
```sh
TMPDIR=$(mktemp -d -t papeer-XXXXX)
@@ -141,10 +175,10 @@ Execute this command in your current shell, or add it to your `.bashrc`.
. <(papeer completion bash)
```
Type `papeer completion bash -h` for more information.
You can replace `bash` by your own shell (zsh, fish or powershell).
Type `papeer completion bash -h` for more information.
# Dependencies
- `cobra` command line interface