handle RSS feed, add CI

.github/workflows/format.yml

@@ -0,0 +1,15 @@
+name: Format
+
+on:
+  push:
+    paths:
+      - '**.go'
+
+jobs:
+  format:
+    runs-on: ubuntu-latest
+    steps:
+    - name: Checkout
+      uses: actions/checkout@v2
+    - name: Check formatting
+      run: if [[ -n "$(gofmt -l .)" ]]; then exit 1; fi

.github/workflows/test.yml

@@ -0,0 +1,15 @@
+name: Test
+
+on:
+  push:
+    paths:
+      - '**.go'
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+    - name: Checkout
+      uses: actions/checkout@v2
+    - name: Test
+      run: make test

Makefile

@@ -0,0 +1,14 @@
+format:
+	gofmt -s -w .
+
+test:
+	go test github.com/lapwat/papeer/book
+
+install:
+	go install
+
+clean:
+	find . -maxdepth 1 -not -name 'README.md' -name '*.md' -delete
+	find . -maxdepth 1 -name '*.epub' -delete
+	find . -maxdepth 1 -name '*.mobi' -delete
+	find . -maxdepth 1 -name 'papeer-v*' -delete

README.md

@@ -1,17 +1,126 @@
+# Papeer
+
+Papeer is a powerful **ereader internet vacuum**. It can scrape any website, removing ads and keeping only the relevant content (formatted text and images). You can export the content to Markdown, EPUB or MOBI files.
+
+# Table of contents
+
+- [Usage](#usage)
+  * [Scrape a web page](#scrape-a-web-page)
+  * [Scrape a whole website](#scrape-a-whole-website)
+    + [`depth` option](#-depth--option)
+    + [`selector` option](#-selector--option)
+    + [Display the table of contents](#display-the-table-of-contents)
+    + [Scrape time](#scrape-time)
+- [Installation](#installation)
+  * [From source](#from-source)
+  * [From binary](#from-binary)
+    + [Linux / MacOS](#linux---macos)
+    + [Windows](#windows)
+  * [MOBI support](#mobi-support)
+- [Autocompletion](#autocompletion)
+- [Dependencies](#dependencies)
+
+# Usage
+
+## Scrape a web page
+
+The `get` command lets you retrieve the content of any web page or RSS feed.
+
 ```
-❯ papeer get --format epub --recursive --delay 500 --limit 10 https://news.ycombinator.com/
+Scrape URL content
-   6s [===============================================>--------------------]  70% Status: 7 out of 10 chapters
+
-   0s [====================================================================] 100% 1. Three ex-US intelligence officers admit hacking for UAE
+Usage:
-   0s [====================================================================] 100% 2. Show HN: Time Travel Debugger
+  papeer get URL [flags]
-   0s [====================================================================] 100% 3. How much faster is Java 17?
+
-   0s [====================================================================] 100% 4. The First Webcam Was Invented to Keep an Eye on a Coffee Pot
+Examples:
-   0s [====================================================================] 100% 5. Nikon's 2021 Photomicrography Competition Winners
+papeer get https://www.eff.org/cyberspace-independence
-   0s [====================================================================] 100% 6. HTTP Status 418 – I'm a teapot
+
-   0s [====================================================================] 100% 7. H3: Hexagonal hierarchical geospatial indexing system
+Flags:
-  --- [--------------------------------------------------------------------]   0% 8. Automatic cipher suite ordering in Go’s crypto/tls
+  -a, --author string      book author
-  --- [--------------------------------------------------------------------]   0% 9. Find engineering roles at over 800 YC-funded startups
+      --delay int          time in milliseconds to wait before downloading next chapter, use with depth/selector (default -1)
-  --- [--------------------------------------------------------------------]   0% 10. Futarchy: Robin Hanson on prediction markets
+  -d, --depth int          scraping depth
-Ebook saved to "Hacker_News.epub"
+  -f, --format string      file format [stdout, md, epub, mobi] (default "md")
+  -h, --help               help for get
+      --images             retrieve images only
+  -i, --include            include URL as first chapter, use with depth/selector
+  -l, --limit int          limit number of chapters, use with depth/selector (default -1)
+  -n, --name string        book name (default: page title)
+  -o, --offset int         skip first chapters, use with depth/selector
+      --output string      file name (default: book name)
+  -q, --quiet              hide progress bar
+  -r, --reverse            reverse chapter order
+  -s, --selector strings   table of contents CSS selector
+  -t, --threads int        download concurrency, use with depth/selector (default -1)
+      --use-link-name      use link name for chapter title
+```
+
+## Scrape a whole website
+
+If a navigation menu is present on a website, you can scrape the content of each page.
+
+You can activate this mode by using the `depth` or `selector` options.
+
+### `depth` option
+
+This option defaults to 0, `papeer` will grab only the main page.
+
+If you specify a value greater than 0, `papeer` will grab pages as deep as the value you specify.
+
+> Using `include` option will include all intermediary levels into the book.
+
+### `selector` option
+
+If this option is not specified, `papeer` will grab only the one page.
+
+If this option is specified, `papeer` will select the links (a HTML tag) present on the main page, then grab each one of them.
+
+You can chain this option to grab several level of pages with diferent selectors for each level.
+
+### Display the table of contents
+
+Before actually scraping a whole website, it is a good idea to use the `list` command. This command is like a **dry run**, which lets you vizualize the content before actually retrieving it. You can use several options to customize the table of contents extraction, such as `selector`, `limit`, `offset`, `reverse` and `include`. Type `papeer list --help` for more information about those options.
+
+```sh
+papeer list https://12factor.net/ -s 'section.concrete>article>h2>a'
+```
+```
+ #  NAME                    URL                                    
+ 1  I. Codebase             https://12factor.net/codebase          
+ 2  II. Dependencies        https://12factor.net/dependencies      
+ 3  III. Config             https://12factor.net/config            
+ 4  IV. Backing services    https://12factor.net/backing-services  
+ 5  V. Build, release, run  https://12factor.net/build-release-run 
+ 6  VI. Processes           https://12factor.net/processes         
+ 7  VII. Port binding       https://12factor.net/port-binding      
+ 8  VIII. Concurrency       https://12factor.net/concurrency       
+ 9  IX. Disposability       https://12factor.net/disposability     
+10  X. Dev/prod parity      https://12factor.net/dev-prod-parity   
+11  XI. Logs                https://12factor.net/logs              
+12  XII. Admin processes    https://12factor.net/admin-processes
+```
+
+### Scrape time
+
+Once you are satisfied with the table of contents listed by the `ls` command, you can actually scrape the content of those pages. You can use the same options that you specified for the `ls` command. You can specify `delay` and `threads` options when using `selector` or `depth` options.
+
+```sh
+papeer get https://12factor.net/ --selector='section.concrete>article>h2>a'
+```
+```
+[======================================>-----------------------------] Chapters 7 / 12
+[====================================================================] 1. I. Codebase
+[====================================================================] 2. II. Dependencies
+[====================================================================] 3. III. Config
+[====================================================================] 4. IV. Backing services
+[====================================================================] 5. V. Build, release, run
+[====================================================================] 6. VI. Processes
+[====================================================================] 7. VII. Port binding
+[--------------------------------------------------------------------] 8. VIII. Concurrency
+[--------------------------------------------------------------------] 9. IX. Disposability
+[--------------------------------------------------------------------] 10. X. Dev/prod parity
+[--------------------------------------------------------------------] 11. XI. Logs
+[--------------------------------------------------------------------] 12. XII. Admin processes
+Markdown saved to "The_Twelve-Factor_App.md"
 ```
 
 # Installation
@@ -24,13 +133,29 @@ go get -u github.com/lapwat/papeer
 
 ## From binary
 
+### Linux / MacOS
+
 ```sh
-curl -L https://github.com/lapwat/papeer/releases/download/v0.0.2/papeer-v0.0.2 > papeer
+# use platform=darwin for MacOS
-chmod +x papeer
+platform=linux
+release=0.5.0
+
+# download and extract
+curl -L https://github.com/lapwat/papeer/releases/download/v$release/papeer-v$release-$platform-amd64.tar.gz > papeer.tar.gz
+tar xzvf papeer.tar.gz
+rm papeer.tar.gz
+
+# move to user binaries
 sudo mv papeer /usr/local/bin
 ```
 
-## Install kindlegen to export websites to MOBI (optional)
+### Windows
+
+Download [latest release](https://github.com/lapwat/papeer/releases/download/v0.5.0/papeer-v0.5.0-windows-amd64.exe.zip).
+
+## MOBI support
+
+Install kindlegen to convert websites, Linux only
 
 ```sh
 TMPDIR=$(mktemp -d -t papeer-XXXXX)
@@ -38,37 +163,7 @@ curl -L https://github.com/lapwat/papeer/releases/download/kindlegen/kindlegen_l
 tar xzvf $TMPDIR/kindlegen.tar.gz -C $TMPDIR
 chmod +x $TMPDIR/kindlegen
 sudo mv $TMPDIR/kindlegen /usr/local/bin
-rm $TMPDIR
+rm -rf $TMPDIR
-```
-
-# Usage
-
-```
-Browse the web in the eink era
-
-Usage:
-  papeer [flags]
-  papeer [command]
-
-Available Commands:
-  completion  generate the autocompletion script for the specified shell
-  get         Scrape URL content
-  help        Help about any command
-  ls          Print table of content
-  version     Print the version number of papeer
-
-Flags:
-  -d, --delay int         wait before downloading next chapter, in milliseconds (default -1)
-  -f, --format string     file format [md, epub, mobi] (default "md")
-  -h, --help              help for papeer
-  -i, --include           include URL as first chapter, in resursive mode
-  -o, --output string     output file
-  -q, --quiet             do not show progress bars
-  -r, --recursive         create one chapter per natigation item
-  -s, --selector string   table of content CSS selector
-      --stdout            print to standard output
-
-Use "papeer [command] --help" for more information about a command.
 ```
 
 # Autocompletion
@@ -79,10 +174,10 @@ Execute this command in your current shell, or add it to your `.bashrc`.
 . <(papeer completion bash)
 ```
 
-Type `papeer completion bash -h` for more information.
-
 You can replace `bash` by your own shell (zsh, fish or powershell).
 
+Type `papeer completion bash -h` for more information.
+
 # Dependencies
 
 - `cobra` command line interface

book/book.go

@@ -1,27 +1,27 @@
 package book
 
-type Book struct {
+type book struct {
 	name     string
 	author   string
 	chapters []chapter
 }
 
-func New(name, author string) Book {
+func New(name, author string) book {
-	return Book{name, author, []chapter{}}
+	return book{name, author, []chapter{}}
 }
 
-func (b *Book) AddChapter(c chapter) {
+func (b *book) AddChapter(c chapter) {
 	b.chapters = append(b.chapters, c)
 }
 
-func (b Book) Name() string {
+func (b book) Name() string {
 	return b.name
 }
 
-func (b Book) Author() string {
+func (b book) Author() string {
 	return b.name
 }
 
-func (b *Book) Chapters() []chapter {
+func (b *book) Chapters() []chapter {
 	return b.chapters
 }

book/chapter.go

@@ -1,13 +1,20 @@
 package book
 
 type chapter struct {
+	body        string
 	name        string
 	author      string
 	content     string
+	subChapters []chapter
+	config      *ScrapeConfig
 }
 
-func NewChapter(name, author, content string) chapter {
+func NewChapter(body, name, author, content string, subChapters []chapter, config *ScrapeConfig) chapter {
-	return chapter{name, author, content}
+	return chapter{body, name, author, content, subChapters, config}
+}
+
+func (c chapter) Body() string {
+	return c.body
 }
 
 func (c chapter) Name() string {
@@ -21,3 +28,7 @@ func (c chapter) Author() string {
 func (c chapter) Content() string {
 	return c.content
 }
+
+func (c chapter) SubChapters() []chapter {
+	return c.subChapters
+}

book/format.go

@@ -0,0 +1,164 @@
+package book
+
+import (
+	"fmt"
+	"log"
+	"os"
+	"os/exec"
+	"strings"
+
+	md "github.com/JohannesKaufmann/html-to-markdown"
+	"github.com/PuerkitoBio/goquery"
+	epub "github.com/bmaupin/go-epub"
+)
+
+func Filename(name string) string {
+	filename := name
+
+	filename = strings.ReplaceAll(filename, " ", "_")
+	filename = strings.ReplaceAll(filename, "/", "")
+
+	return filename
+}
+
+func ToMarkdownString(c chapter) string {
+	markdown := ""
+
+	if c.config.Include {
+		// title
+		markdown += fmt.Sprintf("%s\n", c.Name())
+		markdown += fmt.Sprintf("%s\n\n", strings.Repeat("=", len(c.Name())))
+
+		// convert content to markdown
+		content, err := md.NewConverter("", true, nil).ConvertString(c.Content())
+		if err != nil {
+			log.Fatal(err)
+		}
+		markdown += fmt.Sprintf("%s\n\n\n", content)
+	}
+
+	for _, sc := range c.SubChapters() {
+		// subchapters content
+		markdown += fmt.Sprintf("%s\n\n\n", ToMarkdownString(sc))
+	}
+
+	return markdown
+}
+
+func ToMarkdown(c chapter, filename string) string {
+	if len(filename) == 0 {
+		filename = fmt.Sprintf("%s.md", Filename(c.Name()))
+	}
+
+	markdown := ToMarkdownString(c)
+
+	// write to file
+	f, err := os.Create(filename)
+	if err != nil {
+		log.Fatal(err)
+	}
+	_, err2 := f.WriteString(markdown)
+	if err2 != nil {
+		log.Fatal(err2)
+	}
+	f.Close()
+
+	return filename
+}
+
+func ToEpub(c chapter, filename string) string {
+	if len(filename) == 0 {
+		filename = fmt.Sprintf("%s.epub", Filename(c.Name()))
+	}
+
+	// init ebook
+	e := epub.NewEpub(c.Name())
+	e.SetAuthor(c.Author())
+
+	AppendToEpub(e, c)
+
+	err := e.Write(filename)
+	if err != nil {
+		log.Fatal(err)
+	}
+
+	return filename
+}
+
+func AppendToEpub(e *epub.Epub, c chapter) {
+	content := ""
+
+	if c.config.Include {
+
+		if c.config.ImagesOnly == false {
+			content = c.Content()
+		}
+
+		// parse content
+		doc, err := goquery.NewDocumentFromReader(strings.NewReader(c.Content()))
+		if err != nil {
+			log.Fatal(err)
+		}
+
+		// download images and replace src in img tags of content
+		doc.Find("img").Each(func(i int, s *goquery.Selection) {
+			src, _ := s.Attr("src")
+			src = strings.Split(src, "?")[0] // remove query part
+			imagePath, _ := e.AddImage(src, "")
+
+			if c.config.ImagesOnly {
+				imageTag, _ := goquery.OuterHtml(s)
+				content += strings.Replace(imageTag, src, imagePath, 1)
+			} else {
+				content = strings.Replace(content, src, imagePath, 1)
+			}
+		})
+
+		html := ""
+		// add title only if ImagesOnly = false
+		if c.config.ImagesOnly == false {
+			html += fmt.Sprintf("<h1>%s</h1>", c.Name())
+		}
+		html += content
+
+		//  write to epub file
+		_, err = e.AddSection(html, c.Name(), "", "")
+		if err != nil {
+			log.Fatal(err)
+		}
+
+	}
+
+	for _, sc := range c.SubChapters() {
+		AppendToEpub(e, sc)
+	}
+}
+
+func ToMobi(c chapter, filename string) string {
+	if len(filename) == 0 {
+		filename = fmt.Sprintf("%s.mobi", Filename(c.Name()))
+	} else {
+
+		// add .mobi extension if not specified
+		if strings.HasSuffix(filename, ".mobi") == false {
+			filename = fmt.Sprintf("%s.mobi", filename)
+		}
+
+	}
+
+	filenameEPUB := strings.ReplaceAll(filename, ".mobi", ".epub")
+	ToEpub(c, filenameEPUB)
+
+	exec.Command("kindlegen", filenameEPUB).Run()
+	// exec command always return status 1 even if it succeed
+	// if err != nil {
+	// 	log.Fatal(err)
+	// }
+
+	err := os.Remove(filenameEPUB)
+	if err != nil {
+		log.Fatal(err)
+	}
+
+	return filename
+}

book/format_test.go

@@ -0,0 +1,127 @@
+package book
+
+import (
+	"errors"
+	"os"
+	"testing"
+)
+
+func TestFilename(t *testing.T) {
+
+	got := Filename("This is a chapter / book")
+	want := "This_is_a_chapter__book"
+
+	if got != want {
+		t.Errorf("got %q, wanted %q", got, want)
+	}
+
+}
+
+func TestToMarkdownString(t *testing.T) {
+
+	c := NewChapterFromURL("https://books.lapw.at/", "", []*ScrapeConfig{NewScrapeConfig()}, 0, func(index int, name string) {})
+
+	got := ToMarkdownString(c)
+	want := "Books\n=====\n\n- [Discours de la Méthode](https://books.lapw.at/posts/ren%C3%A9-descartes-discours-de-la-m%C3%A9thode/)clock 98 min read -\n1637\n\n- [The Twelve-Factor App](https://books.lapw.at/posts/adam-wiggins-the-twelve-factor-app/)clock 22 min read -\n2011\n\n\n"
+
+	if got != want {
+		t.Errorf("got %q, wanted %q", got, want)
+	}
+
+}
+
+func TestToMarkdown(t *testing.T) {
+
+	c := NewChapterFromURL("https://books.lapw.at/", "", []*ScrapeConfig{NewScrapeConfig()}, 0, func(index int, name string) {})
+	ToMarkdown(c, "")
+
+	filename := "Books.md"
+	if _, err := os.Stat(filename); errors.Is(err, os.ErrNotExist) {
+		t.Errorf("%s does not exist: %v", filename, err)
+	} else {
+		if err := os.Remove(filename); err != nil {
+			t.Errorf("cannot remove %v: %v", filename, err)
+		}
+	}
+
+}
+
+func TestToMarkdownFilename(t *testing.T) {
+
+	filename := "ebook.md"
+	c := NewChapterFromURL("https://books.lapw.at/", "", []*ScrapeConfig{NewScrapeConfig()}, 0, func(index int, name string) {})
+	ToMarkdown(c, filename)
+
+	if _, err := os.Stat(filename); errors.Is(err, os.ErrNotExist) {
+		t.Errorf("%s does not exist: %v", filename, err)
+	} else {
+		if err := os.Remove(filename); err != nil {
+			t.Errorf("cannot remove %v: %v", filename, err)
+		}
+	}
+
+}
+
+func TestToEpub(t *testing.T) {
+
+	c := NewChapterFromURL("https://books.lapw.at/", "", []*ScrapeConfig{NewScrapeConfig()}, 0, func(index int, name string) {})
+	ToEpub(c, "")
+
+	filename := "Books.epub"
+	if _, err := os.Stat(filename); errors.Is(err, os.ErrNotExist) {
+		t.Errorf("%s does not exist: %v", filename, err)
+	} else {
+		if err := os.Remove(filename); err != nil {
+			t.Errorf("cannot remove %v: %v", filename, err)
+		}
+	}
+
+}
+
+func TestToEpubFilename(t *testing.T) {
+
+	filename := "ebook.epub"
+	c := NewChapterFromURL("https://books.lapw.at/", "", []*ScrapeConfig{NewScrapeConfig()}, 0, func(index int, name string) {})
+	ToEpub(c, filename)
+
+	if _, err := os.Stat(filename); errors.Is(err, os.ErrNotExist) {
+		t.Errorf("%s does not exist: %v", filename, err)
+	} else {
+		if err := os.Remove(filename); err != nil {
+			t.Errorf("cannot remove %v: %v", filename, err)
+		}
+	}
+
+}
+
+func TestToMobi(t *testing.T) {
+
+	filename := "ebook.mobi"
+	c := NewChapterFromURL("https://books.lapw.at/", "", []*ScrapeConfig{NewScrapeConfig()}, 0, func(index int, name string) {})
+	ToMobi(c, filename)
+
+	if _, err := os.Stat(filename); errors.Is(err, os.ErrNotExist) {
+		t.Errorf("%s does not exist: %v", filename, err)
+	} else {
+		if err := os.Remove(filename); err != nil {
+			t.Errorf("cannot remove %v: %v", filename, err)
+		}
+	}
+
+}
+
+func TestToMobiFilename(t *testing.T) {
+
+	filename := "ebook.mobi"
+	c := NewChapterFromURL("https://books.lapw.at/", "", []*ScrapeConfig{NewScrapeConfig()}, 0, func(index int, name string) {})
+	ToMobi(c, filename)
+
+	if _, err := os.Stat(filename); errors.Is(err, os.ErrNotExist) {
+		t.Errorf("%s does not exist: %v", filename, err)
+	} else {
+		if err := os.Remove(filename); err != nil {
+			t.Errorf("cannot remove %v: %v", filename, err)
+		}
+	}
+
+}

book/link.go

@@ -0,0 +1,18 @@
+package book
+
+type link struct {
+	href string
+	text string
+}
+
+func NewLink(href, text string) link {
+	return link{href, text}
+}
+
+func (c link) Href() string {
+	return c.href
+}
+
+func (c link) Text() string {
+	return c.text
+}

book/progress.go

@@ -0,0 +1,59 @@
+package book
+
+import (
+	"fmt"
+	"strings"
+
+	"github.com/gosuri/uiprogress"
+)
+
+type progress struct {
+	global      *uiprogress.Bar
+	individuals []*uiprogress.Bar
+}
+
+func NewProgress(links []link, parent string, depth int) progress {
+	uiprogress.Start()
+
+	global := uiprogress.AddBar(len(links))
+	indentGlobal := strings.Repeat("> ", depth)
+	global.AppendFunc(func(b *uiprogress.Bar) string {
+		return fmt.Sprintf("%v%v (%v / %v)", indentGlobal, parent, b.Current(), len(links))
+	})
+
+	// hide individual bars if more than 50 chapters
+	individuals := []*uiprogress.Bar{}
+	indent := strings.Repeat("- ", depth)
+	if len(links) <= 50 {
+		for index, link := range links {
+			bar := uiprogress.AddBar(1)
+			barText := fmt.Sprintf("%v#%v %v", indent, index+1, link.Text())
+			bar.AppendFunc(func(b *uiprogress.Bar) string {
+				return barText
+			})
+			individuals = append(individuals, bar)
+		}
+	}
+
+	return progress{global, individuals}
+}
+
+func (p *progress) IncrementGlobal() {
+	p.global.Incr()
+}
+
+func (p *progress) Increment(index int) {
+	p.IncrementGlobal()
+	if len(p.individuals) > index {
+		p.individuals[index].Incr()
+	}
+}
+
+func (p *progress) UpdateName(index int, name string) {
+	if len(p.individuals) > index {
+		barText := fmt.Sprintf("%s", name)
+		p.individuals[index].AppendFunc(func(b *uiprogress.Bar) string {
+			return barText
+		})
+	}
+}

book/scraper.go

@@ -1,31 +1,131 @@
 package book
 
 import (
+	"bytes"
 	"fmt"
+	"io"
 	"log"
 	"math"
+	"net/http"
 	urllib "net/url"
 	"strings"
 	"sync"
 	"time"
 
+	"github.com/PuerkitoBio/goquery"
 	readability "github.com/go-shiori/go-readability"
 	colly "github.com/gocolly/colly/v2"
-	"github.com/gosuri/uiprogress"
+	"github.com/mmcdole/gofeed"
 )
 
-type scraper struct {
+type ScrapeConfig struct {
-	url string
+	Depth       int
+	Selector    string
+	Quiet       bool
+	Limit       int
+	Offset      int
+	Reverse     bool
+	Delay       int
+	Threads     int
+	Include     bool
+	ImagesOnly  bool
+	UseLinkName bool
 }
 
-func NewBookFromURL(url, selector string, include bool, limit, delay int) Book {
+func NewScrapeConfig() *ScrapeConfig {
-	home := NewChapterFromURL(url)
+	return &ScrapeConfig{0, "", false, -1, 0, false, -1, -1, true, false, false}
-	b := New(home.Name(), home.Author())
+}
 
-	chapters := tableOfContent(url, selector, limit, delay)
+func NewScrapeConfigs(selectors []string) []*ScrapeConfig {
-	if include {
+	configs := []*ScrapeConfig{}
-		b.AddChapter(home)
+
+	for _, s := range selectors {
+		config := NewScrapeConfig()
+		config.Selector = s
+
+		configs = append(configs, config)
 	}
+
+	return configs
+}
+
+func NewScrapeConfigsAjin() []*ScrapeConfig {
+	config0 := NewScrapeConfig()
+	config0.Depth = 0
+	config0.Selector = ".dt>a"
+	config0.Limit = 3
+	config0.Offset = 0
+	config0.Delay = 5000
+	config0.Include = false
+
+	config1 := NewScrapeConfig()
+	config1.Depth = 1
+	config1.Selector = ".nav_apb>a"
+	config1.Limit = 3
+	config1.Offset = 1
+	config1.Delay = 5000
+	config1.Include = false
+
+	config2 := NewScrapeConfig()
+	config2.Depth = 2
+	config2.ImagesOnly = true
+
+	return []*ScrapeConfig{config0, config1, config2}
+}
+
+func NewScrapeConfigsWikipedia() []*ScrapeConfig {
+	config0 := NewScrapeConfig()
+	config0.Depth = 0
+	config0.Threads = -1
+	config0.Include = true
+
+	config1 := NewScrapeConfig()
+	config1.Depth = 1
+	config1.Include = true
+
+	return []*ScrapeConfig{config0, config1}
+}
+
+func NewScrapeConfigFake() *ScrapeConfig {
+	config := NewScrapeConfig()
+	config.Include = false
+
+	return config
+}
+
+func NewBookFromURL(url string, selector []string, name, author string, include, ImagesOnly, useLinkName, quiet bool, limit, offset, delay, threads int) book {
+	config1 := NewScrapeConfig()
+	config1.ImagesOnly = ImagesOnly
+	config1.UseLinkName = useLinkName
+
+	var chapters []chapter
+	var home chapter
+
+	if len(selector) > 0 {
+		config2 := NewScrapeConfig()
+		config2.Selector = selector[0]
+		config2.Limit = limit
+		config2.Offset = offset
+		config2.Delay = delay
+		config2.Threads = threads
+		config2.Include = include
+		config2.ImagesOnly = ImagesOnly
+		config2.UseLinkName = useLinkName
+		chapters, home = tableOfContent(url, config2, config1, quiet)
+	} else {
+		chapters = []chapter{NewChapterFromURL(url, "", []*ScrapeConfig{config1}, 0, func(index int, name string) {})}
+		home = chapters[0]
+	}
+
+	if len(name) == 0 {
+		name = home.Name()
+	}
+
+	if len(author) == 0 {
+		author = home.Author()
+	}
+
+	b := New(name, author)
 	for _, c := range chapters {
 		b.AddChapter(c)
 	}
@@ -33,123 +133,343 @@ func NewBookFromURL(url, selector string, include bool, limit, delay int) Book {
 	return b
 }
 
-func NewChapterFromURL(url string) chapter {
+func NewChapterFromURL(url, linkName string, configs []*ScrapeConfig, index int, updateProgressBarName func(index int, name string)) chapter {
-	article, err := readability.FromURL(url, 30*time.Second)
+	config := configs[0]
-	if err != nil {
-		log.Fatalf("failed to parse %s, %v\n", url, err)
-	}
 
-	// metadata := fmt.Sprintf("URL     : %s\nTitle   : %s\nAuthor  : %s\nLength  : %d\nExcerpt : %s\nSiteName: %s\nImage   : %s\nFavicon : %s", url, article.Title, article.Byline, article.Length, article.Excerpt, article.SiteName, article.Image, article.Favicon)
-	// fmt.Println(metadata)
-
-	return chapter{article.Title, article.Byline, article.Content}
-}
-
-func tableOfContent(url, selector string, limit, delay int) []chapter {
-	c := colly.NewCollector()
-
-	classesLinks := map[string][]map[string]string{}
-	classesCount := map[string]int{}
-	classMax := ""
-
-	selectorSet := true
-	if selector == "" {
-		selector = "a"
-		selectorSet = false
-	}
-
-	c.OnHTML(selector, func(e *colly.HTMLElement) {
-		href := e.Attr("href")
-		text := strings.TrimSpace(e.Text)
-		class := e.Attr("class")
-
-		if selectorSet || class != "" && text != "" {
-			classesLinks[class] = append(classesLinks[class], map[string]string{
-				"href": href,
-				"text": text,
-			})
-
-			classesCount[class]++
-
-			if classesCount[class] > classesCount[classMax] {
-				classMax = class
-			}
-		}
-
-	})
-	c.Visit(url)
-
-	links := classesLinks[classMax]
-	if limit != -1 {
-		limit = int(math.Min(float64(limit), float64(len(links))))
-		links = links[:limit]
-	}
-
-	chapters := make([]chapter, len(links))
 	base, err := urllib.Parse(url)
 	if err != nil {
 		log.Fatal(err)
 	}
 
-	// init global progress bar
+	// get page body
-	uiprogress.Start()
+	response, err := http.Get(url)
-	barGlobal := uiprogress.AddBar(len(links)).AppendCompleted().PrependElapsed()
+	if err != nil {
-	barGlobal.AppendFunc(func(b *uiprogress.Bar) string {
+		log.Fatal(err)
-		return fmt.Sprintf("Status: %d out of %d chapters", b.Current(), len(links))
+	}
-	})
+	defer response.Body.Close()
 
-	// init progress bars
+	// duplicate response stream
-	bars := []*uiprogress.Bar{}
+	readabilityReader := &bytes.Buffer{}
-	for index, link := range links {
+	bodyReader := io.TeeReader(response.Body, readabilityReader)
-		bar := uiprogress.AddBar(1).AppendCompleted().PrependElapsed()
+
-		barText := fmt.Sprintf("%d. %s", index+1, link["text"])
+	// extract HTML body
-		bar.AppendFunc(func(b *uiprogress.Bar) string {
+	body, err := io.ReadAll(bodyReader)
-			return barText
+
-		})
+	// extract article content and metadata
-		bars = append(bars, bar)
+	article, err := readability.FromReader(readabilityReader, base)
+	if err != nil {
+		log.Fatalf("failed to parse %s, %v\n", url, err)
 	}
 
-	if delay >= 0 {
+	name := linkName
-		for index, link := range links {
+	if config.UseLinkName == false {
-			// and then use it to parse relative URLs
+		name = article.Title
-			u, err := base.Parse(link["href"])
+
+		// notify progressbar with new name
+		updateProgressBarName(index, name)
+	}
+
+	var subchapters []chapter
+	if len(configs) > 1 {
+
+		// retrieve links on page
+		links, _, _, err := GetLinks(base, config.Selector, config.Limit, config.Offset, config.Reverse, false)
 		if err != nil {
 			log.Fatal(err)
 		}
 
-			chapters[index] = NewChapterFromURL(u.String())
+		// init progess bar
-
+		var p progress
-			bars[index].Incr()
+		if config.Quiet == false {
-			barGlobal.Incr()
+			p = NewProgress(links, name, config.Depth)
-
-			// do not wait after downloading last chapter
-			if index < len(links)-1 {
-				time.Sleep(time.Duration(delay) * time.Millisecond)
 		}
 
+		// init chapters list
+		subchapters = make([]chapter, len(links))
+
+		if config.Delay >= 0 {
+
+			// synchronous mode
+			for index, link := range links {
+				// and then use it to parse relative URLs
+				u, err := base.Parse(link.href)
+				if err != nil {
+					log.Fatal(err)
+				}
+
+				sc := NewChapterFromURL(u.String(), link.text, configs[1:], index, p.UpdateName)
+				subchapters[index] = sc
+				if config.Quiet == false {
+					p.Increment(index)
+				}
+
+				time.Sleep(time.Duration(config.Delay) * time.Millisecond)
 			}
 
 		} else {
+			// asynchronous mode
 			var wg sync.WaitGroup
-		for index, link := range links {
+
+			threads := config.Threads
+			if threads == -1 {
+				threads = len(links)
+			}
+			semaphore := make(chan bool, threads)
+
+			for index, l := range links {
 
 				wg.Add(1)
-			go func(index int, link map[string]string) {
+				semaphore <- true
+
+				go func(index int, l link) {
 					defer wg.Done()
 
 					// and then use it to parse relative URLs
-				u, err := base.Parse(link["href"])
+					u, err := base.Parse(l.href)
 					if err != nil {
 						log.Fatal(err)
 					}
 
-				chapters[index] = NewChapterFromURL(u.String())
+					sc := NewChapterFromURL(u.String(), l.text, configs[1:], index, p.UpdateName)
+					subchapters[index] = sc
 
-				bars[index].Incr()
+					if config.Quiet == false {
-				barGlobal.Incr()
+						p.Increment(index)
-			}(index, link)
+					}
+
+					<-semaphore
+				}(index, l)
 			}
 			wg.Wait()
 		}
-	return chapters
+	}
+
+	content := ""
+	if config.Include {
+
+		// we care about the content only if:
+		// - we include this level
+		// - we use the page name
+		content = article.Content
+
+		// extract images
+		if config.ImagesOnly {
+
+			// parse HTML
+			doc, err := goquery.NewDocumentFromReader(strings.NewReader(content))
+			if err != nil {
+				log.Fatal(err)
+			}
+
+			// append every image to content
+			content = ""
+			doc.Find("img").Each(func(i int, s *goquery.Selection) {
+				imageTag, _ := goquery.OuterHtml(s)
+				imageTag = strings.ReplaceAll(imageTag, "\n", "")
+
+				content += imageTag
+			})
+
+		}
+	}
+
+	return chapter{string(body), name, article.Byline, content, subchapters, config}
+}
+
+func tableOfContent(url string, config *ScrapeConfig, subConfig *ScrapeConfig, quiet bool) ([]chapter, chapter) {
+	base, err := urllib.Parse(url)
+	if err != nil {
+		log.Fatal(err)
+	}
+
+	links, _, home, err := GetLinks(base, config.Selector, config.Limit, config.Offset, config.Reverse, config.Include)
+	if err != nil {
+		log.Fatal(err)
+	}
+
+	chapters := make([]chapter, len(links))
+	delay := config.Delay
+
+	var p progress
+	if quiet == false {
+		p = NewProgress(links, "", 0)
+	}
+
+	if delay >= 0 {
+		// synchronous mode
+
+		for index, l := range links {
+			// and then use it to parse relative URLs
+			u, err := base.Parse(l.href)
+			if err != nil {
+				log.Fatal(err)
+			}
+
+			chapters[index] = NewChapterFromURL(u.String(), l.text, []*ScrapeConfig{subConfig}, 0, func(index int, name string) {})
+
+			if quiet == false {
+				p.Increment(index)
+			}
+
+			// short sleep for last chapter to let the progress bar update
+			if index == len(links)-1 {
+				delay = 100
+			}
+
+			time.Sleep(time.Duration(delay) * time.Millisecond)
+		}
+
+	} else {
+		// asynchronous mode
+		var wg sync.WaitGroup
+
+		threads := config.Threads
+		if threads == -1 {
+			threads = len(links)
+		}
+		semaphore := make(chan bool, threads)
+
+		for index, l := range links {
+
+			wg.Add(1)
+			semaphore <- true
+
+			go func(index int, l link) {
+				defer wg.Done()
+
+				// and then use it to parse relative URLs
+				u, err := base.Parse(l.href)
+				if err != nil {
+					log.Fatal(err)
+				}
+
+				chapters[index] = NewChapterFromURL(u.String(), l.text, []*ScrapeConfig{subConfig}, 0, func(index int, name string) {})
+
+				if quiet == false {
+					p.Increment(index)
+				}
+
+				<-semaphore
+			}(index, l)
+		}
+		wg.Wait()
+	}
+
+	return chapters, home
+}
+
+func GetPath(elm *goquery.Selection) string {
+	path := []string{}
+
+	for {
+		selector := strings.ToLower(goquery.NodeName(elm))
+		if len(selector) == 0 {
+			break
+		}
+
+		path = append(path, selector)
+		elm = elm.Parent()
+	}
+
+	join := strings.Join(path, "<")
+	return join
+}
+
+func GetLinks(url *urllib.URL, selector string, limit, offset int, reverse, include bool) ([]link, string, chapter, error) {
+	var links []link
+	var pathMax string
+
+	parser := gofeed.NewParser()
+	feed, err := parser.ParseURL(url.String())
+
+	if err == nil {
+		// RSS feed
+
+		for _, item := range feed.Items {
+			links = append(links, NewLink(item.Link, item.Title))
+		}
+
+		pathMax = "RSS"
+	} else {
+		// HTML website
+
+		selectorSet := true
+		if len(selector) == 0 {
+			selector = "a"
+			selectorSet = false
+		}
+	
+		pathLinks := map[string][]link{}
+		pathCount := map[string]int{}
+		pathMax = ""
+	
+		// visit and count link classes
+		c := colly.NewCollector()
+		c.OnHTML(selector, func(e *colly.HTMLElement) {
+			href := e.Attr("href")
+			text := strings.TrimSpace(e.Text)
+			path := GetPath(e.DOM)
+			key := path
+	
+			if selectorSet {
+	
+				// if selector is set, we use the selector specified by the user
+	
+				key = selector
+				pathLinks[key] = append(pathLinks[key], NewLink(href, text))
+				pathCount[key] += 1
+				pathMax = key
+	
+			} else {
+	
+				// if selector is not set, we compute the selector ourselves
+	
+				class := e.Attr("class")
+				// include the element class to make sure we have the same exact path for every link in the table of content
+				key = fmt.Sprintf("%s.%s", path, class)
+	
+				// we count this key if the link text is not empty
+				if text != "" {
+					pathLinks[key] = append(pathLinks[key], NewLink(href, text))
+					pathCount[key] += len(text)
+	
+					if pathCount[key] > pathCount[pathMax] {
+						pathMax = key
+					}
+				}
+	
+			}
+		})
+		c.Visit(url.String())
+
+		links = pathLinks[pathMax]
+	}
+
+	if len(links) == 0 {
+		return []link{}, pathMax, chapter{}, fmt.Errorf("no link found for selector: %s", selector)
+	}
+
+	end := len(links)
+	if limit != -1 {
+		end = int(math.Min(float64(limit+offset), float64(len(links))))
+	}
+
+	links = links[offset:end]
+
+	home := NewChapterFromURL(url.String(), "", []*ScrapeConfig{NewScrapeConfig()}, 0, func(index int, name string) {})
+
+	// include home page
+	if include {
+		l := NewLink(url.String(), home.Name())
+		links = append([]link{l}, links...)
+	}
+
+	// reverse links
+	if reverse {
+		for i, j := 0, len(links)-1; i < j; i, j = i+1, j-1 {
+			links[i], links[j] = links[j], links[i]
+		}
+	}
+
+	return links, pathMax, home, nil
 }

book/scraper_test.go

@@ -0,0 +1,233 @@
+package book
+
+import (
+	"testing"
+	"time"
+)
+
+func TestBody(t *testing.T) {
+
+	config := NewScrapeConfig()
+	c := NewChapterFromURL("https://books.lapw.at/", "", []*ScrapeConfig{config}, 0, func(index int, name string) {})
+
+	got := c.Body()
+	want := "<!doctype html>\n<html lang=\"en-us\">\n  <head>\n    <title>Books</title>\n    <link rel=\"shortcut icon\" href=\"/favicon.ico\" />\n    <meta charset=\"utf-8\" />\n    <meta name=\"generator\" content=\"Hugo 0.59.1\" />\n    <meta name=\"viewport\" content=\"width=device-width, initial-scale=1\" />\n    <meta name=\"author\" content=\"John Doe\" />\n    <meta name=\"description\" content=\" \" />\n    <link rel=\"stylesheet\" href=\"https://books.lapw.at/css/main.min.88e7083eff65effb7485b6e6f38d10afbec25093a6fac42d734ce9024d3defbd.css\" />\n\n    \n    <meta name=\"twitter:card\" content=\"summary\"/>\n<meta name=\"twitter:title\" content=\"Books\"/>\n<meta name=\"twitter:description\" content=\" \"/>\n\n    <meta property=\"og:title\" content=\"Books\" />\n<meta property=\"og:description\" content=\" \" />\n<meta property=\"og:type\" content=\"website\" />\n<meta property=\"og:url\" content=\"https://books.lapw.at/\" />\n\n\n\n  </head>\n  <body>\n    <header class=\"app-header\">\n      <a href=\"https://books.lapw.at/\"><img class=\"app-header-avatar\" src=\"/book.svg\" alt=\"John Doe\" /></a>\n      <h1>Books</h1>\n      <p> </p>\n      <div class=\"app-header-social\">\n        \n      </div>\n    </header>\n    <main class=\"app-container\">\n      \n  <article>\n    <h1>Books</h1>\n    <ul class=\"posts-list\">\n      \n        <li class=\"posts-list-item\">\n          <a class=\"posts-list-item-title\" href=\"https://books.lapw.at/posts/ren%C3%A9-descartes-discours-de-la-m%C3%A9thode/\">Discours de la Méthode</a>\n          <span class=\"posts-list-item-description\">\n            <svg xmlns=\"http://www.w3.org/2000/svg\" width=\"24\" height=\"24\" viewBox=\"0 0 24 24\" fill=\"none\" stroke=\"currentColor\" stroke-width=\"2\" stroke-linecap=\"round\" stroke-linejoin=\"round\" class=\"icon icon-clock\">\n  <title>clock</title>\n  <circle cx=\"12\" cy=\"12\" r=\"10\"></circle><polyline points=\"12 6 12 12 16 14\"></polyline>\n</svg> 98 min read -\n            1637\n          </span>\n        </li>\n      \n        <li class=\"posts-list-item\">\n          <a class=\"posts-list-item-title\" href=\"https://books.lapw.at/posts/adam-wiggins-the-twelve-factor-app/\">The Twelve-Factor App</a>\n          <span class=\"posts-list-item-description\">\n            <svg xmlns=\"http://www.w3.org/2000/svg\" width=\"24\" height=\"24\" viewBox=\"0 0 24 24\" fill=\"none\" stroke=\"currentColor\" stroke-width=\"2\" stroke-linecap=\"round\" stroke-linejoin=\"round\" class=\"icon icon-clock\">\n  <title>clock</title>\n  <circle cx=\"12\" cy=\"12\" r=\"10\"></circle><polyline points=\"12 6 12 12 16 14\"></polyline>\n</svg> 22 min read -\n            2011\n          </span>\n        </li>\n      \n    </ul>\n    \n\n\n\n  </article>\n\n    </main>\n  </body>\n</html>\n"
+
+	if got != want {
+		t.Errorf("got %v, wanted %v", got, want)
+	}
+
+}
+
+func TestName(t *testing.T) {
+
+	config := NewScrapeConfig()
+	c := NewChapterFromURL("https://books.lapw.at/", "", []*ScrapeConfig{config}, 0, func(index int, name string) {})
+
+	got := c.Name()
+	want := "Books"
+
+	if got != want {
+		t.Errorf("got %v, wanted %v", got, want)
+	}
+
+}
+
+func TestCustomName(t *testing.T) {
+
+	config := NewScrapeConfig()
+	config.UseLinkName = true
+	c := NewChapterFromURL("https://books.lapw.at/", "Custom Name", []*ScrapeConfig{config}, 0, func(index int, name string) {})
+
+	got := c.Name()
+	want := "Custom Name"
+
+	if got != want {
+		t.Errorf("got %v, wanted %v", got, want)
+	}
+
+}
+
+func TestAuthor(t *testing.T) {
+
+	config := NewScrapeConfig()
+	c := NewChapterFromURL("https://books.lapw.at/", "", []*ScrapeConfig{config}, 0, func(index int, name string) {})
+
+	got := c.Author()
+	want := "John Doe"
+
+	if got != want {
+		t.Errorf("got %v, wanted %v", got, want)
+	}
+
+}
+
+func TestContent(t *testing.T) {
+
+	config := NewScrapeConfig()
+	c := NewChapterFromURL("https://books.lapw.at/", "", []*ScrapeConfig{config}, 0, func(index int, name string) {})
+
+	got := c.Content()
+	want := "<div id=\"readability-page-1\" class=\"page\">\n    \n    <main>\n      \n  <article>\n    \n    <ul>\n      \n        <li>\n          <a href=\"https://books.lapw.at/posts/ren%C3%A9-descartes-discours-de-la-m%C3%A9thode/\">Discours de la Méthode</a>\n          <span>\n            <svg xmlns=\"http://www.w3.org/2000/svg\" width=\"24\" height=\"24\" viewBox=\"0 0 24 24\" fill=\"none\" stroke=\"currentColor\" stroke-width=\"2\" stroke-linecap=\"round\" stroke-linejoin=\"round\">\n  <title>clock</title>\n  <circle cx=\"12\" cy=\"12\" r=\"10\"></circle><polyline points=\"12 6 12 12 16 14\"></polyline>\n</svg> 98 min read -\n            1637\n          </span>\n        </li>\n      \n        <li>\n          <a href=\"https://books.lapw.at/posts/adam-wiggins-the-twelve-factor-app/\">The Twelve-Factor App</a>\n          <span>\n            <svg xmlns=\"http://www.w3.org/2000/svg\" width=\"24\" height=\"24\" viewBox=\"0 0 24 24\" fill=\"none\" stroke=\"currentColor\" stroke-width=\"2\" stroke-linecap=\"round\" stroke-linejoin=\"round\">\n  <title>clock</title>\n  <circle cx=\"12\" cy=\"12\" r=\"10\"></circle><polyline points=\"12 6 12 12 16 14\"></polyline>\n</svg> 22 min read -\n            2011\n          </span>\n        </li>\n      \n    </ul>\n    \n\n\n\n  </article>\n\n    </main>\n  \n\n</div>"
+
+	if got != want {
+		t.Errorf("got %v, wanted %v", got, want)
+	}
+
+}
+
+func TestDelay(t *testing.T) {
+
+	config0 := NewScrapeConfig()
+	config0.Delay = 500
+
+	config1 := NewScrapeConfig()
+
+	start := time.Now()
+	NewChapterFromURL("https://books.lapw.at/", "", []*ScrapeConfig{config0, config1}, 0, func(index int, name string) {})
+	elapsed := time.Since(start)
+
+	got := elapsed
+	want := time.Duration(500) * time.Millisecond
+
+	if got < want {
+		t.Errorf("got %v, wanted min %v", got, want)
+	}
+
+}
+
+func TestContentImagesOnly(t *testing.T) {
+
+	config := NewScrapeConfig()
+	config.ImagesOnly = true
+
+	c := NewChapterFromURL("https://books.lapw.at/posts/adam-wiggins-the-twelve-factor-app/", "", []*ScrapeConfig{config}, 0, func(index int, name string) {})
+
+	got := c.Content()
+	want := "<img src=\"https://books.lapw.at/images/codebase-deploys.png\" alt=\"One codebase maps to many deploys\"/><img src=\"https://books.lapw.at/images/attached-resources.png\" alt=\"A production deploy attached to four backing services.\"/><img src=\"https://books.lapw.at/images/release.png\" alt=\"Code becomes a build, which is combined with config to create a release.\"/><img src=\"https://books.lapw.at/images/process-types.png\" alt=\"Scale is expressed as running processes, workload diversity is expressed as process types.\"/>"
+
+	if got != want {
+		t.Errorf("got %v, wanted %v", got, want)
+	}
+
+}
+
+func TestSubChapters(t *testing.T) {
+
+	config0 := NewScrapeConfig()
+	config1 := NewScrapeConfig()
+
+	c := NewChapterFromURL("https://books.lapw.at/", "", []*ScrapeConfig{config0, config1}, 0, func(index int, name string) {})
+
+	got := len(c.SubChapters())
+	want := 2
+
+	if got != want {
+		t.Errorf("got %v, wanted %v", got, want)
+	}
+
+}
+
+func TestSubChaptersRSS(t *testing.T) {
+
+	config0 := NewScrapeConfig()
+	config1 := NewScrapeConfig()
+
+	c := NewChapterFromURL("https://blog.lapw.at/rss", "", []*ScrapeConfig{config0, config1}, 0, func(index int, name string) {})
+
+	got := len(c.SubChapters())
+	want := 8
+
+	if got != want {
+		t.Errorf("got %v, wanted %v", got, want)
+	}
+
+}
+
+func TestSubChaptersSelector(t *testing.T) {
+
+	config0 := NewScrapeConfig()
+	config0.Selector = "section.concrete > article > h2 > a"
+
+	config1 := NewScrapeConfig()
+
+	c := NewChapterFromURL("https://12factor.net/", "", []*ScrapeConfig{config0, config1}, 0, func(index int, name string) {})
+
+	got := len(c.SubChapters())
+	want := 12
+
+	if got != want {
+		t.Errorf("got %v, wanted %v", got, want)
+	}
+
+}
+
+func TestSubChaptersLimit(t *testing.T) {
+
+	config0 := NewScrapeConfig()
+	config0.Limit = 1
+
+	config1 := NewScrapeConfig()
+
+	c := NewChapterFromURL("https://books.lapw.at/", "", []*ScrapeConfig{config0, config1}, 0, func(index int, name string) {})
+
+	got := len(c.SubChapters())
+	want := 1
+
+	if got != want {
+		t.Errorf("got %v, wanted %v", got, want)
+	}
+
+}
+
+func TestSubChaptersLimitOver(t *testing.T) {
+
+	config0 := NewScrapeConfig()
+	config0.Limit = 3
+
+	config1 := NewScrapeConfig()
+
+	c := NewChapterFromURL("https://books.lapw.at/", "", []*ScrapeConfig{config0, config1}, 0, func(index int, name string) {})
+
+	got := len(c.SubChapters())
+	want := 2
+
+	if got != want {
+		t.Errorf("got %v, wanted %v", got, want)
+	}
+
+}
+
+func TestReverse(t *testing.T) {
+
+	config0 := NewScrapeConfig()
+	config0.Reverse = true
+
+	config1 := NewScrapeConfig()
+
+	c := NewChapterFromURL("https://books.lapw.at/", "", []*ScrapeConfig{config0, config1}, 0, func(index int, name string) {})
+
+	got := c.SubChapters()[0].Name()
+	want := "The Twelve-Factor App"
+
+	if got != want {
+		t.Errorf("got %v, wanted %v", got, want)
+	}
+
+}
+
+func TestNotInclude(t *testing.T) {
+
+	config := NewScrapeConfig()
+	config.Include = false
+
+	c := NewChapterFromURL("https://books.lapw.at/", "", []*ScrapeConfig{config}, 0, func(index int, name string) {})
+
+	got := c.Content()
+	want := ""
+
+	if got != want {
+		t.Errorf("got %v, wanted %v", got, want)
+	}
+
+}

cmd/get.go

@@ -3,167 +3,180 @@ package cmd
 import (
 	"errors"
 	"fmt"
-	"log"
-	"os"
-	"os/exec"
 	"strings"
 
-	md "github.com/JohannesKaufmann/html-to-markdown"
+	"github.com/spf13/cobra"
-	epub "github.com/bmaupin/go-epub"
-	cobra "github.com/spf13/cobra"
 
 	"github.com/lapwat/papeer/book"
 )
 
-var quiet, stdout, recursive, include bool
+type GetOptions struct {
-var format, output, selector string
+	// url string
-var limit, delay int
+
+	name   string
+	author string
+	Format string
+	output string
+	images bool
+	// ImagesOnly bool
+	quiet bool
+
+	Selector []string
+	depth    int
+	limit    int
+	offset   int
+	reverse  bool
+	delay    int
+	threads  int
+	// includeUrl bool
+	include     bool
+	useLinkName bool
+}
+
+var getOpts *GetOptions
+
+func init() {
+	getOpts = &GetOptions{}
+
+	getCmd.PersistentFlags().StringVarP(&getOpts.name, "name", "n", "", "book name (default: page title)")
+	getCmd.PersistentFlags().StringVarP(&getOpts.author, "author", "a", "", "book author")
+	getCmd.PersistentFlags().StringVarP(&getOpts.Format, "format", "f", "md", "file format [stdout, md, epub, mobi]")
+	getCmd.PersistentFlags().StringVarP(&getOpts.output, "output", "", "", "file name (default: book name)")
+	getCmd.PersistentFlags().BoolVarP(&getOpts.images, "images", "", false, "retrieve images only")
+	getCmd.PersistentFlags().BoolVarP(&getOpts.quiet, "quiet", "q", false, "hide progress bar")
+
+	// common with list command
+	getCmd.Flags().StringSliceVarP(&getOpts.Selector, "selector", "s", []string{}, "table of contents CSS selector")
+	getCmd.Flags().IntVarP(&getOpts.depth, "depth", "d", 0, "scraping depth")
+	getCmd.Flags().IntVarP(&getOpts.limit, "limit", "l", -1, "limit number of chapters, use with depth/selector")
+	getCmd.Flags().IntVarP(&getOpts.offset, "offset", "o", 0, "skip first chapters, use with depth/selector")
+	getCmd.Flags().BoolVarP(&getOpts.reverse, "reverse", "r", false, "reverse chapter order")
+	getCmd.Flags().IntVarP(&getOpts.delay, "delay", "", -1, "time in milliseconds to wait before downloading next chapter, use with depth/selector")
+	getCmd.Flags().IntVarP(&getOpts.threads, "threads", "t", -1, "download concurrency, use with depth/selector")
+	getCmd.Flags().BoolVarP(&getOpts.include, "include", "i", false, "include URL as first chapter, use with depth/selector")
+	getCmd.Flags().BoolVarP(&getOpts.useLinkName, "use-link-name", "", false, "use link name for chapter title")
+
+	rootCmd.AddCommand(getCmd)
+}
 
 var getCmd = &cobra.Command{
-	Use:   "get",
+	Use:     "get URL",
 	Short:   "Scrape URL content",
+	Example: "papeer get https://www.eff.org/cyberspace-independence",
 	Args: func(cmd *cobra.Command, args []string) error {
 		if len(args) < 1 {
 			return errors.New("requires an URL argument")
 		}
 
 		formatEnum := map[string]bool{
+			"stdout": true,
 			"md":     true,
 			"epub":   true,
 			"mobi":   true,
 		}
-		if formatEnum[format] != true {
+
-			return fmt.Errorf("invalid format specified: %s", format)
+		if formatEnum[getOpts.Format] != true {
+			return fmt.Errorf("invalid format specified: %s", getOpts.Format)
 		}
 
-		if format == "epub" || format == "mobi" {
+		// add .mobi to filename if not specified
-			if stdout {
+		if getOpts.Format == "mobi" {
-				return errors.New("cannot print EPUB/MOBI file to standard output")
+			if len(getOpts.output) > 0 && strings.HasSuffix(getOpts.output, ".mobi") == false {
+				getOpts.output = fmt.Sprintf("%s.mobi", getOpts.output)
 			}
 		}
 
-		if format == "mobi" {
+		if cmd.Flags().Changed("include") && getOpts.depth == 0 && len(getOpts.Selector) == 0 {
-			if len(output) > 0 && strings.HasSuffix(output, ".mobi") == false {
+			return errors.New("cannot use include option if depth/selector is not specified")
-				output = fmt.Sprintf("%s.mobi", output)
-			}
 		}
 
-		if cmd.Flags().Changed("selector") && recursive == false {
+		if cmd.Flags().Changed("limit") && getOpts.depth == 0 && len(getOpts.Selector) == 0 {
-			return errors.New("cannot use selector option if not in recursive mode")
+			return errors.New("cannot use limit option if depth/selector is not specified")
 		}
 
-		if cmd.Flags().Changed("include") && recursive == false {
+		if cmd.Flags().Changed("offset") && getOpts.depth == 0 && len(getOpts.Selector) == 0 {
-			return errors.New("cannot use include option if not in recursive mode")
+			return errors.New("cannot use offset option if depth/selector is not specified")
 		}
 
-		if cmd.Flags().Changed("limit") && recursive == false {
+		if cmd.Flags().Changed("reverse") && getOpts.depth == 0 && len(getOpts.Selector) == 0 {
-			return errors.New("cannot use limit option if not in recursive mode")
+			return errors.New("cannot use reverse option if depth/selector is not specified")
 		}
 
-		if cmd.Flags().Changed("delay") && recursive == false {
+		if cmd.Flags().Changed("delay") && getOpts.depth == 0 && len(getOpts.Selector) == 0 {
-			return errors.New("cannot use delay option if not in recursive mode")
+			return errors.New("cannot use delay option if depth/selector is not specified")
+		}
+
+		if cmd.Flags().Changed("threads") && getOpts.depth == 0 && len(getOpts.Selector) == 0 {
+			return errors.New("cannot use threads option if depth/selector is not specified")
+		}
+
+		if cmd.Flags().Changed("use-link-name") && getOpts.depth == 0 && len(getOpts.Selector) == 0 {
+			return errors.New("cannot use use-link-name option if depth/selector is not specified")
+		}
+
+		if cmd.Flags().Changed("delay") && cmd.Flags().Changed("threads") {
+			return errors.New("cannot use delay and threads options at the same time")
 		}
 
 		return nil
 	},
 	Run: func(cmd *cobra.Command, args []string) {
 		url := args[0]
-		var b book.Book
 
-		if recursive {
+		// fill selector array with empty selectors to match depth
-			b = book.NewBookFromURL(url, selector, include, limit, delay)
+		getOpts.Selector = append(getOpts.Selector, "")
-		} else {
+		for len(getOpts.Selector) < getOpts.depth+1 {
-			c := book.NewChapterFromURL(url)
+			getOpts.Selector = append(getOpts.Selector, "")
-			b = book.New(c.Name(), c.Author())
-			b.AddChapter(c)
 		}
 
-		// if quiet == false {
+		// generate config for each level
-		// 	metadata := fmt.Sprintf("URL     : %s\nTitle   : %s\nAuthor  : %s\nLength  : %d\nExcerpt : %s\nSiteName: %s\nImage   : %s\nFavicon : %s", url, article.Title, article.Byline, article.Length, article.Excerpt, article.SiteName, article.Image, article.Favicon)
+		configs := make([]*book.ScrapeConfig, len(getOpts.Selector))
-		// 	fmt.Println(metadata)
+		for index, s := range getOpts.Selector {
-		// }
+			config := book.NewScrapeConfig()
+			config.Selector = s
+			config.Quiet = getOpts.quiet
+			config.Limit = getOpts.limit
+			config.Offset = getOpts.offset
+			config.Reverse = getOpts.reverse
+			config.Delay = getOpts.delay
+			config.Threads = getOpts.threads
+			config.ImagesOnly = getOpts.images
+			config.Include = getOpts.include
+			config.UseLinkName = getOpts.useLinkName
 
-		if len(output) == 0 {
+			// do not use link name for root level as there is not parent link
-			// set default output
+			if index == 0 {
-			output = strings.ReplaceAll(b.Name(), " ", "_")
+				config.UseLinkName = false
-			output = strings.ReplaceAll(output, "/", "")
-			output = fmt.Sprintf("%s.%s", output, format)
 			}
 
-		if format == "md" {
+			// always include last level by default
-			f, err := os.Create(output)
+			if index == len(getOpts.Selector)-1 {
-			if err != nil {
+				config.Include = true
-				log.Fatal(err)
 			}
 
-			defer f.Close()
+			configs[index] = config
-
-			for _, c := range b.Chapters() {
-				content, err := md.NewConverter("", true, nil).ConvertString(c.Content())
-				if err != nil {
-					log.Fatal(err)
 		}
 
-				if stdout {
+		c := book.NewChapterFromURL(url, "", configs, 0, func(index int, name string) {})
-					fmt.Println(content)
-				} else {
 
-					_, err := f.WriteString(content)
+		if getOpts.Format == "stdout" {
-					if err != nil {
+			markdown := book.ToMarkdownString(c)
-						log.Fatal(err)
+			fmt.Println(markdown)
 		}
 
-				}
+		if getOpts.Format == "md" {
+			filename := book.ToMarkdown(c, getOpts.output)
+			fmt.Printf("Markdown saved to \"%s\"\n", filename)
 		}
 
-			if stdout == false {
+		if getOpts.Format == "epub" {
-				fmt.Printf("Markdown saved to \"%s\"\n", output)
+			filename := book.ToEpub(c, getOpts.output)
-			}
+			fmt.Printf("Ebook saved to \"%s\"\n", filename)
 		}
 
-		if format == "epub" {
+		if getOpts.Format == "mobi" {
-			e := epub.NewEpub(b.Name())
+			filename := book.ToMobi(c, getOpts.output)
-			e.SetAuthor(b.Author())
+			fmt.Printf("Ebook saved to \"%s\"\n", filename)
-
-			for _, c := range b.Chapters() {
-				e.AddSection(c.Content(), c.Name(), "", "")
-			}
-
-			err := e.Write(output)
-			if err != nil {
-				log.Fatal(err)
-			}
-
-			fmt.Printf("Ebook saved to \"%s\"\n", output)
-		}
-
-		if format == "mobi" {
-			e := epub.NewEpub(b.Name())
-			e.SetAuthor(b.Author())
-
-			for _, chapter := range b.Chapters() {
-				e.AddSection(chapter.Content(), chapter.Name(), "", "")
-			}
-
-			outputEPUB := strings.ReplaceAll(output, ".mobi", ".epub")
-
-			err := e.Write(outputEPUB)
-			if err != nil {
-				log.Fatal(err)
-			}
-
-			exec.Command("kindlegen", outputEPUB).Run()
-			// exec command always return status 1 even if it fails
-			// if err != nil {
-			// 	log.Fatal(err)
-			// }
-
-			fmt.Printf("Ebook saved to \"%s\"\n", output)
-
-			err2 := os.Remove(outputEPUB)
-			if err2 != nil {
-				log.Fatal(err)
-			}
 		}
 	},
 }

cmd/list.go

@@ -5,15 +5,54 @@ import (
 	"fmt"
 	"log"
 	urllib "net/url"
+	"os"
 	"strings"
 
-	colly "github.com/gocolly/colly/v2"
+	"github.com/jedib0t/go-pretty/v6/table"
 	cobra "github.com/spf13/cobra"
+
+	"github.com/lapwat/papeer/book"
 )
 
+type ListOptions struct {
+	// url string
+
+	Selector []string
+	depth    int
+	limit    int
+	offset   int
+	reverse  bool
+	delay    int
+	threads  int
+	// includeUrl bool
+	include     bool
+	useLinkName bool
+}
+
+var listOpts *ListOptions
+
+func init() {
+	listOpts = &ListOptions{}
+
+	// common with get command
+	listCmd.Flags().StringSliceVarP(&listOpts.Selector, "selector", "s", []string{}, "table of contents CSS selector")
+	listCmd.Flags().IntVarP(&listOpts.depth, "depth", "d", 0, "scraping depth")
+	listCmd.Flags().IntVarP(&listOpts.limit, "limit", "l", -1, "limit number of chapters, use with depth/selector")
+	listCmd.Flags().IntVarP(&listOpts.offset, "offset", "o", 0, "skip first chapters, use with depth/selector")
+	listCmd.Flags().BoolVarP(&listOpts.reverse, "reverse", "r", false, "reverse chapter order")
+	listCmd.Flags().IntVarP(&listOpts.delay, "delay", "", -1, "time in milliseconds to wait before downloading next chapter, use with depth/selector")
+	listCmd.Flags().IntVarP(&listOpts.threads, "threads", "t", -1, "download concurrency, use with depth/selector")
+	listCmd.Flags().BoolVarP(&listOpts.include, "include", "i", false, "include URL as first chapter, use with depth/selector")
+	listCmd.Flags().BoolVarP(&listOpts.useLinkName, "use-link-name", "", false, "use link name for chapter title")
+
+	rootCmd.AddCommand(listCmd)
+}
+
 var listCmd = &cobra.Command{
-	Use:   "ls",
+	Use:     "list URL",
-	Short: "Print table of content",
+	Aliases: []string{"ls"},
+	Short:   "Print URL table of contents",
+	Example: "papeer list https://12factor.net/ -s 'section.concrete>article>h2>a'",
 	Args: func(cmd *cobra.Command, args []string) error {
 		if len(args) < 1 {
 			return errors.New("requires an URL argument")
@@ -21,49 +60,46 @@ var listCmd = &cobra.Command{
 		return nil
 	},
 	Run: func(cmd *cobra.Command, args []string) {
+		if len(listOpts.Selector) == 0 {
+			listOpts.Selector = []string{""}
+		}
+
 		base, err := urllib.Parse(args[0])
 		if err != nil {
 			log.Fatal(err)
 		}
 
-		if selector == "" {
+		links, path, _, err := book.GetLinks(base, listOpts.Selector[0], listOpts.limit, listOpts.offset, listOpts.reverse, listOpts.include)
-			selector = "a"
-		}
-
-		// visit and count link classes
-		classesLinks := map[string][]map[string]string{}
-		classesCount := map[string]int{}
-		classMax := ""
-
-		c := colly.NewCollector()
-		c.OnHTML(selector, func(e *colly.HTMLElement) {
-			href := e.Attr("href")
-			text := strings.TrimSpace(e.Text)
-			class := e.Attr("class")
-
-			if cmd.Flags().Changed("selector") || class != "" && text != "" {
-				classesLinks[class] = append(classesLinks[class], map[string]string{
-					"href": href,
-					"text": text,
-				})
-
-				classesCount[class]++
-
-				if classesCount[class] > classesCount[classMax] {
-					classMax = class
-				}
-			}
-		})
-
-		c.Visit(base.String())
-		for index, link := range classesLinks[classMax] {
-			u, err := base.Parse(link["href"])
 		if err != nil {
 			log.Fatal(err)
 		}
 
-			fmt.Printf("Chapter %d: %s %s\n", index+1, link["text"], u.String())
+		t := table.NewWriter()
+		t.SetOutputMirror(os.Stdout)
+		t.Style().Options.DrawBorder = false
+		t.Style().Options.SeparateColumns = false
+		t.Style().Options.SeparateHeader = false
+
+		// format selector path
+		pathArray := strings.Split(path, "<")
+		// reverse path
+		for i, j := 0, len(pathArray)-1; i < j; i, j = i+1, j-1 {
+			pathArray[i], pathArray[j] = pathArray[j], pathArray[i]
 		}
+		pathFormatted := strings.Join(pathArray, ">")
+
+		t.AppendHeader(table.Row{"#", "Name", fmt.Sprintf("Url [%s]", pathFormatted)})
+
+		for index, link := range links {
+			u, err := base.Parse(link.Href())
+			if err != nil {
+				log.Fatal(err)
+			}
+
+			t.AppendRow([]interface{}{index + 1, link.Text(), u.String()})
+		}
+
+		t.Render()
 
 	},
 }

cmd/root.go

@@ -21,18 +21,3 @@ func Execute() {
 		os.Exit(1)
 	}
 }
-
-func init() {
-	rootCmd.PersistentFlags().StringVarP(&format, "format", "f", "md", "file format [md, epub, mobi]")
-	rootCmd.PersistentFlags().StringVarP(&output, "output", "o", "", "output file")
-	rootCmd.PersistentFlags().StringVarP(&selector, "selector", "s", "", "table of content CSS selector")
-	rootCmd.PersistentFlags().BoolVarP(&recursive, "recursive", "r", false, "create one chapter per natigation item")
-	rootCmd.PersistentFlags().BoolVarP(&include, "include", "i", false, "include URL as first chapter, in resursive mode")
-	rootCmd.PersistentFlags().BoolVarP(&quiet, "quiet", "q", false, "do not show logs")
-	rootCmd.PersistentFlags().BoolVarP(&stdout, "stdout", "", false, "print to standard output")
-	rootCmd.PersistentFlags().IntVarP(&limit, "limit", "l", -1, "limit number of chapters, in recursive mode")
-	rootCmd.PersistentFlags().IntVarP(&delay, "delay", "d", -1, "time to wait before downloading next chapter, in milliseconds")
-
-	rootCmd.AddCommand(getCmd)
-	rootCmd.AddCommand(listCmd)
-}

cmd/utils.go

@@ -1,5 +0,0 @@
-package cmd
-
-func getTableOfContent() {
-
-}

cmd/version.go

@@ -14,6 +14,6 @@ var versionCmd = &cobra.Command{
 	Use:   "version",
 	Short: "Print the version number of papeer",
 	Run: func(cmd *cobra.Command, args []string) {
-		fmt.Println("papeer v0.0.2")
+		fmt.Println("papeer v0.5.0")
 	},
 }

go.mod

@@ -10,33 +10,45 @@ require (
 )
 
 require (
+	github.com/PuerkitoBio/goquery v1.7.1
 	github.com/gocolly/colly/v2 v2.1.0
 	github.com/gosuri/uilive v0.0.4 // indirect
 	github.com/gosuri/uiprogress v0.0.1
 )
 
 require (
-	github.com/PuerkitoBio/goquery v1.5.1 // indirect
 	github.com/andybalholm/cascadia v1.2.0 // indirect
 	github.com/antchfx/htmlquery v1.2.3 // indirect
-	github.com/antchfx/xmlquery v1.2.4 // indirect
+	github.com/antchfx/xmlquery v1.3.6 // indirect
-	github.com/antchfx/xpath v1.1.8 // indirect
+	github.com/antchfx/xpath v1.2.0 // indirect
 	github.com/go-shiori/dom v0.0.0-20210627111528-4e4722cd0d65 // indirect
 	github.com/gobwas/glob v0.2.3 // indirect
-	github.com/gofrs/uuid v3.1.0+incompatible // indirect
+	github.com/gofrs/uuid v4.0.0+incompatible // indirect
 	github.com/gogs/chardet v0.0.0-20191104214054-4b6791f73a28 // indirect
-	github.com/golang/groupcache v0.0.0-20200121045136-8c9f03a8e57e // indirect
+	github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
 	github.com/golang/protobuf v1.5.2 // indirect
 	github.com/inconshreveable/mousetrap v1.0.0 // indirect
+	github.com/jedib0t/go-pretty/v6 v6.2.4 // indirect
+	github.com/json-iterator/go v1.1.11 // indirect
 	github.com/kennygrant/sanitize v1.2.4 // indirect
-	github.com/mattn/go-isatty v0.0.3 // indirect
+	github.com/mattn/go-isatty v0.0.14 // indirect
+	github.com/mattn/go-runewidth v0.0.13 // indirect
+	github.com/mitchellh/colorstring v0.0.0-20190213212951-d06e56a500db // indirect
+	github.com/mmcdole/gofeed v1.1.3 // indirect
+	github.com/mmcdole/goxpp v0.0.0-20181012175147-0068e33feabf // indirect
+	github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421 // indirect
+	github.com/modern-go/reflect2 v1.0.1 // indirect
+	github.com/rivo/uniseg v0.2.0 // indirect
 	github.com/saintfish/chardet v0.0.0-20120816061221-3af4cd4741ca // indirect
+	github.com/schollz/progressbar/v3 v3.8.3 // indirect
 	github.com/sirupsen/logrus v1.8.1 // indirect
 	github.com/spf13/pflag v1.0.5 // indirect
-	github.com/temoto/robotstxt v1.1.1 // indirect
+	github.com/temoto/robotstxt v1.1.2 // indirect
+	golang.org/x/crypto v0.0.0-20210817164053-32db794688a5 // indirect
 	golang.org/x/net v0.0.0-20210614182718-04defd469f4e // indirect
-	golang.org/x/sys v0.0.0-20210514084401-e8d321eab015 // indirect
+	golang.org/x/sys v0.0.0-20210910150752-751e447fb3d0 // indirect
+	golang.org/x/term v0.0.0-20210615171337-6886f2dfbf5b // indirect
 	golang.org/x/text v0.3.6 // indirect
 	google.golang.org/appengine v1.6.7 // indirect
-	google.golang.org/protobuf v1.26.0 // indirect
+	google.golang.org/protobuf v1.27.1 // indirect
 )

go.sum

@@ -42,8 +42,9 @@ github.com/BurntSushi/xgb v0.0.0-20160522181843-27f122750802/go.mod h1:IVnqGOEym
 github.com/JohannesKaufmann/html-to-markdown v1.3.0 h1:K/p4cq8Ib13hcSVcKQNfKCSWw93CYW5pAjY0fl85has=
 github.com/JohannesKaufmann/html-to-markdown v1.3.0/go.mod h1:JNSClIRYICFDiFhw6RBhBeWGnMSSKVZ6sPQA+TK4tyM=
 github.com/OneOfOne/xxhash v1.2.2/go.mod h1:HSdplMjZKSmBqAxg5vPj2TmRDmfkzw+cTzAElWljhcU=
-github.com/PuerkitoBio/goquery v1.5.1 h1:PSPBGne8NIUWw+/7vFBV+kG2J/5MOjbzc7154OaKCSE=
 github.com/PuerkitoBio/goquery v1.5.1/go.mod h1:GsLWisAFVj4WgDibEWF4pvYnkVQBpKBKeU+7zCJoLcc=
+github.com/PuerkitoBio/goquery v1.7.1 h1:oE+T06D+1T7LNrn91B4aERsRIeCLJ/oPSa6xB9FPnz4=
+github.com/PuerkitoBio/goquery v1.7.1/go.mod h1:XY0pP4kfraEmmV1O7Uf6XyjoslwsneBbgeDjLYuN8xY=
 github.com/alecthomas/template v0.0.0-20160405071501-a0175ee3bccc/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc=
 github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf/go.mod h1:ybxpYRFXyAe+OPACYpWeL0wqObRcbAqCMya13uyzqw0=
 github.com/andybalholm/cascadia v1.1.0/go.mod h1:GsXiBklL0woXo1j/WYWtSYYC4ouU9PqHO0sqidkEA4Y=
@@ -51,11 +52,14 @@ github.com/andybalholm/cascadia v1.2.0 h1:vuRCkM5Ozh/BfmsaTm26kbjm0mIOM3yS5Ek/F5
 github.com/andybalholm/cascadia v1.2.0/go.mod h1:YCyR8vOZT9aZ1CHEd8ap0gMVm2aFgxBp0T0eFw1RUQY=
 github.com/antchfx/htmlquery v1.2.3 h1:sP3NFDneHx2stfNXCKbhHFo8XgNjCACnU/4AO5gWz6M=
 github.com/antchfx/htmlquery v1.2.3/go.mod h1:B0ABL+F5irhhMWg54ymEZinzMSi0Kt3I2if0BLYa3V0=
-github.com/antchfx/xmlquery v1.2.4 h1:T/SH1bYdzdjTMoz2RgsfVKbM5uWh3gjDYYepFqQmFv4=
 github.com/antchfx/xmlquery v1.2.4/go.mod h1:KQQuESaxSlqugE2ZBcM/qn+ebIpt+d+4Xx7YcSGAIrM=
+github.com/antchfx/xmlquery v1.3.6 h1:kaEVzH1mNo/2AJZrhZjAaAUTy2Nn2zxGfYYU8jWfXOo=
+github.com/antchfx/xmlquery v1.3.6/go.mod h1:64w0Xesg2sTaawIdNqMB+7qaW/bSqkQm+ssPaCMWNnc=
 github.com/antchfx/xpath v1.1.6/go.mod h1:Yee4kTMuNiPYJ7nSNorELQMr1J33uOpXDMByNYhvtNk=
-github.com/antchfx/xpath v1.1.8 h1:PcL6bIX42Px5usSx6xRYw/wjB3wYGkj0MJ9MBzEKVgk=
 github.com/antchfx/xpath v1.1.8/go.mod h1:Yee4kTMuNiPYJ7nSNorELQMr1J33uOpXDMByNYhvtNk=
+github.com/antchfx/xpath v1.1.10/go.mod h1:Yee4kTMuNiPYJ7nSNorELQMr1J33uOpXDMByNYhvtNk=
+github.com/antchfx/xpath v1.2.0 h1:mbwv7co+x0RwgeGAOHdrKy89GvHaGvxxBtPK0uF9Zr8=
+github.com/antchfx/xpath v1.2.0/go.mod h1:i54GszH55fYfBmoZXapTHN8T8tkcHfRgLyVwwqzXNcs=
 github.com/antihax/optional v1.0.0/go.mod h1:uupD/76wgC+ih3iEmQUL+0Ugr19nfwCT1kdvxnR2qWY=
 github.com/armon/circbuf v0.0.0-20150827004946-bbbad097214e/go.mod h1:3U/XgcO3hCbHZ8TKRvWD2dDTCfh9M9ya+I9JpbB7O8o=
 github.com/armon/consul-api v0.0.0-20180202201655-eb2c6b5be1b6/go.mod h1:grANhF5doyWs3UAsr3K4I6qtAmlQcZDesFNEHPZAzj8=
@@ -83,6 +87,7 @@ github.com/coreos/go-semver v0.3.0/go.mod h1:nnelYz7RCh+5ahJtPPxZlU+153eP4D4r3Ee
 github.com/coreos/go-systemd v0.0.0-20190321100706-95778dfbb74e/go.mod h1:F5haX7vjVVG0kc13fIWeqUViNPyEJxv/OmvnBo0Yme4=
 github.com/coreos/go-systemd/v22 v22.3.2/go.mod h1:Y58oyj3AT4RCenI/lSvhwexgC+NSVTIJ3seZv2GcEnc=
 github.com/coreos/pkg v0.0.0-20180928190104-399ea9e2e55f/go.mod h1:E3G3o1h8I7cfcXa63jLwjI0eiQQMgzzUDFVpN/nH/eA=
+github.com/cpuguy83/go-md2man/v2 v2.0.0-20190314233015-f79a8a8ca69d/go.mod h1:maD7wRr/U5Z6m/iR4s+kqSMx2CaBsrgA7czyZG/E6dU=
 github.com/cpuguy83/go-md2man/v2 v2.0.0/go.mod h1:maD7wRr/U5Z6m/iR4s+kqSMx2CaBsrgA7czyZG/E6dU=
 github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
 github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
@@ -99,6 +104,7 @@ github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7
 github.com/fatih/color v1.7.0/go.mod h1:Zm6kSWBoL9eyXnKyktHP6abPY2pDugNf5KwzbycvMj4=
 github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo=
 github.com/fsnotify/fsnotify v1.4.9/go.mod h1:znqG4EE+3YCdAaPaxE2ZRY/06pZUdp0tY4IgpuI1SZQ=
+github.com/fzipp/gocyclo v0.3.1/go.mod h1:DJHO6AUmbdqj2ET4Z9iArSuwWgYDRryYt2wASxc7x3E=
 github.com/ghodss/yaml v1.0.0/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04=
 github.com/go-gl/glfw v0.0.0-20190409004039-e6da0acd62b1/go.mod h1:vR7hzQXu2zJy9AVAgeJqvqgH9Q5CA+iKCZ2gyEVpxRU=
 github.com/go-gl/glfw/v3.3/glfw v0.0.0-20191125211704-12ad95a8df72/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8=
@@ -118,8 +124,9 @@ github.com/gocolly/colly v1.2.0/go.mod h1:Hof5T3ZswNVsOHYmba1u03W65HDWgpV5HifSuu
 github.com/gocolly/colly/v2 v2.1.0 h1:k0DuZkDoCsx51bKpRJNEmcxcp+W5N8ziuwGaSDuFoGs=
 github.com/gocolly/colly/v2 v2.1.0/go.mod h1:I2MuhsLjQ+Ex+IzK3afNS8/1qP3AedHOusRPcRdC5o0=
 github.com/godbus/dbus/v5 v5.0.4/go.mod h1:xhWf0FNVPg57R7Z0UbKHbJfkEywrmjJnf7w5xrFpKfA=
-github.com/gofrs/uuid v3.1.0+incompatible h1:q2rtkjaKT4YEr6E1kamy0Ha4RtepWlQBedyHx0uzKwA=
 github.com/gofrs/uuid v3.1.0+incompatible/go.mod h1:b2aQJv3Z4Fp6yNu3cdSllBxTCLRxnplIgP/c0N/04lM=
+github.com/gofrs/uuid v4.0.0+incompatible h1:1SD/1F5pU8p29ybwgQSwpQk+mwdRrXCYuPhW6m+TnJw=
+github.com/gofrs/uuid v4.0.0+incompatible/go.mod h1:b2aQJv3Z4Fp6yNu3cdSllBxTCLRxnplIgP/c0N/04lM=
 github.com/gogo/protobuf v1.1.1/go.mod h1:r8qH/GZQm5c6nD/R0oafs1akxWv10x8SbQlK7atdtwQ=
 github.com/gogo/protobuf v1.2.1/go.mod h1:hp+jE20tsWTFYpLwKvXlhS1hjn+gTNwPg2I6zVXpSg4=
 github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69NZV8Q=
@@ -129,8 +136,9 @@ github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfU
 github.com/golang/groupcache v0.0.0-20190129154638-5b532d6fd5ef/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
 github.com/golang/groupcache v0.0.0-20190702054246-869f871628b6/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
 github.com/golang/groupcache v0.0.0-20191227052852-215e87163ea7/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
-github.com/golang/groupcache v0.0.0-20200121045136-8c9f03a8e57e h1:1r7pUrabqp18hOBcwBwiTsbnFeTZHV9eER/QT5JVZxY=
 github.com/golang/groupcache v0.0.0-20200121045136-8c9f03a8e57e/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
+github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da h1:oI5xCqsCo564l8iNU+DwB5epxmsaqB+rhGL0m5jtYqE=
+github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
 github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
 github.com/golang/mock v1.2.0/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
 github.com/golang/mock v1.3.1/go.mod h1:sBzyDLLjw3U8JLTeZvSv8jJB+tU5PVekmnlKIyFUx0Y=
@@ -225,12 +233,17 @@ github.com/ianlancetaylor/demangle v0.0.0-20200824232613-28f6c0f3b639/go.mod h1:
 github.com/inconshreveable/mousetrap v1.0.0 h1:Z8tu5sraLXCXIcARxBp/8cbvlwVa7Z1NHg9XEKhtSvM=
 github.com/inconshreveable/mousetrap v1.0.0/go.mod h1:PxqpIevigyE2G7u3NXJIT2ANytuPF1OarO4DADm73n8=
 github.com/jawher/mow.cli v1.1.0/go.mod h1:aNaQlc7ozF3vw6IJ2dHjp2ZFiA4ozMIYY6PyuRJwlUg=
+github.com/jedib0t/go-pretty/v6 v6.2.4 h1:wdaj2KHD2W+mz8JgJ/Q6L/T5dB7kyqEFI16eLq7GEmk=
+github.com/jedib0t/go-pretty/v6 v6.2.4/go.mod h1:+nE9fyyHGil+PuISTCrp7avEdo6bqoMwqZnuiK2r2a0=
 github.com/jonboulle/clockwork v0.1.0/go.mod h1:Ii8DK3G1RaLaWxj9trq07+26W01tbo22gdxWY5EU2bo=
+github.com/json-iterator/go v1.1.10/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4=
+github.com/json-iterator/go v1.1.11 h1:uVUAXhF2To8cbw/3xN3pxj6kk7TYKs98NIrTqPlMWAQ=
 github.com/json-iterator/go v1.1.11/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4=
 github.com/jstemmer/go-junit-report v0.0.0-20190106144839-af01ea7f8024/go.mod h1:6v2b51hI/fHJwM22ozAgKL4VKDeJcHhJFhtBdhmNjmU=
 github.com/jstemmer/go-junit-report v0.9.1/go.mod h1:Brl9GWCQeLvo8nXZwPNNblvFj/XSXhF0NWZEnDohbsk=
 github.com/jtolds/gls v4.20.0+incompatible/go.mod h1:QJZ7F/aHp+rZTRtaJ1ow/lLfFfVYBRgL+9YlvaHOwJU=
 github.com/julienschmidt/httprouter v1.2.0/go.mod h1:SYymIcj16QtmaHHD7aYtjjsJG7VTCxuUUipMqKk8s4w=
+github.com/k0kubun/go-ansi v0.0.0-20180517002512-3bf9e2903213/go.mod h1:vNUNkEQ1e29fT/6vq2aBdFsgNPmy8qMdSay1npru+Sw=
 github.com/kennygrant/sanitize v1.2.4 h1:gN25/otpP5vAsO2djbMhF/LQX6R7+O1TB4yv8NzpJ3o=
 github.com/kennygrant/sanitize v1.2.4/go.mod h1:LGsjYYtgxbetdg5owWB2mpgUL6e2nfw2eObZ0u0qvak=
 github.com/kisielk/errcheck v1.1.0/go.mod h1:EZBBE59ingxPouuu3KfxchcWSUPOHkagtvWXihfKN4Q=
@@ -245,11 +258,18 @@ github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
 github.com/magiconair/properties v1.8.0/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ=
 github.com/magiconair/properties v1.8.5/go.mod h1:y3VJvCyxH9uVvJTWEGAELF3aiYNyPKd5NZ3oSwXrF60=
 github.com/mattn/go-colorable v0.0.9/go.mod h1:9vuHe8Xs5qXnSaW/c/ABM9alt+Vo+STaOChaDxuIBZU=
-github.com/mattn/go-isatty v0.0.3 h1:ns/ykhmWi7G9O+8a448SecJU3nSMBXJfqQkl0upE1jI=
 github.com/mattn/go-isatty v0.0.3/go.mod h1:M+lRXTBqGeGNdLjl/ufCoiOlB5xdOkqRJdNxMWT7Zi4=
+github.com/mattn/go-isatty v0.0.14 h1:yVuAays6BHfxijgZPzw+3Zlu5yQgKGP2/hcQbHb7S9Y=
+github.com/mattn/go-isatty v0.0.14/go.mod h1:7GGIvUiUoEMVVmxf/4nioHXj79iQHKdU27kJ6hsGG94=
+github.com/mattn/go-runewidth v0.0.9 h1:Lm995f3rfxdpd6TSmuVCHVb/QhupuXlYr8sCI/QdE+0=
+github.com/mattn/go-runewidth v0.0.9/go.mod h1:H031xJmbD/WCDINGzjvQ9THkh0rPKHF+m2gUSrubnMI=
+github.com/mattn/go-runewidth v0.0.13 h1:lTGmDsbAYt5DmK6OnoV7EuIF1wEIFAcxld6ypU4OSgU=
+github.com/mattn/go-runewidth v0.0.13/go.mod h1:Jdepj2loyihRzMpdS35Xk/zdY8IAYHsh153qUoGf23w=
 github.com/matttproud/golang_protobuf_extensions v1.0.1/go.mod h1:D8He9yQNgCq6Z5Ld7szi9bcBfOoFv/3dc6xSMkL2PC0=
 github.com/miekg/dns v1.0.14/go.mod h1:W1PPwlIAgtquWBMBEV9nkV9Cazfe8ScdGz/Lj7v3Nrg=
 github.com/mitchellh/cli v1.0.0/go.mod h1:hNIlj7HEI86fIcpObd7a0FcrxTWetlwJDGcceTlRvqc=
+github.com/mitchellh/colorstring v0.0.0-20190213212951-d06e56a500db h1:62I3jR2EmQ4l5rM/4FEfDWcRD+abF5XlKShorW5LRoQ=
+github.com/mitchellh/colorstring v0.0.0-20190213212951-d06e56a500db/go.mod h1:l0dey0ia/Uv7NcFFVbCLtqEBQbrT4OCwCSKTEv6enCw=
 github.com/mitchellh/go-homedir v1.0.0/go.mod h1:SfyaCUpYCn1Vlf4IUYiD9fPX4A5wJrkLzIz1N1q0pr0=
 github.com/mitchellh/go-homedir v1.1.0/go.mod h1:SfyaCUpYCn1Vlf4IUYiD9fPX4A5wJrkLzIz1N1q0pr0=
 github.com/mitchellh/go-testing-interface v1.0.0/go.mod h1:kRemZodwjscx+RGhAo8eIhFbs2+BFgRtFPeD/KE+zxI=
@@ -258,8 +278,14 @@ github.com/mitchellh/iochan v1.0.0/go.mod h1:JwYml1nuB7xOzsp52dPpHFffvOCDupsG0Qu
 github.com/mitchellh/mapstructure v0.0.0-20160808181253-ca63d7c062ee/go.mod h1:FVVH3fgwuzCH5S8UJGiWEs2h04kUh9fWfEaFds41c1Y=
 github.com/mitchellh/mapstructure v1.1.2/go.mod h1:FVVH3fgwuzCH5S8UJGiWEs2h04kUh9fWfEaFds41c1Y=
 github.com/mitchellh/mapstructure v1.4.1/go.mod h1:bFUtVrKA4DC2yAKiSyO/QUcy7e+RRV2QTWOzhPopBRo=
+github.com/mmcdole/gofeed v1.1.3 h1:pdrvMb18jMSLidGp8j0pLvc9IGziX4vbmvVqmLH6z8o=
+github.com/mmcdole/gofeed v1.1.3/go.mod h1:QQO3maftbOu+hiVOGOZDRLymqGQCos4zxbA4j89gMrE=
+github.com/mmcdole/goxpp v0.0.0-20181012175147-0068e33feabf h1:sWGE2v+hO0Nd4yFU/S/mDBM5plIU8v/Qhfz41hkDIAI=
+github.com/mmcdole/goxpp v0.0.0-20181012175147-0068e33feabf/go.mod h1:pasqhqstspkosTneA62Nc+2p9SOBBYAPbnmRRWPQ0V8=
+github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421 h1:ZqeYNhU3OHLH3mGKHDcjJRFFRrJa6eAM5H+CtDdOsPc=
 github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
 github.com/modern-go/reflect2 v0.0.0-20180701023420-4b7aa43c6742/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0=
+github.com/modern-go/reflect2 v1.0.1 h1:9f412s+6RmYXLWZSEzVVgPGK7C2PphHj5RJrvfx9AWI=
 github.com/modern-go/reflect2 v1.0.1/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0=
 github.com/mwitkow/go-conntrack v0.0.0-20161129095857-cc309e4a2223/go.mod h1:qRWi+5nqEBWmkhHvq77mSJWrCKwh8bxhgT7d/eI7P4U=
 github.com/oklog/ulid v1.3.1/go.mod h1:CirwcVhetQ6Lv90oh/F+FBtV6XMibvdAFo93nm5qn4U=
@@ -268,6 +294,7 @@ github.com/pelletier/go-toml v1.2.0/go.mod h1:5z9KED0ma1S8pY6P1sdut58dfprrGBbd/9
 github.com/pelletier/go-toml v1.9.3/go.mod h1:u1nR/EPcESfeI/szUZKdtJ0xRNbUoANCkoOuaOx1Y+c=
 github.com/pkg/errors v0.8.0/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
 github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
+github.com/pkg/profile v1.2.1/go.mod h1:hJw3o1OdXxsrSjjVksARp5W95eeEaEfptyVZyv6JUPA=
 github.com/pkg/sftp v1.10.1/go.mod h1:lYOWFsE0bwd1+KfKJaKeuokY15vzFx25BLbzYYoAxZI=
 github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
 github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
@@ -282,6 +309,8 @@ github.com/prometheus/common v0.4.0/go.mod h1:TNfzLD0ON7rHzMJeJkieUDPYmFC7Snx/y8
 github.com/prometheus/procfs v0.0.0-20181005140218-185b4288413d/go.mod h1:c3At6R/oaqEKCNdg8wHV1ftS6bRYblBhIjjI8uT2IGk=
 github.com/prometheus/procfs v0.0.0-20190507164030-5867b95ac084/go.mod h1:TjEm7ze935MbeOT/UhFTIMYKhuLP4wbCsTZCD3I8kEA=
 github.com/prometheus/tsdb v0.7.1/go.mod h1:qhTCs0VvXwvX/y3TZrWD7rabWM+ijKTux40TwIPHuXU=
+github.com/rivo/uniseg v0.2.0 h1:S1pD9weZBuJdFmowNwbpi7BJ8TNftyUImj/0WQi72jY=
+github.com/rivo/uniseg v0.2.0/go.mod h1:J6wj4VEh+S6ZtnVlnTBMWIodfgj8LQOQFoIToxlJtxc=
 github.com/rogpeppe/fastuuid v0.0.0-20150106093220-6724a57986af/go.mod h1:XWv6SoW27p1b0cqNHllgS5HIMJraePCO15w5zCzIWYg=
 github.com/rogpeppe/fastuuid v1.2.0/go.mod h1:jVj6XXZzXRy/MSR5jhDC/2q6DgLz+nrA6LYCDYWNEvQ=
 github.com/rogpeppe/go-internal v1.3.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4=
@@ -289,6 +318,8 @@ github.com/russross/blackfriday/v2 v2.0.1/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQD
 github.com/ryanuber/columnize v0.0.0-20160712163229-9b3edd62028f/go.mod h1:sm1tb6uqfes/u+d4ooFouqFdy9/2g9QGwK3SQygK0Ts=
 github.com/saintfish/chardet v0.0.0-20120816061221-3af4cd4741ca h1:NugYot0LIVPxTvN8n+Kvkn6TrbMyxQiuvKdEwFdR9vI=
 github.com/saintfish/chardet v0.0.0-20120816061221-3af4cd4741ca/go.mod h1:uugorj2VCxiV1x+LzaIdVa9b4S4qGAcH6cbhh4qVxOU=
+github.com/schollz/progressbar/v3 v3.8.3 h1:FnLGl3ewlDUP+YdSwveXBaXs053Mem/du+wr7XSYKl8=
+github.com/schollz/progressbar/v3 v3.8.3/go.mod h1:pWnVCjSBZsT2X3nx9HfRdnCDrpbevliMeoEVhStwHko=
 github.com/sean-/seed v0.0.0-20170313163322-e2103e2c3529/go.mod h1:DxrIzT+xaE7yg65j358z/aeFdxmN0P9QXhEzd20vsDc=
 github.com/sebdah/goldie/v2 v2.5.1 h1:hh70HvG4n3T3MNRJN2z/baxPR8xutxo7JVxyi2svl+s=
 github.com/sebdah/goldie/v2 v2.5.1/go.mod h1:oZ9fp0+se1eapSRjfYbsV/0Hqhbuu3bJVvKI/NNtssI=
@@ -328,10 +359,12 @@ github.com/stretchr/testify v1.6.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/
 github.com/stretchr/testify v1.7.0 h1:nwc3DEeHmmLAfoZucVR881uASk0Mfjw8xYJ99tb5CcY=
 github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
 github.com/subosito/gotenv v1.2.0/go.mod h1:N0PQaV/YGNqwC0u51sEeR/aUtSLEXKX9iv69rRypqCw=
-github.com/temoto/robotstxt v1.1.1 h1:Gh8RCs8ouX3hRSxxK7B1mO5RFByQ4CmJZDwgom++JaA=
 github.com/temoto/robotstxt v1.1.1/go.mod h1:+1AmkuG3IYkh1kv0d2qEB9Le88ehNO0zwOr3ujewlOo=
+github.com/temoto/robotstxt v1.1.2 h1:W2pOjSJ6SWvldyEuiFXNxz3xZ8aiWX5LbfDiOFd7Fxg=
+github.com/temoto/robotstxt v1.1.2/go.mod h1:+1AmkuG3IYkh1kv0d2qEB9Le88ehNO0zwOr3ujewlOo=
 github.com/tmc/grpc-websocket-proxy v0.0.0-20190109142713-0ad062ec5ee5/go.mod h1:ncp9v5uamzpCO7NfCPTXjqaC+bZgJeR0sMTm6dMHP7U=
 github.com/ugorji/go v1.1.4/go.mod h1:uQMGLiO92mf5W77hV/PUCpI3pbzQx3CRekS0kk+RGrc=
+github.com/urfave/cli v1.22.3/go.mod h1:Gos4lmkARVdJ6EkW0WaNv/tZAAMe9V7XWyB60NtXRu0=
 github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2/go.mod h1:UETIi67q53MR2AWcXfiuqkDkRtnGDLqkBTpCHuJHxtU=
 github.com/xordataexchange/crypt v0.0.3-0.20170626215501-b2862e3d0a77/go.mod h1:aYKd//L2LvnjZzWKhF00oedf4jCCReLcmhLdhm1A27Q=
 github.com/yuin/goldmark v1.1.25/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
@@ -366,6 +399,8 @@ golang.org/x/crypto v0.0.0-20190605123033-f99c8df09eb5/go.mod h1:yigFU9vqHzYiE8U
 golang.org/x/crypto v0.0.0-20190820162420-60c769a6c586/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
 golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
 golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
+golang.org/x/crypto v0.0.0-20210817164053-32db794688a5 h1:HWj/xjIHfjYU5nVXpTM0s39J9CbLn7Cc5a7IC5rwsMQ=
+golang.org/x/crypto v0.0.0-20210817164053-32db794688a5/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
 golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
 golang.org/x/exp v0.0.0-20190306152737-a1d7652674e8/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
 golang.org/x/exp v0.0.0-20190510132918-efd6b22b2522/go.mod h1:ZjyILWgesfNpC6sMxTJOJm9Kp84zZh5NQWvqDGG3Qr8=
@@ -435,6 +470,7 @@ golang.org/x/net v0.0.0-20200520182314-0ba52f642ac2/go.mod h1:qpuaurCH72eLCgpAm/
 golang.org/x/net v0.0.0-20200602114024-627f9648deb9/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A=
 golang.org/x/net v0.0.0-20200625001655-4c5254603344/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
 golang.org/x/net v0.0.0-20200707034311-ab3426394381/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
+golang.org/x/net v0.0.0-20200813134508-3edf25e44fcc/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
 golang.org/x/net v0.0.0-20200822124328-c89045814202/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
 golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
 golang.org/x/net v0.0.0-20201031054903-ff519b6c9102/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
@@ -470,6 +506,7 @@ golang.org/x/sync v0.0.0-20200625203802-6e8e738ad208/go.mod h1:RxMgew5VJxzue5/jJ
 golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.0.0-20201207232520-09787c993a3a/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.0.0-20210220032951-036812b2e83c/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
+golang.org/x/sys v0.0.0-20180816055513-1c9583448a9c/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
 golang.org/x/sys v0.0.0-20180823144017-11551d06cbcc/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
 golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
 golang.org/x/sys v0.0.0-20180905080454-ebe1bf3edb33/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
@@ -516,9 +553,15 @@ golang.org/x/sys v0.0.0-20210330210617-4fbd30eecc44/go.mod h1:h1NjWce9XRLGQEsW7w
 golang.org/x/sys v0.0.0-20210403161142-5e06dd20ab57/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20210423082822-04245dca01da/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20210510120138-977fb7262007/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
-golang.org/x/sys v0.0.0-20210514084401-e8d321eab015 h1:hZR0X1kPW+nwyJ9xRxqZk1vx5RUObAPBdKVvXPDUH/E=
 golang.org/x/sys v0.0.0-20210514084401-e8d321eab015/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/sys v0.0.0-20210630005230-0f9fa26af87c h1:F1jZWGFhYfh0Ci55sIpILtKKK8p3i2/krTr0H1rg74I=
+golang.org/x/sys v0.0.0-20210630005230-0f9fa26af87c/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/sys v0.0.0-20210910150752-751e447fb3d0 h1:xrCZDmdtoloIiooiA9q0OQb9r8HejIHYoHGhGCe1pGg=
+golang.org/x/sys v0.0.0-20210910150752-751e447fb3d0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
+golang.org/x/term v0.0.0-20210615171337-6886f2dfbf5b h1:9zKuko04nR4gjZ4+DNjHqRlAJqbJETHwiNKDqTfOjfE=
+golang.org/x/term v0.0.0-20210615171337-6886f2dfbf5b/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
 golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
 golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
 golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
@@ -691,8 +734,9 @@ google.golang.org/protobuf v1.23.1-0.20200526195155-81db48ad09cc/go.mod h1:EGpAD
 google.golang.org/protobuf v1.24.0/go.mod h1:r/3tXBNzIEhYS9I1OUVjXDlt8tc493IdKGjtUeSXeh4=
 google.golang.org/protobuf v1.25.0/go.mod h1:9JNX74DMeImyA3h4bdi1ymwjUzf21/xIlbajtzgsN7c=
 google.golang.org/protobuf v1.26.0-rc.1/go.mod h1:jlhhOSvTdKEhbULTjvd4ARK9grFBp09yW+WbY/TyQbw=
-google.golang.org/protobuf v1.26.0 h1:bxAC2xTBsZGibn2RTntX0oH50xLsqy1OxA9tTL3p/lk=
 google.golang.org/protobuf v1.26.0/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc=
+google.golang.org/protobuf v1.27.1 h1:SnqbnDw1V7RiZcXPx5MEeqPv2s79L9i7BJUlG/+RurQ=
+google.golang.org/protobuf v1.27.1/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc=
 gopkg.in/alecthomas/kingpin.v2 v2.2.6/go.mod h1:FMv+mEhP44yOT+4EoQTLFTRgOQ1FBLkstjWtayDeSgw=
 gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
 gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=

release.sh

@@ -0,0 +1,28 @@
+#!/usr/bin/env bash
+
+if [ "$#" -ne 1 ]; then
+    echo "Illegal number of parameters"
+    echo "Usage: ./release.sh X.X.X"
+    exit 1
+fi
+
+version=$1
+platforms=("linux/amd64" "darwin/amd64" "windows/amd64")
+
+for platform in "${platforms[@]}"
+do
+    platform_split=(${platform//\// })
+    GOOS=${platform_split[0]}
+    GOARCH=${platform_split[1]}
+    output_name=papeer
+
+    if [ $GOOS = "windows" ]; then
+        env GOOS=$GOOS GOARCH=$GOARCH go build -o "$output_name.exe"
+        zip "$output_name-v$version-$GOOS-$GOARCH.exe.zip" "$output_name.exe"
+        rm "$output_name.exe"
+    else
+        env GOOS=$GOOS GOARCH=$GOARCH go build -o "$output_name"
+        tar czvf "$output_name-v$version-$GOOS-$GOARCH.tar.gz" "$output_name"
+        rm "$output_name"
+    fi
+done