fix: display images in epub

README.md

@@ -1,53 +1,6 @@
-```
-❯ papeer get --format=epub --recursive --delay=500 --limit=10 https://news.ycombinator.com/
-[===============================================>--------------------] Chapters 7 / 10
-[====================================================================] 1. Three ex-US intelligence officers admit hacking for UAE
-[====================================================================] 2. Show HN: Time Travel Debugger
-[====================================================================] 3. How much faster is Java 17?
-[====================================================================] 4. The First Webcam Was Invented to Keep an Eye on a Coffee Pot
-[====================================================================] 5. Nikon's 2021 Photomicrography Competition Winners
-[====================================================================] 6. HTTP Status 418 – I'm a teapot
-[====================================================================] 7. H3: Hexagonal hierarchical geospatial indexing system
-[--------------------------------------------------------------------] 8. Automatic cipher suite ordering in Go’s crypto/tls
-[--------------------------------------------------------------------] 9. Find engineering roles at over 800 YC-funded startups
-[--------------------------------------------------------------------] 10. Futarchy: Robin Hanson on prediction markets
-Ebook saved to "Hacker_News.epub"
-```
+# Papeer
 
-# Installation
-
-## From source
-
-```sh
-go get -u github.com/lapwat/papeer
-```
-
-## From binary
-
-### On Linux / MacOS
-
-```sh
-platform=linux
-# platform=darwin for MacOS
-curl -L https://github.com/lapwat/papeer/releases/download/v0.2.0/papeer-v0.2.0-$platform-amd64 > papeer
-chmod +x papeer
-sudo mv papeer /usr/local/bin
-```
-
-### On Windows
-
-Download [latest release](https://github.com/lapwat/papeer/releases/download/v0.2.0/papeer-v0.2.0-windows-amd64.exe).
-
-## Install kindlegen to export websites to MOBI (optional)
-
-```sh
-TMPDIR=$(mktemp -d -t papeer-XXXXX)
-curl -L https://github.com/lapwat/papeer/releases/download/kindlegen/kindlegen_linux_2.6_i386_v2_9.tar.gz > $TMPDIR/kindlegen.tar.gz
-tar xzvf $TMPDIR/kindlegen.tar.gz -C $TMPDIR
-chmod +x $TMPDIR/kindlegen
-sudo mv $TMPDIR/kindlegen /usr/local/bin
-rm -rf $TMPDIR
-```
+Papeer is a tool that lets you scrape content from the internet. It can scrape any web page, keeping only relevant content (formatted text and images) and removing ads and menus. You can export the content to Markdown, EPUB or MOBI files.
 
 # Usage
 
@@ -67,7 +20,7 @@ Available Commands:
 
 Flags:
   -d, --delay int         time to wait before downloading next chapter, in milliseconds (default -1)
-  -f, --format string     file format [md, epub, mobi] (default "md")
+  -f, --format string     file format [stdout, md, epub, mobi] (default "stdout")
   -h, --help              help for papeer
       --images            retrieve images only
   -i, --include           include URL as first chapter, in resursive mode
@@ -75,12 +28,93 @@ Flags:
   -o, --offset int        skip first chapters, in recursive mode
       --output string     output file
   -r, --recursive         create one chapter per natigation item
-  -s, --selector string   table of content CSS selector
-      --stdout            print to standard output
+  -s, --selector string   table of content CSS selector, in resursive mode
+  -t, --threads int       download concurrency, in recursive mode (default -1)
 
 Use "papeer [command] --help" for more information about a command.
 ```
 
+# Examples
+
+## Grab a single page
+
+The `get` command lets you retrieve the content of a web page.
+
+```sh
+papeer get https://www.eff.org/cyberspace-independence
+# A Declaration of the Independence of Cyberspace
+# ===============================================
+
+# Governments of the Industrial World, you weary giants of flesh and steel, I come from Cyberspace, the new home of Mind. On behalf of the future, I ask you of the past to leave us alone. You are not welcome among us. You have no sovereignty where we gather...
+```
+
+## Grab several pages (recursive mode)
+
+The `recursive` option lets you extract the table of content of a website, then scrape the content of each one of its pages.
+
+### Display table of content
+
+Before trying the `recursive` option, it is a good idea to use the `ls` option, which lets you vizualize the content that will be retrieved. You can use several options to customize the table of content extraction, such as `selector`, `limit`, `offset` and `include`. Type `papeer help` for more information about those options.
+
+```sh
+papeer ls https://news.ycombinator.com/ --limit=5
+#  #  NAME                                                      URL                                                                                             
+#  1  Tailwind CSS v3.0                                         https://tailwindcss.com/blog/tailwindcss-v3                                                     
+#  2  A molten salt storage solution using sodium hydroxide     https://sifted.eu/articles/salt-energy-storage-seaborg-hyme/                                    
+#  3  HashiCorp IPO today                                       https://www.hashicorp.com/blog/a-new-chapter-for-hashicorp                                      
+#  4  Stack Graphs                                              https://github.blog/2021-12-09-introducing-stack-graphs/
+#  5  ‘Tipping point’ makes partisan polarization irreversible  https://news.cornell.edu/stories/2021/12/tipping-point-makes-partisan-polarization-irreversible 
+```
+
+### Scrape time
+
+Once you are satisfied with the table of content listed by the `ls` command, you can actually scrape the content of those pages. You can use the same options that you specified for the `ls` command. In recursive mode, you also have the possibility to use `delay` and `threads` options.
+
+```sh
+papeer get https://news.ycombinator.com/ --recursive --delay=500 --limit=5 --format=md
+# [========================================>---------------------------] Chapters 3 / 5
+# [====================================================================] 1. Tailwind CSS v3.0
+# [====================================================================] 2. A molten salt storage solution using sodium hydroxide
+# [====================================================================] 3. HashiCorp IPO today
+# [--------------------------------------------------------------------] 4. Stack Graphs
+# [--------------------------------------------------------------------] 5. ‘Tipping point’ makes partisan polarization irreversible
+# Markdown saved to "Hacker News.md"
+```
+
+# Installation
+
+## From source
+
+```sh
+go get -u github.com/lapwat/papeer
+```
+
+## From binary
+
+### On Linux / MacOS
+
+```sh
+platform=linux # use platform=darwin for MacOS
+curl -L https://github.com/lapwat/papeer/releases/download/v0.3.0/papeer-v0.3.0-$platform-amd64 > papeer
+chmod +x papeer
+sudo mv papeer /usr/local/bin
+```
+
+### On Windows
+
+Download [latest release](https://github.com/lapwat/papeer/releases/download/v0.3.0/papeer-v0.3.0-windows-amd64.exe).
+
+## Install kindlegen to export websites to MOBI (optional)
+
+```sh
+TMPDIR=$(mktemp -d -t papeer-XXXXX)
+curl -L https://github.com/lapwat/papeer/releases/download/kindlegen/kindlegen_linux_2.6_i386_v2_9.tar.gz > $TMPDIR/kindlegen.tar.gz
+tar xzvf $TMPDIR/kindlegen.tar.gz -C $TMPDIR
+chmod +x $TMPDIR/kindlegen
+sudo mv $TMPDIR/kindlegen /usr/local/bin
+rm -rf $TMPDIR
+```
+
 # Autocompletion
 
 Execute this command in your current shell, or add it to your `.bashrc`.
@@ -100,4 +134,4 @@ You can replace `bash` by your own shell (zsh, fish or powershell).
 - `html-to-markdown` convert HTML to Markdown
 - `go-epub` convert HTML to EPUB
 - `colly` query HTML trees
-- `uiprogress` display progress bars
+- `uiprogress` display progress bars

book/progress.go

@@ -19,8 +19,8 @@ func NewProgress(links []link) progress {
 		return fmt.Sprintf("Chapters %d / %d", b.Current(), len(links))
 	})
 
-	individuals := []*uiprogress.Bar{}
 	// hide individual bars if more than 50 chapters
+	individuals := []*uiprogress.Bar{}
 	if len(links) <= 50 {
 		for index, link := range links {
 			bar := uiprogress.AddBar(1)

book/scraper.go

@@ -14,9 +14,9 @@ import (
 	colly "github.com/gocolly/colly/v2"
 )
 
-func NewBookFromURL(url, selector string, recursive, include, images bool, limit, offset, delay int) book {
+func NewBookFromURL(url, selector string, recursive, include, images bool, limit, offset, delay, threads int) book {
 	if recursive {
-		chapters := tableOfContent(url, selector, limit, offset, delay, include, images)
+		chapters := tableOfContent(url, selector, limit, offset, delay, threads, include, images)
 
 		b := New(chapters[0].Name(), chapters[0].Author())
 		for _, c := range chapters {
@@ -41,22 +41,24 @@ func NewChapterFromURL(url string, images bool) chapter {
 	content := strings.ReplaceAll(article.Content, "\n", "")
 
 	if images {
-		// Load the HTML document
+		// parse html content
 		doc, err := goquery.NewDocumentFromReader(strings.NewReader(content))
 		if err != nil {
 			log.Fatal(err)
 		}
 
-		// Find the review items
+		// extract images only
+		content = ""
 		doc.Find("img").Each(func(i int, s *goquery.Selection) {
-			content, _ = goquery.OuterHtml(s)
+			newContent, _ := goquery.OuterHtml(s)
+			content += newContent
 		})
 	}
 
 	return chapter{article.Title, article.Byline, content}
 }
 
-func tableOfContent(url, selector string, limit, offset, delay int, include, images bool) []chapter {
+func tableOfContent(url, selector string, limit, offset, delay, threads int, include, images bool) []chapter {
 	base, err := urllib.Parse(url)
 	if err != nil {
 		log.Fatal(err)
@@ -71,6 +73,7 @@ func tableOfContent(url, selector string, limit, offset, delay int, include, ima
 	progress := NewProgress(links)
 
 	if delay >= 0 {
+		// synchronous mode
 
 		for index, link := range links {
 			// and then use it to parse relative URLs
@@ -91,10 +94,19 @@ func tableOfContent(url, selector string, limit, offset, delay int, include, ima
 		}
 
 	} else {
+		// asynchronous mode
 		var wg sync.WaitGroup
+
+		if threads == -1 {
+			threads = len(links)
+		}
+		semaphore := make(chan bool, threads)
+
 		for index, l := range links {
 
 			wg.Add(1)
+			semaphore <- true
+
 			go func(index int, l link) {
 				defer wg.Done()
 
@@ -107,6 +119,7 @@ func tableOfContent(url, selector string, limit, offset, delay int, include, ima
 				chapters[index] = NewChapterFromURL(u.String(), images)
 				progress.Incr(index)
 
+				<-semaphore
 			}(index, l)
 		}
 		wg.Wait()

cmd/get.go

@@ -9,15 +9,16 @@ import (
 	"strings"
 
 	md "github.com/JohannesKaufmann/html-to-markdown"
+	"github.com/PuerkitoBio/goquery"
 	epub "github.com/bmaupin/go-epub"
-	cobra "github.com/spf13/cobra"
+	"github.com/spf13/cobra"
 
 	"github.com/lapwat/papeer/book"
 )
 
-var stdout, recursive, include, images bool
+var recursive, include, images bool
 var format, output, selector string
-var limit, offset, delay int
+var limit, offset, delay, threads int
 
 var getCmd = &cobra.Command{
 	Use:   "get",
@@ -28,20 +29,16 @@ var getCmd = &cobra.Command{
 		}
 
 		formatEnum := map[string]bool{
-			"md":   true,
-			"epub": true,
-			"mobi": true,
+			"stdout": true,
+			"md":     true,
+			"epub":   true,
+			"mobi":   true,
 		}
 		if formatEnum[format] != true {
 			return fmt.Errorf("invalid format specified: %s", format)
 		}
 
-		if format == "epub" || format == "mobi" {
-			if stdout {
-				return errors.New("cannot print EPUB/MOBI file to standard output")
-			}
-		}
-
+		// add .mobi to filename if not specified
 		if format == "mobi" {
 			if len(output) > 0 && strings.HasSuffix(output, ".mobi") == false {
 				output = fmt.Sprintf("%s.mobi", output)
@@ -68,11 +65,19 @@ var getCmd = &cobra.Command{
 			return errors.New("cannot use delay option if not in recursive mode")
 		}
 
+		if cmd.Flags().Changed("threads") && recursive == false {
+			return errors.New("cannot use threads option if not in recursive mode")
+		}
+
+		if cmd.Flags().Changed("delay") && cmd.Flags().Changed("threads") {
+			return errors.New("cannot use delay and threads options at the same time")
+		}
+
 		return nil
 	},
 	Run: func(cmd *cobra.Command, args []string) {
 		url := args[0]
-		b := book.NewBookFromURL(url, selector, recursive, include, images, limit, offset, delay)
+		b := book.NewBookFromURL(url, selector, recursive, include, images, limit, offset, delay, threads)
 
 		if len(output) == 0 {
 			// set default output
@@ -81,19 +86,10 @@ var getCmd = &cobra.Command{
 			output = fmt.Sprintf("%s.%s", output, format)
 		}
 
-		if format == "md" {
-			var f *os.File
-			var err error
-
-			if !stdout {
-				f, err = os.Create(output)
-				if err != nil {
-					log.Fatal(err)
-				}
-				defer f.Close()
-			}
+		if format == "stdout" {
 
 			for _, c := range b.Chapters() {
+				// convert to markdown
 				content, err := md.NewConverter("", true, nil).ConvertString(c.Content())
 				if err != nil {
 					log.Fatal(err)
@@ -101,20 +97,38 @@ var getCmd = &cobra.Command{
 
 				text := fmt.Sprintf("%s\n%s\n\n%s\n\n\n", c.Name(), strings.Repeat("=", len(c.Name())), content)
 
-				if stdout {
-					fmt.Println(text)
-				} else {
-					_, err := f.WriteString(text)
-					if err != nil {
-						log.Fatal(err)
-					}
+				// write to stdout
+				fmt.Println(text)
+			}
 
+		}
+
+		if format == "md" {
+
+			// create markdown file
+			f, err := os.Create(output)
+			if err != nil {
+				log.Fatal(err)
+			}
+			defer f.Close()
+
+			for _, c := range b.Chapters() {
+				// convert to markdown
+				content, err := md.NewConverter("", true, nil).ConvertString(c.Content())
+				if err != nil {
+					log.Fatal(err)
+				}
+
+				text := fmt.Sprintf("%s\n%s\n\n%s\n\n\n", c.Name(), strings.Repeat("=", len(c.Name())), content)
+
+				// write to markdown file
+				_, err = f.WriteString(text)
+				if err != nil {
+					log.Fatal(err)
 				}
 			}
 
-			if stdout == false {
-				fmt.Printf("Markdown saved to \"%s\"\n", output)
-			}
+			fmt.Printf("Markdown saved to \"%s\"\n", output)
 		}
 
 		if format == "epub" {
@@ -122,16 +136,27 @@ var getCmd = &cobra.Command{
 			e.SetAuthor(b.Author())
 
 			for _, c := range b.Chapters() {
-				if images {
-					e.AddSection(c.Content(), "", "", "")
-				} else {
-					html := fmt.Sprintf("<h1>%s</h1>%s", c.Name(), c.Content())
-
-					_, err := e.AddSection(html, c.Name(), "", "")
-					if err != nil {
-						log.Fatal(err)
-					}
+				// parse content 
+				doc, err := goquery.NewDocumentFromReader(strings.NewReader(c.Content()))
+				if err != nil {
+					log.Fatal(err)
 				}
+
+				// retrieve images and download it
+				contentWithLocalImages := c.Content()
+				doc.Find("img").Each(func(i int, s *goquery.Selection) {
+					src, _ := s.Attr("src")
+					imagePath, _ := e.AddImage(src, "")
+
+					contentWithLocalImages = strings.ReplaceAll(contentWithLocalImages, src, imagePath)
+				})
+
+				html := fmt.Sprintf("<h1>%s</h1>%s", c.Name(), contentWithLocalImages)
+				_, err = e.AddSection(html, c.Name(), "", "")
+				if err != nil {
+					log.Fatal(err)
+				}
+
 			}
 
 			err := e.Write(output)

cmd/root.go

@@ -23,16 +23,16 @@ func Execute() {
 }
 
 func init() {
-	rootCmd.PersistentFlags().StringVarP(&format, "format", "f", "md", "file format [md, epub, mobi]")
+	rootCmd.PersistentFlags().StringVarP(&format, "format", "f", "stdout", "file format [stdout, md, epub, mobi]")
 	rootCmd.PersistentFlags().StringVarP(&output, "output", "", "", "output file")
 	rootCmd.PersistentFlags().StringVarP(&selector, "selector", "s", "", "table of content CSS selector, in resursive mode")
 	rootCmd.PersistentFlags().BoolVarP(&recursive, "recursive", "r", false, "create one chapter per natigation item")
 	rootCmd.PersistentFlags().BoolVarP(&include, "include", "i", false, "include URL as first chapter, in resursive mode")
-	rootCmd.PersistentFlags().BoolVarP(&stdout, "stdout", "", false, "print to standard output")
 	rootCmd.PersistentFlags().BoolVarP(&images, "images", "", false, "retrieve images only")
 	rootCmd.PersistentFlags().IntVarP(&limit, "limit", "l", -1, "limit number of chapters, in recursive mode")
 	rootCmd.PersistentFlags().IntVarP(&offset, "offset", "o", 0, "skip first chapters, in recursive mode")
 	rootCmd.PersistentFlags().IntVarP(&delay, "delay", "d", -1, "time to wait before downloading next chapter, in milliseconds")
+	rootCmd.PersistentFlags().IntVarP(&threads, "threads", "t", -1, "download concurrency, in recursive mode")
 
 	rootCmd.AddCommand(getCmd)
 	rootCmd.AddCommand(listCmd)

cmd/utils.go

@@ -1,5 +0,0 @@
-package cmd
-
-func getTableOfContent() {
-
-}

cmd/version.go

@@ -14,6 +14,6 @@ var versionCmd = &cobra.Command{
 	Use:   "version",
 	Short: "Print the version number of papeer",
 	Run: func(cmd *cobra.Command, args []string) {
-		fmt.Println("papeer v0.1.1")
+		fmt.Println("papeer v0.3.0")
 	},
 }