...

Content server: Fix reading of books with thousands of internal files not working in the Chrome browser. Fixes #1964742 [Error when opening EPUB file which contains many chapters via Web Interface](https://bugs.launchpad.net/calibre/+bug/1964742 )
Apparently chrome's network stack is too dumb to queue multiple ajax requests so do it in our code.
2025-12-15 06:56:13 +01:00 · 2022-03-14 20:11:35 +05:30 · 2022-03-14 20:09:05 +05:30 · 2022-03-14 10:34:05 +05:30 · 2022-03-13 20:18:52 +05:30 · 2022-03-13 20:07:00 +05:30
1168 changed files with 160989 additions and 127480 deletions
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@ -7,7 +7,7 @@ jobs:
        runs-on: ${{ matrix.os }}
        strategy:
            matrix:
-                os: [ubuntu-latest, macos-latest, windows-latest]
+                os: [ubuntu-latest, macos-latest, windows-2019]
        steps:
            - name: Checkout source code
              uses: actions/checkout@master
--- a/.github/workflows/translations.yml
+++ b/.github/workflows/translations.yml
@ -27,9 +27,6 @@ jobs:
            - name: Install calibre dependencies
              run: setup/arch-ci.sh

-            - name: Install translations dependencies
-              run: python -m pip install transifex-client
-
            - name: Bootstrap calibre
              run: runuser -u ci -- python setup.py bootstrap --ephemeral

--- a/.gitignore
+++ b/.gitignore
@ -10,6 +10,7 @@ compile_commands.json
 link_commands.json
 src/calibre/plugins
 resources/images.qrc
+resources/icons.rcc
 manual/generated
 manual/locale
 manual/.doctrees
@ -58,3 +59,4 @@ recipes/debug
 /*env*/
 cmake-build-*
 bypy/b
+bypy/virtual-machines.conf
--- a/2
+++ b/2
@ -1,5 +1,5 @@
 Files: *
-Copyright: Copyright (C) 2008-2020 Kovid Goyal <kovid@kovidgoyal.net>
+Copyright: Copyright (C) 2008-2022 Kovid Goyal <kovid@kovidgoyal.net>
 License: GPL-3
 The full text of the GPL is distributed as in
 /usr/share/common-licenses/GPL-3 on Debian systems.
--- a/Changelog.txt
+++ b/Changelog.txt
@ -6,7 +6,7 @@
 # to the ticket list.
 # Also, each release can have new and improved recipes.

-# {{{ 5.x.0 2021-xx-xx
+# {{{ 5.x.0 2022-xx-xx
 #
 # :: new features
 #
@ -23,7 +23,234 @@
 # - title by author
 # }}}

-{{{ 5.33.0 2021-12-03
+{{{ 5.38.0 2022-03-04
+
+:: new features
+
+- [1852929] E-book viewer: When displaying estimated time to completion for reading a book, remember the reading rate the next time the book is opened
+
+- [1961500] Dark theme: Highlight the current cell in the book list with a lighter background and different foreground to make it more obvious
+
+- [1961639] An option to disable editing composite columns in the main book list when Tabbing through them (Preferences->Look & feel->Edit metadata)
+
+:: bug fixes
+
+- Tag editor: Fix regression in previous release that caused double clicking on tags to not work on non Linux platforms
+
+- [1962365] Copy to library: Fix annotations not being copied
+
+- [1962213] Edit book: Spell check: Fix words after a comment not being checked
+
+- [1960554] PDF Output: Fix conversion failing if there are ToC entries pointing to removed content
+
+- [1961775] E-book viewer: Fix an error when opening books with MathML for the second time if the last read position was at a MathML element
+
+- Edit book: Fix double clicking to select a word also selecting smart quotes surrounding the word
+
+- EPUB 3 metadata: Fix non-integer series index being sometimes represented using exponential notation
+
+:: improved recipes
+- Lenta.ru and aif.ru
+- Indian Express
+- Live Mint
+- Mainichi
+- Japan Times
+
+:: new recipes
+- Hindustan Times by unkn0wn
+- India Legal Magazine by unkn0wn
+- RT на русском by Vuizur
+}}}
+
+{{{ 5.37.0 2022-02-18
+
+:: new features
+
+- [1961129] Book details: Add actions to trim the cover to the right-click menu
+
+- [1960586] Allow removing multiple email addresses at once in Preferences->Sharing by email
+
+- Book details: Use a better mono-spaced font on Windows by default
+
+- Add a tweak in Preferences->Tweaks to change the behavior of the Tab key when completing entries
+
+- [1959928] Edit metadata: In "All in one" mode add an adjustable splitter between the cover and formats boxes
+
+:: bug fixes
+
+- [1960686] Textile output: Dont fail if input document has invalid padding or margin specifications
+
+- [1960446] E-book viewer: Fix image display window not remembering its size and settings when run from within calibre
+
+- E-book viewer: Fix setting to use roman numerals for series not being respected
+
+- Edit book: When saving a copy do not fail if the original file has no write permissions
+
+- [1960180] Embed fonts tool: Create <head> when missing
+
+- Tag editor: Improve performance when very large number of tags present
+
+:: improved recipes
+- Live Mint
+- The Hindu
+- Reuters
+- MMC RTV Slovenija
+- Down To Earth
+- Publico.PT
+}}}
+
+{{{ 5.36.0 2022-02-04
+
+:: new features
+
+- Edit metadata dialog: Allow controlling which custom columns are present in this dialog via Preferences->Look & feel->Edit metadata
+
+- Edit metadata dialog: Allow manually sizing the various sections of the dialog in "All on 1 tab" mode
+
+- Edit book: Spell checking: Update the bundled English and Spanish dictionaries
+
+- [1958773] BibTeX catalogs: Support tags like custom columns
+
+:: bug fixes
+
+- [1959659] Amazon metadata download: Fix paragraphs in the comments being merged
+
+- [1958979] Amazon.de metadata download: Fix published date and series information not being fetched for some books
+
+- Email delivery: Fix sending email via Hotmail not working since this week because Microsoft changed the SMTP server name
+
+- [1959220] Do not remove articles for titles in the Polish language
+
+- [1959207] E-book viewer: When using Read aloud do not automatically lookup the highlighted word until read aloud is paused or stopped
+
+- E-book viewer: Fix Ctrl+p shortcut for printing not working
+
+- [1958882] Show an error when viewing a specific format and the file is missing
+
+- Edit book: Fix renaming of classes that start/end with non word characters not working
+
+- [1958730] Edit book: Preview panel: Fix hyphenation at end of line being rendered as boxes on macOS
+
+- [1959893] Fix incorrect selection size displayed in Trim image dialog when image is scaled down to fit
+
+- [1959782] Edit book: Fix pasting files from another editor instance failing if a file with the same name already exists
+
+- [1959981] When reviewing metadata if the newly downloaded metadata has no language but there is an existing language, ensure it is preserved
+
+:: improved recipes
+- India Today
+- Indian Express
+- Live Mint
+- Al Jazeera in English
+- The Financial Express
+- The Straits Times
+
+:: new recipes
+- title by author
+}}}
+
+{{{ 5.35.0 2022-01-21
+
+:: new features
+
+- [1956006] Coloring/icon rules: Allow creating a rule for date columns that matches *today*
+
+- Kobo driver: Add support for new firmware
+
+- [1954890] Content server: Show total number of results when searching for books
+
+:: bug fixes
+
+- [1958028] E-book viewer: Fix searching for text near the end of a chapter sometimes not working
+
+- [1954714] E-book viewer: Fix auto hyphenation on macOS not rendering the hyphens correctly
+
+- Edit book: Reports: Fix thumbnails of SVG images not rendered
+
+- ODT metadata: Support reading tags from multiple <keyword> elements
+
+- [1958115] LRF Input: Fix a regression in calibre 5 that broke parsing of some LRF files
+
+- [1956097] MOBI output: Dont fail if input document contains invalid % based lengths
+
+- [1955308] AZW3 Input: Handle AZW3 files with incorrect TAGX Offset INDX header fields
+
+- [1956932] Comic conversion: Fix conversion of comic images that are stored as grayscale images in JPEG format not working when converting to PDF with image processing turned off
+
+- [1955967] calibredb catalog: Fix --ids and --search options not working for CSV/XML catalogs
+
+- [1958490] Tag browser: Fix the find box not using all available width
+
+- [1956192] E-book viewer: Remove books that do not exist from the recently opened book list
+
+- Completion popups: Fix display of items containing line breaks
+
+- [1956129] Fix line breaks in custom column descriptions not being rendered in their tooltips
+
+- [1956088] Fix Preferences->Searching->Clear search histories not taking effect till a restart for some search boxes
+
+- [1955732] Hierarchical entries in user category may not merge correctly in tag browser
+
+:: improved recipes
+- Foreign Affairs
+- MIT Technology Review
+- Reuters
+- Clarin
+- General Knowledge Today
+- Popular Science
+
+:: new recipes
+- Dw.de by xav
+- Equestria Daily by Timothee Andres
+}}}
+
+{{{ 5.34.0 2021-12-17
+
+:: new features
+
+- Happy holidays to everyone!
+
+- Driver for the new Nook Glowlight 4
+
+- Edit book: Spell check tool: Add an exclude files button to exclude some files from being checked
+
+- EPUB/MOBI Catalogs: Increase the maximum thumbnail size to 3 inches from 2 inches
+
+- [1953739] Allow creating a shortcut in Preferences->Shortcuts->Edit metadata to paste metadata ignoring the value of the exclude_fields tweak
+
+- [1954715] E-book viewer: Displays links marked up as glossary and bibliography links as popups
+
+- [1954572] Add a tweak in Preferences->Tweaks to provide the sort value for undefined numbers
+
+:: bug fixes
+
+- Edit book: Fix pressing F8 to jump to next misspelled word not working after last word in current file
+
+- [1954889] Fix PDB E-reader output broken in calibre 5
+
+- [1954839] Edit book: Reports: Include descendant selectors that use classes when counting class usage
+
+- [1954726] E-book viewer: Fix an error when opening some books with highlights that span in-line text formatting
+
+- [1954460] MTP driver: Do not send the calibre device db files to the root folder on the Supernote A5 x as it fails
+
+- ToC Editor: Workaround an occasional error when closing on Windows if the file being edited is in a DropBox/antivirus prone folder
+
+- Fix a regression in the previous release that broke creating new keyboard shortcuts
+
+- Comments editor: When flowing the tool bar onto multiple lines do not split up groups of buttons
+
+- Various compatibility fixes for Python 3.10 used by some Linux distributions
+
+:: improved recipes
+- Pocket
+- El Pais
+- American Prospect
+- Mediapart
+
+}}}
+
+{{{ 5.33.2 2021-12-03

 :: new features

@ -53,10 +280,13 @@

 - [1951507] E-book viewer: Fix sorting of highlights incorrect in books that use HTML ids with a hyphen in them

- [1951467] PDF Output: Fix the option to break long words at the ends of lines causing boxes to be rendered at the end fo the line on macOS with some fonts
+- [1951467] PDF Output: Fix the option to break long words at the ends of lines causing boxes to be rendered at the end of the line on macOS with some fonts

 - Google metadata plugin: When searching by ISBN if no results are found retry using an alternate query syntax

+- 5.33.2 fixes a couple of regressions that broke the toolbar in the popup comments editor dialog and rendering of the download
+  metadata button in the edit metadata dialog on Windows, as well as reading files from MTP devices on Windows
+
 :: improved recipes
 - Smithsonian Magazine

--- a/bypy/README.rst
+++ b/bypy/README.rst
@ -17,7 +17,7 @@ First create some empty top level directory and run the following commands::

    git clone https://github.com/kovidgoyal/bypy.git
    git clone https://github.com/kovidgoyal/calibre.git
-    cd calibre
+    cd bypy && git switch qt5 && cd ../calibre

 Now we need to bootstrap calibre, for which all its Linux build dependencies
 must have already been installed (see the `Dependencies
--- a/bypy/macos/main.py
+++ b/bypy/macos/main.py
@ -253,7 +253,7 @@ def get_dependencies(self, path_to_lib):
    @flush
    def get_local_dependencies(self, path_to_lib):
        for x, is_id in self.get_dependencies(path_to_lib):
-            if x.startswith('@rpath/Qt'):
+            if x.startswith('@rpath/Qt') or x.startswith('@rpath/libexpat'):
                yield x, x[len('@rpath/'):], is_id
            elif x in ('libunrar.dylib', 'libstemmer.0.dylib', 'libstemmer.dylib') and not is_id:
                yield x, x, is_id
--- a/bypy/sources.json
+++ b/bypy/sources.json
@ -626,8 +626,8 @@
    {
        "name": "feedparser",
        "unix": {
-            "filename": "feedparser-5.2.1.tar.bz2",
-            "hash": "sha256:ce875495c90ebd74b179855449040003a1beb40cd13d5f037a0654251e260b02",
+            "filename": "feedparser-6.0.8.tar.bz2",
+            "hash": "sha256:5ce0410a05ab248c8c7cfca3a0ea2203968ee9ff4486067379af4827a59f9661",
            "urls": ["pypi"]
        }
    },
--- a/manual/conf.py
+++ b/manual/conf.py
@ -171,14 +171,17 @@ def sort_languages(x):
    return sort_key(type(u'')(name))


+website = 'https://calibre-ebook.com'
 html_context['other_languages'].sort(key=sort_languages)
 html_context['support_text'] = _('Support calibre')
 html_context['support_tooltip'] = _('Contribute to support calibre development')
-html_context['homepage_url'] = 'https://calibre-ebook.com'
+html_context['homepage_url'] = website
 if needs_localization:
    html_context['homepage_url'] = localize_website_link(html_context['homepage_url'])
 extlinks = {
+    'website_base': (website, None),
    'website': (html_context['homepage_url'] + '/%s', None),
+    'download_file': (f'{website}/downloads/%s', '%s'),
 }
 del sort_languages, get_language

@ -236,11 +239,13 @@ def sort_languages(x):
 # If false, no module index is generated.
 # latex_use_modindex = True

+# we use lualatex as it is actively maintained and pdflatex and xelatex fail
+# to render smart quotes and dashes
+latex_engine = 'lualatex'
 latex_logo = 'resources/logo.png'
 latex_show_pagerefs = True
 latex_show_urls = 'footnote'
 latex_elements = {
    'papersize':'letterpaper',
-    'fontenc':r'\usepackage[T2A,T1]{fontenc}',
    'preamble': r'\renewcommand{\pageautorefname}{%s}' % _('page'),
 }
--- a/manual/conversion.rst
+++ b/manual/conversion.rst
@ -652,7 +652,7 @@ calibre can automatically convert ``.docx`` files created by Microsoft Word 2007
 newer. Just add the file to calibre and click convert.

 .. note::
-    There is a `demo .docx file <https://calibre-ebook.com/downloads/demos/demo.docx>`_
+    There is a :download_file:`demo .docx file <demos/demo.docx>`
    that demonstrates the capabilities of the calibre conversion engine. Just
    download it and convert it to EPUB or AZW3 to see what calibre can do.

@ -671,9 +671,8 @@ produce clean HTML that will convert well. Note that Word produces really messy
 HTML, converting it can take a long time, so be patient. If you have a newer
 version of Word available, you can directly save it as .docx as well.

-Another alternative is to use the free OpenOffice. Open your .doc file in
-OpenOffice and save it in OpenOffice's format .odt. calibre can directly convert
-.odt files.
+Another alternative is to use the free LibreOffice. Open your .doc file in
+LibreOffice and save it as .docx, which can be directly converted in calibre.

 Convert TXT documents
 ~~~~~~~~~~~~~~~~~~~~~~
@ -791,10 +790,10 @@ calibre will automatically convert this .cbc file into a e-book with a Table of
 EPUB advanced formatting demo
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

-Various advanced formatting for EPUB files is demonstrated in this `demo file <https://calibre-ebook.com/downloads/demos/demo.epub>`_.
+Various advanced formatting for EPUB files is demonstrated in this :download_file:`demo file <demos/demo.epub>`.
 The file was created from hand coded HTML using calibre and is meant to be used as a template for your own EPUB creation efforts.

-The source HTML it was created from is available `demo.zip <https://calibre-ebook.com/downloads/demos/demo.zip>`_. The settings used to create the
+The source HTML it was created from is available :download_file:`demo.zip <demos/demo.zip>`. The settings used to create the
 EPUB from the ZIP file are::

    ebook-convert demo.zip .epub -vv --authors "Kovid Goyal" --language en --level1-toc '//*[@class="title"]' --disable-font-rescaling --page-breaks-before / --no-default-epub-cover
--- a/manual/creating_plugins.rst
+++ b/manual/creating_plugins.rst
@ -40,7 +40,7 @@ the folder in which you created :file:`__init__.py`::
    are in :file:`/Applications/calibre.app/Contents/MacOS/`.

 You can download the Hello World plugin from
-`helloworld_plugin.zip  <https://calibre-ebook.com/downloads/helloworld_plugin.zip>`_.
+:download_file:`helloworld_plugin.zip`.

 Every time you use calibre to convert a book, the plugin's :meth:`run` method will be called and the
 converted book will have its publisher set to "Hello World". This is a trivial plugin, lets move on to
@ -54,7 +54,7 @@ This plugin will be spread over a few files (to keep the code clean). It will sh
 how to create elements in the calibre user interface and how to access
 and query the books database in calibre.

-You can download this plugin from `interface_demo_plugin.zip <https://calibre-ebook.com/downloads/interface_demo_plugin.zip>`_
+You can download this plugin from :download_file:`interface_demo_plugin.zip`

 .. _import_name_txt:

@ -185,7 +185,7 @@ Edit book plugins

 Now let's change gears for a bit and look at creating a plugin to add tools to
 the calibre book editor. The plugin is available here:
-`editor_demo_plugin.zip <https://calibre-ebook.com/downloads/editor_demo_plugin.zip>`_.
+:download_file:`editor_demo_plugin.zip`.

 The first step, as for all plugins is to create the
 import name empty txt file, as described :ref:`above <import_name_txt>`.
@ -259,7 +259,7 @@ data in a separate process. A simple example plugin follows that shows how
 to do this.

 You can download the plugin from
-`webengine_demo_plugin.zip  <https://calibre-ebook.com/downloads/webengine_demo_plugin.zip>`_.
+:download_file:`webengine_demo_plugin.zip`.

 The important part of the plugin is in two functions:

--- a/manual/develop.rst
+++ b/manual/develop.rst
@ -75,7 +75,7 @@ Getting the code
 ------------------

 You can get the calibre source code in two ways, using a version control system or
-directly downloading a `tarball <https://calibre-ebook.com/dist/src>`_.
+directly downloading a :website_base:`tarball <dist/src>`.

 calibre uses `Git <https://www.git-scm.com/>`_, a distributed version control
 system. Git is available on all the platforms calibre supports.  After
@ -89,7 +89,7 @@ calibre is a very large project with a very long source control history, so the
 above can take a while (10 mins to an hour depending on your internet speed).

 If you want to get the code faster, the source code for the latest release is
-always available as an `archive <https://calibre-ebook.com/dist/src>`_.
+always available as an :website_base:`archive <dist/src>`.

 To update a branch to the latest code, use the command::

@ -210,9 +210,9 @@ Create a plain text file::
    export CALIBRE_DEVELOP_FROM="/Users/kovid/work/calibre/src"
    calibre-debug -g

-Save this file as ``/usr/bin/calibre-develop``, then set its permissions so that it can be executed::
+Save this file as :file:`/usr/local/bin/calibre-develop`, then set its permissions so that it can be executed::

-    chmod +x /usr/bin/calibre-develop
+    chmod +x /usr/local/bin/calibre-develop

 Once you have done this, run::

@ -309,7 +309,7 @@ You can insert the following two lines of code to start an interactive Python se
    ipython(locals())

 When running from the command line, this will start an interactive Python interpreter with access to all
-locally defined variables (variables in the local scope). The interactive prompt even has TAB completion
+locally defined variables (variables in the local scope). The interactive prompt even has :kbd:`Tab` completion
 for object properties and you can use the various Python facilities for introspection, such as
 :func:`dir`, :func:`type`, :func:`repr`, etc.

--- a/manual/drm.rst
+++ b/manual/drm.rst
@ -69,7 +69,7 @@ calibre is open source software while DRM by its very nature is closed. If
 calibre were to support opening or viewing DRM files it could be trivially
 modified to be used as a tool for DRM removal which is illegal under today's
 laws. Open source software and DRM are a clash of principles. While DRM is all
-about controlling the user open source software is about empowering the user.
+about controlling the user, open source software is about empowering the user.
 The two simply can not coexist.


--- a/manual/edit.rst
+++ b/manual/edit.rst
@ -26,8 +26,7 @@ Basic workflow
 ---------------

 .. note::
-    A video tour of the calibre E-book editor is available `here
-    <https://calibre-ebook.com/demo#tutorials>`_.
+    A video tour of the calibre E-book editor is available :website:`here <demo#tutorials>`.

 When you first open a book with the Edit book tool, you will be presented with
 a list of files on the left. These are the individual HTML files, stylesheets,
@ -678,6 +677,14 @@ common in your book and to run a simple search and replace on individual words.
    check tool. If you do not do this and continue to use the Spell check tool,
    you could lose the changes you have made in the editor.

+.. note::
+   To exclude an individual file from being spell checked when running the
+   spell check tool, you can use the :guilabel:`Exclude files` button or
+   add the following comment just under the opening tag in the file::
+
+        <!-- calibre-no-spell-check -->
+
+
 Adding new dictionaries
 ###########################

--- a/manual/faq.rst
+++ b/manual/faq.rst
@ -158,8 +158,7 @@ How do I use some of the advanced features of the conversion tools?
 You can get help on any individual feature of the converters by mousing over
 it in the GUI or running ``ebook-convert dummy.html .epub -h`` at a terminal.
 A good place to start is to look at the following demo file that demonstrates
-some of the advanced features
-`html-demo.zip <https://calibre-ebook.com/downloads/html-demo.zip>`_
+some of the advanced features :download_file:`html-demo.zip <html-demo.zip>`.


 Device integration
@ -196,7 +195,7 @@ We just need some information from you:

 Once you send us the output for a particular operating system, support for the device in that operating system
 will appear in the next release of calibre. To send us the output, open a bug report and attach the output to it.
-See `how to report bugs <https://calibre-ebook.com/bugs>`_.
+See :website:`how to report bugs <bugs>`.

 My device is not being detected by calibre?
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@ -216,8 +215,8 @@ Follow these steps to find the problem:
      fix that by looking under :guilabel:`System Preferences > Security and
      Privacy > Privacy > Files and Folders`.
    * Make sure you are running the latest version of calibre (currently
-      |version|). The latest version can always be downloaded from `the calibre
-      website <https://calibre-ebook.com/download>`_.  You can tell what
+      |version|). The latest version can always be downloaded from :website:`the calibre
+      website <download>`.  You can tell what
      version of calibre you are currently running by looking at the bottom
      line of the main calibre window.
    * Ensure your operating system is seeing the device. That is, the device
@ -415,7 +414,7 @@ antivirus program.

 .. note::
    If you are concerned about giving calibre access to your email
-    account, simply create a new free email account with GMX or Hotmail
+    account, simply create a new free email account with GMX or Outlook
    and use it only for calibre.


@ -603,7 +602,7 @@ Why doesn't calibre have a column for foo?
 calibre is designed to have columns for the most frequently and widely used
 fields. In addition, you can add any columns you like. Columns can be added via
 :guilabel:`Preferences->Interface->Add your own columns`.  Watch the tutorial
-`UI Power tips <https://calibre-ebook.com/demo#tutorials>`_ to learn how to
+:website:`UI Power tips <demo#tutorials>` to learn how to
 create your own columns, or read `this blog post
 <https://blog.calibre-ebook.com/calibre-custom-columns/>`_.

@ -921,13 +920,13 @@ Downloading from the Internet can sometimes result in a corrupted download. If t
      the :guilabel:`Security` tab. Make sure that your user account has full control
      for this folder.

-If you still cannot get the installer to work and you are on Windows, you can use the `calibre portable install <https://calibre-ebook.com/download_portable>`_, which does not need an installer (it is just a zip file).
+If you still cannot get the installer to work and you are on Windows, you can use the :website:`calibre portable install <download_portable>`, which does not need an installer (it is just a ZIP file).

 My antivirus program claims calibre is a virus/trojan?
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

 The first thing to check is that you are downloading calibre from the official
-website: `<https://calibre-ebook.com/download>`_. Make sure you are clicking the
+website: :website:`<download>`. Make sure you are clicking the
 download links on the left, not the advertisements on the right. calibre is a
 very popular program and unscrupulous people try to setup websites offering it
 for download to fool the unwary.
@ -995,8 +994,8 @@ proxies used by calibre in Preferences->Miscellaneous.
 I want some feature added to calibre. What can I do?
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 You have two choices:
- 1. Create a patch by hacking on calibre and send it to me for review and inclusion. See `Development <https://calibre-ebook.com/get-involved>`_.
- 2. `Open a bug requesting the feature <https://calibre-ebook.com/bugs>`_. Remember that while you may think your feature request is extremely important/essential, calibre developers might not agree. Fortunately, calibre is open source, which means you always have the option of implementing your feature yourself, or hiring someone to do it for you. Furthermore, calibre has a comprehensive plugin architecture, so you might be able to develop your feature as a plugin, see :ref:`pluginstutorial`.
+ 1. Create a patch by hacking on calibre and send it to me for review and inclusion. See :website:`Development <get-involved>`.
+ 2. :website:`Open a bug requesting the feature <bugs>`. Remember that while you may think your feature request is extremely important/essential, calibre developers might not agree. Fortunately, calibre is open source, which means you always have the option of implementing your feature yourself, or hiring someone to do it for you. Furthermore, calibre has a comprehensive plugin architecture, so you might be able to develop your feature as a plugin, see :ref:`pluginstutorial`.

 Why doesn't calibre have an automatic update?
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@ -1008,8 +1007,8 @@ For many reasons:
    to see if you want to update once a year or so. There is a check box to
    turn off the update notification, on the update notification itself.

-  * calibre downloads currently use `about 150TB of bandwidth a month
-    <https://calibre-ebook.com/dynamic/downloads>`_. Implementing automatic
+  * calibre downloads currently use :website_base:`about 150TB of bandwidth a month
+    <dynamic/downloads>`. Implementing automatic
    updates would greatly increase that and end up costing thousands of dollars
    a month, which someone has to pay.

@ -1032,7 +1031,7 @@ calibre is licensed under the GNU General Public License v3 (an open source lice
 How do I run calibre from my USB stick?
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

-A portable version of calibre is available `here <https://calibre-ebook.com/download_portable>`_.
+A portable version of calibre is available :website:`here <download_portable>`.

 How do I run parts of calibre like news download and the Content server on my own Linux server?
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
--- a/manual/glossary.rst
+++ b/manual/glossary.rst
@ -26,4 +26,4 @@ Glossary
       **URL** *(Uniform Resource Locator)* for example: ``http://example.com``

    regexp
-         **Regular expressions** provide a concise and flexible means for identifying strings of text of interest, such as particular characters, words, or patterns of characters. See `regexp syntax <https://docs.python.org/library/re.html>`_ for the syntax of regular expressions used in Python.
+         **Regular expressions** provide a concise and flexible means for identifying strings of text of interest, such as particular characters, words, or patterns of characters. See :doc:`the tutorial <regexp>` for an introduction to regular expressions.
--- a/manual/gui.rst
+++ b/manual/gui.rst
@ -495,7 +495,7 @@ be configured to read metadata from the file name instead, via
 :guilabel:`Preferences->Import/export->Adding books->Read metadata from file contents`.

 You can also control how metadata is read from the filename using regular
-expressions (see :doc:`regexp`).  In the :guilabel:`Adding books` section of
+expressions (see :doc:`regexp`). In the :guilabel:`Adding books` section of
 the configuration dialog, you can specify a regular expression that calibre
 will use to try and guess metadata from the names of e-book files that you add
 to the library. The default regular expression is::
@ -566,7 +566,7 @@ The first click on an item will restrict the list of books to those that contain

 Items in the Tag browser have their icons partially colored. The amount of color depends on the average rating of the books in that category. So for example if the books by Isaac Asimov have an average of four stars, the icon for Isaac Asimov in the Tag browser will be 4/5th colored. You can hover your mouse over the icon to see the average rating.

-The outer-level items in the :guilabel:`Tag browser`, such as Authors and Series, are called categories. You can create your own categories, called :guilabel:`User categories`, which are useful for organizing items. For example, you can use the :guilabel:`User categories editor` (click the :guilabel:`Configure` button at the lower-left of the :guilabel:`Tag browser` and choose :guilabel:`Manage authors, series, etc->User categories`) to create a User category called ``Favorite Authors``, then put the items for your favorites into the category. User categories can have sub-categories. For example, the User category ``Favorites.Authors`` is a sub-category of ``Favorites``. You might also have ``Favorites.Series``, in which case there will be two sub-categories under ``Favorites``. Sub-categories can be created by right-clicking on a User category, choosing :guilabel:`Add sub-category to...`, and entering the sub-category name; or by using the :guilabel:`User categories editor` by entering names like the Favorites example above.
+The outer-level items in the :guilabel:`Tag browser`, such as Authors and Series, are called categories. You can create your own categories, called :guilabel:`User categories`, which are useful for organizing items. For example, you can use the :guilabel:`User categories editor` (click the :guilabel:`Configure` button at the lower-left of the :guilabel:`Tag browser` and choose :guilabel:`Manage authors, tags, etc->User categories`) to create a User category called ``Favorite Authors``, then put the items for your favorites into the category. User categories can have sub-categories. For example, the User category ``Favorites.Authors`` is a sub-category of ``Favorites``. You might also have ``Favorites.Series``, in which case there will be two sub-categories under ``Favorites``. Sub-categories can be created by right-clicking on a User category, choosing :guilabel:`Add sub-category to...`, and entering the sub-category name; or by using the :guilabel:`User categories editor` by entering names like the Favorites example above.

 You can search User categories in the same way as built-in categories, by clicking on them. There are four different searches cycled through by clicking:
    1. "everything matching an item in the category" indicated by a single green plus sign.
--- a/manual/metadata.rst
+++ b/manual/metadata.rst
@ -52,7 +52,7 @@ The normal edit metadata dialog also has :guilabel:`Next` and :guilabel:`Previou
 Search and replace
 ^^^^^^^^^^^^^^^^^^^^

-The :guilabel:`Bulk metadata edit` dialog allows you to perform arbitrarily powerful search and replace operations on the selected books. By default it uses a simple text search and replace, but it also support *regular expressions*. For more on regular expressions, see :ref:`regexptutorial`.
+The :guilabel:`Edit metadata for many books` dialog allows you to perform arbitrarily powerful search and replace operations on the selected books. By default it uses a simple text search and replace, but it also support *regular expressions*. For more on regular expressions, see :ref:`regexptutorial`.

 As noted above, there are two search and replace modes: character match and regular expression. Character match will look in the `Search field` you choose for the characters you type in the `search for` box and replace those characters with what you type in the `replace with` box. Each occurrence of the search characters in the field will be replaced. For example, assume the field being searched contains `a bad cat`. If you search for `a` to be replaced with `HELLO`, then the result will be `HELLO bHELLOd cHELLOt`.

--- a/manual/news.rst
+++ b/manual/news.rst
@ -105,7 +105,7 @@ This is Python, so indentation is important. After you've added the lines, it sh

 In the above, ``def print_version(self, url)`` defines a *method* that is called by calibre for every article. ``url`` is the URL of the original article. What ``print_version`` does is take that url and replace it with the new URL that points to the print version of the article. To learn about `Python <https://www.python.org>`_ see the `tutorial <https://docs.python.org/tutorial/>`_.

-Now, click the :guilabel:`Add/update recipe` button and your changes will be saved. Re-download the e-book. You should have a much improved e-book. One of the problems with the new version is that the fonts on the print version webpage are too small. This is automatically fixed when converting to an e-book, but even after the fixing process, the font size of the menus and navigation bar to become too large relative to the article text. To fix this, we will do some more customization, in the next section.
+Now, click the :guilabel:`Add/update recipe` button and your changes will be saved. Re-download the e-book. You should have a much improved e-book. One of the problems with the new version is that the fonts on the print version webpage are too small. This is automatically fixed when converting to an e-book, but even after the fixing process, the font size of the menus and navigation bar become too large relative to the article text. To fix this, we will do some more customization, in the next section.

 Replacing article styles
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
--- a/manual/simple_index.rst
+++ b/manual/simple_index.rst
@ -11,7 +11,7 @@ You've just started calibre. What do you do now? Before calibre can do anything

 Once you've admired the list of books you just added to your heart's content, you'll probably want to read one. In order to do that you'll have to convert the book to a format your reader understands. When first running calibre, the :guilabel:`Welcome wizard` starts and will set up calibre for your reader device. Conversion is a breeze. Just select the book you want to convert then click the "Convert books" button. Ignore all the options for now and click "OK". The little icon in the bottom right corner will start spinning. Once it's finished spinning, your converted book is ready. Click the "View" button to read the book.

-If you want to read the book on your reader, connect it to the computer, wait till calibre detects it (10-20 seconds) and then click the "Send to device" button. Once the icon stops spinning again, disconnect your reader and read away! If you didn't convert the book in the previous step, calibre will auto convert it to the format your reader device understands.
+If you want to read the book on your reader, connect the reader to the computer, wait till calibre detects it (10-20 seconds) and then click the "Send to device" button. Once the icon stops spinning again, disconnect your reader and read away! If you didn't convert the book in the previous step, calibre will auto convert it to the format your reader device understands.

 To get started with more advanced usage, you should read about :doc:`gui`. For even more power and versatility, learn the :doc:`generated/en/cli-index`. You will find the list of :doc:`faq` useful as well.

--- a/manual/template_lang.rst
+++ b/manual/template_lang.rst
@ -161,7 +161,7 @@ The functions intended for use in Single Function Mode are:
 * ``rating_to_stars(use_half_stars)`` -- Returns the rating as string of star (``★``) characters. The value must be a number between 0 and 5. Set use_half_stars to 1 if you want half star characters for fractional numbers available with custom ratings columns.
 * ``re(pattern, replacement)`` -- return the value after applying the regular expression. All instances of ``pattern`` in the value are replaced with ``replacement``. The template language uses case insensitive `Python regular expressions <https://docs.python.org/3/library/re.html>`_.
 * ``select(key)`` -- interpret the value as a comma-separated list of items with each item having the form ``id:value`` (the calibre ``identifier`` format). The function finds the first pair with the id equal to key and returns the corresponding value. If no id matches then the function returns the empty string.
-* ``shorten(left chars, middle text, right chars)`` -- Return a shortened version of the value, consisting of ``left chars`` characters from the beginning of the value, followed by ``middle text``, followed by ``right chars`` characters from the end of the value. ``Left chars`` and ``right chars`` must be non-negative integers. Example: assume you want to display the title with a length of at most 15 characters in length. One template that does this is ``{title:shorten(9,-,5)}``. For a book with the title `Ancient English Laws in the Times of Ivanhoe` the result will be `Ancient E-nhoe`: the first 9 characters of the title, a ``-``, then the last 5 characters. If the value's length is less than ``left chars`` + ``right chars`` + the length of ``middle text`` then the value will be returned unchanged. For example, the title `The Dome` would not be changed.
+* ``shorten(left chars, middle text, right chars)`` -- Return a shortened version of the value, consisting of ``left chars`` characters from the beginning of the value, followed by ``middle text``, followed by ``right chars`` characters from the end of the value. ``Left chars`` and ``right chars`` must be non-negative integers. Example: assume you want to display the title with a length of at most 15 characters in length. One template that does this is ``{title:shorten(9,-,5)}``. For a book with the title `Ancient English Laws in the Times of Ivanhoe` the result will be `Ancient E-anhoe`: the first 9 characters of the title, a ``-``, then the last 5 characters. If the value's length is less than ``left chars`` + ``right chars`` + the length of ``middle text`` then the value will be returned unchanged. For example, the title `The Dome` would not be changed.
 * ``str_in_list(separator, [ string, found_val, ]+ not_found_val)`` -- interpret the value as a list of items separated by ``separator`` then compare ``string`` against each value in the list. The ``string`` is not a regular expression. If ``string`` is equal to any item (ignoring case) then return the corresponding ``found_val``. If ``string`` contains ``separators`` then it is also treated as a list and each subvalue is checked. The ``string`` and ``found_value`` pairs can be repeated as many times as desired, permitting returning different values depending on string's value. If none of the strings match then ``not_found_value`` is returned. The strings are checked in order. The first match is returned.
 * ``subitems(start_index, end_index)`` -- This function breaks apart lists of tag-like hierarchical items such as genres. It interprets the value as a comma-separated list of tag-like items, where each item is a period-separated list. It returns a new list made by extracting from each item the components from ``start_index`` to ``end_index``, then merging the results back together. Duplicates are removed. The first subitem in a period-separated list has an index of zero. If an index is negative then it counts from the end of the list. As a special case, an end_index of zero is assumed to be the length of the list.

@ -678,7 +678,7 @@ The same thing happens for authors, but using a different character for the cut,

 Plugboards affect the metadata written into the book when it is saved to disk or written to the device. Plugboards do not affect the metadata used by ``save to disk`` and ``send to device`` to create the file names. Instead, file names are constructed using the templates entered on the appropriate preferences window.

-Tips:
+Tips
 -----

 * Use the Template Tester to test templates. Add the tester to the context menu for books in the library and/or give it a keyboard shortcut.
--- a/manual/viewer.rst
+++ b/manual/viewer.rst
@ -41,7 +41,7 @@ You can access the viewer controls by either:
 The viewer has two modes, "paged" and "flow". In paged mode the book content
 is presented as pages, similar to a paper book. In flow mode the text is
 presented continuously, like in a web browser. You can switch between them
-using the viewer Preferences under :guilabel:`Page layout` or by pressing the
+using the viewer :guilabel:`Preferences` under :guilabel:`Page layout` or by pressing the
 :kbd:`Ctrl+M` key.


@ -62,7 +62,7 @@ Table of Contents
 ^^^^^^^^^^^^^^^^^^^^

 If the book you are reading defines a Table of Contents, you can access it by
-pressing the :guilabel:`Table of Contents` button.  This will bring up a list
+pressing the :guilabel:`Table of Contents` button. This will bring up a list
 of sections in the book. You can click on any of them to jump to that portion
 of the book.

--- a/recipes/aif_ru.recipe
+++ b/recipes/aif_ru.recipe
@ -1,45 +1,24 @@
 #!/usr/bin/env python
 # vim:fileencoding=utf-8
-from __future__ import with_statement, unicode_literals
+from __future__ import unicode_literals
 from calibre.web.feeds.news import BasicNewsRecipe


-class AdvancedUserRecipe1592177429(BasicNewsRecipe):
-    title = 'Аргументы и Факты'
-    encoding = 'utf8'
-    language = 'ru'
+class AdvancedUserRecipe1645564525(BasicNewsRecipe):
+    title          = 'Аргументы и Факты - aif.ru'
+    __author__     = 'MrBeef12'
    oldest_article = 7
-    max_articles_per_feed = 25
-    verbose = 3
+    max_articles_per_feed = 100
+    auto_cleanup   = True
+    language = 'ru'

-    feeds = [
-        ('AIF', 'https://www.aif.ru/rss/all.php'),
+    feeds          = [
+        ('ПОЛИТИКА', 'https://aif.ru/rss/politics.php'),
+        ('ДЕНЬГИ', 'https://aif.ru/rss/money.php'),
+        ('ОБЩЕСТВО', 'https://aif.ru/rss/society.php'),
+        ('ЗДОРОВЬЕ', 'https://aif.ru/rss/health.php'),
+        ('КУЛЬТУРА', 'https://aif.ru/rss/culture.php'),
+        ('СПОРТ', 'https://aif.ru/rss/sport.php'),
+        ('АВТОМОБИЛИ', 'https://aif.ru/rss/automobile.php'),
+        ('НЕДВИЖИМОСТЬ', 'https://aif.ru/rss/realty.php'),
    ]
-    INDEX = 'https://www.aif.ru/rss/all.php'
-
-    def parse_index(self):
-        feeds = []
-        section_title = 'aif'
-        articles = []
-        soup = self.index_to_soup(self.INDEX)
-        ii = 0
-        for item in soup.findAll('item'):
-            if ii < self.max_articles_per_feed:
-                try:
-                    ii = ii + 1
-                    A = str(item)
-                    i = A.find(u'link')
-                    j = A.find(u'description')
-                    ZZ = item.find('description')
-                    ZZ1 = str(ZZ)  # bs4.element.Tag to str
-                    ZZ2 = ZZ1[24:-19]
-                    AU = A[i:j]
-                    try:
-                        articles.append({'url':AU[6:-2], 'title':ZZ2})
-                    except Exception:
-                        pass
-                except Exception:
-                    self.log("Exception handled!")
-        if articles:
-            feeds.append((section_title, articles))
-        return feeds
--- a/recipes/al_jazeera.recipe
+++ b/recipes/al_jazeera.recipe
@ -4,11 +4,7 @@
 '''
 english.aljazeera.net
 '''
-from calibre.web.feeds.news import BasicNewsRecipe
-
-
-def has_cls(x):
-    return dict(attrs={'class': lambda cls: cls and x in cls.split()})
+from calibre.web.feeds.news import BasicNewsRecipe, classes


 class AlJazeera(BasicNewsRecipe):
@ -31,7 +27,7 @@ class AlJazeera(BasicNewsRecipe):
        'publisher': publisher, 'language': language
    }
    keep_only_tags = [
-        dict(id='article-page'),
+        classes('article-header article-featured-image wysiwyg--all-content'),
    ]

    remove_tags = [
--- a/recipes/aprospect.recipe
+++ b/recipes/aprospect.recipe
@ -1,20 +1,39 @@
-from calibre.web.feeds.news import BasicNewsRecipe
+#!/usr/bin/env  python
+# License: GPLv3 Copyright: 2008, Kovid Goyal <kovid at kovidgoyal.net>
+
+from __future__ import unicode_literals
+from calibre.web.feeds.news import BasicNewsRecipe, classes


 class AmericanProspect(BasicNewsRecipe):
-    title = u'American Prospect'
-    __author__ = u'Michael Heinz, a.peter'
-    version = 2
-
-    oldest_article = 30
+    title = 'American Prospect'
+    __author__ = 'Kovid Goyal'
+    oldest_article = 300
    language = 'en'
    max_articles_per_feed = 100
    recursions = 0
    no_stylesheets = True
    remove_javascript = True
+    encoding = 'utf-8'

    use_embedded_content = False

    no_stylesheets = True
-    auto_cleanup = True
-    feeds = [(u'Articles', u'feed://www.prospect.org/articles_rss.jsp')]
+
+    keep_only_tags = [
+        dict(id=['title', 'content']),
+    ]
+    remove_tags = [
+        classes('slideout-close-btn media-options')
+    ]
+
+    def get_feeds(self):
+        soup = self.index_to_soup('https://prospect.org/archive')
+        for a in soup.findAll('a', href=True):
+            href = a['href']
+            if href.endswith('-issue/'):
+                d = href.strip('/').split('/')[-1]
+                self.timefmt = ' [{}]'.format(d.rpartition('-')[0])
+                self.log('Found magazine URL', href)
+                return [('Articles', href + 'index.rss')]
+        return [('Articles', 'https://prospect.org/api/rss/all.rss')]
--- a/recipes/bbc.recipe
+++ b/recipes/bbc.recipe
@ -111,6 +111,23 @@ def parse_article_json(root, abort_article):
        elif bt == 'text':
            lines.extend(serialize_text(block))
    return '<html><body id="main-content">' + '\n'.join(lines) + '</body></html>'
+
+
+def parse_raw_html(html, abort_article):
+    q = '>window.__INITIAL_DATA__="{'
+    idx = html.find(q)
+    if idx < 0:
+        raise ValueError('Failed to find JSON')
+    data = html[idx + len(q) - 2:]
+    idx = data.find('}";</script>')
+    data = data[:idx+2]
+    data = json.loads(data)
+    root = json.loads(data)
+    return parse_article_json(root, abort_article)
+
+
+if __name__ == '__main__':
+    print(parse_raw_html(open('/t/raw.html').read(), print))
 # }}}


@ -269,12 +286,4 @@ class BBCNews(BasicNewsRecipe):
    resolve_internal_links = True

    def preprocess_raw_html(self, raw_html, url):
-        q = '>window.__INITIAL_DATA__={'
-        idx = raw_html.find(q)
-        if idx < 0:
-            raise ValueError('Failed to find JSON')
-        data = raw_html[idx + len(q) - 1:]
-        idx = data.find('};</script>')
-        data = data[:idx+1]
-        root = json.loads(data)
-        return parse_article_json(root, self.abort_article)
+        return parse_raw_html(raw_html, self.abort_article)
--- a/recipes/bbc_fast.recipe
+++ b/recipes/bbc_fast.recipe
@ -12,18 +12,20 @@
 def serialize_image(block):
    yield '<div>'
    block = block['model']
-    media = block['media']
-    alt = prepare_string_for_xml(media.get('alt') or '', True)
+    img = block['image']
+    alt = prepare_string_for_xml(img.get('alt') or '', True)
    for q in ('originalSrc', 'src'):
-        if q in media:
-            src = prepare_string_for_xml(media[q])
+        if q in img:
+            src = prepare_string_for_xml(img[q])
            break
    else:
-        raise ValueError('No src found in media block: {}'.format(media))
+        raise ValueError('No src found in img block: {}'.format(img))
    yield '<img src="{}" alt="{}"/>'.format(src, alt)
    caption = block.get('caption')
-    if caption:
-        yield '<div>{}</div>'.format(prepare_string_for_xml(caption))
+    if caption and caption.get('type') == 'text':
+        yield '<div>'
+        yield from serialize_paragraph(caption)
+        yield '</div>'
    yield '</div>'


@ -102,13 +104,30 @@ def parse_article_json(root, abort_article):
        lines.append('<h1>{}</h1>'.format(prepare_string_for_xml(article['headline'])))
    if article.get('contributor'):
        lines.extend(serialize_contributor(article['contributor']))
-    for block in article['blocks']:
+    for block in article['content']['model']['blocks']:
        bt = block.get('type')
        if bt == 'image':
            lines.extend(serialize_image(block))
        elif bt == 'text':
            lines.extend(serialize_text(block))
    return '<html><body id="main-content">' + '\n'.join(lines) + '</body></html>'
+
+
+def parse_raw_html(html, abort_article):
+    q = '>window.__INITIAL_DATA__="{'
+    idx = html.find(q)
+    if idx < 0:
+        raise ValueError('Failed to find JSON')
+    data = html[idx + len(q) - 2:]
+    idx = data.find('}";</script>')
+    data = data[:idx+2]
+    data = json.loads(data)
+    root = json.loads(data)
+    return parse_article_json(root, abort_article)
+
+
+if __name__ == '__main__':
+    print(parse_raw_html(open('/t/raw.html').read(), print))
 # }}}


@ -124,10 +143,13 @@ class BBC(BasicNewsRecipe):
    publisher = 'BBC'
    category = 'news, UK, world'
    language = 'en_GB'
-    masthead_url = 'https://news.bbcimg.co.uk/img/1_0_1/cream/hi/news/news-blocks.gif'
    conversion_options = {
        'comments': description, 'tags': category, 'language': language, 'publisher': publisher,
    }
+    # Removes empty feeds - why keep them!?
+    remove_empty_feeds = True
+    ignore_duplicate_articles = {'title', 'url'}
+    resolve_internal_links = True

    feeds = [
        ('Top Stories', 'https://feeds.bbci.co.uk/news/rss.xml'),
@ -150,12 +172,4 @@ class BBC(BasicNewsRecipe):
    ]

    def preprocess_raw_html(self, raw_html, url):
-        q = '>window.__INITIAL_DATA__={'
-        idx = raw_html.find(q)
-        if idx < 0:
-            raise ValueError('Failed to find JSON')
-        data = raw_html[idx + len(q) - 1:]
-        idx = data.find('};</script>')
-        data = data[:idx+1]
-        root = json.loads(data)
-        return parse_article_json(root, self.abort_article)
+        return parse_raw_html(raw_html, self.abort_article)
--- a/recipes/clarin.recipe
+++ b/recipes/clarin.recipe
@ -34,14 +34,11 @@ class Clarin(BasicNewsRecipe):
    needs_subscription = 'optional'
    INDEX = 'http://www.clarin.com'
    LOGIN = 'https://app-pase.clarin.com/pase-registracion/app/pase/ingresarNavegable?execution=e1s1'
-    masthead_url = 'http://www.clarin.com/images/logo_clarin.svg'
    cover_url = strftime('http://tapas.clarin.com/tapa/%Y/%m/%d/%Y%m%d_thumb.jpg')

    compress_news_images = True
    scale_news_images_to_device = True
    compress_news_images_max_size = 10  # kB
-    scale_news_images = True
-    handle_gzip = True

    # To get all the data (images)
    auto_cleanup = False
@ -132,7 +129,7 @@ def get_browser(self):
        return br

    def preprocess_html(self, soup):
-        for img in soup.findAll(['img']):
+        for img in soup.findAll(['img'], attrs={'data-big': True}):
            img['src'] = img['data-big']

        for figCaption in soup.findAll(['figcaption']):
--- a/recipes/down_to_earth.recipe
+++ b/recipes/down_to_earth.recipe
@ -1,8 +1,4 @@
-from __future__ import with_statement
-__license__ = 'GPL 3'
-__copyright__ = '2014, Amit <amitkp.ias@gmail.com>'
-
-from calibre.web.feeds.news import BasicNewsRecipe
+from calibre.web.feeds.news import BasicNewsRecipe, classes


 class My_Feeds(BasicNewsRecipe):
@ -16,19 +12,15 @@ class My_Feeds(BasicNewsRecipe):
    center_navbar = True
    use_embedded_content = False
    remove_empty_feeds = True
-    remove_tags_before = dict(name='div', id='PageContent')
-    remove_tags_after = [dict(name='div'), {'class': 'taxonomy-terms'}]
+    keep_only_tags = [
+        classes('detail-heading content-main news-basic-info news-banner news-detail-content')
+    ]
    remove_tags = [
+        classes('add-comment btn hindi_detail_link single-news-letter'),
        dict(id=['comments', 'breadcrumb', 'node_related_stories']),
        dict(attrs={'class': ['commentCount', 'box']})
    ]

    feeds = [
-        (u'editor', u'http://www.downtoearth.org.in/taxonomy/term/20348/0/feed'),
-        (u'cover story', u'http://www.downtoearth.org.in/taxonomy/term/20345/0/feed'),
-        (u'special report', u'http://www.downtoearth.org.in/taxonomy/term/20384/0/feed'),
-        (u'features', u'http://www.downtoearth.org.in/taxonomy/term/20350/0/feed'),
-        (u'news', u'http://www.downtoearth.org.in/taxonomy/term/20366/0/feed'),
-        (u'debate', u'http://www.downtoearth.org.in/taxonomy/term/20347/0/feed'),
-        (u'natural disasters', u'http://www.downtoearth.org.in/taxonomy/term/20822/0/feed')
+        ('All', 'https://www.downtoearth.org.in/rss/all'),
    ]
--- a/recipes/dw_de.recipe
+++ b/recipes/dw_de.recipe
@ -0,0 +1,29 @@
+#!/usr/bin/env python
+from calibre.web.feeds.news import BasicNewsRecipe
+
+
+class AdvancedUserRecipe1398527969(BasicNewsRecipe):
+    title = u'DW-Deutsch XXL'
+    language = 'de_DE'
+    __author__ = 'xav'
+    oldest_article = 7
+    max_articles_per_feed = 100
+    auto_cleanup = True
+    no_stylesheets = True
+    use_embedded_content = False
+    remove_javascript = True
+    feeds = [(u'Nachrichten', u'http://rss.dw.com/xml/rss-de-news'),
+             (u'Themen des Tages', u'http://rss.dw.com/xml/rss-de-top'),
+             (u'Langsam gesprochene Nachrichten',
+              u'https://rss.dw.com/rdf/DKfeed_lgn_de'),
+             (u'Wissenschaft', u'https://rss.dw.com/xml/rss-de-wissenschaft'),
+             (u'Wirtschaft', u'https://rss.dw.com/xml/rss-de-eco'),
+             (u'Wort der Woche',
+              u'https://rss.dw.com/xml/DKpodcast_wortderwoche_de'),
+             (u'Deutschland entdecken',
+              u'http://rss.dw.com/xml/rss-de-deutschlandentdecken')]
+
+    def print_version(self, url):
+        target = url.rpartition('/')[2]
+        print_url = 'https://www.dw-world.de/popups/popup_printcontent/' + target
+        return print_url
--- a/recipes/el_pais.recipe
+++ b/recipes/el_pais.recipe
@ -42,10 +42,13 @@ class ElPais(BasicNewsRecipe):
                              'articulo-apertura',
                              'articulo__contenedor'
                             ]}),
+        dict(name='div', attrs={'class': 'a_c',}),
+
    ]

    remove_tags = [
-        dict(attrs={'class': [
+        dict(
+            attrs={'class': [
                              'sumario__interior',
                              'articulo-trust',
                              'compartir',
@ -54,7 +57,11 @@ class ElPais(BasicNewsRecipe):
                              'more_info',
                              'articulo-apoyos',
                              'top10',
-                             ]}),
+                             ]
+                    },
+            ),
+        dict(id='cta_id'),
+        dict(name='svg'),
    ]

    feeds = [
--- a/recipes/equestria_daily.recipe
+++ b/recipes/equestria_daily.recipe
@ -0,0 +1,81 @@
+#!/usr/bin/env python
+# vim:fileencoding=utf-8
+from calibre.web.feeds.news import BasicNewsRecipe
+from calibre.utils.date import parse_date, utcnow
+
+
+class AdvancedUserRecipe1639926896(BasicNewsRecipe):
+    __author__ = "Aisteru"
+    __copyright__ = "2021, Timothée Andres <timothee dot andres at gmail dot com>"
+    __license__ = 'GNU General Public License v3 - http://www.gnu.org/copyleft/gpl.html'
+
+    title = "Equestria Daily"
+    description = "Everything new in Equestria and beyond!"
+    language = 'en_US'
+
+    # Max. supported by website: 50
+    max_articles_per_feed = 30
+
+    compress_news_images = True
+    compress_news_images_auto_size = 4
+    no_stylesheets = True
+    keep_only_tags = [{'name': 'div', 'class_': ['post', 'hentry']}]
+    remove_tags = [{'name': 'div', 'class_': 'post-footer'}]
+    extra_css = '.article_date { margin-left: 10px; }'
+
+    # Masthead image dimensions
+    MI_WIDTH = 600
+    MI_HEIGHT = 200
+
+    # To discard posts under a certain section, simply comment the whole line
+    sections = [
+        ("Art", 'Art'),
+        ("News", 'News'),
+        ("Fics", 'Fanfiction'),
+        ("Media", 'Media'),
+        ("Comics", 'Comic'),
+        ("Community", 'Community'),
+        ("Editorial", 'Editorial'),
+    ]
+
+    def get_masthead_url(self):
+        soup = self.index_to_soup('https://www.equestriadaily.com')
+        img = soup.select_one('#header img')
+        return img['src']
+
+    def parse_index(self):
+        results = {}
+        current_date = utcnow()
+
+        def clean_description(description):
+            lines = description.split('\n')
+            return '\n'.join([line.strip() for line in lines if len(line.strip()) > 0])
+
+        for (section_name, section_url_name) in self.sections:
+            soup = self.index_to_soup(
+                f'https://www.equestriadaily.com/search/label/{section_url_name}?max-results={self.max_articles_per_feed}')
+            articles = soup.select('div.post.hentry')
+            previous_post_date = current_date
+
+            for article in articles:
+                article_entry = {}
+
+                header = article.select_one('h3 > a')
+                article_entry['title'] = header.text
+                article_entry['url'] = header['href']
+                article_entry['date'] = article.select_one('span.post-timestamp').text.split('\n')[1]
+                article_entry['description'] = clean_description(article.select_one('div.entry-content').text)
+                article_entry['content'] = ''  # Must be empty
+
+                post_date = previous_post_date
+
+                try:
+                    post_date = parse_date(article_entry['date'])
+                    previous_post_date = post_date
+                except Exception:
+                    pass
+
+                if (current_date - post_date).days <= self.oldest_article:
+                    results.setdefault(section_name, []).append(article_entry)
+
+        return [(section, results[section]) for section in results]
--- a/recipes/fe_india.recipe
+++ b/recipes/fe_india.recipe
@ -19,35 +19,48 @@ class FE_India(BasicNewsRecipe):
    description = 'Financial news from India'
    publisher = 'The Indian Express Limited'
    category = 'news, politics, finances, India'
-    oldest_article = 30
+    oldest_article = 2
    max_articles_per_feed = 200
    no_stylesheets = True
    encoding = 'utf-8'
    use_embedded_content = False
    language = 'en_IN'
    remove_empty_feeds = True
+    ignore_duplicate_articles = {'url'}
    publication_type = 'magazine'

    conversion_options = {
        'comment': description, 'tags': category, 'publisher': publisher, 'language': language
    }

-    keep_only_tags = [classes('post-title place-line leftstory')]
+    keep_only_tags = [classes('wp-block-post-title wp-block-post-excerpt ie-network-post-meta-wrapper wp-block-post-featured-image wp-block-post-content')]
    remove_attributes = ['width', 'height']

    feeds = [
-        ('Latest news', 'https://www.financialexpress.com/feed/'),
+        # https://www.financialexpress.com/syndication/
+        # Print feeds
+        ('Front Page','https://www.financialexpress.com/print/front-page/feed/'),
+        ('Corporate Markets','https://www.financialexpress.com/print/corporate-markets/feed/'),
+        ('Economy','https://www.financialexpress.com/print/economy-print/feed/'),
+        ('Opinion','https://www.financialexpress.com/print/edits-columns/feed/'),
+        ('personal Finance','https://www.financialexpress.com/print/personal-finance-print/feed/'),
+        # ('Brandwagon', 'https://www.financialexpress.com/print/brandwagon/feed/'),
+        # Other Feeds
        ('Economy', 'https://www.financialexpress.com/economy/feed/'),
-        ('Industry', 'https://www.financialexpress.com/industry/feed/'),
        ('Banking & finance', 'https://www.financialexpress.com/industry/banking-finance/feed/'),
-        ('Companies', 'https://www.financialexpress.com/industry/companies/feed/'),
-        ('Jobs', 'https://www.financialexpress.com/industry/jobs/feed/'),
-        ('Tech', 'https://www.financialexpress.com/industry/tech/feed/'),
-        ('Lifestyle', 'https://www.financialexpress.com/industry/lifestyle/feed/'),
-        ('Health', 'https://www.financialexpress.com/industry/health/feed/'),
-        ('Science', 'https://www.financialexpress.com/industry/science/feed/'),
-        ('Sports', 'https://www.financialexpress.com/industry/sports/feed/'),
-        ('Fe Columnist', 'https://www.financialexpress.com/industry/fe-columnist/feed/'),
+        ('Opinion', 'https://www.financialexpress.com/opinion/feed/'),
+        ('Editorial', 'https://www.financialexpress.com/editorial/feed/'),
+        ('Budget', 'https://www.financialexpress.com/budget/feed/'),
+        ('Industry', 'https://www.financialexpress.com/industry/feed/'),
+        ('Market', 'https://www.financialexpress.com/market/feed/'),
+        ('Jobs', 'https://www.financialexpress.com/jobs/feed/'),
+        ('SME', 'https://www.financialexpress.com/industry/sme/feed/'),
+        ('Mutual Funds', 'https://www.financialexpress.com/money/mutual-funds/feed/'),
+        ('Health','https://www.financialexpress.com/lifestyle/health/feed'),
+        # ('Health Care','https://www.financialexpress.com/healthcare/feed'),
+        ('Science','https://www.financialexpress.com/lifestyle/science/feed'),
+        ('Infrastructure','https://www.financialexpress.com/infrastructure/feed'),
+        ('Money','https://www.financialexpress.com/money/feed'),
    ]

    def preprocess_html(self, soup, *a):
--- a/recipes/foreignaffairs.recipe
+++ b/recipes/foreignaffairs.recipe
@ -136,7 +136,7 @@ class ForeignAffairsRecipe(BasicNewsRecipe):
        classes('article-header article-body article-lead-image article-body-text'),
    ]
    remove_tags = [
-        classes('loading-indicator paywall article-footer')
+        classes('loading-indicator paywall article-footer article-tools')
    ]

    conversion_options = {'comments': description, 'tags': category, 'language': 'en',
@ -149,10 +149,12 @@ def parse_index(self):
            soup.head.title.string))[0]
        self.title = "Foreign Affairs ({})".format(date)
        self.timefmt = u' [%s]' % date
-        link = soup.find('link', rel='revision', href=True)['href']
+        link = soup.find('link', rel='canonical', href=True)['href']
        year, volnum, issue_vol = link.split('/')[-3:]
-        self.cover_url = (soup.find('img', {'class': 'subscribe-callout-image'})['data-src']
-                          .split("|", 1)[0].replace('issue_small_1x', 'issue_large_2x'))
+        self.cover_url = soup.find(**classes('subscribe-callout-image'))['data-src'].split("|")[-1]
+        self.cover_url = self.cover_url.split('?')[0]
+        self.cover_url = self.cover_url.replace('.webp', '')
+        self.cover_url = self.cover_url.replace('_webp_issue_small_2x', 'issue_large_2x')

        cls = soup.find('body')['class']
        if isinstance(cls, (list, tuple)):
--- a/recipes/gkt.recipe
+++ b/recipes/gkt.recipe
@ -1,4 +1,4 @@
-from calibre.web.feeds.news import BasicNewsRecipe
+from calibre.web.feeds.news import BasicNewsRecipe, classes


 class GKT(BasicNewsRecipe):
@ -12,33 +12,36 @@ class GKT(BasicNewsRecipe):
    no_javascript = True
    auto_cleanup = True

-    def parse_gkt_section(self, url, ignore_error=False):
-        try:
-            root = self.index_to_soup(url, as_tree=True)
-        except Exception:
-            if ignore_error:
-                return
-            raise
-        for a in root.xpath('//div[@class="posts-listing"]/h1/a[@href]'):
-            title = self.tag_to_string(a).strip()
-            url = a.get('href')
-            if title and url:
-                self.log('\tFound article:', title, 'at', url)
-                yield {'title': title, 'url': url}
-
    def parse_index(self):
-        url = 'http://www.gktoday.in/'
-        root = self.index_to_soup(url, as_tree=True)
-        ans = []
-        h3 = root.xpath('//h3[@class="widget-title"]')[1]
-        for a in h3.getparent().xpath('descendant::li/a[@href]'):
-            category = self.tag_to_string(a).strip()
-            if 'PDF' in category or not category:
-                continue
-            url = a.get('href')
-            self.log('Found section:', category, 'at', url)
-            articles = list(self.parse_gkt_section(url)) + \
-                list(self.parse_gkt_section(url + '/page/2', ignore_error=True))
-            if articles:
-                ans.append((category, articles))
-        return ans
+        securl = 'https://www.gktoday.in/current-affairs/'
+        ans = {}
+
+        def p_tags(h1):
+            for sib in h1.next_siblings:
+                if sib.name == 'h1':
+                    break
+                if sib.name == 'p':
+                    yield sib
+
+        def find_cat(ps):
+            for p in ps:
+                for a in p.findAll('a', rel='tag'):
+                    return self.tag_to_string(a)
+
+        for i in range(1, 6):
+            page = '' if i == 1 else 'page/' + str(i)
+            self.log('Trying:', securl + page)
+            soup = self.index_to_soup(securl + page)
+            container = soup.find(**classes('left_middle_content'))
+            for h1 in container.findAll('h1'):
+                title = self.tag_to_string(h1)
+                a = h1.find('a')
+                if a is None:
+                    continue
+                url = a['href']
+                ps = tuple(p_tags(h1))
+                category = find_cat(ps) or 'Unknown'
+                ans.setdefault(category, []).append({
+                    'title': title, 'url': url, 'description': self.tag_to_string(ps[0])})
+                self.log('\t' + title + ' ' + url)
+        return list(ans.items())
--- a/recipes/hindu.recipe
+++ b/recipes/hindu.recipe
@ -23,7 +23,7 @@ class TheHindu(BasicNewsRecipe):
    __author__ = 'Kovid Goyal'
    max_articles_per_feed = 100
    no_stylesheets = True
-    remove_attributes = ['style']
+    remove_attributes = ['style', 'height', 'width']
    extra_css = '.lead-img-cont { text-align: center; } ' \
                '.lead-img-caption { font-size: small; font-style: italic; } ' \
                '.mobile-author-cont { font-size: small; text-transform: uppercase; } ' \
@ -37,8 +37,9 @@ class TheHindu(BasicNewsRecipe):
    ]

    def get_browser(self):
-        br = BasicNewsRecipe.get_browser(self)
+        br = BasicNewsRecipe.get_browser(self, user_agent='common_words/based')
        br.addheaders += [('Referer', self.epaper_url)]     # needed for fetching cover
+        # br.set_debug_http(True)
        return br

    def get_cover_url(self):
@ -54,6 +55,8 @@ def preprocess_html(self, soup):
                source.extract()
        except Exception:
            pass
+        for img in soup.findAll(attrs={'data-original': True}):
+            img['src'] = img['data-original']
        # Place intro beneath the title, skip duplicates
        try:
            soup.h1.insert_after(soup.find('h2', attrs={'class': 'intro'}))
@ -76,7 +79,7 @@ def populate_article_metadata(self, article, soup, first):

    def articles_from_soup(self, soup):
        ans = []
-        div = soup.find('section', attrs={'id': 'section_'})
+        div = soup.find('section', attrs={'id': 'section_1'})
        if div is None:
            return ans
        for ul in div.findAll('ul', attrs={'class': 'archive-list'}):
@ -99,7 +102,7 @@ def parse_index(self):
        #     {'title':'xxx', 'url':'http://www.thehindu.com/opinion/op-ed/rohingya-bangladeshs-burden-to-bear/article19694058.ece'},
        #     {'title':'yyy', 'url':'http://www.thehindu.com/sci-tech/energy-and-environment/on-river-washed-antique-plains/article19699327.ece'}
        # ])]
-        soup = self.index_to_soup('http://www.thehindu.com/todays-paper/')
+        soup = self.index_to_soup('https://www.thehindu.com/todays-paper/')
        nav_div = soup.find(id='subnav-tpbar-latest')
        section_list = []

--- a/recipes/hindustan_times.recipe
+++ b/recipes/hindustan_times.recipe
@ -0,0 +1,58 @@
+#!/usr/bin/env  python
+
+from calibre.web.feeds.news import BasicNewsRecipe, classes
+
+
+class HindustanTimes(BasicNewsRecipe):
+    title = u'Hindustan Times'
+    description = 'News from India.'
+    language = 'en_IN'
+    __author__ = 'unkn0wn'
+    oldest_article = 1  # days
+    max_articles_per_feed = 50
+    encoding = 'utf-8'
+    use_embedded_content = False
+    no_stylesheets = True
+    remove_attributes = ['style', 'height', 'width']
+    ignore_duplicate_articles = {'url'}
+    extra_css = 'button { display: none; } '
+
+    keep_only_tags = [
+        dict(name='h1'),
+        dict(name='div', attrs={'class':'sortDec'}),
+        dict(name='picture'),
+        dict(name='figcaption'),
+        classes('dateTime storyBy storyDetails detail freemiumText paywall'),
+    ]
+
+    remove_tags = [
+        classes('htsWrapper shareArticle new__newsletter__signup signup__box subscribe freemium-card adMinHeight313'
+                ' storyTopics embed_div shareIcons close-btn'),
+        dict(name='div', attrs={'class':[]}),
+        dict(name='footer'),
+    ]
+
+    feeds = [
+        ('Editorial','https://www.hindustantimes.com/feeds/rss/editorials/rssfeed.xml'),
+        ('Opinion','https://www.hindustantimes.com/feeds/rss/opinion/rssfeed.xml'),
+        ('HT Insight','https://www.hindustantimes.com/feeds/rss/ht-insight/rssfeed.xml'),
+        ('Analysis','https://www.hindustantimes.com/feeds/rss/analysis/rssfeed.xml'),
+        ('India News','https://www.hindustantimes.com/feeds/rss/india-news/rssfeed.xml'),
+        ('World News','https://www.hindustantimes.com/feeds/rss/world-news/rssfeed.xml'),
+        ('Business','https://www.hindustantimes.com/feeds/rss/business/rssfeed.xml'),
+        ('Science','https://www.hindustantimes.com/feeds/rss/science/rssfeed.xml'),
+        ('Education','https://www.hindustantimes.com/feeds/rss/education/rssfeed.xml'),
+        ('Elections','https://www.hindustantimes.com/feeds/rss/elections/rssfeed.xml'),
+        ('Sports','https://www.hindustantimes.com/feeds/rss/sports/rssfeed.xml'),
+        ('Books','https://www.hindustantimes.com/feeds/rss/books/rssfeed.xml'),
+        ('HT Weekend','https://www.hindustantimes.com/feeds/rss/ht-weekend/rssfeed.xml'),
+        # ('Entertainment','https://www.hindustantimes.com/feeds/rss/entertainment/rssfeed.xml'),
+        # ('Lifestyle',''https://www.hindustantimes.com/feeds/rss/lifestyle/rssfeed.xml'),
+        # ('Cities',''https://www.hindustantimes.com/feeds/rss/cities/rssfeed.xml'),
+        # ('Budget',''https://www.hindustantimes.com/feeds/rss/budget/rssfeed.xml')
+        ]
+
+    def preprocess_html(self, soup):
+        for img in soup.findAll('img', attrs={'data-src': True}):
+            img['src'] = img['data-src']
+        return soup
--- a/recipes/india_legal_magazine.recipe
+++ b/recipes/india_legal_magazine.recipe
@ -0,0 +1,40 @@
+#!/usr/bin/env  python
+
+from calibre.web.feeds.news import BasicNewsRecipe, classes
+
+
+class IndiaLegalLive(BasicNewsRecipe):
+    title = 'India Legal Magazine'
+    language = 'en_IN'
+    __author__ = 'unkn0wn'
+    oldest_article = 7  # days
+    max_articles_per_feed = 50
+    encoding = 'utf-8'
+    use_embedded_content = False
+    no_stylesheets = True
+    remove_attributes = ['style', 'height', 'width']
+
+    masthead_url = 'https://d2r2ijn7njrktv.cloudfront.net/IL/uploads/2020/12/03181846/india-legal-live-logo-218x73-1.png'
+
+    def get_cover_url(self):
+        soup = self.index_to_soup('https://www.indialegallive.com/')
+        for citem in soup.findAll('img', src=lambda s: s and s.endswith('shot.jpg')):
+            return citem['src']
+
+    keep_only_tags = [
+        dict(name='h1'),
+        classes(
+            'tdb_single_subtitle tdb_single_date tdb_single_featured_image tdb_single_content'
+        ),
+    ]
+    remove_tags = [
+        dict(name='div', attrs={'style':'position:absolute;top:0;left:-9999px;'}),
+    ]
+
+    feeds = [
+        ('Courts', 'https://www.indialegallive.com/constitutional-law-news/courts-news/rss'),
+        ('Ring Side', 'https://www.indialegallive.com/ringside/rss'),
+        ('Cover Story Articles', 'https://www.indialegallive.com/cover-story-articles/rss'),
+        ('Special', 'https://www.indialegallive.com/special/rss'),
+        ('Columns', 'https://www.indialegallive.com/column-news/rss'),
+    ]
--- a/recipes/india_today.recipe
+++ b/recipes/india_today.recipe
@ -1,4 +1,4 @@
-from calibre.web.feeds.news import BasicNewsRecipe
+from calibre.web.feeds.news import BasicNewsRecipe, classes


 class IndiaToday(BasicNewsRecipe):
@ -7,14 +7,41 @@ class IndiaToday(BasicNewsRecipe):
    __author__ = 'Krittika Goyal'
    oldest_article = 15  # days
    max_articles_per_feed = 25
-
    no_stylesheets = True
-    auto_cleanup = True
+    use_embedded_content = False
+    remove_attributes = ['style']
+
+    keep_only_tags = [
+        dict(name='h1'),
+        classes('story-kicker story-right'),
+        dict(itemProp='articleBody'),
+    ]

    feeds = [
-        ('The Big Story', 'https://www.indiatoday.in/rss/1206614'),
+        ('Editor\'s Note','https://www.indiatoday.in/rss/1206516'),
        ('Cover Story', 'https://www.indiatoday.in/rss/1206509'),
+        ('The Big Story', 'https://www.indiatoday.in/rss/1206614'),
+        ('UP Front','https://www.indiatoday.in/rss/1206609'),
+        ('Liesure','https://www.indiatoday.in/rss/1206551'),
        ('Nation', 'https://www.indiatoday.in/rss/1206514'),
+        ('Health','https://www.indiatoday.in/rss/1206515'),
+        ('Defence','https://www.indiatoday.in/rss/1206517'),
+        ('Guest Column','https://www.indiatoday.in/rss/1206612'),
        ('States', 'https://www.indiatoday.in/rss/1206500'),
        ('Economy', 'https://www.indiatoday.in/rss/1206513'),
+        ('Special Report','https://www.indiatoday.in/rss/1206616'),
+        ('Investigation','https://www.indiatoday.in/rss/1206617'),
+        ('Diplomacy','https://www.indiatoday.in/rss/1206512'),
+        ('Sports','https://www.indiatoday.in/rss/1206518'),
    ]
+
+    def preprocess_raw_html(self, raw_html, url):
+        from calibre.ebooks.BeautifulSoup import BeautifulSoup
+        soup = BeautifulSoup(raw_html)
+        for script in soup.findAll('script'):
+            script.extract()
+        for style in soup.findAll('style'):
+            style.extract()
+        for img in soup.findAll('img', attrs={'data-src': True}):
+            img['src'] = img['data-src']
+        return str(soup)
--- a/recipes/indian_express.recipe
+++ b/recipes/indian_express.recipe
@ -17,38 +17,31 @@ class IndianExpress(BasicNewsRecipe):

    no_stylesheets = True
    use_embedded_content = False
+    remove_attributes = ['style','height','width']
+    ignore_duplicate_articles = {'url'}

    keep_only_tags = [
        classes('heading-part full-details')
    ]
    remove_tags = [
-        classes('share-social appstext story-tags')
+        dict(name='div', attrs={'id':'ie_story_comments'}),
+        dict(name='img', attrs={'src':'https://images.indianexpress.com/2021/06/explained-button-300-ie.jpeg'}),
+        dict(name='a', attrs={'href':'https://indianexpress.com/section/explained/?utm_source=newbanner'}),
+        dict(name='img', attrs={'src':'https://images.indianexpress.com/2021/06/opinion-button-300-ie.jpeg'}),
+        dict(name='a', attrs={'href':'https://indianexpress.com/section/opinion/?utm_source=newbanner'}),
+        classes('share-social appstext story-tags ie-int-campign-ad ie-breadcrumb custom_read_button unitimg copyright')
    ]
    feeds = [
-        ('Front Page',
-         'http://indianexpress.com/print/front-page/feed/'),
-        ('Editorials',
-         'http://indianexpress.com/section/opinion/editorials/feed/'),
-        ('Crime',
-         'http://indianexpress.com/section/india/crime/feed/'),
-        ('Cricket',
-         'http://indianexpress.com/section/sports/cricket/feed/'),
-        ('Health',
-         'http://www.indianexpress.com/lifestyle/health/feed/'),
-        ('Asia',
-         'http://indianexpress.com/section/world/asia/'),
-        ('Politics',
-         'http://indianexpress.com/section/india/politics/feed/'),
-        ('Mumbai',
-         'http://www.indianexpress.com/cities/mumbai/feed/'),
-        ('Op-Ed',
-         'http://indianexpress.com/section/opinion/feed/'),
-        ('Lifestyle',
-         'http://indianexpress.com/section/lifestyle/feed/'),
-        ('Science & Technology',
-         'http://indianexpress.com/section/technology/feed/'),
-        ('Bollywood',
-         'http://indianexpress.com/section/entertainment/bollywood/feed/'),
+        ('Front Page', 'https://indianexpress.com/print/front-page/feed/'),
+        ('Op-Ed', 'http://indianexpress.com/section/opinion/feed/'),
+        ('Science & Technology', 'http://indianexpress.com/section/technology/feed/'),
+        ('Movie Reviews', 'https://indianexpress.com/section/entertainment/movie-review/feed/'),
+        ('Sunday Eye', 'https://indianexpress.com/print/eye/feed/'),
+        ('Explained', 'https://indianexpress.com/section/explained/feed/'),
+        ('Delhi Confidential', 'https://indianexpress.com/section/delhi-confidential/feed'),
+        ('Economy', 'https://indianexpress.com/print/economy/feed'),
+        ('Express Network', 'https://indianexpress.com/print/express-network/'),
+        # Want to add more? go-to:https://indianexpress.com/syndication/
    ]

    def preprocess_html(self, soup):
--- a/recipes/irish_times.recipe
+++ b/recipes/irish_times.recipe
@ -51,26 +51,37 @@ def get_browser(self):
        # To understand the signin logic read signin javascript from submit button from
        # https://www.irishtimes.com/signin

+        br = BasicNewsRecipe.get_browser(self, user_agent='curl/7.80.0')
+        ip_data = json.loads(br.open('https://ipapi.co//json').read())
        br = BasicNewsRecipe.get_browser(self)
        url = 'https://www.irishtimes.com/signin'
        deviceid = str(uuid4()).replace('-', '')
        # Enable debug stuff?
        # br.set_debug_http(True)
        br.open(url).read()
+        from pprint import pprint
+        pprint(ip_data)
+        br.set_cookie('IT_country', ip_data['country_code'], '.irishtimes.com')
+        br.set_cookie('IT_eu', 'true' if ip_data['in_eu'] else 'false', '.irishtimes.com')
        rurl = 'https://www.irishtimes.com/auth-rest-api/v1/paywall/login'
        rq = Request(rurl, headers={
            'Accept': '*/*',
            'Accept-Language': 'en-US,en;q=0.5',
            'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8',
+            'Origin': 'https://www.irishtimes.com',
            'Referer': url,
            'X-Requested-With': 'XMLHttpRequest',
-        },  data=urlencode({'username': self.username, 'password': self.password,'deviceid':deviceid, 'persistent':'on'}))
+            'sec-fetch-site': 'same-origin',
+            'sec-fetch-dest': 'empty',
+            'sec-fetch-mode': 'cors',
+        },  data=urlencode({'username': self.username, 'password': self.password, 'deviceid':deviceid, 'persistent':'on', 'rid': ''}))

        r = br.open(rq)
        raw = r.read()
        data = json.loads(raw)
        # print(data)
        if r.code != 200 or b'user_id' not in raw:
+            pprint(data)
            raise ValueError('Failed to log in check username/password')

        # Set cookie
--- a/recipes/japan_times.recipe
+++ b/recipes/japan_times.recipe
@ -1,58 +1,69 @@
-__license__ = 'GPL v3'
-__copyright__ = '2008-2013, Darko Miletic <darko.miletic at gmail.com>'
-'''
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+
+__license__ = "GPL v3"
+__copyright__ = (
+    "2008-2013, Darko Miletic <darko.miletic at gmail.com>. "
+    "2022, Albert Aparicio Isarn <aaparicio at posteo.net>"
+)
+"""
 japantimes.co.jp
-'''
+"""

 from calibre.web.feeds.news import BasicNewsRecipe


-def classes(classes):
-    q = frozenset(classes.split(' '))
-    return dict(attrs={
-        'class': lambda x: x and frozenset(x.split()).intersection(q)})
-
-
 class JapanTimes(BasicNewsRecipe):
-    title = 'The Japan Times'
-    __author__ = 'Darko Miletic'
-    description = "Daily news and features on Japan from the most widely read English-language newspaper in Japan. Coverage includes national news, business news, sports news, commentary and features on living in Japan, entertainment, the arts, education and more."  # noqa
-    language = 'en_JP'
-    category = 'news, politics, japan'
-    publisher = 'The Japan Times'
+    title = "The Japan Times"
+    __author__ = "Albert Aparicio Isarn (original recipe by Darko Miletic)"
+    description = (
+        "The latest news from Japan Times, Japan's leading English-language daily newspaper"
+    )
+    language = "en_JP"
+    category = "news, politics, japan"
+    publisher = "The Japan Times"
    oldest_article = 2
    max_articles_per_feed = 150
    no_stylesheets = True
+    remove_javascript = True
    use_embedded_content = False
-    encoding = 'utf8'
-    publication_type = 'newspaper'
-    extra_css = 'body{font-family: Geneva,Arial,Helvetica,sans-serif}'
+    encoding = "utf8"
+    publication_type = "newspaper"
+    masthead_url = "https://cdn-japantimes.com/wp-content/themes/jt_theme/library/img/japantimes-logo-tagline.png"
+    extra_css = "body{font-family: Geneva,Arial,Helvetica,sans-serif}"

    conversion_options = {
-        'comment': description, 'tags': category, 'publisher': publisher, 'language': language
+        "comment": description,
+        "tags": category,
+        "publisher": publisher,
+        "language": language,
    }

-    remove_tags_after = dict(name='div', attrs={'class': 'entry'}),
-    keep_only_tags = [dict(name='div', attrs={'class': 'padding_block'})]
+    remove_tags_before = {"name": "h1"}
+    remove_tags_after = {"name": "ul", "attrs": {"class": "single-sns-area"}}
+    keep_only_tags = [
+        {"name": "div", "attrs": {"class": "padding_block"}},
+        # {"name": "h5", "attrs": {"class": "writer", "role": "author"}},
+        # {"name": "p", "attrs": {"class": "credit"}},
+    ]
    remove_tags = [
-        dict(name=['iframe', 'embed', 'object', 'base', 'form']), dict(attrs={'class': [
-            'meta_extras', 'related_articles']}), dict(attrs={'id': 'content_footer_menu'}),
-        dict(id='no_js_blocker'),
-        classes('single-sns-area jt-related-stories'),
+        {"name": "div", "id": "no_js_blocker", "attrs": {"class": "padding_block"}},
+        {"name": "div", "attrs": {"class": "single-upper-meta"}},
+        {"name": "ul", "attrs": {"class": "single-sns-area"}},
    ]
    feeds = [
-
-    (u'News', u'http://www.japantimes.co.jp/news/feed/'),
-    (u'Opinion', u'http://www.japantimes.co.jp/opinion/feed/'),
-    (u'Life', u'http://www.japantimes.co.jp/opinion/feed/'),
-    (u'Community', u'http://www.japantimes.co.jp/community/feed/'),
-    (u'Culture', u'http://www.japantimes.co.jp/culture/feed/'),
-    (u'Sports', u'http://www.japantimes.co.jp/sports/feed/')
+        (u"Top Stories", u"https://www.japantimes.co.jp/feed/topstories/"),
+        (u"News", u"https://www.japantimes.co.jp/news/feed/"),
+        (u"Opinion", u"https://www.japantimes.co.jp/opinion/feed/"),
+        (u"Life", u"https://www.japantimes.co.jp/life/feed/"),
+        (u"Community", u"https://www.japantimes.co.jp/community/feed/"),
+        (u"Culture", u"https://www.japantimes.co.jp/culture/feed/"),
+        (u"Sports", u"https://www.japantimes.co.jp/sports/feed/"),
    ]

    def get_article_url(self, article):
        rurl = BasicNewsRecipe.get_article_url(self, article)
-        return rurl.partition('?')[0]
+        return rurl.partition("?")[0]

    def preprocess_raw_html(self, raw, url):
-        return '<html><head>' + raw[raw.find('</head>'):]
+        return "<html><head>" + raw[raw.find("</head>") :]
--- a/recipes/johm.recipe
+++ b/recipes/johm.recipe
@ -17,10 +17,10 @@ class JournalofHospitalMedicine(BasicNewsRecipe):

    def get_browser(self):
        br = BasicNewsRecipe.get_browser(self)
-        br.open('http://www3.interscience.wiley.com/cgi-bin/home')
+        br.open('https://onlinelibrary.wiley.com/')
        br.select_form(nr=0)
-        br['j_username'] = self.username
-        br['j_password'] = self.password
+        br['login'] = self.username
+        br['password'] = self.password
        response = br.submit()
        raw = response.read()
        if '<h2>LOGGED IN</h2>' not in raw:
@ -29,7 +29,7 @@ def get_browser(self):

    # TO GET ARTICLE TOC
    def johm_get_index(self):
-        return self.index_to_soup('http://onlinelibrary.wiley.com/journal/10.1002/(ISSN)1553-5606/currentissue')
+        return self.index_to_soup('https://shmpublications.onlinelibrary.wiley.com/toc/15535606/current')

    # To parse artice toc
    def parse_index(self):
--- a/recipes/lenta_ru.recipe
+++ b/recipes/lenta_ru.recipe
@ -1,181 +1,17 @@
 #!/usr/bin/env python
-
-'''
-Lenta.ru
-'''
-
-from calibre.web.feeds.feedparser import parse
-from calibre.ebooks.BeautifulSoup import Tag
+# vim:fileencoding=utf-8
+from __future__ import unicode_literals
 from calibre.web.feeds.news import BasicNewsRecipe
-import re


-def new_tag(soup, name, attrs=()):
-    impl = getattr(soup, 'new_tag', None)
-    if impl is not None:
-        return impl(name, attrs=dict(attrs))
-    return Tag(soup, name, attrs=attrs or None)
-
-
-class LentaRURecipe(BasicNewsRecipe):
-    title = u'Lenta.ru: \u041d\u043e\u0432\u043e\u0441\u0442\u0438'
-    __author__ = 'Nikolai Kotchetkov'
-    publisher = 'lenta.ru'
-    category = 'news, Russia'
-    description = u'''\u0415\u0436\u0435\u0434\u043d\u0435\u0432\u043d\u0430\u044f
-        \u0438\u043d\u0442\u0435\u0440\u043d\u0435\u0442-\u0433\u0430\u0437\u0435\u0442\u0430.
-        \u041d\u043e\u0432\u043e\u0441\u0442\u0438 \u0441\u043e
-        \u0432\u0441\u0435\u0433\u043e \u043c\u0438\u0440\u0430 \u043d\u0430
-        \u0440\u0443\u0441\u0441\u043a\u043e\u043c
-        \u044f\u0437\u044b\u043a\u0435'''
-    description = u'Ежедневная интернет-газета. Новости со всего мира на русском языке'
-    oldest_article = 3
+class AdvancedUserRecipe1645563203(BasicNewsRecipe):
+    title          = 'Lenta.ru - Новости'
+    __author__ = 'MrBeef12'
+    oldest_article = 7
    max_articles_per_feed = 100
-
-    masthead_url = u'http://img.lenta.ru/i/logowrambler.gif'
-    cover_url = u'http://img.lenta.ru/i/logowrambler.gif'
-
-    # Add feed names if you want them to be sorted (feeds of this list appear
-    # first)
-    sortOrder = [u'_default', u'В России', u'б.СССР', u'В мире']
-
-    encoding = 'cp1251'
+    auto_cleanup   = True
    language = 'ru'
-    no_stylesheets = True
-    remove_javascript = True
-    recursions = 0

-    conversion_options = {
-        'comment': description, 'tags': category, 'publisher': publisher, 'language': language
-    }
-
-    keep_only_tags = [dict(name='td', attrs={'class': ['statya', 'content']})]
-
-    remove_tags_after = [dict(name='p', attrs={'class': 'links'}), dict(
-        name='div', attrs={'id': 'readers-block'})]
-
-    remove_tags = [dict(name='table', attrs={'class': ['vrezka', 'content']}), dict(name='div', attrs={
-        'class': 'b240'}), dict(name='div', attrs={'id': 'readers-block'}), dict(name='p', attrs={'class': 'links'})]
-
-    feeds = [u'http://lenta.ru/rss/']
-
-    extra_css = 'h1 {font-size: 1.2em; margin: 0em 0em 0em 0em;} h2 {font-size: 1.0em; margin: 0em 0em 0em 0em;} h3 {font-size: 0.8em; margin: 0em 0em 0em 0em;}'  # noqa
-
-    def parse_index(self):
-        try:
-            feedData = parse(self.feeds[0])
-            if not feedData:
-                raise NotImplementedError
-            self.log("parse_index: Feed loaded successfully.")
-
-            def get_virtual_feed_articles(feed):
-                if feed in feeds:
-                    return feeds[feed][1]
-                self.log("Adding new feed: ", feed)
-                articles = []
-                feeds[feed] = (feed, articles)
-                return articles
-
-            feeds = {}
-
-            # Iterate feed items and distribute articles using tags
-            for item in feedData.entries:
-                link = item.get('link', '')
-                title = item.get('title', '')
-                if '' == link or '' == title:
-                    continue
-                article = {'title': title, 'url': link, 'description': item.get(
-                    'description', ''), 'date': item.get('date', ''), 'content': ''}
-                if not item.get('tags'):
-                    get_virtual_feed_articles('_default').append(article)
-                    continue
-                for tag in item.tags:
-                    addedToDefault = False
-                    term = tag.get('term', '')
-                    if '' == term:
-                        if (not addedToDefault):
-                            get_virtual_feed_articles(
-                                '_default').append(article)
-                        continue
-                    get_virtual_feed_articles(term).append(article)
-
-            # Get feed list
-            # Select sorted feeds first of all
-            result = []
-            for feedName in self.sortOrder:
-                if (not feeds.get(feedName)):
-                    continue
-                result.append(feeds[feedName])
-                del feeds[feedName]
-            result = result + feeds.values()
-
-            return result
-
-        except Exception as err:
-            self.log(err)
-            raise NotImplementedError
-
-    def preprocess_html(self, soup):
-        return self.adeify_images(soup)
-
-    def postprocess_html(self, soup, first_fetch):
-
-        contents = new_tag(soup, 'div')
-
-        # Extract tags with given attributes
-        extractElements = {'div': [{'id': 'readers-block'}]}
-
-        # Remove all elements that were not extracted before
-        for tag, attrs in extractElements.items():
-            for attr in attrs:
-                garbage = soup.findAll(tag, attr)
-                if garbage:
-                    for pieceOfGarbage in garbage:
-                        pieceOfGarbage.extract()
-
-        # Find article text using header
-        # and add all elements to contents
-        element = soup.find({'h1': True, 'h2': True})
-        if (element):
-            element.name = 'h1'
-        while element:
-            nextElement = element.nextSibling
-            element.extract()
-            contents.insert(len(contents.contents), element)
-            element = nextElement
-
-        # Place article date after header
-        dates = soup.findAll(text=re.compile(
-            r'\d{2}\.\d{2}\.\d{4}, \d{2}:\d{2}:\d{2}'))
-        if dates:
-            for date in dates:
-                for string in date:
-                    parent = date.parent
-                    if (parent and isinstance(parent, Tag) and 'div' == parent.name and 'dt' == ''.join(parent['class'])):
-                        # Date div found
-                        parent.extract()
-                        parent[
-                            'style'] = 'font-size: 0.5em; color: gray; font-family: monospace;'
-                        contents.insert(1, parent)
-                        break
-
-        # Place article picture after date
-        pic = soup.find('img')
-        if pic:
-            picDiv = new_tag(soup, 'div')
-            picDiv['style'] = 'width: 100%; text-align: center;'
-            pic.extract()
-            picDiv.insert(0, pic)
-            title = pic.get('title', None)
-            if title:
-                titleDiv = new_tag(soup, 'div')
-                titleDiv['style'] = 'font-size: 0.5em;'
-                titleDiv.insert(0, title)
-                picDiv.insert(1, titleDiv)
-            contents.insert(2, picDiv)
-
-        body = soup.find('td', {'class': ['statya', 'content']})
-        if body:
-            body.replaceWith(contents)
-
-        return soup
+    feeds          = [
+        ('Lenta.ru - Новости России и мира сегодня', 'https://lenta.ru/rss/'),
+    ]
--- a/recipes/livemint.recipe
+++ b/recipes/livemint.recipe
@ -1,36 +1,81 @@
-from calibre.web.feeds.news import BasicNewsRecipe
+#!/usr/bin/env  python
+
+from calibre.web.feeds.news import BasicNewsRecipe, classes
+from datetime import date
+
+is_saturday = date.today().weekday() == 5


 class LiveMint(BasicNewsRecipe):
    title = u'Live Mint'
+    description = 'Financial News from India.'
    language = 'en_IN'
    __author__ = 'Krittika Goyal'
    oldest_article = 1  # days
    max_articles_per_feed = 50
+    encoding = 'utf-8'
    use_embedded_content = False
-
    no_stylesheets = True
-    auto_cleanup = True
+    remove_attributes = ['style', 'height', 'width']

-    feeds = [
-        ('News',
-         'http://www.livemint.com/rss/news'),
-        ('Technology',
-         'http://www.livemint.com/rss/technology'),
-        ('Companies',
-         'http://www.livemint.com/rss/companies'),
-        ('Consumer',
-         'http://www.livemint.com/rss/consumer'),
-        ('Opinion',
-         'http://www.livemint.com/rss/opinion'),
-        ('Money',
-         'http://www.livemint.com/rss/money'),
-        ('Industry',
-         'http://www.livemint.com/rss/industry'),
-        ('Economy Politics',
-         'http://www.livemint.com/rss/economy_politics'),
-        ('Lounge',
-         'http://www.livemint.com/rss/lounge'),
-        ('Sports',
-         'http://www.livemint.com/rss/sports'),
-    ]
+    if is_saturday:
+        keep_only_tags = [
+            dict(name='h1'),
+            dict(name='h2', attrs={'id':'story-summary-0'}),
+            dict(name='picture'),
+            dict(name='div', attrs={'class':'innerBanCaption'}),
+            dict(name='div', attrs={'id':'date-display-before-content'}),
+            dict(name='div', attrs={'class':'storyContent'}),
+        ]
+        remove_tags = [
+            classes(
+                'sidebarAdv similarStoriesClass moreFromSecClass'
+            )
+        ]
+        feeds = [
+            ('News', 'https://lifestyle.livemint.com/rss/news'),
+            ('Food','https://lifestyle.livemint.com/rss/food'),
+            ('Fashion','https://lifestyle.livemint.com/rss/fashion'),
+            ('How to Lounge','https://lifestyle.livemint.com/rss/how-to-lounge'),
+            ('Smart Living','https://lifestyle.livemint.com/rss/smart-living'),
+        ]
+    else:
+        keep_only_tags = [
+            dict(name='h1'),
+            dict(name='picture'),
+            dict(name='figcaption'),
+            classes('articleInfo FirstEle summary highlights paywall'),
+        ]
+        remove_tags = [
+            classes(
+                'trendingSimilarHeight moreNews mobAppDownload label msgError msgOk'
+            )
+        ]
+
+        feeds = [
+            ('Companies', 'https://www.livemint.com/rss/companies'),
+            ('Opinion', 'https://www.livemint.com/rss/opinion'),
+            ('Money', 'https://www.livemint.com/rss/money'),
+            ('Economy', 'https://www.livemint.com/rss/economy/'),
+            ('Politics', 'https://www.livemint.com/rss/politics'),
+            ('Science', 'https://www.livemint.com/rss/science'),
+            ('Industry', 'https://www.livemint.com/rss/industry'),
+            ('Education', 'https://www.livemint.com/rss/education'),
+            ('Sports', 'https://www.livemint.com/rss/sports'),
+            ('Technology', 'https://www.livemint.com/rss/technology'),
+            ('News', 'https://www.livemint.com/rss/news'),
+            ('Mutual Funds', 'https://www.livemint.com/rss/Mutual Funds'),
+            ('Markets', 'https://www.livemint.com/rss/markets'),
+            ('AI', 'https://www.livemint.com/rss/AI'),
+            ('Insurance', 'https://www.livemint.com/rss/insurance'),
+            ('Budget', 'https://www.livemint.com/rss/budget'),
+            ('Elections', 'https://www.livemint.com/rss/elections'),
+        ]
+
+    def preprocess_html(self, soup):
+        for img in soup.findAll('img', attrs={'data-src': True}):
+            img['src'] = img['data-src']
+        if is_saturday:
+            for img in soup.findAll('img', attrs={'data-img': True}):
+                img['src'] = img['data-img']
+        return soup
--- a/recipes/mainichi_en.recipe
+++ b/recipes/mainichi_en.recipe
@ -1,64 +1,142 @@
-__license__ = 'GPL v3'
-__copyright__ = '2010, Hiroshi Miura <miurahr@linux.com>'
-'''
-www.mainichi.jp
-'''
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+
+__license__ = "GPL v3"
+__copyright__ = (
+    "2010, Hiroshi Miura <miurahr@linux.com>. "
+    "2021, Albert Aparicio Isarn <aaparicio at posteo.net>"
+)
+
+"""
+www.mainichi.jp/english
+"""
+
+from datetime import datetime

-import re
 from calibre.web.feeds.news import BasicNewsRecipe


 class MainichiEnglishNews(BasicNewsRecipe):
-    title = u'The Mainichi'
-    __author__ = 'Hiroshi Miura'
+    title = u"The Mainichi"
+    __author__ = "Albert Aparicio Isarn (old version by Hiroshi Miura)"
+
+    description = "Japanese traditional newspaper Mainichi news in English"
+    publisher = "Mainichi News"
+    publication_type = "newspaper"
+    category = "news, japan"
+    language = "en_JP"
+
+    index = "http://mainichi.jp/english/"
+    masthead_url = index + "images/themainichi.png"
+
    oldest_article = 2
    max_articles_per_feed = 40
-    description = 'Japanese traditional newspaper Mainichi news in English'
-    publisher = 'Mainichi News'
-    category = 'news, japan'
-    language = 'en_JP'
-    index = 'http://mainichi.jp/english/english/index.html'
+    no_stylesheets = True
    remove_javascript = True
-    masthead_url = 'http://mainichi.jp/english/images/themainichi.png'

-    remove_tags_before = {'class': "NewsTitle"}
-    remove_tags_after = {'class': "NewsBody clr"}
+    remove_tags_before = {"id": "main-cont"}
+    remove_tags_after = {"class": "main-text"}
+    remove_tags = [{"name": "div", "id": "tools"}, {"name": "div", "class": "sub"}]

-    def parse_feeds(self):
+    def get_pickup_section(self, soup):
+        # Topmost story
+        top = soup.find("section", attrs={"class": "pickup section"})
+        top_link = top.find("p", attrs={"class": "midashi"}).find("a")

-        feeds = BasicNewsRecipe.parse_feeds(self)
+        try:
+            top_date = (
+                soup.find("div", attrs={"id": "main"})
+                .find("div", attrs={"class": "date-box"})
+                .find("p", attrs={"class": "date"})
+                .string
+            )

-        for curfeed in feeds:
-            delList = []
-            for a, curarticle in enumerate(curfeed.articles):
-                if re.search(r'pheedo.jp', curarticle.url):
-                    delList.append(curarticle)
-                if re.search(r'rssad.jp', curarticle.url):
-                    delList.append(curarticle)
-            if len(delList) > 0:
-                for d in delList:
-                    index = curfeed.articles.index(d)
-                    curfeed.articles[index:index + 1] = []
+            top_date_formatted = datetime.strptime(top_date, "%A, %B %d, %Y").strftime("%Y/%m/%d")
+        except AttributeError:
+            # If date not present, assume it is from today
+            top_date_formatted = datetime.now().strftime("%Y/%m/%d")

-        return feeds
+        top_description = top.find("p", attrs={"class": "txt"}).text
+
+        return [
+            {
+                "title": top_link.string,
+                "date": top_date_formatted,
+                "url": "https:" + top_link["href"],
+                "description": top_description,
+            }
+        ]
+
+    def retrieve_news_from_column(self, column):
+        column_news = []
+
+        for item in column.findAll("li"):
+            if item:
+                itema = item.find("a")
+                date_item = itema.find("p", attrs={"class": "date"})
+
+                column_news.append(
+                    {
+                        "title": itema.find("span").string,
+                        "date": date_item.string.strip("（）") if date_item else "",
+                        "url": "https:" + itema["href"],
+                        "description": "",
+                    }
+                )
+
+        return column_news
+
+    def get_top_stories(self, soup):
+        top_stories = self.get_pickup_section(soup)
+
+        news_section = soup.find("section", attrs={"class": "newslist"})
+        top_news = news_section.find("div", attrs={"class": "main-box"}).find("ul")
+
+        top_stories.extend(self.retrieve_news_from_column(top_news))
+
+        return top_stories
+
+    def get_editor_picks(self, soup):
+        editor_picks = []
+
+        news_section = soup.find("section", attrs={"class": "newslist"})
+        news = news_section.find("div", attrs={"class": "sub-box"}).find("ul")
+
+        editor_picks.extend(self.retrieve_news_from_column(news))
+
+        return editor_picks
+
+    def get_section(self, section):
+        soup = self.index_to_soup(self.index + section + "index.html")
+
+        section_news_items = self.get_pickup_section(soup)
+
+        news_columns = (
+            soup.find("section", attrs={"class": "newslist section"})
+            .find("div", attrs={"class": "col-set"})
+            .find("ul")
+        )
+
+        section_news_items.extend(self.retrieve_news_from_column(news_columns))
+
+        return section_news_items

    def parse_index(self):
-        feeds = []
-        soup = self.index_to_soup(self.index)
-        for section in soup.findAll('section'):
-            newsarticles = []
-            section_name = 'news'
-            hds = section.find('div', attrs={'class': 'CategoryHead clr'})
-            if hds:
-                section_item = hds.find('h1')
-                if section_item:
-                    section_name = section_item.find('a').string
-                items = section.find('ul', attrs={'class': 'MaiLink'})
-                for item in items.findAll('li'):
-                    if item:
-                        itema = item.find('a')
-                        newsarticles.append({
-                            'title': itema.string, 'date': '', 'url': itema['href'], 'description': ''
-                        })
-                feeds.append((section_name, newsarticles))
+        soup = self.index_to_soup(self.index + "index.html")
+
+        feeds = [
+            ("Top Stories", self.get_top_stories(soup)),
+            ("Editor's Picks", self.get_editor_picks(soup)),
+            # ("Latest Articles", self.get_section(self.index + "latest"+"index.html")),
+            ("Japan", self.get_section("japan")),
+            ("World", self.get_section("world")),
+            ("Business", self.get_section("business")),
+            ("Sports", self.get_section("sports")),
+            ("Science", self.get_section("science")),
+            ("Entertainment", self.get_section("entertainment")),
+            ("Opinion", self.get_section("opinion")),
+            ("Lifestyle", self.get_section("lifestyle")),
+            ("Obituaries", self.get_section("obituaries")),
+        ]
+
        return feeds
--- a/recipes/mediapart.recipe
+++ b/recipes/mediapart.recipe
@ -45,9 +45,12 @@ class Mediapart(BasicNewsRecipe):
    keep_only_tags = [
        dict(name='h1'),
        dict(name='div', **classes('author')),
-        classes('introduction content-article')
+        classes('news__heading__top__intro news__body__center__article')
+    ]
+    remove_tags = [
+        classes('login-subscribe print-source_url'),
+        dict(name='svg'),
    ]
-    remove_tags = [classes('login-subscribe print-source_url')]
    conversion_options = {'smarten_punctuation': True}

    masthead_url = "https://raw.githubusercontent.com/lhoupert/calibre_contrib/main/mediapart_masthead.png"
--- a/recipes/mit_technology_review.recipe
+++ b/recipes/mit_technology_review.recipe
@ -7,8 +7,7 @@
 '''
 technologyreview.com
 '''
-from calibre.web.feeds.news import BasicNewsRecipe
-import re
+from calibre.web.feeds.news import BasicNewsRecipe, prefixed_classes


 def absurl(x):
@ -38,26 +37,14 @@ class MitTechnologyReview(BasicNewsRecipe):
    tags = 'news, technology, science'
    no_stylesheets = True

-    """
-    regex for class names
-    """
-    articleHeaderRegex= '^.*contentHeader__wrapper.*$'
-    editorLetterHeaderRegex = "^.*contentHeader--vertical__wrapper.*$"
-    articleContentRegex = "^.*contentbody__wrapper.*$"
-    imagePlaceHolderRegex = "^.*image__placeholder.*$"
-    advertisementRegex = "^.*sliderAd__wrapper.*$"
-
    keep_only_tags = [
-        dict(name='header',  attrs={'class': re.compile(editorLetterHeaderRegex, re.IGNORECASE)}),
-        dict(name='header',  attrs={'class': re.compile(articleHeaderRegex, re.IGNORECASE)}),
-        dict(name='div',  attrs={'class': re.compile(articleContentRegex, re.IGNORECASE)})
+        prefixed_classes('contentHeader contentArticleHeader contentBody')
    ]
    remove_tags = [
        dict(name="aside"),
        dict(name="svg"),
        dict(name="blockquote"),
-        dict(name="img", attrs={'class': re.compile(imagePlaceHolderRegex, re.IGNORECASE)}),
-        dict(name="div", attrs={'class': re.compile(advertisementRegex, re.IGNORECASE)}),
+        prefixed_classes('image__placeholder sliderAd__wrapper'),
    ]

    def parse_index(self):
--- a/recipes/mmc_rtv.recipe
+++ b/recipes/mmc_rtv.recipe
@ -1,58 +1,48 @@
-__license__ = 'GPL v3'
-__copyright__ = '2010, BlonG'
-'''
-www.rtvslo.si
-'''
+#!/usr/bin/env python
+# News source: https://www.rtvslo.si
+# License: GPLv3
+# Copyright: 2022, TadejS
+
+
 from calibre.web.feeds.news import BasicNewsRecipe


 class MMCRTV(BasicNewsRecipe):
    title = u'MMC RTV Slovenija'
-    __author__ = u'BlonG'
+    __author__ = u'TadejS'
    description = u"Prvi interaktivni multimedijski portal, MMC RTV Slovenija"
    oldest_article = 3
-    max_articles_per_feed = 20
+    max_articles_per_feed = 100
    language = 'sl'
    no_stylesheets = True
    use_embedded_content = False
+    encoding = 'utf-8'
+    publication_type = 'newspaper'

-    cover_url = 'https://sites.google.com/site/javno2010/home/rtv_slo_cover.jpg'
-
-    extra_css = '''
-            h1{font-family:Arial,Helvetica,sans-serif; font-weight:bold;font-size:large;}
-            h2{font-family:Arial,Helvetica,sans-serif; font-weight:normal;font-size:small;}
-            p{font-family:Arial,Helvetica,sans-serif;font-size:small;}
-            body{font-family:Helvetica,Arial,sans-serif;font-size:small;}
-            '''
-
-    def print_version(self, url):
-        split_url = url.split("/")
-        print_url = 'http://www.rtvslo.si/index.php?c_mod=news&op=print&id=' + \
-            split_url[-1]
-        return print_url
+    cover_url = 'https://img.rtvslo.si/_static/novi/logo/tvmmc-light-bg.png'

    keep_only_tags = [
-        dict(name='div', attrs={'class': 'title'}),
-        dict(name='div', attrs={'id': 'newsbody'}),
-        dict(name='div', attrs={'id': 'newsblocks'}),
+        dict(name='header', attrs={'class': 'article-header'}),
+        dict(name='div', attrs={'class': 'article-body'}),
+    ]
+    remove_tags=[
+        dict(name='div', attrs={'class':'gallery-grid'}),
+        dict(name='div', attrs={'class':'gallery'}),
+        dict(name='div', attrs={'class':'exposed-article'}),
+        dict(name='div', attrs={'class':'d-lg-none'}),
+        dict(name='div', attrs={'class':'section-heading'}),
    ]
-#    remove_tags=[
-# 40            dict(name='div', attrs={'id':'newsblocks'}),
-#            ]

    feeds = [
-        (u'Slovenija', u'http://www.rtvslo.si/feeds/01.xml'),
-        (u'Svet', u'http://www.rtvslo.si/feeds/02.xml'),
-        (u'Evropska unija', u'http://www.rtvslo.si/feeds/16.xml'),
-        (u'Gospodarstvo', u'http://www.rtvslo.si/feeds/04.xml'),
-        (u'\u010crna kronika', u'http://www.rtvslo.si/feeds/08.xml'),
-        (u'Okolje', u'http://www.rtvslo.si/feeds/12.xml'),
-        (u'Znanost in tehnologija', u'http://www.rtvslo.si/feeds/09.xml'),
-        (u'Zabava', u'http://www.rtvslo.si/feeds/06.xml'),
-        (u'Ture avanture', u'http://www.rtvslo.si/feeds/28.xml'),
+        (u'Slovenija', u'https://www.rtvslo.si/feeds/01.xml'),
+        (u'Evropska unija', u'https://www.rtvslo.si/feeds/16.xml'),
+        (u'Svet', u'https://www.rtvslo.si/feeds/02.xml'),
+        (u'Gospodarstvo', u'https://www.rtvslo.si/feeds/04.xml'),
+        (u'Okolje', u'https://www.rtvslo.si/feeds/12.xml'),
+        (u'Znanost in tehnologija', u'https://www.rtvslo.si/feeds/09.xml'),
+        (u'Kultura', u'https://www.rtvslo.si/feeds/05.xml'),
+        (u'Šport', u'https://www.rtvslo.si/feeds/03.xml'),
+        (u'Zabava', u'https://www.rtvslo.si/feeds/06.xml'),
+        (u'Ture avanture', u'https://www.rtvslo.si/feeds/28.xml'),
+        (u'Črna kronika', u'https://www.rtvslo.si/feeds/08.xml'),
    ]
-
-#    def preprocess_html(self, soup):
-#            newsblocks = soup.find('div',attrs = ['id':'newsblocks'])
-#            soup.find('div', attrs = {'id':'newsbody'}).insert(-1, newsblocks)
-#            return soup
--- a/recipes/nytimes.recipe
+++ b/recipes/nytimes.recipe
@ -186,7 +186,8 @@ def parse_todays_page(self):
        soup = self.read_nyt_metadata()
        script = soup.findAll('script', text=lambda x: x and 'window.__preloadedData' in x)[0]
        script = type(u'')(script)
-        data = json.loads(script[script.find('{'):script.rfind(';')].strip().rstrip(';'))['initialState']
+        json_data = script[script.find('{'):script.rfind(';')].strip().rstrip(';')
+        data = json.loads(json_data.replace(':undefined', ':null'))['initialState']
        containers, sections = {}, {}
        article_map = {}
        gc_pat = re.compile(r'groupings.(\d+).containers.(\d+)')
--- a/recipes/nytimes_sub.recipe
+++ b/recipes/nytimes_sub.recipe
@ -186,7 +186,8 @@ def parse_todays_page(self):
        soup = self.read_nyt_metadata()
        script = soup.findAll('script', text=lambda x: x and 'window.__preloadedData' in x)[0]
        script = type(u'')(script)
-        data = json.loads(script[script.find('{'):script.rfind(';')].strip().rstrip(';'))['initialState']
+        json_data = script[script.find('{'):script.rfind(';')].strip().rstrip(';')
+        data = json.loads(json_data.replace(':undefined', ':null'))['initialState']
        containers, sections = {}, {}
        article_map = {}
        gc_pat = re.compile(r'groupings.(\d+).containers.(\d+)')
--- a/recipes/nytimesbook.recipe
+++ b/recipes/nytimesbook.recipe
@ -100,7 +100,7 @@ def process_image_block(lines, block, data):


 def json_to_html(raw):
-    data = json.loads(raw)
+    data = json.loads(raw.replace(':undefined', ':null'))
    data = data['initialState']
    article = next(iter(data.values()))
    body = data[article['sprinkledBody']['id']]
--- a/recipes/open_magazine.recipe
+++ b/recipes/open_magazine.recipe
@ -0,0 +1,48 @@
+#!/usr/bin/env  python
+
+from calibre.web.feeds.news import BasicNewsRecipe, classes
+
+
+class OpenMagazine(BasicNewsRecipe):
+    title = u'Open Magazine'
+    description = 'The weekly current affairs and features magazine.'
+    language = 'en_IN'
+    __author__ = 'unkn0wn'
+    oldest_article = 7  # days
+    max_articles_per_feed = 50
+    encoding = 'utf-8'
+    use_embedded_content = False
+    no_stylesheets = True
+    remove_attributes = ['style', 'height', 'width']
+    masthead_url = 'https://openthemagazine.com/wp-content/themes/open/images/logo.png'
+    ignore_duplicate_articles = {'url'}
+    extra_css = '[id^="caption-attachment"] {font-size: small;font-style: italic;}'
+
+    def get_cover_url(self):
+        soup = self.index_to_soup('https://openthemagazine.com/magazine/')
+        tag = soup.find(attrs={'class': 'mb-2 right-image'})
+        if tag:
+            self.cover_url = tag.find('img')['src']
+        return super().get_cover_url()
+
+    keep_only_tags = [
+        classes('post-data post-thumb post-meta post-excerp'),
+    ]
+
+    remove_tags = [
+        classes('top-social menu-social post-author blurb-social button-links'),
+    ]
+
+    remove_tags_after = [
+        classes('post-meta post-excerp'),
+    ]
+
+    feeds = [
+        ('Cover Story', 'https://openthemagazine.com/cover-story/feed/'),
+        ('Columns', 'https://openthemagazine.com/columns/feed'),
+        ('Essays', 'https://openthemagazine.com/essays/feed'),
+        ('Features', 'https://openthemagazine.com/features/feed'),
+        ('Books', 'https://openthemagazine.com/lounge/books/feed'),
+        ('Art & Culture', 'https://openthemagazine.com/art-culture/feed'),
+        ('Cinema', 'https://openthemagazine.com/cinema/feed'),
+    ]
--- a/recipes/popscience.recipe
+++ b/recipes/popscience.recipe
@ -23,18 +23,18 @@ class AdvancedUserRecipe1282101454(BasicNewsRecipe):
    ignore_duplicate_articles = {'url'}
    no_stylesheets = True
    keep_only_tags = [
-        classes('content-body article-header featured-img'),
+        classes('Article-header Article-excerpt Article-author Article-thumbnail Article-bodyText'),
    ]

    def parse_section_index(self, slug):
        soup = self.index_to_soup('https://www.popsci.com/{}/'.format(slug))
-        main = soup.find(**classes('main-module'))
-        for div in main.findAll(**classes('main-item')):
-            a = div.find('a', href=True, **classes('linkable'))
+        main = soup.find(**classes('main-content'))
+        for div in main.findAll(**classes('Post')):
+            a = div.find('a', href=True, **classes('Post-link'))
            url = a['href']
-            title = self.tag_to_string(a.find(**classes('title')))
+            title = self.tag_to_string(div.find(**classes('Post-title')))
            desc = ''
-            dek = a.find(**classes('dek'))
+            dek = div.find(**classes('Post-excerpt'))
            if dek is not None:
                desc = self.tag_to_string(dek)
            self.log(' ', title, url)
--- a/recipes/publico.recipe
+++ b/recipes/publico.recipe
@ -9,6 +9,7 @@
 '''

 from calibre.web.feeds.news import BasicNewsRecipe
+from polyglot.urllib import urlencode


 def classes(classes):
@ -29,6 +30,7 @@ class PublicoPT(BasicNewsRecipe):
    language = 'pt'
    remove_empty_feeds = True
    extra_css = ' body{font-family: Arial,Helvetica,sans-serif } img{margin-bottom: 0.4em} '
+    needs_subscription = True

    keep_only_tags = [
        dict(id='story-content story-header'.split()),
@ -53,6 +55,23 @@ class PublicoPT(BasicNewsRecipe):
        (u'Tecnologia', u'http://feeds.feedburner.com/PublicoTecnologia')
    ]

+    def get_browser(self):
+        br = BasicNewsRecipe.get_browser(self)
+        if self.username is not None and self.password is not None:
+            postdata = urlencode({
+                'username': self.username,
+                'password': self.password
+            })
+            br.open(
+                'https://www.publico.pt/api/user/login',
+                data=postdata,
+                timeout=self.timeout
+            )
+
+            br.set_handle_refresh(True)
+
+        return br
+
    def preprocess_html(self, soup):
        for img in soup.findAll('img', attrs={'data-media-viewer':True}):
            img['src'] = img['data-media-viewer']
--- a/recipes/readitlater.recipe
+++ b/recipes/readitlater.recipe
@ -1,17 +1,10 @@
 """
-Pocket Calibre Recipe v1.4
+Pocket Calibre Recipe v1.5
 """
 from calibre import strftime
 from calibre.web.feeds.news import BasicNewsRecipe
-from string import Template
 import json
 import operator
-import re
-import tempfile
-try:
-    from urllib.parse import urlencode
-except ImportError:
-    from urllib import urlencode
 try:
    from urllib.error import HTTPError, URLError
 except ImportError:
@ -39,7 +32,7 @@ class Pocket(BasicNewsRecipe):
    # Settings people change
    oldest_article = 7.0
    max_articles_per_feed = 50
-    minimum_articles = 10
+    minimum_articles = 1
    mark_as_read_after_dl = True  # Set this to False for testing
    sort_method = 'oldest'  # MUST be either 'oldest' or 'newest'
    # To filter by tag this needs to be a single tag in quotes; IE 'calibre'
@ -49,7 +42,7 @@ class Pocket(BasicNewsRecipe):
    no_stylesheets = True
    use_embedded_content = False
    needs_subscription = True
-    articles_are_obfuscated = True
+    articles_are_obfuscated = False
    apikey = '19eg0e47pbT32z4793Tf021k99Afl889'
    index_url = u'https://getpocket.com'
    read_api_url = index_url + u'/v3/get'
@ -118,7 +111,7 @@ def parse_index(self):
                'item_id':      pocket_article[0],
                'title':        pocket_article[1]['resolved_title'],
                'date':         pocket_article[1]['time_updated'],
-                'url':          u'{0}/a/read/{1}'.format(self.index_url, pocket_article[0]),
+                'url':          pocket_article[1]['resolved_url'],
                'real_url':     pocket_article[1]['resolved_url'],
                'description':  pocket_article[1]['excerpt'],
                'sort':         pocket_article[1]['sort_id']
@ -126,49 +119,6 @@ def parse_index(self):
        self.articles = sorted(self.articles, key=operator.itemgetter('sort'))
        return [("My Pocket Articles for {0}".format(strftime('[%I:%M %p]')), self.articles)]

-    def get_textview(self, url):
-        """
-        Since Pocket's v3 API they removed access to textview. They also
-         redesigned their page to make it much harder to scrape their textview.
-         We need to pull the article, retrieve the formcheck id, then use it
-         to querty for the json version
-        This function will break when pocket hates us
-        """
-        ajax_url = self.index_url + u'/a/x/getArticle.php'
-        soup = self.index_to_soup(url)
-        fc_tag = soup.find('script', text=re.compile("formCheck"))
-        fc_id = re.search(r"formCheck = \'([\d\w]+)\';", fc_tag).group(1)
-        article_id = url.split("/")[-1]
-        data = urlencode({'itemId': article_id, 'formCheck': fc_id})
-        try:
-            response = self.browser.open(ajax_url, data)
-        except HTTPError as e:
-            self.log.exception("unable to get textview {0}".format(e.info()))
-            raise e
-        return json.load(response)['article']
-
-    def get_obfuscated_article(self, url):
-        """
-        Our get_textview returns parsed json so prettify it to something well
-        parsed by calibre.
-        """
-        article = self.get_textview(url)
-        template = Template('<h1>$title</h1><div class="body">$body</div>')
-        with tempfile.NamedTemporaryFile(delete=False) as tf:
-            tmpbody = article['article']
-            for img in article['images']:
-                imgdiv = '<div id="RIL_IMG_{0}" class="RIL_IMG"></div>'.format(
-                    article['images'][img]['image_id'])
-                imgtag = r'<img src="{0}" \>'.format(
-                    article['images'][img]['src'])
-                tmpbody = tmpbody.replace(imgdiv, imgtag)
-
-            tf.write(template.safe_substitute(
-                title=article['title'],
-                body=tmpbody
-            ))
-        return tf.name
-
    def mark_as_read(self, mark_list):
        actions_list = []
        for article_id in mark_list:
--- a/recipes/respekt_magazine.recipe
+++ b/recipes/respekt_magazine.recipe
@ -12,14 +12,14 @@
 # This imports the version bundled with Calibre
 import lxml
 from lxml.builder import E
-respekt_url = 'http://www.respekt.cz'
+respekt_url = 'https://www.respekt.cz'


 class respektRecipe(BasicNewsRecipe):
    __author__  = 'Tomáš Hnyk'
    publisher = u'Respekt Publishing a. s.'
    description = u'Articles from the print edition'
-    title = u'Respekt Magazine – Print'
+    title = u'Respekt Magazine — Print'
    encoding = 'utf-8'
    language = 'cs'
    delay = 0.001
@ -74,10 +74,10 @@ def preprocess_raw_html(self, raw_html, url):
        return raw_html

    def parse_index(self):
-        raw1 = self.index_to_soup('http://www.respekt.cz/tydenik/', raw=True)
+        raw1 = self.index_to_soup('https://www.respekt.cz/tydenik/', raw=True)
        root1 = lxml.html.fromstring(raw1)
        current_edition_url = root1.xpath("//div[@class='heroissue']/a")[0].items()[0][1]
-        raw2 = self.index_to_soup('http://www.respekt.cz/' + current_edition_url, raw=True)
+        raw2 = self.index_to_soup('https://www.respekt.cz/' + current_edition_url, raw=True)
        root2 = lxml.html.fromstring(raw2)
        self.cover_url = root2.xpath("//i[contains(@class, 'heroissue-cover')]")[0].get("data-src")
        # Fetch date
--- a/recipes/reuters.recipe
+++ b/recipes/reuters.recipe
@ -4,17 +4,18 @@

 from __future__ import absolute_import, division, print_function, unicode_literals

-from calibre.web.feeds.news import BasicNewsRecipe, classes
+import json
+from calibre.web.feeds.news import BasicNewsRecipe

 country = 'us'
 country_defs = {
    'us': ('www.reuters.com', {
-        'Business': 'finance',
-        'Markets': 'finance/markets',
        'World': 'world',
-        'Politics': 'politics',
-        'Tech': 'news/technology',
-        'Wealth': 'finance/wealth',
+        'Business': 'business',
+        'Markets': 'markets',
+        'Tech': 'technology',
+        'Sports': 'lifestyle/sports',
+        'Wealth': 'markets/wealth',
    })
 }

@ -32,6 +33,25 @@ def matcher(x):
    return {'attrs': {'class': matcher}}


+def extract_article_list(raw):
+    if isinstance(raw, bytes):
+        raw = raw.decode('utf-8')
+    # open('/t/raw.html', 'w').write(raw)
+    idx = raw.index(';Fusion.globalContent={')
+    d = raw[idx:]
+    d = d[d.index('{'):]
+    data = json.JSONDecoder().raw_decode(d)[0]
+    # from pprint import pformat
+    # print(pformat(data), file=open('/t/raw.py', 'w'))
+    k = 'arcResult' if 'arcResult' in data else 'result'
+    for article in data[k]['articles']:
+        yield {'title': article['title'], 'description': article['description'], 'url': article['canonical_url']}
+
+
+# if __name__ == '__main__':
+#     print(list(extract_article_list(open('/t/World News _ Latest Top Stories _ Reuters.html').read())))
+
+
 class Reuters(BasicNewsRecipe):
    title = 'Reuters'
    description = 'News from all over'
@ -39,29 +59,32 @@ class Reuters(BasicNewsRecipe):
    language = 'en'

    keep_only_tags = [
-        prefixed_classes('ArticlePage-article-header ArticlePage-article-body'),
+        prefixed_classes('article-header__heading___ article-header__author___ article-body__content___'),
    ]
    remove_tags = [
-        prefixed_classes('ArticleBody-read-time-and-social Slideshow-expand-button- TwoColumnsLayout-footer-'),
+        prefixed_classes(
+            'context-widget__tabs___'
+            ' ArticleBody-read-time-and-social Slideshow-expand-button- TwoColumnsLayout-footer- RegistrationPrompt__container___'
+            ' SocialEmbed__inner___'
+        ),
        dict(name=['button', 'link']),
    ]
+    remove_attributes = ['style']
+    extra_css = '''
+    img { max-width: 100%; }
+    '''

    def preprocess_html(self, soup, *a):
-        meta = soup.find(attrs={'name': "sailthru.image.full"})
-        if meta is not None:
-            url = meta['content']
-            body = soup.find(**prefixed_classes('ArticlePage-article-body'))
-            if body is not None:
-                div = soup.new_tag('div')
-                div.append(soup.new_tag('img', src=url))
-                body.insert(0, div)
+        for noscript in soup.findAll('noscript'):
+            if noscript.findAll('img'):
+                noscript.name = 'div'
        return soup

    def parse_index(self):
        base, sections = country_defs[country]
        ans = []

-        for section_title in sorted(sections):
+        for section_title in sections:
            slug = sections[section_title]
            self.log(section_title)
            articles = list(self.parse_reuters_section(base, slug))
@ -73,15 +96,8 @@ def parse_index(self):

    def parse_reuters_section(self, base, slug):
        url = 'https://' + base + '/' + slug
-        try:
-            soup = self.index_to_soup(url)
-        except Exception:
-            self.log.error('Failed to load Reuters section:', url)
-            return
-        for div in soup.findAll(**classes('news-headline-list')):
-            h3 = div.find(**classes('story-title'))
-            a = h3.parent
-            title = self.tag_to_string(h3)
-            url = 'https://{}{}'.format(base, a['href'])
-            self.log('\t', title, url)
-            yield {'title': title, 'url': url}
+        raw = self.index_to_soup(url, raw=True)
+        for article in extract_article_list(raw):
+            article['url'] = 'https://{}{}'.format(base, article['url'])
+            yield article
+            self.log('\t', article['title'], article['url'])
--- a/recipes/rt.recipe
+++ b/recipes/rt.recipe
@ -0,0 +1,25 @@
+#!/usr/bin/env python
+# vim:fileencoding=utf-8
+from calibre.web.feeds.news import BasicNewsRecipe
+
+
+class RtRecipe(BasicNewsRecipe):
+    title          = 'RT на русском'
+    __author__ = 'Vuizur'
+    oldest_article = 7
+    max_articles_per_feed = 100
+    auto_cleanup   = False
+    language = 'ru'
+
+    remove_tags_before = dict(name='h1')
+    remove_tags_after = dict(name='a', attrs={'class':'tags-trends__link'})
+    remove_tags = [
+        dict(name='div', attrs={'class':'error-on-page'}),
+        dict(name='div', attrs={'class':'short-url'}),
+        dict(name='div', attrs={'class':'follows-channel'}),
+        dict(name='a', attrs={'class':'tags-trends__link'})
+    ]
+
+    feeds          = [
+        ('RT на русском', 'https://russian.rt.com/rss'),
+    ]
--- a/recipes/smith.recipe
+++ b/recipes/smith.recipe
@ -4,7 +4,7 @@
 CATEGORIES = {
    'smart-news': 'Smart News',
    'history': 'History',
-    'science': 'Science',
+    'science-nature': 'Science',
    'innovation': 'Innovation',
    'arts-culture': 'Arts & Culture',
    'travel': 'Travel',
@ -58,7 +58,7 @@ def parse_index(self):
        ans = []
        for slug, title in CATEGORIES.items():
            url = 'https://www.smithsonianmag.com/category/' + slug + '/'
-            self.log('Parsing section:', title)
+            self.log('Parsing section:', title, 'at:', url)
            articles = list(self.parse_section(url))
            if articles:
                ans.append((title, articles))
--- a/recipes/straitstimes.recipe
+++ b/recipes/straitstimes.recipe
@ -31,27 +31,24 @@ class StraitsTimes(BasicNewsRecipe):
        'comments': description, 'tags': category, 'language': language, 'publisher': publisher
    }
    keep_only_tags = [
-        classes('node-header node-subheadline group-byline-info group-updated-timestamp group-image-frame field-name-body')
+        classes('node-header node-subheadline group-byline-info group-updated-timestamp group-image-frame field-name-body article-content-rawhtml')
    ]
    remove_tags = [
-        classes('st_telegram_boilerplate'),
+        classes('st_telegram_boilerplate dropdown-menu ads'),
        dict(name='source'),
    ]

    feeds = [
-              (u'Top of the News' , u'http://www.straitstimes.com/print/top-of-the-news/rss.xml')
-              ,(u'World'           , u'http://www.straitstimes.com/print/world/rss.xml')
-              ,(u'Home'            , u'http://www.straitstimes.com/print/home/rss.xml')
-              ,(u'Business'        , u'http://www.straitstimes.com/print/business/rss.xml')
-              ,(u'Life'            , u'http://www.straitstimes.com/print/life/rss.xml')
-              ,(u'Science'         , u'http://www.straitstimes.com/print/science/rss.xml')
-              ,(u'Digital'         , u'http://www.straitstimes.com/print/digital/rss.xml')
-              ,(u'Insight'         , u'http://www.straitstimes.com/print/insight/rss.xml')
-              ,(u'Opinion'         , u'http://www.straitstimes.com/print/opinion/rss.xml')
-              ,(u'Forum'           , u'http://www.straitstimes.com/print/forum/rss.xml')
-              ,(u'Big Picture'     , u'http://www.straitstimes.com/print/big-picture/rss.xml')
-              ,(u'Community'       , u'http://www.straitstimes.com/print/community/rss.xml')
-              ,(u'Education'       , u'http://www.straitstimes.com/print/education/rss.xml')
+        (u'World'            , u'https://www.straitstimes.com/news/world/rss.xml')
+        ,(u'Business'        , u'https://www.straitstimes.com/news/business/rss.xml')
+        ,(u'Life'            , u'https://www.straitstimes.com/news/life/rss.xml')
+        ,(u'Tech'            , u'https://www.straitstimes.com/news/tech/rss.xml')
+        ,(u'Opinion'         , u'https://www.straitstimes.com/news/opinion/rss.xml')
+        ,(u'Life'            , u'https://www.straitstimes.com/news/life/rss.xml')
+        ,(u'Singapore'       , u'https://www.straitstimes.com/news/singapore/rss.xml')
+        ,(u'Asia'     	     , u'https://www.straitstimes.com/news/asia/rss.xml')
+        ,(u'Multimedia'      , u'https://www.straitstimes.com/news/multimedia/rss.xml')
+        ,(u'Sport'           , u'https://www.straitstimes.com/news/sport/rss.xml')
    ]

    def preprocess_html(self, soup):
--- a/recipes/swarajya.recipe
+++ b/recipes/swarajya.recipe
@ -0,0 +1,44 @@
+from calibre.web.feeds.news import BasicNewsRecipe, classes
+
+
+class SwarajyaMag(BasicNewsRecipe):
+    title = u'Swarajya Magazine'
+    __author__ = 'unkn0wn'
+    description = 'Swarajya - a big tent for liberal right of centre discourse that reaches out, engages and caters to the new India.'
+    language = 'en_IN'
+    no_stylesheets = True
+    remove_javascript = True
+    use_embedded_content = False
+    remove_attributes = ['height', 'width']
+    encoding = 'utf-8'
+
+    keep_only_tags = [
+        classes('_2PqtR _1sMRD ntw8h author-bio'),
+    ]
+
+    remove_tags = [
+        classes('_JscD _2r17a'),
+    ]
+
+    def preprocess_html(self, soup):
+        for img in soup.findAll('img', attrs={'data-src': True}):
+            img['src'] = img['data-src'].split('?')[0]
+        return soup
+
+    def parse_index(self):
+        soup = self.index_to_soup('https://swarajyamag.com/all-issues')
+        a = soup.find('a', href=lambda x: x and x.startswith('/issue/'))
+        url = a['href']
+        self.log('Downloading issue:', url)
+        self.cover_url = a.find('img', attrs={'data-src': True})['data-src']
+        soup = self.index_to_soup('https://swarajyamag.com' + url)
+        ans = []
+
+        for a in soup.findAll(**classes('_2eOQr')):
+            url = a['href']
+            if url.startswith('/'):
+                url = 'https://swarajyamag.com' + url
+            title = self.tag_to_string(a)
+            self.log(title, ' at ', url)
+            ans.append({'title': title, 'url': url})
+        return [('Articles', ans)]
--- a/recipes/the_federalist.recipe
+++ b/recipes/the_federalist.recipe
@ -16,29 +16,53 @@ class Federalist(BasicNewsRecipe):
    title = 'The Federalist'
    __author__ = 'Kovid Goyal'
    language = 'en'
-    oldest_article = 7
+    oldest_article = 10
    max_articles_per_feed = 100
    no_stylesheets = True
    encoding = 'utf-8'
    use_embedded_content = False
    remove_attributes = ['xmlns', 'lang', 'style', 'width', 'height']
-    extra_css = '''
-    .shortbio { margin: 1em; padding: 1em; font-style: italic }
-    '''
+
+    extra_css = """
+        .shortbio,.article-excerpt{font-style: italic}
+        .article-author-details,.article-author-description,.article-meta-author,.article-meta-date,.article-thumbnail-caption{font-size: small}
+    """

    keep_only_tags = [
-        classes('entry-header'),
-        classes('wp-post-image post-categories entry-content shortbio byline-month byline-standard alpha-byline'),
+        classes(
+            'title-lg article-thumbnail post-categories article-excerpt article-author-details'
+            ' article-meta-author article-meta-date article-content article-body shortbio entry-header'
+            ' byline-month byline-standard alpha-byline article-author-description article-author-details'),
    ]
    remove_tags = [
        dict(name=['meta', 'link']),
-        classes('auth-ad attachment-post-thumbnail attachment-author-bio'),
+        classes('auth-ad article-share article-tags attachment-post-thumbnail attachment-author-bio'),
    ]

    feeds = [
-        ('All', 'http://thefederalist.com/feed/'),
+        ('All', 'https://thefederalist.com/feed/'),
    ]

+    # def parse_index(self):
+    #     return [('Articles', [
+    #         {
+    #         'title': 'test',
+    #         'url': 'https://thefederalist.com/2022/03/09/propaganda-press-wield-bidens-russia-blame-game-to-gaslight-americans-about-expensive-gas/'},
+    #         {
+    #         'title': 'test2',
+    #         'url': 'https://thefederalist.com/2022/03/10/white-house-will-blame-anyone-but-biden-for-februarys-7-9-inflation-jump/',
+    #         }
+    #     ])]
+
+    def preprocess_raw_html_(self, raw_html, url):
+        soup = self.index_to_soup(raw_html)
+        # this website puts article-thumbnail images inside article-body in
+        # some articles and outside it in others, so we have to special case it
+        for ab in soup.findAll(**classes('article-body')):
+            for img in ab.findAll(**classes('article-thumbnail')):
+                del img['class']
+        return str(soup)
+
    def preprocess_html(self, soup):
        for img in soup.findAll('img', attrs={'data-lazy-src': True}):
            img['src'] = img['data-lazy-src']
@ -49,9 +73,3 @@ def preprocess_html(self, soup):
                img.extract()
            seen.add(src)
        return soup
-
-    # def parse_index(self):
-    #     return [('Articles', [
-    #         {'title':'img', 'url':'http://thefederalist.com/2018/05/09/venezuelas-economic-problems-caused-socialism-not-falling-oil-prices/'},
-    #         {'title':'xxx', 'url':'http://thefederalist.com/2018/05/04/fans-take-on-marvel-dc-and-the-comic-book-industrys-sjw-self-destruction/'},
-    #     ])]
--- a/resources/common-english-words.txt
+++ b/resources/common-english-words.txt
--- a/resources/default_tweaks.py
+++ b/resources/default_tweaks.py
@ -38,11 +38,9 @@
 use_series_auto_increment_tweak_when_importing = False

 #: Add separator after completing an author name
-# Should the completion separator be append
-# to the end of the completed text to
-# automatically begin a new completion operation
-# for authors.
-# Can be either True or False
+# Set this if the completion separator should be appended to the end of the
+# completed text to automatically begin a new completion operation for authors.
+# It can be either True or False
 authors_completer_append_separator = False

 #: Author sort name algorithm
@ -250,6 +248,8 @@
        # French
        'fra'  : (r'Le\s+', r'La\s+', r"L'", u'L´', u'L’', r'Les\s+', r'Un\s+', r'Une\s+',
                  r'Des\s+', r'De\s+La\s+', r'De\s+', r"D'", u'D´', u'L’'),
+        # Polish
+        'pol': (),
        # Italian
        'ita': ('Lo\\s+', 'Il\\s+', "L'", 'L´', 'La\\s+', 'Gli\\s+',
                'I\\s+', 'Le\\s+', 'Uno\\s+', 'Un\\s+', 'Una\\s+', "Un'",
@ -426,37 +426,6 @@
 # Example: locale_for_sorting = 'nb' -- sort using Norwegian rules.
 locale_for_sorting =  ''

-#: Number of columns for custom metadata in the edit metadata dialog
-# Set whether to use one or two columns for custom metadata when editing
-# metadata  one book at a time. If True, then the fields are laid out using two
-# columns. If False, one column is used.
-metadata_single_use_2_cols_for_custom_fields = True
-
-#: Order of custom column(s) in edit metadata
-# Controls the order that custom columns are listed in edit metadata single
-# and bulk. The columns listed in the tweak are displayed first and in the
-# order provided. Any columns not listed are displayed after the listed ones,
-# in alphabetical order. Do note that this tweak does not change the size of
-# the edit widgets. Putting comments widgets in this list may result in some
-# odd widget spacing when using two-column mode.
-# Enter a comma-separated list of custom field lookup names, as in
-# metadata_edit_custom_column_order = ['#genre', '#mytags', '#etc']
-metadata_edit_custom_column_order = []
-
-#: Edit metadata custom column label width and elision point
-# Set the width of custom column labels shown in the edit metadata dialogs.
-# If metadata_edit_elide_labels is True then labels wider than the width
-# will be elided, otherwise they will be word wrapped. The maximum width is
-# computed by multiplying the average width of characters in the font by the
-# appropriate number.
-# Set the elision point to 'middle' to put the ellipsis (…) in the middle of
-# the label, 'right' to put it at the right end of the label, and 'left' to
-# put it at the left end.
-metadata_edit_elide_labels = True
-metadata_edit_bulk_cc_label_length = 25
-metadata_edit_single_cc_label_length = 12
-metadata_edit_elision_point = 'right'
-
 #: The number of seconds to wait before sending emails
 # The number of seconds to wait before sending emails when using a
 # public email server like GMX/Hotmail/Gmail. Default is: 5 minutes
@ -466,7 +435,7 @@
 # to be public relays here. Any relay host ending with one of the suffixes
 # in the list below will be considered a public email server.
 public_smtp_relay_delay = 301
-public_smtp_relay_host_suffixes = ['gmail.com', 'live.com', 'gmx.com']
+public_smtp_relay_host_suffixes = ['gmail.com', 'live.com', 'gmx.com', 'outlook.com']

 #: The maximum width and height for covers saved in the calibre library
 # All covers in the calibre library will be resized, preserving aspect ratio,
@ -531,7 +500,14 @@
 # completions you will now have to press Tab to select one before pressing
 # Enter. Which technique you prefer will depend on the state of metadata in
 # your library and your personal editing style.
+#
+# If preselect_first_completion is False and you want Tab to accept what you
+# typed instead of the first completion then set tab_accepts_uncompleted_text
+# to True. If you do this then to select from the completions you must press
+# the Down or Up arrow keys. The tweak tab_accepts_uncompleted_text is ignored
+# if preselect_first_completion is True
 preselect_first_completion = False
+tab_accepts_uncompleted_text = False

 #: Completion mode when editing authors/tags/series/etc.
 # By default, when completing items, calibre will show you all the candidates
@ -582,6 +558,8 @@
 # Edit metadata->Copy metadata/Paste metadata actions. For example,
 # exclude_fields_on_paste = ['cover', 'timestamp', '#mycolumn']
 # to prevent pasting of the cover, Date and custom column, mycolumn.
+# You can also add a shortcut in Preferences->Shortcuts->Edit metadata
+# to paste metadata ignoring this tweak.
 exclude_fields_on_paste = []

 #: Skip internet connected check
@ -594,3 +572,16 @@
 # Sets the width of the tab stop in the template editor in "average characters".
 # For example, a value of 1 results in a space with the width of one average character.
 template_editor_tab_stop_width = 4
+
+#: Value for undefined numbers when sorting
+# Sets the value to use for undefined numbers when sorting.
+# For example, the value -10 sorts undefined numbers as if they were set to -10.
+# Use 'maximum' for the largest possible number. Use 'minimum' for the smallest
+# possible number. Quotes are optional if entering a number.
+# Examples:
+#   value_for_undefined_numbers_when_sorting = -100
+#   value_for_undefined_numbers_when_sorting = '2'
+#   value_for_undefined_numbers_when_sorting = -0.01
+#   value_for_undefined_numbers_when_sorting = 'minimum'
+#   value_for_undefined_numbers_when_sorting = 'maximum'
+value_for_undefined_numbers_when_sorting = 0
--- a/resources/dictionaries/en-GB/en-GB.aff
+++ b/resources/dictionaries/en-GB/en-GB.aff
--- a/resources/dictionaries/en-GB/en-GB.dic
+++ b/resources/dictionaries/en-GB/en-GB.dic
--- a/resources/dictionaries/en-US/en-US.aff
+++ b/resources/dictionaries/en-US/en-US.aff
--- a/resources/dictionaries/en-US/en-US.dic
+++ b/resources/dictionaries/en-US/en-US.dic
--- a/resources/dictionaries/es-ES/es-ES.aff
+++ b/resources/dictionaries/es-ES/es-ES.aff
@ -1,4 +1,5 @@
 SET UTF-8
+FLAG UTF-8
 TRY aeroinsctldumpbgfvhzóíjáqéñxyúükwAEROINSCTLDUMPBGFVHZÓÍJÁQÉÑXYÚÜKW
 REP 20
 REP ás az
@ -105,15 +106,22 @@ PFX u e tras e[^s]
 PFX v Y 2
 PFX v 0 contra [^r]
 PFX v 0 contrar r
-SFX A Y 9
+PFX w Y 1
+PFX w 0 ex	 .
+SFX A Y 14
 SFX A r ción/S ar
 SFX A er ición/S [^cn]er
 SFX A er ición/S [^e]cer
 SFX A ecer ición/S ecer
 SFX A er ición/S [^o]ner
 SFX A ner sición/S oner
-SFX A r ción/S [^c]ir
-SFX A r ción/S [^u]cir
+SFX A r ción/S [^bc]ir
+SFX A bir pción/S ebir
+SFX A bir pción/S [^ch]ibir
+SFX A ibir epción/S cibir
+SFX A r ción/S hibir
+SFX A ecir ición/S [^ae]decir
+SFX A ecir icción/S [ae]decir
 SFX A ir ción/S ucir
 SFX B Y 3
 SFX B r dura/S [aií]r
@ -194,13 +202,14 @@ SFX M o eza/S nto
 SFX M o eza/S [aie]sto
 SFX M o ez/S usto
 SFX M o uez/S go
-SFX N Y 18
+SFX N Y 19
 SFX N a illa/S [bdfhjlmnprstv]a
 SFX N o illo/S [bdhjlmnprstv]o
 SFX N 0 cilla/S ve
 SFX N za cilla/S za
 SFX N 0 ecilla/S d
-SFX N ón oncillo/S ón
+SFX N ión ioncilla/S [cglnstx]ión
+SFX N ón oncillo/S [^i]ón
 SFX N 0 cillo/S or
 SFX N zo cillo/S zo
 SFX N 0 cillo/S [djlnu]e
@ -261,7 +270,7 @@ SFX T r ble/S ar
 SFX T er ible/S [^aeo]er
 SFX T er íble/S [aeo]er
 SFX T r ble/S ir
-SFX U Y 18
+SFX U Y 19
 SFX U a ita/S [bdfhjlmnprstv]a
 SFX U o ito/S [bdhjlmnprstv]o
 SFX U 0 cita/S ve
@ -269,6 +278,7 @@ SFX U za cita/S za
 SFX U 0 ecita/S d
 SFX U ón oncito/S ón
 SFX U 0 cito/S or
+SFX U o íto/S eo
 SFX U zo cito/S zo
 SFX U 0 cito/S [djlnu]e
 SFX U z cecita/S z
@ -280,8 +290,7 @@ SFX U co quito/S co
 SFX U go guito/S go
 SFX U ca quita/S ca
 SFX U ga guita/S ga
-SFX R Y 240
-SFX R ar ás ar
+SFX R Y 235
 SFX R r mos [aei]r
 SFX R ar áis ar
 SFX R r ba ar
@ -337,12 +346,10 @@ SFX R 0 es ar
 SFX R ar áremos ar
 SFX R 0 eis ar
 SFX R 0 en ar
-SFX R ar á ar
 SFX R r d [aei]r
 SFX R r ndo ar
 SFX R ar ándose ar
 SFX R 0 se [aeií]r
-SFX R er és er
 SFX R er éis er
 SFX R er ía er
 SFX R er ías er
@ -396,7 +403,6 @@ SFX R er yendo [aeo]er
 SFX R er yéndose [aeo]er
 SFX R r ndo ñer
 SFX R er éndose ñer
-SFX R er é er
 SFX R ir ís ir
 SFX R ir ía ir
 SFX R ir ías ir
@ -520,7 +526,6 @@ SFX R r endo [gq]uir
 SFX R r éndose [gq]uir
 SFX R ir yendo [^gq]uir
 SFX R ir yéndose [^gq]uir
-SFX R ir í ir
 SFX E Y 73
 SFX E ar o ar
 SFX E r s [ae]r
@ -595,10 +600,9 @@ SFX E cir zan cir
 SFX E gir jan gir
 SFX E uir an guir
 SFX E quir can quir
-SFX I Y 741
+SFX I Y 738
 SFX I ertar ierto ertar
 SFX I ertar iertas ertar
-SFX I ar ás ar
 SFX I ertar ierta ertar
 SFX I ertar iertan ertar
 SFX I ertar ierte ertar
@ -1138,7 +1142,6 @@ SFX I usar úses usar
 SFX I usar úsen usar
 SFX I cer zco [ae]cer
 SFX I r s [ae]cer
-SFX I er és er
 SFX I r 0 [ae]cer
 SFX I r n [ae]cer
 SFX I cer zca [ae]cer
@ -1274,7 +1277,6 @@ SFX I er igan oer
 SFX I er yan oer
 SFX I ir go sir
 SFX I ir es sir
-SFX I ir ís ir
 SFX I ir e sir
 SFX I ir en sir
 SFX I ir ga sir
@ -1337,14 +1339,13 @@ SFX I ir yas uir
 SFX I ir yamos uir
 SFX I ir yáis uir
 SFX I ir yan uir
-SFX X Y 1141
+SFX X Y 1137
 SFX X er go oner
 SFX X r s oner
 SFX X r 0 oner
 SFX X r mos [ei]r
 SFX X er éis [^v]er
 SFX X er éis [es]ver
-SFX X er és [^v]er
 SFX X r n oner
 SFX X erer iero erer
 SFX X erer ieres erer
@ -2017,26 +2018,26 @@ SFX X aber upierais aber
 SFX X aber upieseis aber
 SFX X aber upieran aber
 SFX X aber upiesen aber
-SFX X r ra ñer 
-SFX X r se ñer 
-SFX X r ras ñer 
-SFX X r ses ñer 
-SFX X er éramos ñer 
-SFX X er ésemos ñer 
-SFX X r rais ñer 
-SFX X r seis ñer 
-SFX X r ran ñer 
-SFX X r sen ñer 
-SFX X er iera [lv]er 
-SFX X er iese [lv]er 
-SFX X er ieras [lv]er 
-SFX X er ieses [lv]er 
-SFX X er iéramos [lv]er 
-SFX X er iésemos [lv]er 
-SFX X er ierais [lv]er 
-SFX X er ieseis [lv]er 
-SFX X er ieran [lv]er 
-SFX X er iesen [lv]er 
+SFX X r ra ñer
+SFX X r se ñer
+SFX X r ras ñer
+SFX X r ses ñer
+SFX X er éramos ñer
+SFX X er ésemos ñer
+SFX X r rais ñer
+SFX X r seis ñer
+SFX X r ran ñer
+SFX X r sen ñer
+SFX X er iera [lv]er
+SFX X er iese [lv]er
+SFX X er ieras [lv]er
+SFX X er ieses [lv]er
+SFX X er iéramos [lv]er
+SFX X er iésemos [lv]er
+SFX X er ierais [lv]er
+SFX X er ieseis [lv]er
+SFX X er ieran [lv]er
+SFX X er iesen [lv]er
 SFX X erir iriera erir
 SFX X erir iriese erir
 SFX X erir irieras erir
@ -2297,16 +2298,16 @@ SFX X aber upieres aber
 SFX X aber upiéremos aber
 SFX X aber upiereis aber
 SFX X aber upieren aber
-SFX X r re ñer 
-SFX X r res ñer 
-SFX X er éremos ñer 
-SFX X r reis ñer 
-SFX X r ren ñer 
-SFX X er iere [lv]er 
-SFX X er ieres [lv]er 
-SFX X er iéremos [lv]er 
-SFX X er iereis [lv]er 
-SFX X er ieren [lv]er 
+SFX X r re ñer
+SFX X r res ñer
+SFX X er éremos ñer
+SFX X r reis ñer
+SFX X r ren ñer
+SFX X er iere [lv]er
+SFX X er ieres [lv]er
+SFX X er iéremos [lv]er
+SFX X er iereis [lv]er
+SFX X er ieren [lv]er
 SFX X erir iriere erir
 SFX X erir irieres erir
 SFX X erir iriéremos erir
@ -2423,9 +2424,6 @@ SFX X ír yéremos oír
 SFX X ír yereis oír
 SFX X ír yeren oír
 SFX X r d [ei]r
-SFX X er é er
-SFX X ir í ir
-SFX X r 0 ír
 SFX X er iendo [^añ]er
 SFX X er iéndose [^añ]er
 SFX X er yendo aer
@ -3065,10 +3063,6 @@ SFX Ì abar ábala abar
 SFX Ì abar ábalas abar
 SFX Ì abar ábalo abar
 SFX Ì abar ábalos abar
-SFX Ì r la r
-SFX Ì r las r
-SFX Ì r lo r
-SFX Ì r los r
 SFX Ì r dla r
 SFX Ì r dlas r
 SFX Ì r dlo r
@ -3445,11 +3439,13 @@ SFX Ì utar útalas utar
 SFX Ì utar útala utar
 SFX Ì utar útalos utar
 SFX Ì utar útalo utar
-SFX Í Y 164
+SFX Ì uzar úzalas uzar
+SFX Ì uzar úzala uzar
+SFX Ì uzar úzalos uzar
+SFX Ì uzar úzalo uzar
+SFX Í Y 172
 SFX Í abar ábame abar
 SFX Í abar ábanos abar
-SFX Í r me r
-SFX Í r nos r
 SFX Í r dme r
 SFX Í r dnos r
 SFX Í ablar áblame ablar
@ -3490,6 +3486,8 @@ SFX Í argar árgame argar
 SFX Í argar árganos argar
 SFX Í arrar árrame arrar
 SFX Í arrar árranos arrar
+SFX Í asar ásame asar
+SFX Í asar ásanos asar
 SFX Í atar átame atar
 SFX Í atar átanos atar
 SFX Í avar ávame avar
@ -3504,6 +3502,8 @@ SFX Í eer éeme eer
 SFX Í eer éenos eer
 SFX Í egar égame egar
 SFX Í egar éganos egar
+SFX Í eger égeme eger
+SFX Í eger égenos eger
 SFX Í egrar égrame egrar
 SFX Í egrar égranos egrar
 SFX Í ejar éjame ejar
@ -3528,6 +3528,8 @@ SFX Í ercar ércame ercar
 SFX Í ercar ércanos ercar
 SFX Í esar ésame esar
 SFX Í esar ésanos esar
+SFX Í escar éscame escar
+SFX Í escar éscanos escar
 SFX Í estar éstame estar
 SFX Í estar éstanos estar
 SFX Í etar étame etar
@ -3538,6 +3540,8 @@ SFX Í evar évame evar
 SFX Í evar évanos evar
 SFX Í ibir íbeme ibir
 SFX Í ibir íbenos ibir
+SFX Í ibrar íbrame ibrar
+SFX Í ibrar íbranos ibrar
 SFX Í icar ícame icar
 SFX Í icar ícanos icar
 SFX Í iciar íciame iciar
@ -3558,6 +3562,8 @@ SFX Í obrar óbrame obrar
 SFX Í obrar óbranos obrar
 SFX Í ocar ócame ocar
 SFX Í ocar ócanos ocar
+SFX Í ocer óceme ocer
+SFX Í ocer ócenos ocer
 SFX Í oger ógeme oger
 SFX Í oger ógenos oger
 SFX Í ojar ójame ojar
@ -3613,8 +3619,6 @@ SFX Í uscar úscanos uscar
 SFX Î Y 86
 SFX Î abar ábale abar
 SFX Î abar ábales abar
-SFX Î r le r
-SFX Î r les r
 SFX Î r dle r
 SFX Î r dles r
 SFX Î ablar áblale ablar
@ -3655,6 +3659,8 @@ SFX Î eter étele eter
 SFX Î eter ételes eter
 SFX Î evar évale evar
 SFX Î evar évales evar
+SFX Î ibir íbele ibir
+SFX Î ibir íbeles ibir
 SFX Î icar ícale icar
 SFX Î icar ícales icar
 SFX Î iciar íciale iciar
@ -3697,7 +3703,7 @@ SFX Î untar úntales untar
 SFX Î untar úntale untar
 SFX Î uscar úscales uscar
 SFX Î uscar úscale uscar
-SFX Ï Y 300
+SFX Ï Y 288
 SFX Ï adir ádemela adir
 SFX Ï adir ádemelas adir
 SFX Ï adir ádemelo adir
@ -3710,18 +3716,6 @@ SFX Ï adir ádesela adir
 SFX Ï adir ádeselas adir
 SFX Ï adir ádeselo adir
 SFX Ï adir ádeselos adir
-SFX Ï ir ímela ir
-SFX Ï ir ímelas ir
-SFX Ï ir ímelo ir
-SFX Ï ir ímelos ir
-SFX Ï ir ínosla ir
-SFX Ï ir ínoslas ir
-SFX Ï ir ínoslo ir
-SFX Ï ir ínoslos ir
-SFX Ï ir ísela ir
-SFX Ï ir íselas ir
-SFX Ï ir íselo ir
-SFX Ï ir íselos ir
 SFX Ï ir ídmela ir
 SFX Ï ir ídmelas ir
 SFX Ï ir ídmelo ir
@ -3746,18 +3740,6 @@ SFX Ï ambiar ámbiasela ambiar
 SFX Ï ambiar ámbiaselas ambiar
 SFX Ï ambiar ámbiaselo ambiar
 SFX Ï ambiar ámbiaselos ambiar
-SFX Ï ar ámela ar
-SFX Ï ar ámelas ar
-SFX Ï ar ámelo ar
-SFX Ï ar ámelos ar
-SFX Ï ar ánosla ar
-SFX Ï ar ánoslas ar
-SFX Ï ar ánoslo ar
-SFX Ï ar ánoslos ar
-SFX Ï ar ásela ar
-SFX Ï ar áselas ar
-SFX Ï ar áselo ar
-SFX Ï ar áselos ar
 SFX Ï ar ádmela ar
 SFX Ï ar ádmelas ar
 SFX Ï ar ádmelo ar
@ -3782,6 +3764,18 @@ SFX Ï ancar áncasela ancar
 SFX Ï ancar áncaselas ancar
 SFX Ï ancar áncaselo ancar
 SFX Ï ancar áncaselos ancar
+SFX Ï andar ándamela andar
+SFX Ï andar ándamelas andar
+SFX Ï andar ándamelo andar
+SFX Ï andar ándamelos andar
+SFX Ï andar ándanosla andar
+SFX Ï andar ándanoslas andar
+SFX Ï andar ándanoslo andar
+SFX Ï andar ándanoslos andar
+SFX Ï andar ándasela andar
+SFX Ï andar ándaselas andar
+SFX Ï andar ándaselo andar
+SFX Ï andar ándaselos andar
 SFX Ï antar ántamela antar
 SFX Ï antar ántamelas antar
 SFX Ï antar ántamelo antar
@ -3806,6 +3800,18 @@ SFX Ï anzar ánzasela anzar
 SFX Ï anzar ánzaselas anzar
 SFX Ï anzar ánzaselo anzar
 SFX Ï anzar ánzaselos anzar
+SFX Ï ardar árdamela ardar
+SFX Ï ardar árdamelas ardar
+SFX Ï ardar árdamelo ardar
+SFX Ï ardar árdamelos ardar
+SFX Ï ardar árdanosla ardar
+SFX Ï ardar árdanoslas ardar
+SFX Ï ardar árdanoslo ardar
+SFX Ï ardar árdanoslos ardar
+SFX Ï ardar árdasela ardar
+SFX Ï ardar árdaselas ardar
+SFX Ï ardar árdaselo ardar
+SFX Ï ardar árdaselos ardar
 SFX Ï asar ásamela asar
 SFX Ï asar ásamelas asar
 SFX Ï asar ásamelo asar
@ -3842,18 +3848,6 @@ SFX Ï eder édesela eder
 SFX Ï eder édeselas eder
 SFX Ï eder édeselo eder
 SFX Ï eder édeselos eder
-SFX Ï er émela er
-SFX Ï er émelas er
-SFX Ï er émelo er
-SFX Ï er émelos er
-SFX Ï er énosla er
-SFX Ï er énoslas er
-SFX Ï er énoslo er
-SFX Ï er énoslos er
-SFX Ï er ésela er
-SFX Ï er éselas er
-SFX Ï er éselo er
-SFX Ï er éselos er
 SFX Ï er édmela er
 SFX Ï er édmelas er
 SFX Ï er édmelo er
@ -3998,15 +3992,11 @@ SFX Ï urar úraselas urar
 SFX Ï urar úrasela urar
 SFX Ï urar úraselos urar
 SFX Ï urar úraselo urar
-SFX Ð Y 60
+SFX Ð Y 56
 SFX Ð egar iégala egar
 SFX Ð egar iégalas egar
 SFX Ð egar iégalo egar
 SFX Ð egar iégalos egar
-SFX Ð r la r
-SFX Ð r las r
-SFX Ð r lo r
-SFX Ð r los r
 SFX Ð r dla r
 SFX Ð r dlas r
 SFX Ð r dlo r
@ -4059,11 +4049,9 @@ SFX Ð ostrar uéstrala ostrar
 SFX Ð ostrar uéstralas ostrar
 SFX Ð ostrar uéstralo ostrar
 SFX Ð ostrar uéstralos ostrar
-SFX Ñ Y 26
+SFX Ñ Y 24
 SFX Ñ ecer éceme ecer
 SFX Ñ ecer écenos ecer
-SFX Ñ r me r
-SFX Ñ r nos r
 SFX Ñ r dme r
 SFX Ñ r dnos r
 SFX Ñ ender iéndeme ender
@ -4086,11 +4074,9 @@ SFX Ñ order uérdeme order
 SFX Ñ order uérdenos order
 SFX Ñ ostrar uéstrame ostrar
 SFX Ñ ostrar uéstranos ostrar
-SFX Ò Y 22
+SFX Ò Y 20
 SFX Ò ecer écele ecer
 SFX Ò ecer éceles ecer
-SFX Ò r le r
-SFX Ò r les r
 SFX Ò r dle r
 SFX Ò r dles r
 SFX Ò egar iégale egar
@ -4109,7 +4095,7 @@ SFX Ò order uérdele order
 SFX Ò order uérdeles order
 SFX Ò ostrar uéstrale ostrar
 SFX Ò ostrar uéstrales ostrar
-SFX Ó Y 120
+SFX Ó Y 96
 SFX Ó ecer écemela ecer
 SFX Ó ecer écemelas ecer
 SFX Ó ecer écemelo ecer
@ -4122,18 +4108,6 @@ SFX Ó ecer écesela ecer
 SFX Ó ecer éceselas ecer
 SFX Ó ecer éceselo ecer
 SFX Ó ecer éceselos ecer
-SFX Ó er émela er
-SFX Ó er émelas er
-SFX Ó er émelo er
-SFX Ó er émelos er
-SFX Ó er énosla er
-SFX Ó er énoslas er
-SFX Ó er énoslo er
-SFX Ó er énoslos er
-SFX Ó er ésela er
-SFX Ó er éselas er
-SFX Ó er éselo er
-SFX Ó er éselos er
 SFX Ó er édmela er
 SFX Ó er édmelas er
 SFX Ó er édmelo er
@ -4158,18 +4132,6 @@ SFX Ó iar íasela iar
 SFX Ó iar íaselas iar
 SFX Ó iar íaselo iar
 SFX Ó iar íaselos iar
-SFX Ó ar ámela ar
-SFX Ó ar ámelas ar
-SFX Ó ar ámelo ar
-SFX Ó ar ámelos ar
-SFX Ó ar ánosla ar
-SFX Ó ar ánoslas ar
-SFX Ó ar ánoslo ar
-SFX Ó ar ánoslos ar
-SFX Ó ar ásela ar
-SFX Ó ar áselas ar
-SFX Ó ar áselo ar
-SFX Ó ar áselos ar
 SFX Ó ar ádmela ar
 SFX Ó ar ádmelas ar
 SFX Ó ar ádmelo ar
@ -4235,10 +4197,6 @@ SFX Ô aer áela aer
 SFX Ô aer áelas aer
 SFX Ô aer áelo aer
 SFX Ô aer áelos aer
-SFX Ô r la r
-SFX Ô r las r
-SFX Ô r lo r
-SFX Ô r los r
 SFX Ô r dla r
 SFX Ô r dlas r
 SFX Ô r dlo r
@ -4255,6 +4213,10 @@ SFX Ô edir ídela edir
 SFX Ô edir ídelas edir
 SFX Ô edir ídelo edir
 SFX Ô edir ídelos edir
+SFX Ô egir ígela egir
+SFX Ô egir ígelas egir
+SFX Ô egir ígelo egir
+SFX Ô egir ígelos egir
 SFX Ô eguir íguela eguir
 SFX Ô eguir íguelas eguir
 SFX Ô eguir íguelo eguir
@ -4291,11 +4253,9 @@ SFX Ô uir úyelas uir
 SFX Ô uir úyela uir
 SFX Ô uir úyelos uir
 SFX Ô uir úyelo uir
-SFX Õ Y 28
+SFX Õ Y 26
 SFX Õ aber ábeme aber
 SFX Õ aber ábenos aber
-SFX Õ r me r
-SFX Õ r nos r
 SFX Õ r dme r
 SFX Õ r dnos r
 SFX Õ aer áeme aer
@ -4320,11 +4280,9 @@ SFX Õ r me ver
 SFX Õ r nos ver
 SFX Õ ucir úceme ucir
 SFX Õ ucir úcenos ucir
-SFX Ö Y 16
+SFX Ö Y 14
 SFX Ö aber ábele aber
 SFX Ö aber ábeles aber
-SFX Ö r le r
-SFX Ö r les r
 SFX Ö r dle r
 SFX Ö r dles r
 SFX Ö aer áele aer
@ -4337,7 +4295,7 @@ SFX Ö er le ner
 SFX Ö er les ner
 SFX Ö r le ver
 SFX Ö r les ver
-SFX Ø Y 84
+SFX Ø Y 60
 SFX Ø aer áemela aer
 SFX Ø aer áemelas aer
 SFX Ø aer áemelo aer
@ -4350,18 +4308,6 @@ SFX Ø aer áesela aer
 SFX Ø aer áeselas aer
 SFX Ø aer áeselo aer
 SFX Ø aer áeselos aer
-SFX Ø er émela er
-SFX Ø er émelas er
-SFX Ø er émelo er
-SFX Ø er émelos er
-SFX Ø er énosla er
-SFX Ø er énoslas er
-SFX Ø er énoslo er
-SFX Ø er énoslos er
-SFX Ø er ésela er
-SFX Ø er éselas er
-SFX Ø er éselo er
-SFX Ø er éselos er
 SFX Ø er édmela er
 SFX Ø er édmelas er
 SFX Ø er édmelo er
@ -4386,18 +4332,6 @@ SFX Ø edir ídesela edir
 SFX Ø edir ídeselas edir
 SFX Ø edir ídeselo edir
 SFX Ø edir ídeselos edir
-SFX Ø ir ímela ir
-SFX Ø ir ímelas ir
-SFX Ø ir ímelo ir
-SFX Ø ir ímelos ir
-SFX Ø ir ínosla ir
-SFX Ø ir ínoslas ir
-SFX Ø ir ínoslo ir
-SFX Ø ir ínoslos ir
-SFX Ø ir ísela ir
-SFX Ø ir íselas ir
-SFX Ø ir íselo ir
-SFX Ø ir íselos ir
 SFX Ø ir ídmela ir
 SFX Ø ir ídmelas ir
 SFX Ø ir ídmelo ir
@ -6285,9 +6219,8 @@ SFX ó r éndome ucir
 SFX ó r éndonos ucir
 SFX ó r éndoos ucir
 SFX ó r éndote ucir
-SFX ô Y 136
+SFX ô Y 137
 SFX ô abar ábate abar
-SFX ô r te [aei]r
 SFX ô r os r
 SFX ô acar ácate acar
 SFX ô achar áchate achar
@ -6331,6 +6264,7 @@ SFX ô azar ázate azar
 SFX ô ear éate ear
 SFX ô ecar écate ecar
 SFX ô echar échate echar
+SFX ô ectar éctate ectar
 SFX ô edar édate edar
 SFX ô egar égate egar
 SFX ô eger égete eger
@ -6373,6 +6307,7 @@ SFX ô intar íntate intar
 SFX ô irar írate irar
 SFX ô ir íos ir
 SFX ô istar ístate istar
+SFX ô istrar ístrate istrar
 SFX ô itar ítate itar
 SFX ô itir ítete itir
 SFX ô izar ízate izar
@ -6422,9 +6357,8 @@ SFX ô urlar úrlate urlar
 SFX ô uscar úscate uscar
 SFX ô ustar ústate ustar
 SFX ô ustrar ústrate ustrar
-SFX õ Y 31
+SFX õ Y 30
 SFX õ aer áete aer
-SFX õ r te [aei]r
 SFX õ r os [^i]r
 SFX õ ecer écete ecer
 SFX õ egar iégate egar
@ -6454,9 +6388,8 @@ SFX õ over uévete over
 SFX õ uar úate uar
 SFX õ unir únete unir
 SFX õ ir íos ir
-SFX ö Y 18
+SFX ö Y 17
 SFX ö aer áete aer
-SFX ö r te [eií]r
 SFX ö r os [^i]r
 SFX ö aler álete aler
 SFX ö cer zte cer
--- a/resources/dictionaries/es-ES/es-ES.dic
+++ b/resources/dictionaries/es-ES/es-ES.dic
--- a/resources/mime.types
+++ b/resources/mime.types
@ -875,7 +875,6 @@ application/x-ustar				ustar
 application/x-wais-source				src
 application/x-wingz				wz
 application/x-x509-ca-cert				crt der
-application/x-xcf				xcf
 application/x-xfig				fig
 application/x-xpinstall				xpi
 application/x400-bp
@ -1146,6 +1145,7 @@ image/x-portable-graymap				pgm
 image/x-portable-pixmap				ppm
 image/x-rgb				rgb
 image/x-xbitmap				xbm
+image/x-xcf				xcf
 image/x-xpixmap				xpm
 image/x-xwindowdump				xwd
 message/cpim
--- a/resources/pdf-preprint.js
+++ b/resources/pdf-preprint.js
@ -35,6 +35,8 @@
    // is U+2010 but that does not render with the default Times font on macOS as of Monterey
    // and Qt 15.5 See https://bugs.launchpad.net/bugs/1951467 and can be easily reproduced
    // by converting a plain text file with the --pdf-hyphenate option
+    // https://bugs.chromium.org/p/chromium/issues/detail?id=1267606 (fix released Feb 1 2022 v98)
+    // See also settings.pyj
    if (HYPHEN_CHAR) {
        for (const elem of document.getElementsByTagName('*')) {
            if (elem.style) {
--- a/setup/init.py
+++ b/setup/init.py
@ -1,5 +1,4 @@
 #!/usr/bin/env python
-# vim:fileencoding=UTF-8:ts=4:sw=4:sta:et:sts=4:ai


 __license__   = 'GPL v3'
@ -92,7 +91,7 @@ def build_cache_dir():
        _cache_dir_built = True
        try:
            os.mkdir(ans)
-        except EnvironmentError as err:
+        except OSError as err:
            if err.errno != errno.EEXIST:
                raise
    return ans
@ -100,7 +99,7 @@ def build_cache_dir():

 def require_git_master(branch='master'):
    if subprocess.check_output(['git', 'symbolic-ref', '--short', 'HEAD']).decode('utf-8').strip() != branch:
-        raise SystemExit('You must be in the {} git branch'.format(branch))
+        raise SystemExit(f'You must be in the {branch} git branch')


 def require_clean_git():
@ -225,7 +224,7 @@ def run_cmd(self, cmd, opts):
        st = time.time()
        self.running(cmd)
        cmd.run(opts)
-        self.info('* %s took %.1f seconds' % (command_names[cmd], time.time() - st))
+        self.info(f'* {command_names[cmd]} took {time.time() - st:.1f} seconds')
        if os.environ.get('CI'):
            self.info('::endgroup::')

--- a/setup/browser_data.py
+++ b/setup/browser_data.py
@ -1,5 +1,4 @@
 #!/usr/bin/env python
-# vim:fileencoding=utf-8
 # License: GPLv3 Copyright: 2017, Kovid Goyal <kovid at kovidgoyal.net>


--- a/setup/build.py
+++ b/setup/build.py
@ -1,11 +1,10 @@
 #!/usr/bin/env python
-# vim:fileencoding=UTF-8:ts=4:sw=4:sta:et:sts=4:ai

 __license__   = 'GPL v3'
 __copyright__ = '2009, Kovid Goyal <kovid@kovidgoyal.net>'
 __docformat__ = 'restructuredtext en'

-import textwrap, os, shlex, subprocess, glob, shutil, sys, json, errno
+import textwrap, os, shlex, subprocess, glob, shutil, sys, json, errno, sysconfig
 from collections import namedtuple

 from setup import Command, islinux, isbsd, isfreebsd, ismacos, ishaiku, SRC, iswindows
@ -28,7 +27,7 @@ def init_symbol_name(name):


 def absolutize(paths):
-    return list(set([x if os.path.isabs(x) else os.path.join(SRC, x.replace('/', os.sep)) for x in paths]))
+    return list({x if os.path.isabs(x) else os.path.join(SRC, x.replace('/', os.sep)) for x in paths})


 class Extension:
@ -167,9 +166,20 @@ def read_extensions():
    return ans


+def get_python_include_paths():
+    ans = []
+    for name in sysconfig.get_path_names():
+        if 'include' in name:
+            ans.append(name)
+
+    def gp(x):
+        return sysconfig.get_path(x)
+
+    return sorted(frozenset(filter(None, map(gp, sorted(ans)))))
+
+
 def init_env(debug=False, sanitize=False):
    from setup.build_environment import win_ld, is64bit, win_inc, win_lib, NMAKE, win_cc
-    from distutils import sysconfig
    linker = None
    if isunix:
        cc = os.environ.get('CC', 'gcc')
@ -202,17 +212,20 @@ def init_env(debug=False, sanitize=False):
        ldflags.append('-shared')

    if islinux or isbsd or ishaiku:
-        cflags.append('-I'+sysconfig.get_python_inc())
-        # getattr(..., 'abiflags') is for PY2 compat, since PY2 has no abiflags
-        # member
-        ldflags.append('-lpython{}{}'.format(
-            sysconfig.get_config_var('VERSION'), getattr(sys, 'abiflags', '')))
+        cflags.extend('-I' + x for x in get_python_include_paths())
+        ldlib = sysconfig.get_config_var('LIBDIR')
+        if ldlib:
+            ldflags += ['-L' + ldlib]
+        ldlib = sysconfig.get_config_var('VERSION')
+        if ldlib:
+            ldflags += ['-lpython' + ldlib + sys.abiflags]
+        ldflags += (sysconfig.get_config_var('LINKFORSHARED') or '').split()

    if ismacos:
        cflags.append('-D_OSX')
        ldflags.extend('-bundle -undefined dynamic_lookup'.split())
        cflags.extend(['-fno-common', '-dynamic'])
-        cflags.append('-I'+sysconfig.get_python_inc())
+        cflags.extend('-I' + x for x in get_python_include_paths())

    if iswindows:
        cc = cxx = win_cc
@ -233,8 +246,8 @@ def init_env(debug=False, sanitize=False):
        for p in win_lib:
            if p:
                ldflags.append('/LIBPATH:'+p)
-        cflags.append('-I%s'%sysconfig.get_python_inc())
-        ldflags.append('/LIBPATH:'+os.path.join(sysconfig.PREFIX, 'libs'))
+        cflags.extend('-I' + x for x in get_python_include_paths())
+        ldflags.append('/LIBPATH:'+os.path.join(sysconfig.get_config_var('prefix'), 'libs'))
        linker = win_ld
    return namedtuple('Environment', 'cc cxx cflags ldflags linker make')(
        cc=cc, cxx=cxx, cflags=cflags, ldflags=ldflags, linker=linker, make=NMAKE if iswindows else 'make')
--- a/setup/build_environment.py
+++ b/setup/build_environment.py
@ -1,5 +1,4 @@
 #!/usr/bin/env python
-# vim:fileencoding=UTF-8:ts=4:sw=4:sta:et:sts=4:ai


 __license__   = 'GPL v3'
--- a/setup/changelog.py
+++ b/setup/changelog.py
@ -1,5 +1,4 @@
 #!/usr/bin/env python
-# vim:fileencoding=utf-8
 # License: GPL v3 Copyright: 2020, Kovid Goyal <kovid at kovidgoyal.net>

 from datetime import date
--- a/setup/check.py
+++ b/setup/check.py
@ -1,5 +1,4 @@
 #!/usr/bin/env python
-# vim:fileencoding=UTF-8:ts=4:sw=4:sta:et:sts=4:ai


 __license__   = 'GPL v3'
@ -16,7 +15,19 @@ def __init__(self, filename, lineno, msg):
        self.filename, self.lineno, self.msg = filename, lineno, msg

    def __str__(self):
-        return '%s:%s: %s' % (self.filename, self.lineno, self.msg)
+        return f'{self.filename}:{self.lineno}: {self.msg}'
+
+
+def checkable_python_files(SRC):
+    for dname in ('odf', 'calibre'):
+        for x in os.walk(os.path.join(SRC, dname)):
+            for f in x[-1]:
+                y = os.path.join(x[0], f)
+                if (f.endswith('.py') and f not in (
+                        'dict_data.py', 'unicodepoints.py', 'krcodepoints.py',
+                        'jacodepoints.py', 'vncodepoints.py', 'zhcodepoints.py') and
+                        'prs500/driver.py' not in y) and not f.endswith('_ui.py'):
+                    yield y


 class Check(Command):
@ -26,15 +37,7 @@ class Check(Command):
    CACHE = 'check.json'

    def get_files(self):
-        for dname in ('odf', 'calibre'):
-            for x in os.walk(self.j(self.SRC, dname)):
-                for f in x[-1]:
-                    y = self.j(x[0], f)
-                    if (f.endswith('.py') and f not in (
-                            'dict_data.py', 'unicodepoints.py', 'krcodepoints.py',
-                            'jacodepoints.py', 'vncodepoints.py', 'zhcodepoints.py') and
-                            'prs500/driver.py' not in y) and not f.endswith('_ui.py'):
-                        yield y
+        yield from checkable_python_files(self.SRC)

        for x in os.walk(self.j(self.d(self.SRC), 'recipes')):
            for f in x[-1]:
@ -91,7 +94,7 @@ def run(self, opts):
        try:
            with open(self.cache_file, 'rb') as f:
                cache = json.load(f)
-        except EnvironmentError as err:
+        except OSError as err:
            if err.errno != errno.ENOENT:
                raise
        dirty_files = tuple(f for f in self.get_files() if not self.is_cache_valid(f, cache))
@ -114,6 +117,24 @@ def report_errors(self, errors):
    def clean(self):
        try:
            os.remove(self.cache_file)
-        except EnvironmentError as err:
+        except OSError as err:
            if err.errno != errno.ENOENT:
                raise
+
+
+class UpgradeSourceCode(Command):
+
+    description = 'Upgrade python source code'
+
+    def run(self, opts):
+        files = []
+        for f in os.listdir(os.path.dirname(os.path.abspath(__file__))):
+            q = os.path.join('setup', f)
+            if f.endswith('.py') and f not in ('linux-installer.py',) and not os.path.isdir(q):
+                files.append(q)
+        for path in checkable_python_files(self.SRC):
+            q = path.replace(os.sep, '/')
+            if '/metadata/sources/' in q or '/store/stores/' in q:
+                continue
+            files.append(q)
+        subprocess.call(['pyupgrade', '--py37-plus'] + files)
--- a/setup/commands.py
+++ b/setup/commands.py
@ -1,5 +1,4 @@
 #!/usr/bin/env python
-# vim:fileencoding=UTF-8:ts=4:sw=4:sta:et:sts=4:ai


 __license__   = 'GPL v3'
@ -13,7 +12,7 @@
    'git_version',
    'develop', 'install',
    'kakasi', 'rapydscript', 'cacerts', 'recent_uas', 'resources',
-    'check', 'test', 'test_rs',
+    'check', 'test', 'test_rs', 'upgrade_source_code',
    'sdist', 'bootstrap', 'extdev',
    'manual', 'tag_release',
    'upload_to_server',
@ -66,8 +65,9 @@
 from setup.gui import GUI
 gui = GUI()

-from setup.check import Check
+from setup.check import Check, UpgradeSourceCode
 check = Check()
+upgrade_source_code = UpgradeSourceCode()

 from setup.test import Test, TestRS
 test = Test()
--- a/setup/csslint.py
+++ b/setup/csslint.py
@ -1,5 +1,4 @@
 #!/usr/bin/env python
-# vim:fileencoding=utf-8
 # License: GPL v3 Copyright: 2020, Kovid Goyal <kovid at kovidgoyal.net>

 import os
--- a/setup/file-hosting-bw.py
+++ b/setup/file-hosting-bw.py
@ -1,5 +1,4 @@
 #!/usr/bin/env python
-# vim:fileencoding=utf-8


 __license__ = 'GPL v3'
--- a/setup/git_post_checkout_hook.py
+++ b/setup/git_post_checkout_hook.py
@ -1,5 +1,4 @@
 #!/usr/bin/env python
-# vim:fileencoding=utf-8

 __license__ = 'GPL v3'
 __copyright__ = '2013, Kovid Goyal <kovid at kovidgoyal.net>'
@ -10,7 +9,7 @@


 def get_branch_name(rev):
-    return subprocess.check_output(['git', 'name-rev', '--name-only', rev]).decode('utf-8').strip()
+    return subprocess.check_output(['git', 'name-rev', '--name-only', '--refs=refs/heads/*', rev]).decode('utf-8').strip()


 base = os.path.dirname(os.path.dirname(os.path.realpath(__file__)))
--- a/setup/git_post_rewrite_hook.py
+++ b/setup/git_post_rewrite_hook.py
@ -1,5 +1,4 @@
 #!/usr/bin/env python
-# vim:fileencoding=utf-8

 __license__ = 'GPL v3'
 __copyright__ = '2014, Kovid Goyal <kovid at kovidgoyal.net>'
--- a/setup/git_pre_commit_hook.py
+++ b/setup/git_pre_commit_hook.py
@ -1,5 +1,4 @@
 #!/usr/bin/env python
-# vim:fileencoding=utf-8
 # License: GPLv3 Copyright: 2008, Kovid Goyal <kovid at kovidgoyal.net>


@ -55,7 +54,7 @@ def __call__(self, match):
            print('Working on bug:', summary)
            if int(bug) > 100000 and action != 'See':
                self.close_bug(bug, action)
-                return match.group() + ' [%s](%s)' % (summary, LAUNCHPAD_BUG % bug)
+                return match.group() + f' [{summary}]({LAUNCHPAD_BUG % bug})'
            return match.group() + ' (%s)' % summary
        return match.group()

@ -66,7 +65,7 @@ def close_bug(self, bug, action):
            'calibre is usually released every alternate Friday.'
        )
        action += 'ed'
-        msg = '%s in branch %s. %s' % (action, 'master', suffix)
+        msg = '{} in branch {}. {}'.format(action, 'master', suffix)
        msg = msg.replace('Fixesed', 'Fixed')
        msg += '\n\n status fixreleased'

--- a/setup/git_version.py
+++ b/setup/git_version.py
@ -1,5 +1,4 @@
 #!/usr/bin/env python
-# vim:fileencoding=utf-8
 # License: GPLv3 Copyright: 2019, Eli Schwartz <eschwartz@archlinux.org>


--- a/setup/gui.py
+++ b/setup/gui.py
@ -1,5 +1,4 @@
 #!/usr/bin/env python
-# vim:fileencoding=UTF-8:ts=4:sw=4:sta:et:sts=4:ai


 __license__   = 'GPL v3'
--- a/setup/hosting.py
+++ b/setup/hosting.py
@ -1,5 +1,4 @@
 #!/usr/bin/env python
-# vim:fileencoding=UTF-8:ts=4:sw=4:sta:et:sts=4:ai

 __license__ = 'GPL v3'
 __copyright__ = '2011, Kovid Goyal <kovid@kovidgoyal.net>'
@ -150,7 +149,7 @@ def __call__(self):
        existing_assets = self.existing_assets(release['id'])
        for path, desc in self.files.items():
            self.info('')
-            url = self.API + 'repos/%s/%s/releases/assets/{}' % (
+            url = self.API + 'repos/{}/{}/releases/assets/{{}}'.format(
                self.username, self.reponame
            )
            fname = os.path.basename(path)
@ -206,7 +205,7 @@ def clean_older_releases(self, releases):

    def do_upload(self, url, path, desc, fname):
        mime_type = mimetypes.guess_type(fname)[0] or 'application/octet-stream'
-        self.info('Uploading to GitHub: %s (%s)' % (fname, mime_type))
+        self.info(f'Uploading to GitHub: {fname} ({mime_type})')
        with ReadFileWithProgressReporting(path) as f:
            return self.requests.post(
                url,
@ -229,7 +228,7 @@ def already_exists(self, r):
        return error_code == 'already_exists'

    def existing_assets(self, release_id):
-        url = self.API + 'repos/%s/%s/releases/%s/assets' % (
+        url = self.API + 'repos/{}/{}/releases/{}/assets'.format(
            self.username, self.reponame, release_id
        )
        r = self.requests.get(url)
@ -238,7 +237,7 @@ def existing_assets(self, release_id):
        return {asset['name']: asset['id'] for asset in r.json()}

    def releases(self):
-        url = self.API + 'repos/%s/%s/releases' % (self.username, self.reponame)
+        url = self.API + f'repos/{self.username}/{self.reponame}/releases'
        r = self.requests.get(url)
        if r.status_code != 200:
            self.fail(r, 'Failed to list releases')
@ -250,7 +249,7 @@ def create_release(self, releases):
            # Check for existing release
            if release['tag_name'] == self.current_tag_name:
                return release
-        url = self.API + 'repos/%s/%s/releases' % (self.username, self.reponame)
+        url = self.API + f'repos/{self.username}/{self.reponame}/releases'
        r = self.requests.post(
            url,
            data=json.dumps({
@ -275,7 +274,7 @@ def generate_index():  # {{{
    releases = set()
    for x in os.listdir('.'):
        if os.path.isdir(x) and '.' in x:
-            releases.add(tuple((int(y) for y in x.split('.'))))
+            releases.add(tuple(int(y) for y in x.split('.')))
    rmap = OrderedDict()
    for rnum in sorted(releases, reverse=True):
        series = rnum[:2] if rnum[0] == 0 else rnum[:1]
@ -301,10 +300,10 @@ def generate_index():  # {{{
        body.append(
            '<li><a href="{0}.html" title="Releases in the {0}.x series">{0}.x</a>\xa0\xa0\xa0<span style="font-size:smaller">[{1} releases]</span></li>'
            .format(  # noqa
-                '.'.join(map(type(''), series)), len(rmap[series])
+                '.'.join(map(str, series)), len(rmap[series])
            )
        )
-    body = '<ul>{0}</ul>'.format(' '.join(body))
+    body = '<ul>{}</ul>'.format(' '.join(body))
    index = template.format(
        title='Previous calibre releases',
        style=style,
@ -315,13 +314,13 @@ def generate_index():  # {{{
        f.write(index.encode('utf-8'))

    for series, releases in rmap.items():
-        sname = '.'.join(map(type(''), series))
+        sname = '.'.join(map(str, series))
        body = [
            '<li><a href="{0}/" title="Release {0}">{0}</a></li>'.format(
-                '.'.join(map(type(''), r))
+                '.'.join(map(str, r))
            ) for r in releases
        ]
-        body = '<ul class="release-list">{0}</ul>'.format(' '.join(body))
+        body = '<ul class="release-list">{}</ul>'.format(' '.join(body))
        index = template.format(
            title='Previous calibre releases (%s.x)' % sname,
            style=style,
@ -332,7 +331,7 @@ def generate_index():  # {{{
            f.write(index.encode('utf-8'))

        for r in releases:
-            rname = '.'.join(map(type(''), r))
+            rname = '.'.join(map(str, r))
            os.chdir(rname)
            try:
                body = []
@ -346,7 +345,7 @@ def generate_index():  # {{{
                        ) for x in windows
                    ]
                    body.append(
-                        '<dt>Windows</dt><dd><ul>{0}</ul></dd>'.format(
+                        '<dt>Windows</dt><dd><ul>{}</ul></dd>'.format(
                            ' '.join(windows)
                        )
                    )
@ -373,7 +372,7 @@ def generate_index():  # {{{
                        ) for x in linux
                    ]
                    body.append(
-                        '<dt>Linux</dt><dd><ul>{0}</ul></dd>'.format(
+                        '<dt>Linux</dt><dd><ul>{}</ul></dd>'.format(
                            ' '.join(linux)
                        )
                    )
@ -384,7 +383,7 @@ def generate_index():  # {{{
                        .format(source[0], 'Source code (all platforms)')
                    )

-                body = '<dl>{0}</dl>'.format(''.join(body))
+                body = '<dl>{}</dl>'.format(''.join(body))
                index = template.format(
                    title='calibre release (%s)' % rname,
                    style=style,
--- a/setup/hyphenation.py
+++ b/setup/hyphenation.py
@ -1,7 +1,5 @@
 #!/usr/bin/env python
-# vim:fileencoding=UTF-8:ts=4:sw=4:sta:et:sts=4:fdm=marker:ai
 # License: GPLv3 Copyright: 2019, Kovid Goyal <kovid at kovidgoyal.net>
-from __future__ import absolute_import, division, print_function, unicode_literals

 import glob
 import hashlib
@ -100,7 +98,7 @@ def run(self, opts):
            for dic in dics:
                with open(os.path.join(output_dir, dic), 'rb') as f:
                    m.update(f.read())
-            hsh = type('')(m.hexdigest())
+            hsh = str(m.hexdigest())
            buf = BytesIO()
            with tarfile.TarFile(fileobj=buf, mode='w') as tf:
                for dic in dics:
--- a/setup/install.py
+++ b/setup/install.py
@ -1,5 +1,4 @@
 #!/usr/bin/env python
-# vim:fileencoding=UTF-8:ts=4:sw=4:sta:et:sts=4:ai
 # License: GPLv3 Copyright: 2009, Kovid Goyal <kovid at kovidgoyal.net>


@ -155,12 +154,15 @@ def run(self, opts):
        self.success()

    def install_env_module(self):
-        import distutils.sysconfig as s
-        libdir = s.get_python_lib(prefix=self.opts.staging_root)
+        import sysconfig
+        libdir = os.path.join(
+            self.opts.staging_root, sysconfig.get_config_var('PLATLIBDIR') or 'lib',
+            os.path.basename(sysconfig.get_config_var('DESTLIB') or sysconfig.get_config_var('LIBDEST') or f'python{sysconfig.get_python_version()}'),
+            'site-packages')
        try:
            if not os.path.exists(libdir):
                os.makedirs(libdir)
-        except EnvironmentError:
+        except OSError:
            self.warn('Cannot install calibre environment module to: '+libdir)
        else:
            path = os.path.join(libdir, 'init_calibre.py')
@ -364,7 +366,7 @@ def add_options(self, parser):
    def pre_sub_commands(self, opts):
        tdir = self.j(self.d(self.SRC), 'translations')
        clone_cmd = [
-            'git', 'clone', 'https://github.com/{}.git'.format(self.TRANSLATIONS_REPO), 'translations']
+            'git', 'clone', f'https://github.com/{self.TRANSLATIONS_REPO}.git', 'translations']
        if opts.ephemeral:
            if os.path.exists(tdir):
                shutil.rmtree(tdir)
--- a/setup/installers.py
+++ b/setup/installers.py
@ -1,5 +1,4 @@
 #!/usr/bin/env python
-# vim:fileencoding=utf-8
 # License: GPLv3 Copyright: 2016, Kovid Goyal <kovid at kovidgoyal.net>


@ -13,7 +12,9 @@

 def get_paths():
    base = d(d(os.path.abspath(__file__)))
-    bypy = os.path.join(d(base), 'bypy')
+    traditional_bypy_location = os.path.join(d(base), 'bypy')
+    compat_bypy_location = os.path.join(base, 'bypy', 'b', 'bypy-old')
+    bypy = compat_bypy_location if os.path.exists(compat_bypy_location) else traditional_bypy_location
    bypy = os.environ.get('BYPY_LOCATION', bypy)
    if not os.path.isdir(bypy):
        raise SystemExit(
@ -84,7 +85,7 @@ def build_single(which='windows', bitness='64', shutdown=True, sign_installers=T
        dest = os.path.join(base, 'dist', x)
        try:
            os.remove(dest)
-        except EnvironmentError:
+        except OSError:
            pass
        os.link(src, dest)
    if shutdown:
@ -246,9 +247,9 @@ def run(self, opts):
        try:
            path = path.format(ext)
            src = os.path.join(ext_dir, os.path.basename(path))
-            subprocess.check_call(['ssh', '-S', control_path, host, 'chmod', '+w', '"{}"'.format(path)])
+            subprocess.check_call(['ssh', '-S', control_path, host, 'chmod', '+w', f'"{path}"'])
            with open(src, 'rb') as f:
-                p = subprocess.Popen(['ssh', '-S', control_path, host, 'cat - > "{}"'.format(path)], stdin=subprocess.PIPE)
+                p = subprocess.Popen(['ssh', '-S', control_path, host, f'cat - > "{path}"'], stdin=subprocess.PIPE)
                p.communicate(f.read())
            if p.wait() != 0:
                raise SystemExit(1)
--- a/setup/lc_data.py
+++ b/setup/lc_data.py
@ -1,5 +1,4 @@
 #!/usr/bin/env python3
-# vim:fileencoding=utf-8


 __license__ = 'GPL v3'
--- a/setup/liberation.py
+++ b/setup/liberation.py
@ -1,7 +1,5 @@
 #!/usr/bin/env python
-# vim:fileencoding=UTF-8:ts=4:sw=4:sta:et:sts=4:fdm=marker:ai
 # License: GPLv3 Copyright: 2019, Kovid Goyal <kovid at kovidgoyal.net>
-from __future__ import absolute_import, division, print_function, unicode_literals

 import glob
 import os
--- a/setup/mathjax.py
+++ b/setup/mathjax.py
@ -1,5 +1,4 @@
 #!/usr/bin/env python
-# vim:fileencoding=UTF-8:ts=4:sw=4:sta:et:sts=4:fdm=marker:ai


 __license__   = 'GPL v3'
--- a/setup/parallel_build.py
+++ b/setup/parallel_build.py
@ -1,5 +1,4 @@
 #!/usr/bin/env python
-# vim:fileencoding=utf-8


 __license__ = 'GPL v3'
--- a/setup/plugins_mirror.py
+++ b/setup/plugins_mirror.py
@ -1,7 +1,5 @@
 #!/usr/bin/env python
-# vim:fileencoding=utf-8
 # License: GPLv3 Copyright: 2013, Kovid Goyal <kovid at kovidgoyal.net>
-from __future__ import absolute_import, division, print_function, unicode_literals


 # Imports {{{
@ -128,7 +126,7 @@ def parse_index(raw=None):  # {{{

        thread_id = url_to_plugin_id(url, deprecated)
        if thread_id in seen:
-            raise ValueError('thread_id for %s and %s is the same: %s' % (seen[thread_id], name, thread_id))
+            raise ValueError(f'thread_id for {seen[thread_id]} and {name} is the same: {thread_id}')
        seen[thread_id] = name
        entry = IndexEntry(name, url, donate, history, uninstall, deprecated, thread_id)
        yield entry
@ -147,7 +145,7 @@ def load_plugins_index():
    try:
        with open(PLUGINS, 'rb') as f:
            raw = f.read()
-    except IOError as err:
+    except OSError as err:
        if err.errno == errno.ENOENT:
            return {}
        raise
@ -173,13 +171,13 @@ def convert_node(fields, x, names={}, import_data=None):
        return dict(zip(keys, values))
    elif name == 'Call':
        if len(x.args) != 1 and len(x.keywords) != 0:
-            raise TypeError('Unsupported function call for fields: %s' % (fields,))
+            raise TypeError(f'Unsupported function call for fields: {fields}')
        return tuple(map(conv, x.args))[0]
    elif name == 'Name':
        if x.id not in names:
            if import_data is not None and x.id in import_data[0]:
                return get_import_data(x.id, import_data[0][x.id], *import_data[1:])
-            raise ValueError('Could not find name %s for fields: %s' % (x.id, fields))
+            raise ValueError(f'Could not find name {x.id} for fields: {fields}')
        return names[x.id]
    elif name == 'BinOp':
        if x.right.__class__.__name__ == 'Str':
@ -188,7 +186,7 @@ def convert_node(fields, x, names={}, import_data=None):
            return x.right.value
    elif name == 'Attribute':
        return conv(getattr(conv(x.value), x.attr))
-    raise TypeError('Unknown datatype %s for fields: %s' % (x, fields))
+    raise TypeError(f'Unknown datatype {x} for fields: {fields}')


 Alias = namedtuple('Alias', 'name asname')
@ -221,7 +219,7 @@ def get_import_data(name, mod, zf, names):
                    return convert_node({x}, node.value)
        if is_module_import:
            return module
-        raise ValueError('Failed to find name: %r in module: %r' % (name, mod))
+        raise ValueError(f'Failed to find name: {name!r} in module: {mod!r}')
    else:
        raise ValueError('Failed to find module: %r' % mod)

@ -457,7 +455,7 @@ def fetch_plugins(old_index):


 def plugin_to_index(plugin, count):
-    title = '<h3><img src="plugin-icon.png"><a href=%s title="Plugin forum thread">%s</a></h3>' % (  # noqa
+    title = '<h3><img src="plugin-icon.png"><a href={} title="Plugin forum thread">{}</a></h3>'.format(  # noqa
        quoteattr(plugin['thread_url']), escape(plugin['name']))
    released = datetime(*tuple(map(int, re.split(r'\D', plugin['last_modified'])))[:6]).strftime('%e %b, %Y').lstrip()
    details = [
@ -478,12 +476,12 @@ def plugin_to_index(plugin, count):
        block.append('<li>%s</li>' % li)
    block = '<ul>%s</ul>' % ('\n'.join(block))
    downloads = ('\xa0<span class="download-count">[%d total downloads]</span>' % count) if count else ''
-    zipfile = '<div class="end"><a href=%s title="Download plugin" download=%s>Download plugin \u2193</a>%s</div>' % (
+    zipfile = '<div class="end"><a href={} title="Download plugin" download={}>Download plugin \u2193</a>{}</div>'.format(
        quoteattr(plugin['file']), quoteattr(plugin['name'] + '.zip'), downloads)
    desc = plugin['description'] or ''
    if desc:
        desc = '<p>%s</p>' % desc
-    return '%s\n%s\n%s\n%s\n\n' % (title, desc, block, zipfile)
+    return f'{title}\n{desc}\n{block}\n{zipfile}\n\n'


 def create_index(index, raw_stats):
@ -526,14 +524,14 @@ def create_index(index, raw_stats):
    try:
        with open('index.html', 'rb') as f:
            oraw = f.read()
-    except EnvironmentError:
+    except OSError:
        oraw = None
    if raw != oraw:
        atomic_write(raw, 'index.html')

    def plugin_stats(x):
        name, count = x
-        return '<tr><td>%s</td><td>%s</td></tr>\n' % (escape(name), count)
+        return f'<tr><td>{escape(name)}</td><td>{count}</td></tr>\n'

    pstats = list(map(plugin_stats, sorted(stats.items(), reverse=True, key=lambda x:x[1])))
    stats = '''\
@ -560,7 +558,7 @@ def plugin_stats(x):
    try:
        with open('stats.html', 'rb') as f:
            oraw = f.read()
-    except EnvironmentError:
+    except OSError:
        oraw = None
    if raw != oraw:
        atomic_write(raw, 'stats.html')
@ -574,7 +572,7 @@ def singleinstance():
    s = _singleinstance = socket.socket(socket.AF_UNIX)
    try:
        s.bind(b'\0calibre-plugins-mirror-singleinstance')
-    except socket.error as err:
+    except OSError as err:
        if getattr(err, 'errno', None) == errno.EADDRINUSE:
            return False
        raise
@ -590,7 +588,7 @@ def update_stats():
        try:
            with open('stats.json', 'rb') as f:
                stats = json.load(f)
-        except EnvironmentError as err:
+        except OSError as err:
            if err.errno != errno.ENOENT:
                raise
        if os.geteuid() != 0:
@ -688,7 +686,7 @@ def test_parse():  # {{{
    new_entries = tuple(parse_index(raw))
    for i, entry in enumerate(old_entries):
        if entry != new_entries[i]:
-            print('The new entry: %s != %s' % (new_entries[i], entry))
+            print(f'The new entry: {new_entries[i]} != {entry}')
            raise SystemExit(1)
    pool = ThreadPool(processes=20)
    urls = [e.url for e in new_entries]
@ -705,7 +703,7 @@ def test_parse():  # {{{
                break
        new_url, aname = parse_plugin_zip_url(raw)
        if new_url != full_url:
-            print('new url (%s): %s != %s for plugin at: %s' % (aname, new_url, full_url, url))
+            print(f'new url ({aname}): {new_url} != {full_url} for plugin at: {url}')
            raise SystemExit(1)

 # }}}
--- a/setup/publish.py
+++ b/setup/publish.py
@ -1,5 +1,4 @@
 #!/usr/bin/env python
-# vim:fileencoding=UTF-8:ts=4:sw=4:sta:et:sts=4:ai


 __license__ = 'GPL v3'
@ -225,7 +224,7 @@ def replace_with_symlinks(self, lang_dir):
            orig = self.j(self.d(base), r)
            try:
                sz = os.stat(orig).st_size
-            except EnvironmentError:
+            except OSError:
                continue
            if sz == os.stat(f).st_size and filecmp._do_cmp(f, orig):
                os.remove(f)
@ -260,7 +259,7 @@ def build_man_pages(self, dest, compress=False):
        os.environ['ALL_USER_MANUAL_LANGUAGES'] = ' '.join(languages)
        try:
            os.makedirs(dest)
-        except EnvironmentError:
+        except OSError:
            pass
        jobs = []
        for l in languages:
@ -268,7 +267,7 @@ def build_man_pages(self, dest, compress=False):
                [sys.executable, self.j(base, 'build.py'), '--man-pages', l, dest],
                '\n\n**************** Building translations for: %s' % l)
            )
-        self.info('\tCreating man pages in {} for {} languages...'.format(dest, len(jobs)))
+        self.info(f'\tCreating man pages in {dest} for {len(jobs)} languages...')
        subprocess.check_call(jobs[0].cmd)
        if not parallel_build(jobs[1:], self.info, verbose=False):
            raise SystemExit(1)
@ -307,4 +306,4 @@ def run(self, opts):
        subprocess.check_call(
            'git tag -s v{0} -m "version-{0}"'.format(__version__).split()
        )
-        subprocess.check_call('git push origin v{0}'.format(__version__).split())
+        subprocess.check_call(f'git push origin v{__version__}'.split())
--- a/setup/resources.py
+++ b/setup/resources.py
@ -1,5 +1,4 @@
 #!/usr/bin/env python
-# vim:fileencoding=UTF-8:ts=4:sw=4:sta:et:sts=4:ai


 __license__   = 'GPL v3'
@ -16,17 +15,13 @@

 def get_opts_from_parser(parser):
    def do_opt(opt):
-        for x in opt._long_opts:
-            yield x
-        for x in opt._short_opts:
-            yield x
+        yield from opt._long_opts
+        yield from opt._short_opts
    for o in parser.option_list:
-        for x in do_opt(o):
-            yield x
+        yield from do_opt(o)
    for g in parser.option_groups:
        for o in g.option_list:
-            for x in do_opt(o):
-                yield x
+            yield from do_opt(o)


 class Kakasi(Command):  # {{{
@ -145,7 +140,7 @@ def run(self, opts):
        try:
            with open(self.CA_PATH, 'rb') as f:
                raw = f.read()
-        except EnvironmentError as err:
+        except OSError as err:
            if err.errno != errno.ENOENT:
                raise
            raw = b''
@ -298,7 +293,7 @@ def run(self, opts):
            except Exception:
                continue
            src = src.replace('def ' + func.__name__, 'def replace')
-            imports = ['from %s import %s' % (x.__module__, x.__name__) for x in func.imports]
+            imports = [f'from {x.__module__} import {x.__name__}' for x in func.imports]
            if imports:
                src = '\n'.join(imports) + '\n\n' + src
            function_dict[func.name] = src
--- a/setup/revendor.py
+++ b/setup/revendor.py
@ -1,5 +1,4 @@
 #!/usr/bin/env python
-# vim:fileencoding=utf-8
 # License: GPL v3 Copyright: 2019, Eli Schwartz <eschwartz@archlinux.org>

 import os
--- a/setup/run-calibre-worker.py
+++ b/setup/run-calibre-worker.py
@ -1,5 +1,4 @@
 #!/usr/bin/env python
-# vim:fileencoding=utf-8
 # License: GPLv3 Copyright: 2016, Kovid Goyal <kovid at kovidgoyal.net>


--- a/Show more
+++ b/Show more