Commit graph

227 commits

Author SHA1 Message Date
FichteFoll
1a23eab8b6
Use https for lyrics.wikia.com, when supported 2019-06-05 23:00:52 +02:00
Carl Suster
e4c03fd63f Fix deprecated placement of inline regex flags
https://bugs.python.org/issue22493
2019-03-31 19:44:49 +11:00
Adrian Sampson
bac8faad78 Resolve W605: invalid escape sequence
This came up in lots of regexes that weren't using "raw" literals.
2018-08-13 10:41:01 -04:00
Abra
3348a466f4 Make lyrics plugin group songs by 'albumartist' rather than 'artist'
when writing ReST
2018-05-14 11:18:34 +04:00
rachmadaniHaryono
e90a547642 chg: dev: fix list item remove error 2018-05-03 09:46:03 +08:00
Adrian Sampson
224d782c2c Fix #2771: handle errors in genius lyrics source 2018-01-30 22:37:44 -05:00
Adrian Sampson
277d81b4d6 lyrics: Don't write ReST by default! 2018-01-30 22:33:32 -05:00
Adrian Sampson
e7417e3683 lyrics: Don't crash when BeautifulSoup isn't found 2018-01-30 22:31:15 -05:00
Lucas Magno
fc2d379fb5 Comply with PEP8 2017-10-09 06:22:42 -03:00
Lucas Magno
1b35a5df0d Fetch lyrics from Genius through scraper 2017-10-08 09:13:51 -03:00
Adrian Sampson
c06eca7e58 Merge pull request #2634 from anarcat/musixmatch-block-detect
lyrics: detect MusixMatch blocking
2017-07-18 17:13:31 -04:00
Antoine Beaupré
5ef68783a8
strip trailing and leading extra dashes
those are introduced if non-word characters are found, and are ugly
2017-07-18 16:33:22 -04:00
Antoine Beaupré
b4b5473093
add pointer to slugify in slug 2017-07-18 16:14:10 -04:00
Antoine Beaupré
a8afabea80
move slug utility function to top-level
it's a generic utility function that can be reused, there's nothing
class-specific about it.
2017-07-18 16:12:48 -04:00
Antoine Beaupré
5e8d17a4fc
lyrics: detect MusixMatch blocking
we just look for the bad string in the HTML. this has the downside
that we may consider songs that have those exact lyrics (you never
know, really) may trigger this warning as well and we would fail to
fetch those songs.

we also fail if lyrics contain another magic string that seems to come
up when you do fill in the CAPTCHA after being blocked.
2017-07-17 12:21:55 -04:00
Antoine Beaupré
458f3636f4
compare artists based on the slug
this is necessary because otherwise artists with different string
representations but the same slug would overwrite one another

this outlines more clearly the code duplication between the rst code
and the slugify function, something which can be fixed later.
2017-07-17 11:59:14 -04:00
Antoine Beaupré
9c36a41ea8
slight refactoring: strip album only once 2017-07-17 11:50:15 -04:00
Antoine Beaupré
9894e8752b
ignore trailing/leading whitespace when comparing artists 2017-07-17 11:49:35 -04:00
Antoine Beaupré
36f84bfedd
add missing trailing newline after lyrics block
this would yield a warning for every song
2017-07-17 11:44:06 -04:00
Adrian Sampson
b303d5beb0 Slightly more complete sentences in comments 2017-07-17 10:59:04 -04:00
Antoine Beaupré
93966ed4ee
strip whitespace in titles
this would cause problems with songs that had trailing spaces with the
index directive
2017-07-17 09:00:22 -04:00
Antoine Beaupré
b6e42ee2e8
fix another unicode error
the unicode strings are not binary - rely on Python to do the right
thing here instead of encoding a string we know is already properly
encoded
2017-07-17 08:55:09 -04:00
Antoine Beaupré
6d58110bd2
move heredocs to top-level globals 2017-07-17 08:50:19 -04:00
Antoine Beaupré
7e0a48a46d
s/rest/rest/ 2017-07-17 08:49:40 -04:00
Adrian Sampson
9de94378b9 An even shorter metavariable 2017-07-16 10:14:49 -04:00
Adrian Sampson
813cf97686 Better metavariable for lyrics --help output 2017-07-16 10:10:41 -04:00
Antoine Beaupré
5d8c15980e
fix flake8 warning 2017-07-15 16:24:07 -04:00
Antoine Beaupré
0bcd16f1ab
deal with encoding issues in python3
when we encode explicitly, we return bytes, so open files as binary
2017-07-15 16:21:41 -04:00
Antoine Beaupré
f667428758
write sphinx base files
we write the artists files in a subdirectory, to avoid infinite
recursions or flooding the current directory needlessly.

this way, the user has a good base structure and can just chain the
command into sphinx to continue building the next format, after
possible tweaks.
2017-07-15 15:43:16 -04:00
Antoine Beaupré
e6adb5e7da
cosmetic: do not use needless heredoc 2017-07-15 15:33:35 -04:00
Antoine Beaupré
91de8aac84
move rst writer to a different function
this simplifies and clarifies the code, although we need to call the writerst function twice to wrap up at the end of the loop
2017-07-15 15:33:35 -04:00
Antoine Beaupré
d330353e1c
rename the skip option to local
skip was a misnomer: we actually skip "unfetched" lyrics. this means
it's somewhat of a double-negative and really confusing.

--local is clearer, although less in opposition with --force
2017-07-15 14:19:25 -04:00
Antoine Beaupré
ac32ae574c
optimize: write only 3 times per file
this makes the code more readable and reduces the number of syscalls
to write files
2017-07-15 09:23:59 -04:00
Antoine Beaupré
469c03a7bf
deal properly with empty album titles 2017-07-15 09:23:59 -04:00
Antoine Beaupré
63aa3b3165
write to separate rst files
this makes the ePUB easier to parse by e-readers, because they do not
need to load one giant HTML file, but one per author. it also makes
sphinx rendering more efficient and interactive
2017-07-15 09:23:58 -04:00
Antoine Beaupré
0fbfa1feae
render RST instead of HTML
ReStructuredText has the advantage over HTML that it can be rendered
easily to multiple formats (HTML, ePUB, PDF) and it supports indexes.

the output needs to be fed into a file and integrated into an existing
Sphinx document, of course.
2017-07-14 17:34:55 -04:00
Antoine Beaupré
9f3e5b28b4
output lyrics in HTML, allow skipping
the idea here is to format the lyrics output a little better so that
it can (for example) be shown as a web page or an ebook.

the new skip option allows for faster generation of the output in the
(most common) case where not all lyrics are available.
2017-07-14 15:31:22 -04:00
Fabrice Laporte
409f070970 Remove lyrics.com source 2017-05-03 22:54:09 +02:00
Fabrice Laporte
7dab9f339e Restore beets module import 2017-05-02 23:48:20 +02:00
Fabrice Laporte
07af27e44b Lyrics are last paragraph with class 'mxm-lyrics__content'
Remove ‘data-reactid’ from marker.
2017-05-02 23:40:25 +02:00
Fabrice Laporte
3e38a33c4a Fix PEP8 2017-05-02 23:37:20 +02:00
Fabrice Laporte
11eb90c758 Fix PEP8 2017-05-02 07:46:51 +02:00
Fabrice Laporte
3e3ad6974c Fix PEP8 2017-05-02 07:30:40 +02:00
Fabrice Laporte
a165d6c00b Fix MusiXmatch text extraction markers 2017-05-01 23:40:09 +02:00
Fabrice Laporte
2bf58a61c3 Decode string with Unicode escape 2017-04-30 23:14:23 +02:00
Adrian Sampson
0a4709f7ef lyrics: Tolerate empty Google response (#2437) 2017-02-13 16:54:56 -05:00
Adrian Sampson
8087e82891 lyrics: Use Requests for Google backend (fix #2437) 2017-02-12 10:30:22 -05:00
Adrian Sampson
d389ac15e1 Use HTTPS for MS translator API (from #2247) 2017-01-02 21:00:01 -05:00
Adrian Sampson
fbc0f322f6 Merge branch 'tigranl-https_fix' 2017-01-02 20:54:17 -05:00
Adrian Sampson
f941fd42de Always use SSL on servers that don't require SNI
I did a little audit using the `openssl` command-line tool to find the servers
that don't require SNI. Here's what I found:

icbrainz.org: SNI
images.weserv.nl: inconclusive, but docs say yes SNI
coverartarchive.org: SNI
webservice.fanart.tv: *no* SNI
dbpedia.org: *no* SNI
en.wikipedia.org: *no* SNI
ws.audioscrobbler.com: *no* SNI
api.microsofttranslator.com: *no* SNI

In summary, *only* MusicBrainz and CoverArtArchive were found to require SNI.
So I'm using SSL unconditionally on all the other sites.
2017-01-02 20:39:10 -05:00