Fixes#5229, is part of #5361 and relates to #5285.
I have to admit thsi was a fairly tough task - I initially assumed that
the problem lies
with how the tests are setup, and that we're probably missing some
`teardown_beets` calls
here and there.
Unfortunately, it was not so simple. I came across several issues that
gave rise to
leftover temporary files:
1. `fetchart`, `artresizer` and `play` handling of temporary files.
These plugins created
isolated temporary files outside of the directories that tests clean up.
You will find
I added a couple of functions (namely `get_module_tempdir`) that force
these plugins to
create files in directories determined by their module names. This way
we can clean up
after them using the new `CleanupModulesMixin`.
2. Tests that ran temporary directories setup twice, running
`_common.TestCase.setUp` and
`test.helper.TestHelper.setup_beets`. Both of these ran `self.temp_dir =
mkdtemp()`,
therefore the directories created by the initial setup persisted since
those have been
overridden and thus unreachable in the teardown. Here, I removed the
`setUp` calls, see
- `test/plugins/test_embedart.py`
- `test/test_importer.py`
- and `test/test_plugins.py` where `setup_beets` was called twice
3. `test/test_config_command.py` attempted to manage the temporary
directory by itself,
where I found that `tearDown` failed to remove the directory for four
tests. Could not
figure out the cause, and found that delegating this task to
`TestHelper` fixed the
issue.
4. Mediafile fixture removal depended on calling
`remove_mediafile_fixtures` method, which
`test/plugins/test_zero.py` failed to do. I made the fixtures to be
created within the
same `temp_dir` directory that gets removed in the teardown, so now they
are taken care
of automatically.
In summary, see the test modules that left files behind:
```
Temp files created by test/__init__.py
Temp files created by test/plugins/__init__.py
Temp files created by test/plugins/lyrics_download_samples.py
Temp files created by test/plugins/test_acousticbrainz.py
Temp files created by test/plugins/test_advancedrewrite.py
Temp files created by test/plugins/test_albumtypes.py
Temp files created by test/plugins/test_art.py
/tmp/tmp11nicahe.jpg
/tmp/tmp1bjmodum.png
/tmp/tmped7nhls4.jpg
/tmp/tmpflnzr9wz.jpg
/tmp/tmpjngkauqs.png
/tmp/tmpkzy9mn6t.jpg
/tmp/tmpph_wmuea.jpg
/tmp/tmps6gk58i_.jpg
/tmp/tmpz2eji_o4.jpg
Temp files created by test/plugins/test_aura.py
Temp files created by test/plugins/test_bareasc.py
/tmp/tmphl3kzhug
/tmp/tmpnh2q6v02
/tmp/tmpppw5qrhz
Temp files created by test/plugins/test_beatport.py
Temp files created by test/plugins/test_bucket.py
Temp files created by test/plugins/test_convert.py
Temp files created by test/plugins/test_discogs.py
Temp files created by test/plugins/test_edit.py
Temp files created by test/plugins/test_embedart.py
/tmp/tmp1ayvqzhx
/tmp/tmp58k6mdfx.jpg
/tmp/tmp64c2lqiv
/tmp/tmp6nar4kr5
/tmp/tmp6u0d5dex
/tmp/tmpacoq7w_f
/tmp/tmpajnr_sxr
/tmp/tmpasj16beh
/tmp/tmpboyaixb5
/tmp/tmpcrmcyt5r
/tmp/tmpdomje5g3
/tmp/tmplu3o6t6g
/tmp/tmpns_xvkns
/tmp/tmpo87o1h6o.jpg
/tmp/tmpqem39h_j
/tmp/tmprlzm18pb
/tmp/tmpt22v4u6x
/tmp/tmptp3rxdgv
Temp files created by test/plugins/test_embyupdate.py
Temp files created by test/plugins/test_export.py
Temp files created by test/plugins/test_fetchart.py
Temp files created by test/plugins/test_filefilter.py
Temp files created by test/plugins/test_ftintitle.py
Temp files created by test/plugins/test_hook.py
Temp files created by test/plugins/test_ihate.py
Temp files created by test/plugins/test_importadded.py
Temp files created by test/plugins/test_importfeeds.py
Temp files created by test/plugins/test_info.py
Temp files created by test/plugins/test_ipfs.py
Temp files created by test/plugins/test_keyfinder.py
Temp files created by test/plugins/test_lastgenre.py
Temp files created by test/plugins/test_limit.py
Temp files created by test/plugins/test_lyrics.py
Temp files created by test/plugins/test_mbsubmit.py
Temp files created by test/plugins/test_mbsync.py
Temp files created by test/plugins/test_mpdstats.py
Temp files created by test/plugins/test_parentwork.py
Temp files created by test/plugins/test_permissions.py
Temp files created by test/plugins/test_player.py
Temp files created by test/plugins/test_playlist.py
Temp files created by test/plugins/test_play.py
/tmp/tmp6ohknmve.m3u
/tmp/tmp8rw2z_j4.m3u
/tmp/tmp9vi27ypx.m3u
/tmp/tmpa_s66jh8.m3u
/tmp/tmpb7h3cn3n.m3u
/tmp/tmpexbmqvry.m3u
/tmp/tmpinbqrt80.m3u
/tmp/tmpql02hax5.m3u
/tmp/tmpvbdzprsf.m3u
/tmp/tmpzipim36x.m3u
Temp files created by test/plugins/test_plexupdate.py
Temp files created by test/plugins/test_plugin_mediafield.py
Temp files created by test/plugins/test_random.py
Temp files created by test/plugins/test_replaygain.py
Temp files created by test/plugins/test_smartplaylist.py
Temp files created by test/plugins/test_spotify.py
Temp files created by test/plugins/test_subsonicupdate.py
Temp files created by test/plugins/test_the.py
Temp files created by test/plugins/test_thumbnails.py
Temp files created by test/plugins/test_types_plugin.py
Temp files created by test/plugins/test_web.py
Temp files created by test/plugins/test_zero.py
/tmp/tmp3ub9xmzy
Temp files created by test/rsrc/beetsplug/test.py
Temp files created by test/rsrc/convert_stub.py
Temp files created by test/testall.py
Temp files created by test/test_art_resize.py
/tmp/tmp3p7p60ih.jpg
/tmp/tmp8exclgit.jpg
/tmp/tmpkrrjsitl.jpg
/tmp/tmpw6n8ee8e.jpg
/tmp/tmpygws_0aw.jpg
Temp files created by test/test_autotag.py
Temp files created by test/test_config_command.py
/tmp/tmp333f0r2j
/tmp/tmphr356z5r
/tmp/tmporp4rag2
/tmp/tmpy7sjqdsw
Temp files created by test/test_datequery.py
Temp files created by test/test_dbcore.py
Temp files created by test/test_files.py
Temp files created by test/test_hidden.py
Temp files created by test/test_importer.py
/tmp/tmp0m363gfb
/tmp/tmp2n3i13mc
/tmp/tmpxk3v304s
Temp files created by test/test_library.py
Temp files created by test/test_logging.py
Temp files created by test/test_m3ufile.py
Temp files created by test/test_mb.py
Temp files created by test/test_metasync.py
Temp files created by test/test_pipeline.py
Temp files created by test/test_plugins.py
/tmp/tmp6pxhx67u
/tmp/tmpb8pqi9ui
/tmp/tmpcx_658g7
/tmp/tmp_giqb9jz
/tmp/tmpgm9xk94_
/tmp/tmpk60l6bt3
/tmp/tmpqoj4la68
/tmp/tmptcdu20rp
/tmp/tmpvr7k5shn
/tmp/tmpwnfnzs91
Temp files created by test/test_query.py
Temp files created by test/test_sort.py
Temp files created by test/test_template.py
Temp files created by test/test_ui_commands.py
/tmp/tmpns2u94w6
Temp files created by test/test_ui_importer.py
Temp files created by test/test_ui_init.py
Temp files created by test/test_ui.py
Temp files created by test/test_util.py
Temp files created by test/test_vfs.py
```
And that's what we have right now:
```
Temp files created by test/__init__.py
Temp files created by test/plugins/__init__.py
Temp files created by test/plugins/lyrics_download_samples.py
Temp files created by test/plugins/test_acousticbrainz.py
Temp files created by test/plugins/test_advancedrewrite.py
Temp files created by test/plugins/test_albumtypes.py
Temp files created by test/plugins/test_art.py
Temp files created by test/plugins/test_aura.py
Temp files created by test/plugins/test_bareasc.py
Temp files created by test/plugins/test_beatport.py
Temp files created by test/plugins/test_bucket.py
Temp files created by test/plugins/test_convert.py
Temp files created by test/plugins/test_discogs.py
Temp files created by test/plugins/test_edit.py
Temp files created by test/plugins/test_embedart.py
Temp files created by test/plugins/test_embyupdate.py
Temp files created by test/plugins/test_export.py
Temp files created by test/plugins/test_fetchart.py
Temp files created by test/plugins/test_filefilter.py
Temp files created by test/plugins/test_ftintitle.py
Temp files created by test/plugins/test_hook.py
Temp files created by test/plugins/test_ihate.py
Temp files created by test/plugins/test_importadded.py
Temp files created by test/plugins/test_importfeeds.py
Temp files created by test/plugins/test_info.py
Temp files created by test/plugins/test_ipfs.py
Temp files created by test/plugins/test_keyfinder.py
Temp files created by test/plugins/test_lastgenre.py
Temp files created by test/plugins/test_limit.py
Temp files created by test/plugins/test_lyrics.py
Temp files created by test/plugins/test_mbsubmit.py
Temp files created by test/plugins/test_mbsync.py
Temp files created by test/plugins/test_mpdstats.py
Temp files created by test/plugins/test_parentwork.py
Temp files created by test/plugins/test_permissions.py
Temp files created by test/plugins/test_player.py
Temp files created by test/plugins/test_playlist.py
Temp files created by test/plugins/test_play.py
Temp files created by test/plugins/test_plexupdate.py
Temp files created by test/plugins/test_plugin_mediafield.py
Temp files created by test/plugins/test_random.py
Temp files created by test/plugins/test_replaygain.py
Temp files created by test/plugins/test_smartplaylist.py
Temp files created by test/plugins/test_spotify.py
Temp files created by test/plugins/test_subsonicupdate.py
Temp files created by test/plugins/test_the.py
Temp files created by test/plugins/test_thumbnails.py
Temp files created by test/plugins/test_types_plugin.py
Temp files created by test/plugins/test_web.py
Temp files created by test/plugins/test_zero.py
Temp files created by test/rsrc/beetsplug/test.py
Temp files created by test/rsrc/convert_stub.py
Temp files created by test/testall.py
Temp files created by test/test_art_resize.py
Temp files created by test/test_autotag.py
Temp files created by test/test_config_command.py
Temp files created by test/test_datequery.py
Temp files created by test/test_dbcore.py
Temp files created by test/test_files.py
Temp files created by test/test_hidden.py
Temp files created by test/test_importer.py
Temp files created by test/test_library.py
Temp files created by test/test_logging.py
Temp files created by test/test_m3ufile.py
Temp files created by test/test_mb.py
Temp files created by test/test_metasync.py
Temp files created by test/test_pipeline.py
Temp files created by test/test_plugins.py
Temp files created by test/test_query.py
Temp files created by test/test_sort.py
Temp files created by test/test_template.py
Temp files created by test/test_ui_commands.py
Temp files created by test/test_ui_importer.py
Temp files created by test/test_ui_init.py
Temp files created by test/test_ui.py
Temp files created by test/test_util.py
Temp files created by test/test_vfs.py
```
Note that the command which provides the output is now available through
`poe`.
Due to some weird race conditions the temporary directories were not
getting torn down for four of the tests. I failed to figure out why, and
I found that using TestHelper to manage the temporary directory somehow
fixes this.
This way they get automatically removed with removal of temp_dir.
After this change `test/plugins/test_zero.py` does not any more leave
any mediafield fixtures hanging around.
And move the definition to a shared module.
The problem was that EmbedartCliTest ran `_common.TestCase.setUp`
method which initialised temporary directory for the tests AND ran
`helper.TestHelper.setup_beets` method which initialised another set of
temporary directories. This meant that the first set of directories
could not be tracked down for the cleanup.
This check existed in the original implementation, but I had removed it
due to a misinterpretation of the 'os' documentation, thinking that the
field was guaranteed to exist.
Fixes#4360
This PR enables querying albums by track fields and tracks by album
fields, and speeds up querying albums by `path` field.
It originally was part of #5240, however we found that the changes
related to the flexible attributes caused degradation in performance. So
this PR contains the first part of #5240 which joined `items` and
`albums` tables in queries.
Fixes: #5325
As @arsaboo underlined, this change makes interaction with `beets`
unbearably slow for systems that have SQLite version below `3.38.0`.
This is what Ubuntu ships with by default.
I have also tested other commands, like `beet update`, and found that
even with up to date SQLite version, this command takes twice as long to
execute as before.
Thus, I think that the performance gain we see using some of the more
complex queries is not worth the performance loss for the most common
operations. Therefore, at least for now, I am reverting this change an
leaving this work on the `only-fast-filtering` branch to be revised.
Another and (hopefully) final attempt to improve querying speed.
Fixes#4360Fixes#3515
and possibly more issues to do with slow queries.
This PR supersedes #4746.
## What's been done
The `album` and `item` tables are joined, and corresponding data from
`item_attributes` and `album_attributes` is merged and made available
for filtering. This enables to achieve the following:
- [x] Faster album path queries, `beet list -a path::some/path`
- [x] Faster flexible attributes queries, both albums and tracks, `beet
list play_count:10`
- [x] (New) Ability to filter albums with track-level (and vice-versa)
**db** field queries, `beet list -a title:something`
- [x] (New) Ability to filter tracks with album-level **flexible** field
queries, `beet list artpath:cover`
- [x] (New) Ability to filter albums with track-level **flexible** field
queries, `beet list -a art_source:something`
## Benchmarks
You can see that now querying speed is more or less constant regardless
of the query, and the speed is mostly influenced by how many results
need to be printed out
Compare this with what we had previously
## Later
https://github.com/beetbox/beets/issues/5318https://github.com/beetbox/beets/issues/5319
