Fixes#4360
This PR enables querying albums by track fields and tracks by album
fields, and speeds up querying albums by `path` field.
It originally was part of #5240, however we found that the changes
related to the flexible attributes caused degradation in performance. So
this PR contains the first part of #5240 which joined `items` and
`albums` tables in queries.
Fixes: #5325
As @arsaboo underlined, this change makes interaction with `beets`
unbearably slow for systems that have SQLite version below `3.38.0`.
This is what Ubuntu ships with by default.
I have also tested other commands, like `beet update`, and found that
even with up to date SQLite version, this command takes twice as long to
execute as before.
Thus, I think that the performance gain we see using some of the more
complex queries is not worth the performance loss for the most common
operations. Therefore, at least for now, I am reverting this change an
leaving this work on the `only-fast-filtering` branch to be revised.
Another and (hopefully) final attempt to improve querying speed.
Fixes#4360Fixes#3515
and possibly more issues to do with slow queries.
This PR supersedes #4746.
## What's been done
The `album` and `item` tables are joined, and corresponding data from
`item_attributes` and `album_attributes` is merged and made available
for filtering. This enables to achieve the following:
- [x] Faster album path queries, `beet list -a path::some/path`
- [x] Faster flexible attributes queries, both albums and tracks, `beet
list play_count:10`
- [x] (New) Ability to filter albums with track-level (and vice-versa)
**db** field queries, `beet list -a title:something`
- [x] (New) Ability to filter tracks with album-level **flexible** field
queries, `beet list artpath:cover`
- [x] (New) Ability to filter albums with track-level **flexible** field
queries, `beet list -a art_source:something`
## Benchmarks
You can see that now querying speed is more or less constant regardless
of the query, and the speed is mostly influenced by how many results
need to be printed out
Compare this with what we had previously
## Later
https://github.com/beetbox/beets/issues/5318https://github.com/beetbox/beets/issues/5319
While the commit merged yesterday fixed the coverage upload to
coveralls, it broke the source files preview on the platform.
This was due to the missing actions/checkout step in the coverage upload
job.
In order to include the table name for fields in this query, use the
`field_query` method.
Since `AnyFieldQuery` is just an `OrQuery` under the hood, remove it and
construct `OrQuery` explicitly instead.
It seems like previously filtering by flexible attributes did not work
- I'd receive '{"data": []}' trying to GET `/aura/tracks?filter[play_count]=11`
Now this works, not only for tracks, but for `/aura/artists` and
`/aura/albums` too.
Additionally, this improves `/aura/tracks` response time significantly.
I tried loading the default of 500 tracks from my library:
On `master`, it took ~20s
After this commit, it takes under 1s.
Unify query creation logic from
- queryparse.py:construct_query_part,
- Model.field_query,
- DefaultTemplateFunctions._tmpl_unique
to a single implementation under `LibModel.field_query` class method.
This method should be used for query resolution for model (flex)fields.
Allow filtering item attributes in album queries and vice versa by
merging `flex_attrs` from Album and Item together as `all_flex_attrs`.
This field is only used for filtering and is discarded after.
For a flexible attribute query, replace the `col_name` property with
a function call that extracts that attribute from the `field_attrs`
field introduced in the earlier commit.
Additionally, for boolean, numeric and date queries CAST the value to
NUMERIC SQLite affinity to ensure that our queries like 'flex:1..5' and
'flex:true' continue working fine.
This removes the concept of 'slow query', since every query for any
field now has an SQL clause.
Use `json_group_object` SQLite function to aggregate flexible attributes
into `flex_attrs` field.
Register SQLite converter `json.loads` to automatically convert the JSON
string to a Python dictionary. Remove the code that had this task
previously.
Replace py3_path by standard os.fsdecode and use native os.cpu_count.
These functions had been written before builtins achieving the same have
been made available (in Python 3.2 and 3.4).
Now that they are available though, this PR updates the codebase to use
builtin implementations rather than the custom ones.
