From 0e65800fbc1485e84ea74091b9a4f03f2b8040ba Mon Sep 17 00:00:00 2001 From: Carl Suster Date: Tue, 4 Jun 2019 13:18:36 +1000 Subject: [PATCH 1/6] Expand library API docs --- docs/dev/api.rst | 221 +++++++++++++++++++++++++++++++++++++---------- 1 file changed, 177 insertions(+), 44 deletions(-) diff --git a/docs/dev/api.rst b/docs/dev/api.rst index d9e68481d..09b9b2f85 100644 --- a/docs/dev/api.rst +++ b/docs/dev/api.rst @@ -1,12 +1,11 @@ -API Documentation -================= +Library Database API +==================== .. currentmodule:: beets.library -This page describes the internal API of beets' core. It's a work in -progress---since beets is an application first and a library second, its API -has been mainly undocumented until recently. Please file bugs if you run -across incomplete or incorrect docs here. +This page describes the internal API of beets' core database features. It +doesn't exhaustively document the API, but is aimed at giving an overview of +the architecture to orient anyone who wants to dive into the code. The :class:`Library` object is the central repository for data in beets. It represents a database containing songs, which are :class:`Item` instances, and @@ -15,8 +14,24 @@ groups of items, which are :class:`Album` instances. The Library Class ----------------- +The :class:`Library` is typically instantiated as a singleton. A single +invocation of beets usually has only one :class:`Library`. It's powered by +:class:`dbcore.Database` under the hood, which handles the `SQLite`_ +abstraction, something like a very minimal `ORM`_. The library is also +responsible for handling queries to retrieve stored objects. + .. autoclass:: Library(path, directory[, path_formats[, replacements]]) + .. automethod:: __init__ + + You can add new items or albums to the library: + + .. automethod:: add + + .. automethod:: add_album + + And there are methods for querying the database: + .. automethod:: items .. automethod:: albums @@ -25,60 +40,178 @@ The Library Class .. automethod:: get_album - .. automethod:: add - - .. automethod:: add_album + Any modifications must go through a :class:`Transaction` which you get can + using this method: .. automethod:: transaction +.. _SQLite: http://sqlite.org/ +.. _ORM: http://en.wikipedia.org/wiki/Object-relational_mapping + + +Model Classes +------------- + +The two model entities in beets libraries, :class:`Item` and :class:`Album`, +share a base class, :class:`LibModel`, that provides common functionality and +ORM-like abstraction. + +Model base +'''''''''' + +Models use dirty-flags to track when the object's metadata goes out of +sync with the database. The dirty dictionary maps field names to booleans +indicating whether the field has been written since the object was last +synchronized (via load or store) with the database. + +.. autoclass:: LibModel + + .. automethod:: all_keys + + .. automethod:: __init__ + + .. autoattribute:: _types + + .. autoattribute:: _fields + + There are CRUD-like methods for interacting with the database: + + .. automethod:: store + + .. automethod:: load + + .. automethod:: remove + + .. automethod:: add + + The fields model classes can be accessed using attributes (dots, as in + ``item.artist``) or items (brackets, as in ``item['artist']``). + The base class :class:`dbcore.Model` has a ``dict``-like interface, so + normal the normal mapping API is supported: + + .. automethod:: keys + + .. automethod:: update + + .. automethod:: items + + .. automethod:: get + +Item +'''' + +Each :class:`Item` object represents a song or track. (We use the more generic +term item because, one day, beets might support non-music media.) An item can +either be purely abstract, in which case it's just a bag of metadata fields, +or it can have an associated file (indicated by ``item.path``). + +In terms of the underlying SQLite database, items are backed by a single table +called items with one column per metadata fields. The metadata fields currently +in use are listed in ``library.py`` in ``Item._fields``. + +To read and write a file's tags, we use the `MediaFile`_ library. +To make changes to either the database or the tags on a file, you +update an item's fields (e.g., ``item.title = "Let It Be"``) and then call +``item.write()``. + +.. _MediaFile: http://mediafile.readthedocs.io/ + +.. autoclass:: Item + + .. automethod:: __init__ + + .. automethod:: from_path + + .. automethod:: get_album + + .. automethod:: destination + + The methods ``read()`` and ``write()`` are complementary: one reads a + file's tags and updates the item's metadata fields accordingly while the + other takes the item's fields and writes them to the file's tags. + + .. automethod:: read + + .. automethod:: write + + .. automethod:: try_write + + .. automethod:: try_sync + + The :class:`Item` class supplements the normal model interface so that they + interacting with the filesystem as well: + + .. automethod:: move + + .. automethod:: remove + + Items also track their modification times (mtimes) to help detect when they + become out of sync with on-disk metadata. + + .. automethod:: current_mtime + +Album +''''' + +An :class:`Album` is a collection of Items in the database. Every item in the +database has either zero or one associated albums (accessible via +``item.album_id``). An item that has no associated album is called a +singleton. + +An :class:`Album` object keeps track of album-level metadata, which is (mostly) +a subset of the track-level metadata. The album-level metadata fields are +listed in ``Album._fields``. +For those fields that are both item-level and album-level (e.g., ``year`` or +``albumartist``), every item in an album should share the same value. Albums +use an SQLite table called ``albums``, in which each column is an album +metadata field. + +.. autoclass:: Album + + .. automethod:: __init__ + + .. automethod:: item_dir + + To get or change an album's metadata, use its fields (e.g., + ``print(album.year)`` or ``album.year = 2012``). Changing fields in this + way updates the album itself and also changes the same field in all + associated items: + + .. autoattribute:: item_keys + + .. automethod:: store + + .. automethod:: try_sync + + .. automethod:: move + + .. automethod:: remove + + Albums also manage album art, image files that are associated with each + album: + + .. automethod:: set_art + + .. automethod:: move_art + + .. automethod:: art_destination + Transactions '''''''''''' The :class:`Library` class provides the basic methods necessary to access and manipulate its contents. To perform more complicated operations atomically, or to interact directly with the underlying SQLite database, you must use a -*transaction*. For example:: +*transaction* (see this `blog post`_ for motivation). For example:: lib = Library() with lib.transaction() as tx: items = lib.items(query) lib.add_album(list(items)) +.. _blog post: http://beets.io/blog/sqlite-nightmare.html + .. currentmodule:: beets.dbcore.db .. autoclass:: Transaction :members: - -Model Classes -------------- - -The two model entities in beets libraries, :class:`Item` and :class:`Album`, -share a base class, :class:`Model`, that provides common functionality and -ORM-like abstraction. - -The fields model classes can be accessed using attributes (dots, as in -``item.artist``) or items (brackets, as in ``item['artist']``). The -:class:`Model` base class provides some methods that resemble `dict` -objects. - -Model base -'''''''''' - -.. currentmodule:: beets.dbcore - -.. autoclass:: Model - :members: - -Item -'''' - -.. currentmodule:: beets.library - -.. autoclass:: Item - :members: - -Album -''''' - -.. autoclass:: Album - :members: From 984aa223c69b8c0831231919647d3bba96cb4839 Mon Sep 17 00:00:00 2001 From: Carl Suster Date: Wed, 5 Jun 2019 13:03:36 +1000 Subject: [PATCH 2/6] docs: highlight model field API --- docs/dev/api.rst | 26 +++++++++++++++----------- 1 file changed, 15 insertions(+), 11 deletions(-) diff --git a/docs/dev/api.rst b/docs/dev/api.rst index 09b9b2f85..62dfd392c 100644 --- a/docs/dev/api.rst +++ b/docs/dev/api.rst @@ -45,16 +45,22 @@ responsible for handling queries to retrieve stored objects. .. automethod:: transaction -.. _SQLite: http://sqlite.org/ -.. _ORM: http://en.wikipedia.org/wiki/Object-relational_mapping +.. _SQLite: https://sqlite.org/ +.. _ORM: https://en.wikipedia.org/wiki/Object-relational_mapping Model Classes ------------- The two model entities in beets libraries, :class:`Item` and :class:`Album`, -share a base class, :class:`LibModel`, that provides common functionality and -ORM-like abstraction. +share a base class, :class:`LibModel`, that provides common functionality. That +class itself specialises :class:`dbcore.Model` which provides an ORM-like +abstraction. + +To get or change the metadata of a model (an item or album), either access its +attributes (e.g., ``print(album.year)`` or ``album.year = 2012``) or use the +``dict``-like interface (e.g. ``item['artist']``). + Model base '''''''''' @@ -84,8 +90,6 @@ synchronized (via load or store) with the database. .. automethod:: add - The fields model classes can be accessed using attributes (dots, as in - ``item.artist``) or items (brackets, as in ``item['artist']``). The base class :class:`dbcore.Model` has a ``dict``-like interface, so normal the normal mapping API is supported: @@ -114,7 +118,7 @@ To make changes to either the database or the tags on a file, you update an item's fields (e.g., ``item.title = "Let It Be"``) and then call ``item.write()``. -.. _MediaFile: http://mediafile.readthedocs.io/ +.. _MediaFile: https://mediafile.readthedocs.io/ .. autoclass:: Item @@ -157,6 +161,8 @@ An :class:`Album` is a collection of Items in the database. Every item in the database has either zero or one associated albums (accessible via ``item.album_id``). An item that has no associated album is called a singleton. +Changing fields on an album (e.g. ``album.year = 2012``) updates the album +itself and also changes the same field in all associated items. An :class:`Album` object keeps track of album-level metadata, which is (mostly) a subset of the track-level metadata. The album-level metadata fields are @@ -172,10 +178,8 @@ metadata field. .. automethod:: item_dir - To get or change an album's metadata, use its fields (e.g., - ``print(album.year)`` or ``album.year = 2012``). Changing fields in this - way updates the album itself and also changes the same field in all - associated items: + Albums extend the normal model interface to also forward changes to their + items: .. autoattribute:: item_keys From e27c6e480b4920ce4a7aa8c966e94fa3659a57e0 Mon Sep 17 00:00:00 2001 From: Carl Suster Date: Wed, 5 Jun 2019 13:10:10 +1000 Subject: [PATCH 3/6] docs: add query API reference --- docs/dev/api.rst | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/docs/dev/api.rst b/docs/dev/api.rst index 62dfd392c..6ccdfd57e 100644 --- a/docs/dev/api.rst +++ b/docs/dev/api.rst @@ -219,3 +219,32 @@ to interact directly with the underlying SQLite database, you must use a .. autoclass:: Transaction :members: + + +Queries +------- + +To access albums and items in a library, we use :doc:`/reference/query`. +In beets, the :class:`Query` abstract base class represents a criterion that +matches items or albums in the database. +Every subclass of :class:`Query` must implement two methods, which implement +two different ways of identifying matching items/albums. + +The ``clause()`` method should return an SQLite ``WHERE`` clause that matches +appropriate albums/items. This allows for efficient batch queries. +Correspondingly, the ``match(item)`` method should take an :class:`Item` object +and return a boolean, indicating whether or not a specific item matches the +criterion. This alternate implementation allows clients to determine whether +items that have already been fetched from the database match the query. + +There are many different types of queries. Just as an example, +:class:`FieldQuery` determines whether a certain field matches a certain value +(an equality query). +:class:`AndQuery` (like its abstract superclass, :class:`CollectionQuery`) +takes a set of other query objects and bundles them together, matching only +albums/items that match all constituent queries. + +Beets has a human-writable plain-text query syntax that can be parsed into +:class:`Query` objects. Calling ``AndQuery.from_strings`` parses a list of +query parts into a query object that can then be used with :class:`Library` +objects. From 918024a4658a82828b1e83a7e6a093d6c0f31fc9 Mon Sep 17 00:00:00 2001 From: Carl Suster Date: Wed, 5 Jun 2019 13:16:12 +1000 Subject: [PATCH 4/6] docs: document mtime management --- docs/dev/api.rst | 39 ++++++++++++++++++++++++++++++++++----- 1 file changed, 34 insertions(+), 5 deletions(-) diff --git a/docs/dev/api.rst b/docs/dev/api.rst index 6ccdfd57e..ec81c169b 100644 --- a/docs/dev/api.rst +++ b/docs/dev/api.rst @@ -120,6 +120,38 @@ update an item's fields (e.g., ``item.title = "Let It Be"``) and then call .. _MediaFile: https://mediafile.readthedocs.io/ +Items also track their modification times (mtimes) to help detect when they +become out of sync with on-disk metadata, mainly to speed up the +:ref:`update-cmd` (which needs to check whether the database is in sync with +the filesystem). This feature turns out to be sort of complicated. + +For any :class:`Item`, there are two mtimes: the on-disk mtime (maintained by +the OS) and the database mtime (maintained by beets). Correspondingly, there is +on-disk metadata (ID3 tags, for example) and DB metadata. The goal with the +mtime is to ensure that the on-disk and DB mtimes match when the on-disk and DB +metadata are in sync; this lets beets do a quick mtime check and avoid +rereading files in some circumstances. + +Specifically, beets attempts to maintain the following invariant: + + If the on-disk metadata differs from the DB metadata, then the on-disk + mtime must be greater than the DB mtime. + +As a result, it is always valid for the DB mtime to be zero (assuming that real +disk mtimes are always positive). However, whenever possible, beets tries to +set ``db_mtime = disk_mtime`` at points where it knows the metadata is +synchronized. When it is possible that the metadata is out of sync, beets can +then just set ``db_mtime = 0`` to return to a consistent state. + +This leads to the following implementation policy: + + * On every write of disk metadata (``Item.write()``), the DB mtime is updated + to match the post-write disk mtime. + * Same for metadata reads (``Item.read()``). + * On every modification to DB metadata (``item.field = ...``), the DB mtime + is reset to zero. + + .. autoclass:: Item .. automethod:: __init__ @@ -130,6 +162,8 @@ update an item's fields (e.g., ``item.title = "Let It Be"``) and then call .. automethod:: destination + .. automethod:: current_mtime + The methods ``read()`` and ``write()`` are complementary: one reads a file's tags and updates the item's metadata fields accordingly while the other takes the item's fields and writes them to the file's tags. @@ -149,11 +183,6 @@ update an item's fields (e.g., ``item.title = "Let It Be"``) and then call .. automethod:: remove - Items also track their modification times (mtimes) to help detect when they - become out of sync with on-disk metadata. - - .. automethod:: current_mtime - Album ''''' From de78151eea5928aab1bd434efb5792b4df9501dc Mon Sep 17 00:00:00 2001 From: Carl Suster Date: Wed, 5 Jun 2019 13:18:46 +1000 Subject: [PATCH 5/6] docs: rename api -> library --- docs/dev/index.rst | 2 +- docs/dev/{api.rst => library.rst} | 0 2 files changed, 1 insertion(+), 1 deletion(-) rename docs/dev/{api.rst => library.rst} (100%) diff --git a/docs/dev/index.rst b/docs/dev/index.rst index a47d6c8f2..ebeccc535 100644 --- a/docs/dev/index.rst +++ b/docs/dev/index.rst @@ -12,4 +12,4 @@ and write metadata tags in media files. .. toctree:: plugins - api + library diff --git a/docs/dev/api.rst b/docs/dev/library.rst similarity index 100% rename from docs/dev/api.rst rename to docs/dev/library.rst From 6769da29ae105d5d12fd71cb7b8eaba629742d6a Mon Sep 17 00:00:00 2001 From: Carl Suster Date: Wed, 5 Jun 2019 13:28:06 +1000 Subject: [PATCH 6/6] docs: add dev importer and cli text from wiki --- docs/dev/cli.rst | 9 +++++++++ docs/dev/importer.rst | 19 +++++++++++++++++++ docs/dev/index.rst | 2 ++ 3 files changed, 30 insertions(+) create mode 100644 docs/dev/cli.rst create mode 100644 docs/dev/importer.rst diff --git a/docs/dev/cli.rst b/docs/dev/cli.rst new file mode 100644 index 000000000..77d3af5a5 --- /dev/null +++ b/docs/dev/cli.rst @@ -0,0 +1,9 @@ +Providing a CLI +=============== + +The ``beets.ui`` module houses interactions with the user via a terminal, the +:doc:`/reference/cli`. +The main function is called when the user types beet on the command line. +The CLI functionality is organized into commands, some of which are built-in +and some of which are provided by plugins. The built-in commands are all +implemented in the ``beets.ui.commands`` submodule. diff --git a/docs/dev/importer.rst b/docs/dev/importer.rst new file mode 100644 index 000000000..5182c7134 --- /dev/null +++ b/docs/dev/importer.rst @@ -0,0 +1,19 @@ +Music Importer +============== + +The importer component is responsible for the user-centric workflow that adds +music to a library. This is one of the first aspects that a user experiences +when using beets: it finds music in the filesystem, groups it into albums, +finds corresponding metadata in MusicBrainz, asks the user for intervention, +applies changes, and moves/copies files. A description of its user interface is +given in :doc:`/guides/tagger`. + +The workflow is implemented in the ``beets.importer`` module and is +distinct from the core logic for matching MusicBrainz metadata (in the +``beets.autotag`` module). The workflow is also decoupled from the command-line +interface with the hope that, eventually, other (graphical) interfaces can be +bolted onto the same importer implementation. + +The importer is multithreaded and follows the pipeline pattern. Each pipeline +stage is a Python coroutine. The ``beets.util.pipeline`` module houses +a generic, reusable implementation of a multithreaded pipeline. diff --git a/docs/dev/index.rst b/docs/dev/index.rst index ebeccc535..f1465494d 100644 --- a/docs/dev/index.rst +++ b/docs/dev/index.rst @@ -13,3 +13,5 @@ and write metadata tags in media files. plugins library + importer + cli