mirror of
https://github.com/beetbox/beets.git
synced 2026-02-16 04:17:07 +01:00
I got a little bit nerdsniped by the problems observed in #5027. In short, my high-level diagnosis in https://github.com/beetbox/beets/pull/5027#issuecomment-1857953929 seems to have been correct: other tests were suppressing the legitimate failure of a flaky test. I found the problem by running other tests before the problem test, like this: ``` $ pytest -k 'test_nonexistant_db or test_delete_removes_item' test/test_ui.py ``` When running `test_nonexistant_db` alone, it fails. When running it like this with another test that goes first, it passes. That's the problem. However, `test_delete_removes_item` is just one example that works to make this problem happen. It appeared that _any_ test in a class that used our `_common.TestCase` base class had this power. I tracked down the issue to our `DummyIO` utility, which was having an unintentional effect even when it was never actually used. Here's the solution. Instead of restoring `sys.stdin` to `sys.__stdin__`, we now restore it to whatever it was before we installed out dummy I/O hooks. This is relevant in pytest, for example, which installs its *own* `sys.stdin`, which we were then clobbering. This was leading to the suppression of test failures observed in #5021 and addressed in #5027. The CI will fail for this PR because it now (correctly) exposes a failing test. Hopefully by combining this with the fixes in the works in #5027, we'll be back to a passing test suite. 😃 @Phil305, could you perhaps help validate that hypothesis? Edit: @snejus: I've now consolidated test I/O handling by removing the legacy `control_stdin`/`capture_stdout` context managers and the custom `DummyOut` stream, replacing them with a pytest-driven `io` fixture that: - provides controllable `stdin` via a lightweight `DummyIn` - captures `stdout` via `capteesys` - attaches a `DummyIO` helper to test classes as `self.io` |
||
|---|---|---|
| .. | ||
| autotag | ||
| plugins | ||
| rsrc | ||
| ui | ||
| util | ||
| __init__.py | ||
| conftest.py | ||
| test_art_resize.py | ||
| test_autotag.py | ||
| test_datequery.py | ||
| test_dbcore.py | ||
| test_files.py | ||
| test_hidden.py | ||
| test_importer.py | ||
| test_library.py | ||
| test_logging.py | ||
| test_m3ufile.py | ||
| test_metadata_plugins.py | ||
| test_metasync.py | ||
| test_pipeline.py | ||
| test_plugins.py | ||
| test_query.py | ||
| test_release.py | ||
| test_sort.py | ||
| test_template.py | ||
| test_types.py | ||
| test_util.py | ||
| testall.py | ||