Commit graph

335 commits

Author SHA1 Message Date
WithoutPants
51f4dd8a59
Delete identical studio stash id rows 2022-09-30 18:10:55 +10:00
WithoutPants
30f7a05ebf
Delete identical rows from performer_stash_ids 2022-09-30 17:25:40 +10:00
WithoutPants
8f594e7fed
Fix migration 2022-09-30 16:23:50 +10:00
WithoutPants
c8cbb36fd5
Ignore multiple identical rows in performers_scenes 2022-09-30 15:59:29 +10:00
DingDongSoLong4
25bc750295
Performance improvements (#2925)
* Add sqlite_stat4 build tag
* Simplify studio filter criterion queries
* Prevent useList loading data before filter initialized
2022-09-30 10:49:51 +10:00
7dJx1qP
d274f86390
Add backup directory path setting (#2953)
* add backup directory path setting
* Don't default backup path
* handle migration backup path input when given filename or path

Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
2022-09-30 10:00:50 +10:00
WithoutPants
ad7fbce5f7
Rebuild association tables, ensure file-system-based galleries cannot be changed (#2955)
* Re-create tables to include primary keys
* Filesystem-based galleries cannot change images
2022-09-30 09:18:58 +10:00
WithoutPants
d4e706daef
Only set video algorithm after migration 12 (#2946) 2022-09-26 11:27:53 +10:00
WithoutPants
0848b02e93
Various bug fixes (#2945)
* Only update fingerprints if changed
* Fix panic when loading primary file fails
* Fix gallery/scene association
* Fix display of scene gallery in card
* Use natural_cs collation with paths for title sorting
2022-09-25 12:07:55 +10:00
WithoutPants
cffcd9f4b8
Various bug fixes (#2935)
* Sort scene/image/gallery tags by name
* Calculate md5 if missing
* Prevent multiple folder create logs
2022-09-20 17:02:14 +10:00
WithoutPants
3fa7b470e7
More timestamp corrections (#2933)
* Fix incorrect timestamp updates
* Correct folder time fields
* Add migration with new indexes
* Correct mod_time format
* Add mod_time to data massage
2022-09-20 13:52:37 +10:00
WithoutPants
1207629a76
Fix incorrectly formatted timestamps (#2918)
* Update updated_at when adding file to object
* Use models.SQLTimestamp for timestamps
* Add data massage to fix incorrect timestamps
2022-09-19 14:53:46 +10:00
WithoutPants
2564351265
Use post commit hook for post-create plugin hooks (#2920) 2022-09-19 14:53:06 +10:00
WithoutPants
0359ce2ed8
Fix bulk add movie to scenes and autotag transaction error (#2928)
* Fix bulk add movie to scene
* Fix already in transaction error for autotag
2022-09-19 14:52:40 +10:00
WithoutPants
648247aa00
Split by whitespace for path includes/excludes (#2919) 2022-09-16 14:45:08 +10:00
WithoutPants
8b79eaca67
[Files Refactor] Use batching for pre/post-migration (#2906)
* Use batching for pre/post-migration
* Clarify release notes
2022-09-14 10:57:00 +10:00
DingDongSoLong4
5c383da5ec
Optimize database after migration (#2904) 2022-09-14 09:15:36 +10:00
WithoutPants
9e08edc76f
[Files Refactor] Don't require fingerprint calculation post-migrate (#2892) 2022-09-07 14:21:10 +10:00
WithoutPants
0c513a604d Fix stack overflow creating windows network folders (#2886) 2022-09-06 07:04:52 +00:00
WithoutPants
6b0bcdea88 [Files Refactor] Set primary flag when cleaning (#2880)
* Ensure single primary per object
* Set primary file during clean
* Only show migration notes for actual migrations
2022-09-06 07:04:52 +00:00
WithoutPants
94d39da706 [Files Refactor] Bug fixes (#2868)
* Return error if multiple rows returned for id
* Add missing LoadFiles calls
* Show id if path is empty
2022-09-06 07:04:52 +00:00
WithoutPants
273cf0383d [Files Refactor] Performance tuning (#2865)
* Don't load image files by default
* Don't load gallery files by default
* Don't load scene files by default
* Retry locked transactions forever
* Don't show release notes if config not loaded
* Don't translate path slashes in export
2022-09-06 07:04:52 +00:00
WithoutPants
0b534d89c6 [Files Refactor] Import export fixup (#2763)
* Adjust json schema
* Remove mappings file from export
* Import file/folder support
* Update documentation
* Make gallery filenames unique
2022-09-06 07:04:52 +00:00
WithoutPants
1222b7b87b Fix files not being cleaned correctly (#2862) 2022-09-06 07:04:52 +00:00
WithoutPants
7b439556c0 [Files Refactor] Rollback platform-agnostic paths (#2852)
* Rollback platform agnostic path storage
* Add release note for database change
2022-09-06 07:04:52 +00:00
WithoutPants
f4825fadf4 [Files refactor] Bug fixes (#2849)
* Fix scene sorting
* Fix folder-based gallery path sorting
* Fix gallery path filter
* Fix stash-box performer submission
* Fix identify logging
* Remove govet from linter
2022-09-06 07:03:43 +00:00
WithoutPants
00608c167a [Files Refactor] Performance tuning (#2819)
* Load scene relationships on demand
* Load image relationships on demand
* Load gallery relationships on demand
* Add dataloaden
* Use dataloaders
* Use where in for other find many functions
2022-09-06 07:03:42 +00:00
WithoutPants
9b31b20fed [Files Refactor] Performance tuning (#2813)
* Do database txn in same thread. Retry on locked db
* Remove captions from slimscenedata
* Fix tracing
* Use where in instead of individual selects
* Remove scenes_query view
* Remove image query view
* Remove gallery query view
* Use where in for FindMany
* Don't interrupt scanning zip files
* Fix image filesize sort
2022-09-06 07:03:42 +00:00
WithoutPants
87167736f6 [Files Refactor] bug fixes (#2811)
* Fix scan options not saving
* Fix duration stat calculation
2022-09-06 07:03:42 +00:00
WithoutPants
569c3a872a [Files Refactor] Performance tuning (#2809)
* Use cache during migration
* Avoid use of query views
* Use FindMany to find related objects
* Log slow queries
* Add folders to generated files
* Use SlimScene for scene queries
* Include filename in migration error message
2022-09-06 07:03:42 +00:00
WithoutPants
5843fdcecc [Files Refactor] Migration fix (#2796)
* Fix large wal file during migration
* Fix migration dropping / from network share paths
2022-09-06 07:03:42 +00:00
WithoutPants
bc47932343 [Files Refactor] Performance tuning (#2784)
* Improve image query performance
* Tune queries
* Fix db generator
* Don't show release notes in setup
* Further tune indexes
* Log when creating screenshot
2022-09-06 07:03:42 +00:00
WithoutPants
abb574205a Files refactor fixes (#2743)
* Fix destroy gallery not destroying file
* Re-add minModTime functionality
* Deprecate useFileMetadata and stripFileExtension
* Optimise files post migration
* Decorate moved files. Use first missing file in move
* Include path in thumbnail generation error log
* Fix stash-box draft submission
* Don't destroy files unless deleting
* Call handler for files with no associated objects
* Fix moved zips causing error on scan
2022-09-06 07:03:42 +00:00
WithoutPants
461068462c [Files Refactor] Filter and sort by file count (#2744)
* Add filtering on file count
* Add sorting by file count
2022-09-06 07:03:42 +00:00
WithoutPants
5495d72849 File storage rewrite (#2676)
* Restructure data layer part 2 (#2599)
* Refactor and separate image model
* Refactor image query builder
* Handle relationships in image query builder
* Remove relationship management methods
* Refactor gallery model/query builder
* Add scenes to gallery model
* Convert scene model
* Refactor scene models
* Remove unused methods
* Add unit tests for gallery
* Add image tests
* Add scene tests
* Convert unnecessary scene value pointers to values
* Convert unnecessary pointer values to values
* Refactor scene partial
* Add scene partial tests
* Refactor ImagePartial
* Add image partial tests
* Refactor gallery partial update
* Add partial gallery update tests
* Use zero/null package for null values
* Add files and scan system
* Add sqlite implementation for files/folders
* Add unit tests for files/folders
* Image refactors
* Update image data layer
* Refactor gallery model and creation
* Refactor scene model
* Refactor scenes
* Don't set title from filename
* Allow galleries to freely add/remove images
* Add multiple scene file support to graphql and UI
* Add multiple file support for images in graphql/UI
* Add multiple file for galleries in graphql/UI
* Remove use of some deprecated fields
* Remove scene path usage
* Remove gallery path usage
* Remove path from image
* Move funscript to video file
* Refactor caption detection
* Migrate existing data
* Add post commit/rollback hook system
* Lint. Comment out import/export tests
* Add WithDatabase read only wrapper
* Prepend tasks to list
* Add 32 pre-migration
* Add warnings in release and migration notes
2022-09-06 07:03:42 +00:00
WithoutPants
964b559309 Restructure data layer (#2532)
* Add new txn manager interface
* Add txn management to sqlite
* Rename get to getByID
* Add contexts to repository methods
* Update query builders
* Add context to reader writer interfaces
* Use repository in resolver
* Tighten interfaces
* Tighten interfaces in dlna
* Tighten interfaces in match package
* Tighten interfaces in scraper package
* Tighten interfaces in scan code
* Tighten interfaces on autotag package
* Remove ReaderWriter usage
* Merge database package into sqlite
2022-09-06 07:03:40 +00:00
kermieisinthehouse
30879389ec
Fix golangci OOM (#2889)
* Fix golangci OOM

* Fix all lints
2022-09-05 22:12:59 -07:00
CJ
9264c15540
Customize recommendations (#2592)
* refactored common code in recommendation row
* Implement front page options in config
* Allow customisation from front page
* Rename recommendations to front page
* Add generic UI settings
* Support adding premade filters

Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
2022-06-14 10:34:04 +10:00
cj
c1a096a1a6
Caption support (#2462)
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
2022-05-06 11:59:28 +10:00
WithoutPants
61d9f57ce9
Add ignore autotag flag (#2439)
* Add autoTagIgnored to database schema
* Graphql changes
* UI changes
* Add field to edit performers dialog
* Apply flag to autotag behaviour
2022-04-04 20:03:39 +10:00
WithoutPants
02ee791796
Fix missing date filter (#2434) 2022-03-29 06:45:46 +11:00
WithoutPants
f581687198
Improve performance of gallery and image queries (#2422)
* Revert to use FindByGalleryID in gallery resolver
* Sort by path in FindByGalleryID
* Optimise queries
2022-03-25 12:00:51 +11:00
WithoutPants
f69bd8a94f
Restructure go project (#2356)
* Move main to cmd
* Move api to internal
* Move logger and manager to internal
* Move shell hiding code to separate package
* Decouple job from desktop and utils
* Decouple session from config
* Move static into internal
* Decouple config from dlna
* Move desktop to internal
* Move dlna to internal
* Decouple remaining packages from config
* Move config into internal
* Move jsonschema and paths to models
* Make ffmpeg functions private
* Move file utility methods into fsutil package
* Move symwalk into fsutil
* Move single-use util functions into client package
* Move slice functions to separate packages
* Add env var to suppress windowsgui arg
* Move hash functions into separate package
* Move identify to internal
* Move autotag to internal
* Touch UI when generating backend
2022-03-17 11:33:59 +11:00
WithoutPants
d88515abcd
Autotag optimisation (#2368)
* Add duration to autotag finish message
* No sorting scene/image/gallery where not specified
* Use an LRU cache for sqlite regexp function
* Compile path separator regex once
* Cache objects with single letter first names
* Move finished auto-tag log
* Add more verbose logging
* Add new changelog
2022-03-09 12:01:56 +11:00
WithoutPants
18665863d6
Remove single unicode character from autotag query (#2363)
* Remove single unicode character from autotag query
* Compile regex once where possible
* Fix CPU profiling
* Only match unicode characters if in path
2022-03-07 13:26:24 +11:00
WithoutPants
1ab5be162e
Handle unicode characters in autotag (#2336) 2022-02-28 13:12:43 +11:00
kermieisinthehouse
4dd0bbc294
Add Several Media Performer Detail Filters, Scene Filters and Sort (#2257)
Co-authored-by: bnkai <48220860+bnkai@users.noreply.github.com>
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
2022-02-16 11:11:57 +11:00
InfiniteTF
b58883c074
Fix image querying (#2119)
* Fix image querying
* Add unit tests

Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
2021-12-15 13:56:03 +11:00
agentfisk
e8447c520a
Heatmap and speed sorting/filtering for interactive scenes [rewrite] (#2096)
* add InteractiveSpeed to scene model
* add InteractiveHeatmapSpeedGenerator
* add GenerateInteractiveHeatmapSpeedTask
* add InteractiveHeatmapSpeedTask to GenerateJob
* add InteractiveHeatmap on sceneRoutes
* delete heatmap when scene is destroyed
* render interactive heatmap in GridCard
* render InteractiveSpeed on SceneCard
* render InteractiveSpeed in SceneFileInfoPanel
* InteractiveSpeed filters
2021-12-13 13:41:07 +11:00
Esteban Sanchez
70d9a05580
Use inner join when getting images in a gallery (#2083)
* Added joinType to join struct
* Added addInnerJoin function to perform INNER JOIN type of joins
* Added innerJoin function to perform INNER JOIN type of joins
* Use inner joins when querying images in a gallery
* Renamed addJoin to addLeftJoin
2021-12-06 12:30:40 +11:00
WithoutPants
2277d0a919
Add search string parsing (#1982)
* Add search string parsing
* Add manual page
2021-11-22 14:59:22 +11:00
InfiniteTF
c571687c99
Resolve performer/studio stashIDs for scraped scenes (#2006)
* Resolve performer/studio stashIDs for scraped scenes
* Check endpoint when matching stashids
2021-11-15 07:51:52 +11:00
gitgiggety
25274e2596
Support Is (not) null for all multi criterions (#1785)
* Support Is (not) null for all multi criterions

Add support for the Is null and Is not null modifiers for all cases of
the MultiCriterionInput and HierarchicalMultiCriterionInput. This
partially overlaps the "X Count" filter which sometimes is available
(because it would be the same as "X Count equals 0" and "X Count greater
than 0") but this also enables it for other criterions like the "Parent
Studio" filter for studios or just the "Studios" filter for scenes /
images / galleries, the "Movies" filter for scenes etc.

* Don't crash UI on bad saved filter
* Add missing code for tag parent/child

Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
2021-11-07 09:34:33 +11:00
7dJx1qP
e961ba4459
Fix tag hierarchy validation (#1926)
* update tag hierarchy validation
* refactor MergeHierarchy
* update tag hierarchy error message
* rename tag hierarchy function
* add tag path to error message
* Rename EnsureHierarchy to ValidateHierarchy

Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
2021-11-07 09:33:46 +11:00
kermieisinthehouse
5de06d4b62
Convert filesize to float64 for arm32 (#1895)
* Convert filesize to float64 for arm32
2021-10-27 11:55:11 +11:00
kermieisinthehouse
4dd56c3d82
Show duration and filesize in results (#1776)
* Add new query interface
* Refactor query builder
* Change Query interface
* Return duration and filesize in scene query
* Adjust UI for scene metadata
* Introduce new image query interface
* Change image Query interface
* Add megapixels and size to image query
* Update image UI

Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
2021-10-25 11:40:13 +11:00
SmallCoccinelle
8b7720e3bf
Enable the appendAssign lint check (#1865)
* Make copies of buffers

Avoid reusing one of the incoming arrays as a append extension, and
make a copy of the data. It's cleaner in the long run and possibly
easier for the GC to maintain.

* Avoid appendAssign problems in tag code

Reuse the existing slice when appending.

* Fix appendAssign in encoder_scene_preview_chunk

Appending and creating a new slice is somewhat unintuitive since the
underlying slice might be extended to satisfy the new capacity. This
sometimes leads to faulty logic.

Rewrite the code so it reuses `args` for all appending, and builds one
array clearly in the code. It follows the general style of the function
where `args` is being built in small incremental batches and avoids
the introduction of new names.

* Enable the appendAssign check

This makes us pass all gocritic warnings.

Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
2021-10-20 16:34:19 +11:00
SmallCoccinelle
214a15bc40
Fixups + enable the commentFormatting linter (#1866)
* Add a space after // comments

For consistency, the commentFormatting lint checker suggests a space
after each // comment block. This commit handles all the spots in
the code where that is needed.

* Rewrite documentation on functions

Use the Go idiom of commenting:

* First sentence declares the purpose.
* First word is the name being declared

The reason this style is preferred is such that grep is able to find
names the user might be interested in. Consider e.g.,

    go doc -all pkg/ffmpeg | grep -i transcode

in which case a match will tell you the name of the function you are
interested in.

* Remove old code comment-blocks

There are some commented out old code blocks in the code base. These are
either 3 years old, or 2 years old. By now, I don't think their use is
going to come back any time soon, and Git will track old pieces of
deleted code anyway.

Opt for deletion.

* Reorder imports

Split stdlib imports from non-stdlib imports in files we are touching.

* Use a range over an iteration variable

Probably more go-idiomatic, and the code needed comment-fixing anyway.

* Use time.After rather than rolling our own

The idiom here is common enough that the stdlib contains a function for
it. Use the stdlib function over our own variant.

* Enable the commentFormatting linter
2021-10-20 16:10:46 +11:00
SmallCoccinelle
e14bb8432c
Enable gocritic (#1848)
* Don't capitalize local variables

ValidCodecs -> validCodecs

* Capitalize deprecation markers

A deprecated marker should be capitalized.

* Use re.MustCompile for static regexes

If the regex fails to compile, it's a programmer error, and should be
treated as such. The regex is entirely static.

* Simplify else-if constructions

Rewrite

   else { if cond {}}

to

   else if cond {}

* Use a switch statement to analyze formats

Break an if-else chain. While here, simplify code flow.

Also introduce a proper static error for unsupported image formats,
paving the way for being able to check against the error.

* Rewrite ifElse chains into switch statements

The "Effective Go" https://golang.org/doc/effective_go#switch document
mentions it is more idiomatic to write if-else chains as switches when
it is possible.

Find all the plain rewrite occurrences in the code base and rewrite.
In some cases, the if-else chains are replaced by a switch scrutinizer.
That is, the code sequence

  if x == 1 {
      ..
  } else if x == 2 {
      ..
  } else if x == 3 {
      ...
  }

can be rewritten into

  switch x {
  case 1:
    ..
  case 2:
    ..
  case 3:
    ..
  }

which is clearer for the compiler: it can decide if the switch is
better served by a jump-table then a branch-chain.

* Rewrite switches, introduce static errors

Introduce two new static errors:

* `ErrNotImplmented`
* `ErrNotSupported`

And use these rather than forming new generative errors whenever the
code is called. Code can now test on the errors (since they are static
and the pointers to them wont change).

Also rewrite ifElse chains into switches in this part of the code base.

* Introduce a StashBoxError in configuration

Since all stashbox errors are the same, treat them as such in the code
base. While here, rewrite an ifElse chain.

In the future, it might be beneifical to refactor configuration errors
into one error which can handle missing fields, which context the error
occurs in and so on. But for now, try to get an overview of the error
categories by hoisting them into static errors.

* Get rid of an else-block in transaction handling

If we succesfully `recover()`, we then always `panic()`. This means the
rest of the code is not reachable, so we can avoid having an else-block
here.

It also solves an ifElse-chain style check in the code base.

* Use strings.ReplaceAll

Rewrite

    strings.Replace(s, o, n, -1)

into

    strings.ReplaceAll(s, o, n)

To make it consistent and clear that we are doing an all-replace in the
string rather than replacing parts of it. It's more of a nitpick since
there are no implementation differences: the stdlib implementation is
just to supply -1.

* Rewrite via gocritic's assignOp

Statements of the form

    x = x + e

is rewritten into

    x += e

where applicable.

* Formatting

* Review comments handled

Stash-box is a proper noun.

Rewrite a switch into an if-chain which returns on the first error
encountered.

* Use context.TODO() over context.Background()

Patch in the same vein as everything else: use the TODO() marker so we
can search for it later and link it into the context tree/tentacle once
it reaches down to this level in the code base.

* Tell the linter to ignore a section in manager_tasks.go

The section is less readable, so mark it with a nolint for now. Because
the rewrite enables a ifElseChain, also mark that as nolint for now.

* Use strings.ReplaceAll over strings.Replace

* Apply an ifElse rewrite

else { if .. { .. } } rewrite into else if { .. }

* Use switch-statements over ifElseChains

Rewrite chains of if-else into switch statements. Where applicable,
add an early nil-guard to simplify case analysis. Also, in
ScanTask's Start(..), invert the logic to outdent the whole block, and
help the reader: if it's not a scene, the function flow is now far more
local to the top of the function, and it's clear that the rest of the
function has to do with scene management.

* Enable gocritic on the code base.

Disable appendAssign for now since we aren't passing that check yet.

* Document the nolint additions

* Document StashBoxBatchPerformerTagInput
2021-10-18 14:12:40 +11:00
SmallCoccinelle
c6f6205e4f
Errorlint sweep + minor linter tweaks (#1796)
* Replace error assertions with Go 1.13 style

Use `errors.As(..)` over type assertions. This enables better use of
wrapped errors in the future, and lets us pass some errorlint checks
in the process.

The rewrite is entirely mechanical, and uses a standard idiom for
doing so.

* Use Go 1.13's errors.Is(..)

Rather than directly checking for error equality, use errors.Is(..).

This protects against error wrapping issues in the future.

Even though something like sql.ErrNoRows doesn't need the wrapping, do
so anyway, for the sake of consistency throughout the code base.

The change almost lets us pass the `errorlint` Go checker except for
a missing case in `js.go` which is to be handled separately; it isn't
mechanical, like these changes are.

* Remove goconst

goconst isn't a useful linter in many cases, because it's false positive
rate is high. It's 100% for the current code base.

* Avoid direct comparison of errors in recover()

Assert that we are catching an error from recover(). If we are,
check that the error caught matches errStop.

* Enable the "errorlint" checker

Configure the checker to avoid checking for errorf wraps. These are
often false positives since the suggestion is to blanket wrap errors
with %w, and that exposes the underlying API which you might not want
to do.

The other warnings are good however, and with the current patch stack,
the code base passes all these checks as well.

* Configure rowserrcheck

The project uses sqlx. Configure rowserrcheck to include said package.

* Mechanically rewrite a large set of errors

Mechanically search for errors that look like

    fmt.Errorf("...%s", err.Error())

and rewrite those into

    fmt.Errorf("...%v", err)

The `fmt` package is error-aware and knows how to call err.Error()
itself.

The rationale is that this is more idiomatic Go; it paves the
way for using error wrapping later with %w in some sites.

This patch only addresses the entirely mechanical rewriting caught by
a project-side search/replace. There are more individual sites not
addressed by this patch.
2021-10-12 14:03:08 +11:00
WithoutPants
e9d48683f8
Autotag scraper (#1817)
* Refactor scraper structures
* Move matching code into new package
* Add autotag scraper
* Always check first letter of auto-tag names
* Account for nulls

Co-authored-by: Kermie <kermie@isinthe.house>
2021-10-11 23:06:06 +11:00
kermieisinthehouse
c31c7c3c99
Fix exclude filter query performance (#1815)
* Fix query performance
* Reorder changelog changes by size to prepare for release
2021-10-10 19:02:26 +11:00
bnkai
ea7a4f8d33
Fix Studio -> Stash ID filter (#1793) 2021-10-03 11:00:49 +11:00
WithoutPants
ca0a8b00ec
Only group by if using having clauses (#1750)
* Only group by if using having clauses
* Change tag sorting SQL
* Add sorting unit tests
2021-10-01 16:24:58 +10:00
gitgiggety
dabf5acefe
Filter tag by hierarchy (#1746)
* Add API support for filtering tags by parent / children
* Add parent & child tags filters for tags to UI
* Add API support for filtering tags by parent / child count
* Add parent & child count filters for tags to UI
* Update db generator
* Add missing build tag
* Add unit tests

Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
2021-10-01 11:50:06 +10:00
SmallCoccinelle
bc6d04dc2a
Remove new-from-rev from the linter (#1783)
* Fix all revive warnings in the code base

All of these are of the form

```
var Identifier Type = Expr
```

where the `Type` is known from the output of `Expr` and can be omitted
as a result.

* Handle unchecked errors

* Remove new-from-rev

Since the project passes all linter checks now, including older
revisions, we can remove new-from-rev. While here, reorder the linter
config file, and move the enabled linters up and settings down.

* Fix failing test cases

Studio & Performer export tests use local time rather than UTC. This
fixes the test cases so integration test
passes.
2021-09-28 09:29:45 +10:00
WithoutPants
b957a87a78
Fix scene marker sort by scene updated SQL error (#1777) 2021-09-27 12:26:38 +10:00
gitgiggety
be94e52f21
Add movie count to performer and studio card (#1760)
* Add movies and movie_count properties to Performer type

Extend the GraphQL API to allow getting the movies and movie count by
performer.

* Add movies count to performer card

* Add movies and movie_count properties to Studio type

Extend the GraphQL API to allow getting the movies and movie count by
studio.

* Add movies count to studio card
2021-09-27 11:31:49 +10:00
Eng Zer Jun
62af723017
refactor: move from io/ioutil to io and os package (#1772)
The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil. This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2021-09-27 10:55:23 +10:00
SmallCoccinelle
a9e2a590b2
Lint checks phase 2 (#1747)
* Log 3 unchecked errors

Rather than ignore errors, log them at
the WARNING log level.

The server has been functioning without these, so assume they are not at
the ERROR level.

* Log errors in concurrency test

If we can't initialize the configuration, treat the test as a failure.

* Undo the errcheck on configurations for now.

* Handle unchecked errors in pkg/manager

* Resolve unchecked errors

* Handle DLNA/DMS unchecked errors

* Handle error checking in concurrency test

Generalize config initialization, so we can initialize a configuration
without writing it to disk.

Use this in the test case, since otherwise the test fails to write.

* Handle the remaining unchecked errors

* Heed gosimple in update test

* Use one-line if-initializer statements

While here, fix a wrong variable capture error.

* testing.T doesn't support %w

use %v instead which is supported.

* Remove unused query builder functions

The Int/String criterion handler functions are now generalized.

Thus, there's no need to keep these functions around anymore.

* Mark filterBuilder.addRecursiveWith nolint

The function is useful in the future and no other refactors are looking
nice.

Keep the function around, but tell the linter to ignore it.

* Remove utils.Btoi

There are no users of this utility function

* Return error on scan failure

If we fail to scan the row when looking for the
unique checksum index, then report the error upwards.

* Fix comments on exported functions

* Fix typos

* Fix startup error
2021-09-23 17:15:50 +10:00
bnkai
af6232ec97
Fix Query for Scene Markers (#1743) 2021-09-20 09:52:19 +10:00
WithoutPants
501ed7c2c2
Make hierarchical criterion depth input optional (#1733) 2021-09-16 20:41:07 +10:00
gitgiggety
f3119a6c38
Add AND, OR and NOT support to studio filter (#1726) 2021-09-16 12:09:23 +10:00
WithoutPants
13a289a4a8
Fix studio scene count sort (#1714) 2021-09-09 19:44:02 +10:00
gitgiggety
04e5ac9c2f
Studio aliases (#1660)
* Add migration to create studio aliases table
* Refactor studioQueryBuilder.Query to use filterBuilder
* Expand GraphQL API with aliases support for studio
* Add aliases support for studios to the UI
* List aliases in details panel
* Allow editing aliases in edit panel
* Add 'aliases' filter when searching
* Find studios by alias in filter / select
* Add auto-tagging based on studio aliases
* Support studio aliases for filename parsing
* Support importing and exporting of studio aliases
* Search for studio alias as well during scraping
2021-09-09 18:13:42 +10:00
gitgiggety
c91ffe1e58
Tag hierarchy (#1519)
* Add migration script for tag relations table
* Expand hierarchical filter features

Expand the features of the hierarchical multi input filter with support
for using a relations table, which only has parent_id and child_id
columns, and support adding an additional intermediate table to join on,
for example for scenes and tags which are linked by the scenes_tags
table as well.

* Add hierarchical filtering for tags
* Add hierarchical tags support to scene markers

Refactor filtering of scene markers to filterBuilder and in the process
add support for hierarchical tags as well.

* List parent and child tags on tag details page
* Support setting parent and child tags

Add support for setting parent and child tags during tag creation and
tag updates.

* Validate no loops are created in tags hierarchy
* Update tag merging to support tag hierarcy
* Add unit tests for tags.EnsureUniqueHierarchy
* Fix applying recursive to with clause

The SQL `RECURSIVE` of a `WITH` clause only needs to be applied once,
imediately after the `WITH`. So this fixes the query building to do just
that, automatically applying the `RECURSIVE` keyword when any added with
clause is added as recursive.

* Rename hierarchical root id column
* Rewrite hierarchical filtering for performance

Completely rewrite the hierarchical filtering to optimize for
performance. Doing the recursive query in combination with a complex
query seems to break SQLite optimizing some things which means that the
recursive part might be 2,5 second slower than adding a static
`VALUES()` list. This is mostly noticable in case of the tag hierarchy
where setting an exclusion with any depth (or depth: all) being applied
has this performance impact of 2,5 second. "Include" also suffered this
issue, but some rewritten query by joining in the *_tags table in one
pass and applying a `WHERE x IS NOT NULL` filter did seem to optimize
that case. But that optimization isn't applied to the `IS NULL` filter
of "exclude". Running a simple query beforehand to get all (recursive)
items and then applying them to the query doesn't have this performance
penalty.

* Remove UI references to child studios and tags
* Add parents to tag export
* Support importing of parent relationship for tags
* Assign stable ids to parent / child badges
* Silence Apollo warning on parents/children fields on tags

Silence warning triggered by Apollo GraphQL by explicitly instructing it
to use the incoming parents/children values. By default it already does
this, but it triggers a warning as it might be unintended that it uses
the incoming values (instead of for example merging both arrays).
Setting merge to false still applies the same behaviour (use only
incoming values) but silences the warning as it's explicitly configured
to work like this.

* Rework detecting unique tag hierarchy

Completely rework the unique tag hierarchy to detect invalid hierarchies
for which a tag is "added in the middle". So when there are tags A <- B
and A <- C, you could previously edit tag B and add tag C as a sub tag
without it being noticed as parent A being applied twice (to tag C).
While afterwards saving tag C would fail as tag A was applied as parent
twice. The updated code correctly detects this scenario as well.

Furthermore the error messaging has been reworked a bit and the message
now mentions both the direct parent / sub tag as well as the tag which
would results in the error. So in aboves example it would now show the
message that tag C can't be applied because tag A already is a parent.

* Update relations on cached tags when needed

Update the relations on cached tags when a tag is created / updated /
deleted so these always reflect the correct state. Otherwise (re)opening
a tag might still show the old relations untill the page is fully
reloaded or the list is navigated. But this obviously is strange when
you for example have tag A, create or update tag B to have a relation to
tag A, and from tags B page click through to tag A and it doesn't show
that it is linked to tag B.
2021-09-09 14:58:43 +10:00
SmallCoccinelle
4b00d24248
Remove unused (#1709)
* Remove stuff which isn't being used

Some fields, functions and structs aren't in use by the project. Remove
them for janitorial reasons.

* Remove more unused code

All of these functions are currently not in use. Clean up the code by
removal, since the version control has the code if need be.

* Remove unused functions

There's a large set of unused functions and variables in the code base.
Remove these, so it clearer what code to support going forward.

Dead code has been eliminated.

Where applicable, comment const-sections in tests, so reserved
identifiers are still known.

* Fix use-def of tsURL

The first def of tsURL doesn't matter because there's no use before
we hit the 2nd def.

* Remove dead code assignment

Setting logFile = "" is effectively dead code, because there's no use
of it later.

* Comment out found

The variable 'found' is dead in the function (because no post-process
action is following it). Comment it for now.

* Comment dead code in tests

These might provide hints as to what isn't covered at the moment.

* Dead code removal

In the case of constants where iota is involved, move the iota so it
matches the current key values.

This avoids problems with persistently stored key IDs.
2021-09-09 14:10:08 +10:00
kermieisinthehouse
265d5f4c70
Apple Silicon Support, Bump Go to 1.17, refactor docker/build/x86_64/Dockerfile (#1646)
* Bump Go to 1.17, refactor build/x86_64 Dockerfile to make better use of multi-stage
* Bump to 1.17 from 1.16
* Bump packr version, provide needed legacy env var
* Add apple silicon support, fix macos build chain
* Update unused travis ci
2021-09-08 15:30:15 +10:00
SmallCoccinelle
4545da9af0
Avoid redundant break statements (#1705)
In Go, a switch-case automatically breaks on end. This makes
the break statement redundant.
2021-09-08 11:23:21 +10:00
SmallCoccinelle
e7f6cb22b7
Error strings noncapitalized (#1704)
* Fix error string capitalization

Error strings often follow another string. Hence, they should not be
capitalized, unless referencing a name.

* Uncapitalize more error strings

While here, use %v on the error directly, which makes it easier to wrap
the error later with %w if need be.

* Uncapitalize more error strings

While here, rename Url to URL as a nitpick.
2021-09-08 11:23:10 +10:00
gitgiggety
7a468413da
Add movies tab to Studios and Performers page (#1675)
* Add movies tab to Studios page
* Add performers filter to movies
* Add movies tab to performers page
2021-09-07 12:16:33 +10:00
gitgiggety
1774a3600c
Fix Performers Is Missing: stash ID filter (#1681)
* Fix Performers Is Missing: stash ID filter.

Fixes #1679
2021-08-30 11:46:41 +10:00
FleetingOrchard
50cb6a9c79
Add duration statistics to stats page (#1626) 2021-08-26 13:37:08 +10:00
gitgiggety
4c144db510
Performers listing improvements (#1671)
* Add sorting on image and gallery count to performers
* Make performer table headers translatable
* Add image and gallery count to performers table
* Make sure list table container fits the table

Make the table container minimally as wide as the table. This fixes the
table "overflowing" to the left and right and the left not being
accessible.

* Remove unnecessary truncation in tables

IMO there is no need to truncate the title in the scenes table and the
name in the performers table. This because both tables also contain an
image which means that multiple lines should be possible without really
extending the height of the row. Furthermore both tables contain other
values which might be way longer and also aren't wrapped (like tags and
performers for scenes, and aliases for performers).
2021-08-26 09:41:18 +10:00
kermieisinthehouse
7b3b2ae9ba
Order sceneDuplicateChecker results by size (#1639) 2021-08-18 13:11:57 +10:00
gitgiggety
fc6cafa15f
Fix hierarchical criteria performance issue (#1643)
* Fix hierarchical criteria performance issue

Don't apply recursive clause to hierarchical criteria when the depth is
set to 0 (i.e.: no recursion is needed).

This undoes the current performance penalty on for example the studios
page. This as reported in #1519, using a database of 4M images, 30K
scenes and 500 studios. Without this fix loading the studios overview,
with the default of 40 items per page, takes 6 to 7 seconds. With this
fix it only takes 0,07 seconds reverting the performance back to the
pre-hierarchical filtering performance (tested against 508f7b84 which
was the last commit before #1397 was merged).
2021-08-17 13:43:22 +10:00
gitgiggety
7cb3d05535
Add (not) between modifiers for number criterion (#1559)
* Add (not) between modifiers for number criterion
* Extract list filters into dedicated components

Extract the filters from the AddFiltersDialog into custom components.
This allows for further refactorring where components will be bound to
criterions.

* Add placeholders to number and duration criterions
* Add backwards compatibility for saved filters

Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
2021-08-12 10:24:16 +10:00
gitgiggety
9d641c64e3
Actually implement TagFilter.marker_count (#1603)
* Actually implement TagFilter.marker_count

The marker_count filter/criterion as defined in TagFilterType isn't
actually implemented. This adds an implementation for it.

Do note this implementation _might_ have performance issues because of
using OR (in the join). Another implentation would be to remove both
joins and use:
```SQL
COUNT(
    SELECT id FROM scene_markers WHERE primary_tag_id = tags.id
  UNION
    SELECT scene_marker_id FROM scene_markers_tags WHERE tag_id = tags.id
)
```
Note this doesn't require a DISTINCT as UNION already removes any
duplicate records.

* Restore marker count filter and sorting

Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
2021-08-04 13:39:24 +10:00
Jekora
ede8cca631
[Feature] Better resolution search (#1568)
* Fix width in database test setup
* Added more filters on resolution field
* added test to verify resolution range is defined for every resolution
* Refactor UI code

Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
2021-08-02 13:22:39 +10:00
gitgiggety
a13f43c13b
Always wrap filter conditions in parentheses (#1577)
* Always wrap filter conditions in parentheses

Fixes #1571
2021-07-14 18:29:59 +10:00
WithoutPants
4165e7779f
Add various filter criteria (#1505)
* Add various filter criteria
* Add tag name criterion
2021-06-23 09:10:20 +10:00
WithoutPants
5fdfbaa7f1
Query bug fixes (#1510)
* Fix joins being dropped
* Fix missing scene stash_id criterion
* Refactor criterion handlers
* Add tag alias filter
* Remove handleCriterionFunc
2021-06-21 15:48:28 +10:00
gitgiggety
d7439b4832
Optimize studio filter on performers (#1515) 2021-06-21 14:17:43 +10:00
WithoutPants
dc7584d77e
Saved filters (#1474)
* Refactor list filter
* Filter/criterion refactor
* Rename option value to type
* Remove None from options
* Add saved filter button
* Integrate default filters
2021-06-16 14:53:32 +10:00
WithoutPants
4fe4da6c01
Merge tags functionality (#1481)
* Add API to merge tags

Add new API endpoint, `tagsMerge(source, destination)` to merge multiple
tags into a single one. The "sources" must be provided as a list of ids
and the destination as a single id. All usages of the source tags
(scenes, markers (primary and additional), images, galleries and
performers) will be updated to the destination tag, all aliases of the
source tags will be updated to the destination, and the name of the
source will be added as alias to the destination as well.

* Add merge tag UI
* Add unit tests
* Update test mocks
* Update internationalisation
* Add changelog entry

Co-authored-by: gitgiggety <gitgiggety@outlook.com>
2021-06-16 14:33:54 +10:00
WithoutPants
f843359ba3
Handle auto-tagging where filename has no whitespace in name (#1488) 2021-06-08 10:47:22 +10:00
WithoutPants
c53799c25b
Fix Performer Studio filtering (#1483)
* Fix performer studio filtering

* Fix studio filter hook
2021-06-06 15:05:05 +10:00
bnkai
ad0a9d0707
Fix movies sorting, scene studio editing (#1478)
* Fix movies:sort_by->scenes_count, scene:edit->remove studio
2021-06-04 09:21:17 +10:00
gitgiggety
7164bb28ac
Filter studio hierarchy (#1397)
* Add basic support for hierarchical filters

Add a new `hierarchicalMultiCriterionHandlerBuilder` filter type which
can / will be used for filtering hierarchical things like the
parent/child relation of the studios.
On the frontend side a new IHierarchicalLabeledIdCriterion criterion
type has been added to accompany this new filter type.

* Refactor movieQueryBuilder to use filterBuilder

Refactor the movieQueryBuilder to use the filterBuilder just as scene,
image and gallery as well.

* Support specifying depth for studios filter

Add an optional depth field to the studios filter for scenes, images,
galleries and movies. When specified that number of included
(grant)children are shown as well. In other words: this adds support for
showing scenes set to child studios when searching on the parent studio.

Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
2021-06-03 20:52:19 +10:00
bnkai
c5fed1bbdc
Fix sort by movies_scene_number (#1448) 2021-05-28 09:01:03 +10:00
WithoutPants
c70faa2a53
Tag aliases (#1412)
* Add Tag Update/UpdateFull
* Tag alias implementation
* Refactor tag page
* Add aliases in UI
* Include tag aliases in q filter
* Include aliases in tag select
* Add aliases to auto-tagger
* Use aliases in scraper
* Add tag aliases for filename parser
2021-05-26 14:36:05 +10:00
peolic
4bad988373
Remove performer "is missing scenes" filter (#1414) 2021-05-25 10:45:26 +10:00
peolic
0472cd9996
Fix performer filters and sort keys (#1413)
* Fix "performer is missing image" filter
* Fix `scene_count` sort
* Add unit tests

Co-authored-by: bnkai <48220860+bnkai@users.noreply.github.com>
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
2021-05-24 17:45:51 +10:00
UnluckyChemical765
547f6d79ad
Add Handy / Funscript support (#1377)
* Add funscript route to scenes

Adds a /scene/:id/funscript route which serves a funscript file, if present.

Current convention is that these are files stored with the same path, but with the extension ".funscript".

* Look for funscript during scan

This is stored in the Scene record and used to drive UI changes for funscript support.

Currently, that's limited to a funscript link in the Scene's file info.

* Add filtering and sorting for interactive
* Add Handy connection key to interface config
* Add Handy client and placeholder component.

Uses defucilis/thehandy, but not thehandy-react as I had difficulty integrating the context with the existing components.

Instead, the expensive calculation for the server time offset is put in localStorage for reuse.

A debounce was added when scrubbing the video, as otherwise it spammed the Handy API with updates to the current offset.
2021-05-24 13:34:28 +10:00
WithoutPants
33999d3e93
Studio Performers page (#1405)
* Refactor performer filter
* Add performer studio criterion
* Add Studio Performers page
2021-05-22 17:07:03 +10:00
WithoutPants
3f97b3a1cb
Remove unnecessary graphql fields (#1370)
* Remove unnecessary graphql fields
* Optimise joined queries
* Tag resolver query optimisation
2021-05-09 19:25:57 +10:00
InfiniteTF
896c3874af
Stash-Box Performer Tagger (#1277)
* Add bulk stash-box performer task
* Add stash-box performer scraper to scrape with menu
2021-05-03 14:21:20 +10:00
WithoutPants
a3609079bb
Autotag support for images and galleries (#1345)
* Add compound queries for images and galleries
* Implement image and gallery auto tagging
2021-05-03 13:09:46 +10:00
julien0221
502d99de1b
Added new filters (date and title) to galleries (#1344)
* Added new filters (date and title) to galleries
* Added image_count on filter for galleries
2021-04-29 11:31:51 +10:00
julien0221
70b66d91a0
Added rating to performers and studios (#1308) 2021-04-26 13:48:32 +10:00
WithoutPants
2eb2d865dc
Auto tag rewrite (#1324) 2021-04-26 12:51:31 +10:00
julien0221
f66010a367
Fixed 0 for weight when a new performer is created and fixed the search is null (#1336) 2021-04-26 12:13:50 +10:00
WithoutPants
bf3f658091
Movie scene sort (#1325)
* Add movie_scene_number sort order
* Sort movie scenes by scene number by default
2021-04-22 12:22:51 +10:00
peolic
9200f167bf
Add studio *_count filters and sort options (#1307) 2021-04-20 16:48:36 +10:00
julien0221
d673c4ce03
added details, deathdate, hair color, weight to performers and added details to studios (#1274)
* added details to performers and studios
* added deathdate, hair_color and weight to performers
* Simplify performer/studio create mutations
* Add changelog and recategorised

Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
2021-04-16 16:06:35 +10:00
WithoutPants
ea54a67798
Add scene/image/gallery popover count buttons for performer/studio/tag cards (#1293)
* Add counts to graphql schema
* Add count resolvers and query refactor
* Add count popover buttons
2021-04-15 10:46:31 +10:00
julien0221
6a4421f8e1
Whitespace is not trimmed from the end of query strings (#1263)
* fixed whitespace not trimmed query string
* fixed whitespace trimming on backend
* added query trim tests and fixed double space
2021-04-13 10:32:52 +10:00
WithoutPants
f6ffda7504
Setup and migration UI refactor (#1190)
* Make config instance-based
* Remove config dependency in paths
* Refactor config init
* Allow startup without database
* Get system status at UI initialise
* Add setup wizard
* Cache and Metadata optional. Database mandatory
* Handle metadata not set during full import/export
* Add links
* Remove config check middleware
* Stash not mandatory
* Panic on missing mandatory config fields
* Redirect setup to main page if setup not required
* Add migration UI
* Remove unused stuff
* Move UI initialisation into App
* Don't create metadata paths on RefreshConfig
* Add folder selector for generated in setup
* Env variable to set and create config file.
Make docker images use a fixed config file.
* Set config file during setup
2021-04-12 09:31:33 +10:00
InfiniteTF
c38660d209
Add phash generation and dupe checking (#1158) 2021-04-12 09:04:40 +10:00
WithoutPants
a2582047ca
Join count filter criteria (#1254)
Co-authored-by: mrbrdo <mrbrdo@gmail.com>
Co-authored-by: peolic <66393006+peolic@users.noreply.github.com>
2021-04-09 18:46:00 +10:00
julien0221
25311247ed
added an url filter option in scenes (#1266)
* added an url filter option in scenes
* added url filter on gallery, movies, performers and studios
* Add empty string filter to stringCriterionHandler
* Add unit tests

Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
2021-04-09 15:05:11 +10:00
peolic
7acae34ed4
Fix performer search columns (#1236)
* Fix performer search columns
* Update changelog
* Move changelog to new version

Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
2021-03-30 14:04:57 +11:00
WithoutPants
960f843259
Fix filter building with sub-filters (#1212)
* Fix bracketing on sub-filters
* Add vscode to gitignore
2021-03-18 21:45:18 +11:00
WithoutPants
d93011a828
Add write mutex and max connection lifetime (#1211) 2021-03-18 21:45:01 +11:00
InfiniteTF
58243cded0
Remove slim graphql endpoints (#1207) 2021-03-17 11:17:01 +11:00
WithoutPants
7e6127975d
Handle NULL in regex criteria (#1208) 2021-03-16 11:13:14 +11:00
WithoutPants
23d85655a8
Refactor tag query (#1194) 2021-03-11 22:17:37 +11:00
WithoutPants
a0676d5c30
Performer tags (#1132)
* Add scraping support for performer tags
* Add performer count to tag cards
* Refactor sqlite test setup
* Add performer tag filtering in gallery and image
* Add bulk update performer
* Add Performers tab to tag page
* Add count filters and sort bys for tags
* Move scene count to icon in performer card #1148
2021-03-10 12:25:51 +11:00
WithoutPants
1850a2b533
Add sqlite filter builder. Add AND, OR, NOT filters to scene filter (#1115)
* Add resolution enum extension
* Add filter builder
* Use filterBuilder for scene query
* Optimise joins
* Add binary operators to scene query
* Use Query for auto-tag
2021-03-02 11:27:36 +11:00
gitgiggety
4825de7d35
Fix SQL error in 8K resolution filter (#1159) 2021-03-01 16:06:57 +11:00
WithoutPants
7cfff46d02
Gallery filter fix (#1147)
* Fix gallery performer and tags filters
* Add unit tests
2021-03-01 13:30:40 +11:00
InfiniteTF
7e0db2aad4
Change NULL filters to filter empty strings as well (#1137) 2021-03-01 11:48:25 +11:00
WithoutPants
e4d91a0226
String regex filter criteria and selective autotag (#1082)
* Add regex string filter criterion
* Use query interface for auto tagging
* Use Query interface for filename parser
* Remove query regex interfaces
* Add selective auto tag
* Use page size 0 as no limit
2021-02-02 07:57:56 +11:00
InfiniteTF
4fd022a93b
Decouple galleries from scenes (#1057) 2021-02-02 07:56:54 +11:00
WithoutPants
1e04deb3d4
Data layer restructuring (#997)
* Move query builders to sqlite package
* Add transaction system
* Wrap model resolvers in transaction
* Add error return value for StringSliceToIntSlice
* Update/refactor mutation resolvers
* Convert query builders
* Remove unused join types
* Add stash id unit tests
* Use WAL journal mode
2021-01-18 12:23:20 +11:00