* add descriptions to tags
* display tag description and tag image on hover
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
* Fire handlers when file updated or moved
* Create galleries as needed
* Clean empty galleries
* Handle cleaning zip folders when path changed
* Fix gallery association on duplicate images
* Re-create missing folder-based galleries
* Only update fingerprints if changed
* Fix panic when loading primary file fails
* Fix gallery/scene association
* Fix display of scene gallery in card
* Use natural_cs collation with paths for title sorting
* Don't recalculate MD5 if not enabled
Remove MD5 if oshash has changed and MD5 was not calculated.
* Fix panic in paged DLNA
* Prevent identical hashes in stash-box drafts
* Fix incorrect timestamp updates
* Correct folder time fields
* Add migration with new indexes
* Correct mod_time format
* Add mod_time to data massage
* Load scene relationships on demand
* Load image relationships on demand
* Load gallery relationships on demand
* Add dataloaden
* Use dataloaders
* Use where in for other find many functions
* Do database txn in same thread. Retry on locked db
* Remove captions from slimscenedata
* Fix tracing
* Use where in instead of individual selects
* Remove scenes_query view
* Remove image query view
* Remove gallery query view
* Use where in for FindMany
* Don't interrupt scanning zip files
* Fix image filesize sort
* Use cache during migration
* Avoid use of query views
* Use FindMany to find related objects
* Log slow queries
* Add folders to generated files
* Use SlimScene for scene queries
* Include filename in migration error message
* Fix destroy gallery not destroying file
* Re-add minModTime functionality
* Deprecate useFileMetadata and stripFileExtension
* Optimise files post migration
* Decorate moved files. Use first missing file in move
* Include path in thumbnail generation error log
* Fix stash-box draft submission
* Don't destroy files unless deleting
* Call handler for files with no associated objects
* Fix moved zips causing error on scan
* Restructure data layer part 2 (#2599)
* Refactor and separate image model
* Refactor image query builder
* Handle relationships in image query builder
* Remove relationship management methods
* Refactor gallery model/query builder
* Add scenes to gallery model
* Convert scene model
* Refactor scene models
* Remove unused methods
* Add unit tests for gallery
* Add image tests
* Add scene tests
* Convert unnecessary scene value pointers to values
* Convert unnecessary pointer values to values
* Refactor scene partial
* Add scene partial tests
* Refactor ImagePartial
* Add image partial tests
* Refactor gallery partial update
* Add partial gallery update tests
* Use zero/null package for null values
* Add files and scan system
* Add sqlite implementation for files/folders
* Add unit tests for files/folders
* Image refactors
* Update image data layer
* Refactor gallery model and creation
* Refactor scene model
* Refactor scenes
* Don't set title from filename
* Allow galleries to freely add/remove images
* Add multiple scene file support to graphql and UI
* Add multiple file support for images in graphql/UI
* Add multiple file for galleries in graphql/UI
* Remove use of some deprecated fields
* Remove scene path usage
* Remove gallery path usage
* Remove path from image
* Move funscript to video file
* Refactor caption detection
* Migrate existing data
* Add post commit/rollback hook system
* Lint. Comment out import/export tests
* Add WithDatabase read only wrapper
* Prepend tasks to list
* Add 32 pre-migration
* Add warnings in release and migration notes
* refactored common code in recommendation row
* Implement front page options in config
* Allow customisation from front page
* Rename recommendations to front page
* Add generic UI settings
* Support adding premade filters
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
* Make the script scraper context-aware
Connect the context to the command execution. This means command
execution can be aborted if the context is canceled. The context is
usually bound to user-interaction, i.e., a scraper operation issued
by the user. Hence, it seems correct to abort a command if the user
aborts.
* Enable errchkjson
Some json marshal calls are *safe* in that they can never fail. This is
conditional on the types of the the data being encoded. errchkjson finds
those calls which are unsafe, and also not checked for errors.
Add logging warnings to the place where unsafe encodings might happen.
This can help uncover usage bugs early in stash if they are tripped,
making debugging easier.
While here, keep the checker enabled in the linter to capture future
uses of json marshalling.
* Pass the context for zip file scanning.
* Pass the context in scanning
* Pass context, replace context.TODO()
Where applicable, pass the context down toward the lower functions in
the call stack. Replace uses of context.TODO() with the passed context.
This makes the code more context-aware, and you can rely on aborting
contexts to clean up subsystems to a far greater extent now.
I've left the cases where there is a context in a struct. My gut feeling
is that they have solutions that are nice, but they require more deep
thinking to unveil how to handle it.
* Remove context from task-structs
As a rule, contexts are better passed explicitly to functions than they
are passed implicitly via structs. In the case of tasks, we already
have a valid context in scope when creating the struct, so remove ctx
from the struct and use the scoped context instead.
With this change it is clear that the scanning functions are under a
context, and the task-starting caller has jurisdiction over the context
and its lifetime. A reader of the code don't have to figure out where
the context are coming from anymore.
While here, connect context.TODO() to the newly scoped context in most
of the scan code.
* Remove context from autotag struct too
* Make more context-passing explicit
In all of these cases, there is an applicable context which is close
in the call-tree. Hook up to this context.
* Simplify context passing in manager
The managers context handling generally wants to use an outer context
if applicable. However, the code doesn't pass it explicitly, but stores
it in a struct. Pull out the context from the struct and use it to
explicitly pass it.
At a later point in time, we probably want to handle this by handing
over the job to a different (program-lifetime) context for background
jobs, but this will do for a start.
* Upgrade gqlgen to v0.17.2
This enables builds on Go 1.18. github.com/vektah/gqlparser is upgraded
to the newest version too.
Getting this to work is a bit of a hazzle. I had to first remove
vendoring from the repository, perform the upgrade and then re-introduce
the vendor directory. I think gqlgens analysis went wrong for some
reason on the upgrade. It would seem a clean-room installation fixed it.
* Bump project to 1.18
* Update all packages, address gqlgenc breaking changes
* Let `go mod tidy` handle the go.mod file
* Upgrade linter to 1.45.2
* Introduce v1.45.2 of the linter
The linter now correctly warns on `strings.Title` because it isn't
unicode-aware. Fix this by using the suggested fix from x/text/cases
to produce unicode-aware strings.
The mapping isn't entirely 1-1 as this new approach has a larger iface:
it spans all of unicode rather than just ASCII. It coincides for ASCII
however, so things should be largely the same.
* Ready ourselves for errchkjson and contextcheck.
* Revert dockerfile golang version changes for now
Co-authored-by: Kermie <kermie@isinthe.house>
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
* Move main to cmd
* Move api to internal
* Move logger and manager to internal
* Move shell hiding code to separate package
* Decouple job from desktop and utils
* Decouple session from config
* Move static into internal
* Decouple config from dlna
* Move desktop to internal
* Move dlna to internal
* Decouple remaining packages from config
* Move config into internal
* Move jsonschema and paths to models
* Make ffmpeg functions private
* Move file utility methods into fsutil package
* Move symwalk into fsutil
* Move single-use util functions into client package
* Move slice functions to separate packages
* Add env var to suppress windowsgui arg
* Move hash functions into separate package
* Move identify to internal
* Move autotag to internal
* Touch UI when generating backend
* Continue identify if source fails
* Handle empty result set correctly
* Parse null values from scraper script correctly
* Omit warning when json selector value missing
* Return nil when scraped item not found
* Fix graphql validation errors
* Add duration to autotag finish message
* No sorting scene/image/gallery where not specified
* Use an LRU cache for sqlite regexp function
* Compile path separator regex once
* Cache objects with single letter first names
* Move finished auto-tag log
* Add more verbose logging
* Add new changelog
* Remove single unicode character from autotag query
* Compile regex once where possible
* Fix CPU profiling
* Only match unicode characters if in path
* Delete funscripts while deleting scene
* Indicate that funscripts will be deleted
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
* add InteractiveSpeed to scene model
* add InteractiveHeatmapSpeedGenerator
* add GenerateInteractiveHeatmapSpeedTask
* add InteractiveHeatmapSpeedTask to GenerateJob
* add InteractiveHeatmap on sceneRoutes
* delete heatmap when scene is destroyed
* render interactive heatmap in GridCard
* render InteractiveSpeed on SceneCard
* render InteractiveSpeed in SceneFileInfoPanel
* InteractiveSpeed filters
* Added joinType to join struct
* Added addInnerJoin function to perform INNER JOIN type of joins
* Added innerJoin function to perform INNER JOIN type of joins
* Use inner joins when querying images in a gallery
* Renamed addJoin to addLeftJoin
* Support a maxAge input on metadata scans.
Extend the GraphQL world with a Duration scalar. It is parsed as a
typical Go duration, i.e., "4h" is 4 hours. Alternatively, one can
pass an integer which is interpreted as seconds.
Extend Mutation.metadataScan(input: $input) to support a new optional
value, maxAge. If set, the scanner will exit early if the file it
is looking at has an mtime older than the cutOff point generated by
now() - maxAge
This speeds up scanning in the case where the user knows how old the
changes on disk are, by exiting the scan early if that is the case.
* Change maxAge into minModTime
Introduce a `Timestamp` scalar, so we have a scalar we control. Let
it accept three formats:
* RFC3339Nano
* @UNIX where UNIX is a unix-timestamp: seconds after 01-01-1970
* '<4h': a timestamp relative to the current server time
This scalar parses to a time.Time.
Use MinModTime in the scanner to filter out a large number of scan
analyzes by exiting the scan operation early.
* Heed the linter, perform errcheck
* Rename test vars for consistency.
* Code review: move minModTime into queuefiles
* Remove the ability to input Unix timestamps
Test failures on the CI-system explains why this is undesirable. It is
not clear what timezone one is operating in when entering a unix
timestamp. We could go with UTC, but it is so much easier to require an
RFC3339 timestamp, which avoids this problem entirely.
* Move the minModTime field into filters
Create a new filter input object for metadata scans, and push the
minModTime field in there. If we come up with new filters, they can
be added to that input object rather than cluttering the main input
object.
* Use utils.ParseDateStringAsTime
Replace time.Parse with utils.ParseDateStringAsTime
While here, add some more test cases for that parser.
* Push scrapeByURL into scrapers
Replace ScrapePerfomerByURL, ScrapeMovie..., ... with ScrapeByURL in
the scraperActionImpl interface. This allows us to delete a lot of
repeated code in the scrapers and replace the central part with a
switch on the scraper type.
* Fold name scraping into one call
Follow up on scraper refactoring. Name scrapers use the same code path.
This allows us to restructure some code and kill some functions, adding
variance to the name scraping code. It allows us to remove some code
repetition as well.
* Do not export loop refs.
* Simplify fragment scraping
Generalize fragment scrapers into ScrapeByFragment. This simplifies
fragment code flows into a simpler pathing which should be easier
to handle in the future.
* Eliminate more context.TODO()
In a number of cases, we have a context now. Use the context rather than
TODO() for those cases in order to make those operations cancellable.
* Pass the context for the stashbox scraper
This removes all context.TODO() in the path of the stashbox scraper,
and replaces it with the context that's present on each of the paths.
* Pass the context into subscrapers
Mostly a mechanical update, where we pass in the context for
subscraping. This removes the final context.TODO() in the scraper
code.
* Warn on unknown fields from scripts
A common mistake for new script writers are that they return fields
not known to stash. For instance the name "description" is used rather
than "details".
Decode disallowing unknown fields. If this fails, use a tee-reader to
fall back to the old behavior, but print a warning for the user in this
case. Thus, we retain the old behavior, but print warnings for scripts
which fails the more strict unknown-fields detection.
* Nil-check before running the postprocessing chain
Fixes panics when scraping returns nil values.
* Lift nil-ness in post-postprocessing
If the struct we are trying to post-process is nil, we shouldn't
enter the postprocessing flow at all. Pass the struct as a value
rather than a pointer, eliminating nil-checks as we go. Use the
top-level postProcess call to make the nil-check and then abort there
if the object we are looking at is nil.
* Allow conversion routines to handle values
If we have a non-pointer type in the interface, we should also convert
those into ScrapedContent. Otherwise we get errors on deprecated
functions.
* Add scan dialog
* Add Auto Tag dialog
* Refactor and combine Generate dialog
* Add clean dialog
* Support scan task default setting
* Support saving auto tag defaults
* Support for generate defaults
* Simplify scraper listing
Introduce an enum, scraper.Kind, which explains what we are looking
for. Make it possible to match this from a scraper struct.
Use the enum to rewrite all the listing code to use the same code path.
* Use a map, nitpick ScrapePerformerList
Let the cache store a map from ID of a scraper to the scraper. This
improves lookups when there are many scrapers, making it practically
O(1) rather than O(n). If many scrapers are stored, this is faster.
Since range expressions work unchanged, we don't have to change much,
and things will still work.
make Kind a Stringer
Rename ScraperPerformerList -> ScraperPerformerQuery since that name
is used in the other scrapers, and we value consistency.
Tune ScraperPerformerQuery:
* Return static errors
* Use the new functionality
* When loading scrapers, do so directly
Rather than first walking the directory structure to obtain file paths,
fold the load directly in the the filepath walk. This makes the code
for more direct.
* Use static ErrNotFound
If a scraper isn't found, return one static error. This paves the way
for eventually doing our own error-presenter in gqlgen.
* Store the cache in the Resolver state
Putting the scraperCache directly in the resolver avoids the need to
call manager.GetInstance() all over the place to get access to the
scraper cache. The cache is stored by pointer, so it should be safe,
since the cache will just update its internal state rather than being
overwritten.
We can now utilize the resolver state to grab the cache where needed.
While here, pass context.Context from the resolver down into a function,
which removes a context.TODO()
* Introduce ScrapedContent
Create a union in the GraphQL schema for all scraped content. This
simplifies the internal implementation because we get variance on
the output content type.
Introduce a new type ScrapedContentType which signifies the scraped
content you want as a caller.
Use these to generalize the List interface and the URL scraping
interface.
* Simplify the scraper API
Introduce a new interface for scraping. This interface is then
used in the upper half of the scraper code, to make the code use one
code flow rather than multiple code flows. Variance is currently at
the old scraper structure.
Add extending interfaces for the different ways of invoking scrapes.
Use interface conversions to convert a scraper from the cache to a
scraper supporting the extra methods.
The return path returns models.ScrapedContent.
Write a general postProcess function in the scraper, handling all
ScrapedContent via type switching. This consolidates all postprocessing
code flows.
Introduce marhsallers in the resolver code for converting ScrapedContent
into the underlying concrete types. Use this to plug the existing
fields in the Query resolver, so everything still works.
* ScrapedContent: add more marshalling functions
Handle all marshalling of ScrapedContent through marhsalling functions.
Removes some hand-rolled early variants of it, and replaces it with
a canonical code flow.
* Support loadByName via scraper_s
In order to temporarily plug a hole in the current implementation, we
use the older implementation as a hook to get the newer implementation
to run.
Later on, this can serve as a guide for how to implement the lower level
bits inside the scrapers themselves. For now, it just enables support.
* Plug the remaining scraper functions for now
Since we would like to have a scraper which works in between refactors,
plug the lower level parts of the scraper for now. It avoids us having
to tackle this part just yet.
* Move postprocessing to its own file
There's enough postprocessing to clutter the main scrapers.go file.
Move all of this into a new file, postprocessing to make the API
simpler. It now lives in scrapers.go.
* Scraper: Invoke API consistency
scraper.Cache.ScrapeByName -> ScrapeName
* Fix scraping scenes by URL
Simple typo. While here, also make a single marshaller nil-aware.
* Introduce scraper groups, consolidate loadByURL
Rename `scraper_s` into `group`. A group is a group of scrapers with
the same identity. This corresponds to a single YAML file for a scraper
configuration. It defines a group which supports different types of
scraping contexts.
Move config into the group, and lift txnManager and globalConfig to
the group.
Because we now return models.ScrapedContent we can use interfaces to
get variance from the different underlying scrapers. Use a type
switch for the URL matcher candidates. And then again for the scrapers.
This consolidates all URL scraping paths into one.
While here, remove the urlMatcher interface which isn't needed. Also
clean up the remaining interfaces for url scraping and delete code
which has no purpose anymore.
* Consolidate fragment scraping in one code path
While here, abide the linters checks.
* Refactor loadByFragment
Give it the same treatment as loadByURL:
Step 1: find a scraperActionImpl which works for the data.
Step 2: use that to scrape
Most of this is simple analysis on the data at hand. It can be pushed
down further in a later commit, but for now we leave it here.
* Remove configScraper, autotag is a scraper
Remove the remains of the configScraper struct. It now lives on in the
group struct. Kill the remaining interfaces from the old implementation
while here.
Remove group.specification since it can now be handled by a simple
func call to spec().
Work through the autotag scraper. It now implements the scraper
interface, so it can be used as a scraper. This also simplifies the
autotag scraper quite a bit since it doens't have to implement a number
of unsupported func calls.
* Simplify the fragment scraper flow
* Pass the context
Eliminate a round of context.TODO() in the scraper code by passing
the calling context down into the subsystem. This will gracefully
allow for termination of remote calls if the client goes away for some
reason in GraphQL requests.
* Improve listScrapers in the schema
Support lists of types we accept.
* Be graceful on nil values in conversion
Supporting nil-values make the API more robust in the
case of partial results in a multi-scrape situation.
* Improve listScrapers: output at-most-once
Use the ID of a scraper to reduce the output set. If a scraper has
been included, don't include it again.
* Consolidate all API level errors into resolver.go
* Reorder files and functions:
scrapers.go -> cache.go:
It almost contains nothing but the cache code.
Move errors into scraper.go from here because
It is a better place to have them living right now
group.go:
All of the group structure. This can now go from
scraper.go, making it more lean. Move group create
from config_scraper to here.
config.go:
Move the `(c config) spec()` call to here.
config_scraper.go:
Empty file by now
* Name-update the scraper interfaces
Use 'via' rather than 'loadBy'.
The scrape happens via a given scrape method, so I think this is a nice
name for it.
* Rename scrapers for consistency.
While here, improve the error formatting, so different errors come
back differently.
* Nuke the freeones field from the GraphQL schema
* Fix autotag interfacing, refactor
The autotag scraper uses a pointer receiver, but the rest of the code
we use for scraping doesn't expect a pointer-receiver. Hence, to fix
the autotag scraper, we change it to be a value receiver, like the
rest of the code.
Fix: viaScene, and viaGallery.
While here, remove a couple of pointer-receiver methods which can be
trivially rewritten into plain functions.
* Protect against pointer interfaces
The underlying code can be a bit inconsistent in what it returns.
Introduce pointer-types in the postprocessing layer and handle them
accordingly for now. Once a better understanding of the lower levels
are understood, we can lift this.
* Move ErrConversion into the models package.
The conversion error pertains to the logic of converting models.
Because of this, it should move there, so it is centralized.
* Be consistent in scraper resolver error handling
If we have a static error
Err = errors.New(..)
Then use it wrapped at the start:
fmt.Errorf("%w: ...context...", Err)
This reads better.
While here, avoid using the underlying Atoi errors: they are verbose,
and like 99% of the time, the user know what is wrong from the input
string, so just give that back.
Also, remove the scraper id from the error contexts: it is implicit,
and the error wouldn't change if we used a different scraper, which
the error message would imply.
* Mark the list*Scrapers() API as deprecated
The same functionality is now present in listScrapers.
* Improve error formatting
Think about how each error is going to be used and tweak them to be
nicer.
* Return a sorted list of scrapers
This helps testing, it's closer to what we had, caches like stable data,
and it is easier for humans. It also makes the output stable, because
map iteration is randomized.
* Fix listScrapers calls to return in ID-order
Since we need the ordering to be by ID in all situations, it is easier
to just generalize the cache listScrapers call to support multiple
scraper types.
This avoids a de-dupe map up the chain, since every scraper is only
considered once. Sorting now happens in the cache listScrapers call.
Use this generalized function in all resolvers, which are now simple
passthroughs.
* Remove UpdateConfig from the scraper cache.
This isn't needed, so get rid of it.
* Pull a context into identify
Scraping scenes in the identify tasks now use a context from up the
call chain.
* Do not store the scraper cache in the resolver.
Scraper caches are updated through
manager.singleton•RefreshScraperCache, so we can't keep a pointer to
it in the resolver. Instead, solve this by adding a fetcher method to
the resolver type. This keeps it local to the resolver, while handling
the problem of updating caches in the configuration.
* Separate overrides from config
* Don't allow changing overridden value
* Write default host and port to config file
* Use existing library value. Hide generated if set
* Support Is (not) null for all multi criterions
Add support for the Is null and Is not null modifiers for all cases of
the MultiCriterionInput and HierarchicalMultiCriterionInput. This
partially overlaps the "X Count" filter which sometimes is available
(because it would be the same as "X Count equals 0" and "X Count greater
than 0") but this also enables it for other criterions like the "Parent
Studio" filter for studios or just the "Studios" filter for scenes /
images / galleries, the "Movies" filter for scenes etc.
* Don't crash UI on bad saved filter
* Add missing code for tag parent/child
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
* update tag hierarchy validation
* refactor MergeHierarchy
* update tag hierarchy error message
* rename tag hierarchy function
* add tag path to error message
* Rename EnsureHierarchy to ValidateHierarchy
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
* add delete file and generated files by default config options
* add alert message with files to be deleted
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
The version checking code performs its own error management and will
not pass errors to the caller. Hence, it needs to be aware of the types
of errors which can be returned.
In particular, the context.Canceled error will be returned if the
context is aborted through cancelation. This happens when the request
is terminated by tapping CTRL-C or if the browser request is terminated
while we are sitting waiting for the GH API.
* Docker CI builds: half the size, less than half the build time
* Add an "Official Build" Designator
* Fix .git constantly invalidating build cache, use distro ffmpeg
* Fix official build detection, add some compiler image docs
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
* Make copies of buffers
Avoid reusing one of the incoming arrays as a append extension, and
make a copy of the data. It's cleaner in the long run and possibly
easier for the GC to maintain.
* Avoid appendAssign problems in tag code
Reuse the existing slice when appending.
* Fix appendAssign in encoder_scene_preview_chunk
Appending and creating a new slice is somewhat unintuitive since the
underlying slice might be extended to satisfy the new capacity. This
sometimes leads to faulty logic.
Rewrite the code so it reuses `args` for all appending, and builds one
array clearly in the code. It follows the general style of the function
where `args` is being built in small incremental batches and avoids
the introduction of new names.
* Enable the appendAssign check
This makes us pass all gocritic warnings.
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
* Add Cookies directly to the request
Rather than maintaining a cookie jar on a one-shot HTTP client, maintain
the jar ourselves: make a new jar, then use it to select the right
cookies.
The cookies are set on the request rather than on the client. This will
retain the current behavior as we are always throwing the client away
after each use.
This patch enables the lifting of the http client as well over time.
* Introduce a cached scraper HTTP client
The scraper cache is augmented with an *http.Client. These are safe for
concurrent use, so the pointer can safely be passed around. Push this
into scraper configurations where applicable, next to the txnManagers.
When we issue a loadUrl request, do so on the cached *http.Client,
which will reuse existing idle connections in the client if any are
present.
* Set MaxIdleConnsPerHost. Closes#1850
We allow for up to 8 idle connections to a single host. This should
make concurrent operation toward the same host reuse connections, even
for sizeable concurrency.
The number isn't bumped excessively high. We should probably limit
concurrency toward a single site anyway, since we'll be able to overrun
a site with queries quite easily if we have many concurrent goroutines
issuing requests at the same time.
* Reinstate driverOptions / useCDP check
Use DeMorgan's laws to invert the logic and exit early. Fixes tests
breaking.
* Documentation fixup.
* Use the scraper http.Client when fetching images
Fold image fetchers onto the cached scraper http.Client as well. This
makes the scraper have a single http.Client cache for all its
operations.
Thread the client upwards to the relevant attachment points: either the
cache, or a stash_box instance, which is extended to include a pointer
to the client.
Style roughly follows that of txnManagers.
* Use the same http Client as the GraphQL client use
Rather than using http.DefaultClient, use the same client as the
GraphQL client use in the stash_box subsystem. This localizes the
client used in the subsystem into the constructing New.. call.
* Hoist HTTP client construction
Create a function for initializaing the HTTP Client we use. While here
hoist magic numbers into constants. Introduce a proper static redirect
error and use it in the client code as well.
* Reinstate printCookies
This is a debugging function, and it might still come in handy in the
future at some point.
* Nitpick comment.
* Minor tidy
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
* Add a space after // comments
For consistency, the commentFormatting lint checker suggests a space
after each // comment block. This commit handles all the spots in
the code where that is needed.
* Rewrite documentation on functions
Use the Go idiom of commenting:
* First sentence declares the purpose.
* First word is the name being declared
The reason this style is preferred is such that grep is able to find
names the user might be interested in. Consider e.g.,
go doc -all pkg/ffmpeg | grep -i transcode
in which case a match will tell you the name of the function you are
interested in.
* Remove old code comment-blocks
There are some commented out old code blocks in the code base. These are
either 3 years old, or 2 years old. By now, I don't think their use is
going to come back any time soon, and Git will track old pieces of
deleted code anyway.
Opt for deletion.
* Reorder imports
Split stdlib imports from non-stdlib imports in files we are touching.
* Use a range over an iteration variable
Probably more go-idiomatic, and the code needed comment-fixing anyway.
* Use time.After rather than rolling our own
The idiom here is common enough that the stdlib contains a function for
it. Use the stdlib function over our own variant.
* Enable the commentFormatting linter
* Don't capitalize local variables
ValidCodecs -> validCodecs
* Capitalize deprecation markers
A deprecated marker should be capitalized.
* Use re.MustCompile for static regexes
If the regex fails to compile, it's a programmer error, and should be
treated as such. The regex is entirely static.
* Simplify else-if constructions
Rewrite
else { if cond {}}
to
else if cond {}
* Use a switch statement to analyze formats
Break an if-else chain. While here, simplify code flow.
Also introduce a proper static error for unsupported image formats,
paving the way for being able to check against the error.
* Rewrite ifElse chains into switch statements
The "Effective Go" https://golang.org/doc/effective_go#switch document
mentions it is more idiomatic to write if-else chains as switches when
it is possible.
Find all the plain rewrite occurrences in the code base and rewrite.
In some cases, the if-else chains are replaced by a switch scrutinizer.
That is, the code sequence
if x == 1 {
..
} else if x == 2 {
..
} else if x == 3 {
...
}
can be rewritten into
switch x {
case 1:
..
case 2:
..
case 3:
..
}
which is clearer for the compiler: it can decide if the switch is
better served by a jump-table then a branch-chain.
* Rewrite switches, introduce static errors
Introduce two new static errors:
* `ErrNotImplmented`
* `ErrNotSupported`
And use these rather than forming new generative errors whenever the
code is called. Code can now test on the errors (since they are static
and the pointers to them wont change).
Also rewrite ifElse chains into switches in this part of the code base.
* Introduce a StashBoxError in configuration
Since all stashbox errors are the same, treat them as such in the code
base. While here, rewrite an ifElse chain.
In the future, it might be beneifical to refactor configuration errors
into one error which can handle missing fields, which context the error
occurs in and so on. But for now, try to get an overview of the error
categories by hoisting them into static errors.
* Get rid of an else-block in transaction handling
If we succesfully `recover()`, we then always `panic()`. This means the
rest of the code is not reachable, so we can avoid having an else-block
here.
It also solves an ifElse-chain style check in the code base.
* Use strings.ReplaceAll
Rewrite
strings.Replace(s, o, n, -1)
into
strings.ReplaceAll(s, o, n)
To make it consistent and clear that we are doing an all-replace in the
string rather than replacing parts of it. It's more of a nitpick since
there are no implementation differences: the stdlib implementation is
just to supply -1.
* Rewrite via gocritic's assignOp
Statements of the form
x = x + e
is rewritten into
x += e
where applicable.
* Formatting
* Review comments handled
Stash-box is a proper noun.
Rewrite a switch into an if-chain which returns on the first error
encountered.
* Use context.TODO() over context.Background()
Patch in the same vein as everything else: use the TODO() marker so we
can search for it later and link it into the context tree/tentacle once
it reaches down to this level in the code base.
* Tell the linter to ignore a section in manager_tasks.go
The section is less readable, so mark it with a nolint for now. Because
the rewrite enables a ifElseChain, also mark that as nolint for now.
* Use strings.ReplaceAll over strings.Replace
* Apply an ifElse rewrite
else { if .. { .. } } rewrite into else if { .. }
* Use switch-statements over ifElseChains
Rewrite chains of if-else into switch statements. Where applicable,
add an early nil-guard to simplify case analysis. Also, in
ScanTask's Start(..), invert the logic to outdent the whole block, and
help the reader: if it's not a scene, the function flow is now far more
local to the top of the function, and it's clear that the rest of the
function has to do with scene management.
* Enable gocritic on the code base.
Disable appendAssign for now since we aren't passing that check yet.
* Document the nolint additions
* Document StashBoxBatchPerformerTagInput
* Use the request context
The code uses context.Background() in a flow where there is a
http.Request. Use the requests context instead.
* Use a true context in the plugin example
Let AddTag/RemoveTag take a context and use that context throughout
the example.
* Avoid the use of context.Background
Prefer context.TODO over context.Background deep in the call chain.
This marks the site as something which we need to context-handle
later, and also makes it clear to the reader that the context is
sort-of temporary in the code base.
While here, be consistent in handling the `act` variable in each
branch of the if .. { .. } .. check.
* Prefer context.TODO over context.Background
For the different scraping operations here, there is a context
higher up the call chain, which we ought to use. Mark the call-sites
as TODO for now, so we can come back later on a sweep of which parts
can be context-lifted.
* Thread context upwards
Initialization requires context for transactions. Thread the context
upward the call chain.
At the intialization call, add a context.TODO since we can't break this
yet. The singleton assumption prevents us from pulling it up into main for
now.
* make tasks context-aware
Change the task interface to understand contexts.
Pass the context down in some of the branches where it is needed.
* Make QueryStashBoxScene context-aware
This call naturally sits inside the request-context. Use it.
* Introduce a context in the JS plugin code
This allows us to use a context for HTTP calls inside the system.
Mark the context with a TODO at top level for now.
* Nitpick error formatting
Use %v rather than %s for error interfaces.
Do not begin an error strong with a capital letter.
* Avoid the use of http.Get in FFMPEG download chain
Since http.Get has no context, it isn't possible to break out or have
policy induced. The call will block until the GET completes. Rewrite
to use a http Request and provide a context.
Thread the context through the call chain for now. provide
context.TODO() at the top level of the initialization chain.
* Make getRemoteCDPWSAddress aware of contexts
Eliminate a call to http.Get and replace it with a context-aware
variant.
Push the context upwards in the call chain, but plug it before the
scraper interface so we don't have to rewrite said interface yet.
Plugged with context.TODO()
* Scraper: make the getImage function context-aware
Use a context, and pass it upwards. Plug it with context.TODO()
up the chain before the rewrite gets too much out of hand for now.
Minor tweaks along the way, remove a call to context.Background()
deep in the call chain.
* Make NOTIFY request context-aware
The call sits inside a Request-handler. So it's natural to use the
requests context as the context for the outgoing HTTP request.
* Use a context in the url scraper code
We are sitting in code which has a context, so utilize it for the
request as well.
* Use a context when checking versions
When we check the version of stash on Github, use a context. Thread
the context up to the initialization routine of the HTTP/GraphQL
server and plug it with a context.TODO() for now.
This paves the way for providing a context to the HTTP server code in a
future patch.
* Make utils func ReadImage context-aware
In almost all of the cases, there is a context in the call chain which
is a natural use. This is true for all the GraphQL mutations.
The exception is in task_stash_box_tag, so plug that task with
context.TODO() for now.
* Make stash-box get context-aware
Thread a context through the call chain until we hit the Client API.
Plug it with context.TODO() there for now.
* Enable the noctx linter
The code is now free of any uncontexted HTTP request. This means we
pass the noctx linter, and we can enable it in the code base.
* Add collation to directory listings. Closes#1806
Introduce a new `locale` arg to the `Query.directory` field. Set "en"
as the default for the field for backward compatibility. Use the given
locale, sending it through a language matcher, and use `x/text` as the
collation engine for the matched language.
Augment the file `ListDirs` call to optionally take a Collator. If the
Collator is given, sort file listings according to the collators rules.
While here, document the GraphQL schema a bit more.
Add matchers by looking at the current front-end locales, and make sure
each of these occur in the matcher list.
* Language matcher touchups
* Avoid having `en-US` twice.
* Introduce `en-AU`.
* Pass IgnoreCase and Numeric collation
Allow the collator to be configured with options. Pass the options
IgnoreCase and Numeric to the collator.
* Replace error assertions with Go 1.13 style
Use `errors.As(..)` over type assertions. This enables better use of
wrapped errors in the future, and lets us pass some errorlint checks
in the process.
The rewrite is entirely mechanical, and uses a standard idiom for
doing so.
* Use Go 1.13's errors.Is(..)
Rather than directly checking for error equality, use errors.Is(..).
This protects against error wrapping issues in the future.
Even though something like sql.ErrNoRows doesn't need the wrapping, do
so anyway, for the sake of consistency throughout the code base.
The change almost lets us pass the `errorlint` Go checker except for
a missing case in `js.go` which is to be handled separately; it isn't
mechanical, like these changes are.
* Remove goconst
goconst isn't a useful linter in many cases, because it's false positive
rate is high. It's 100% for the current code base.
* Avoid direct comparison of errors in recover()
Assert that we are catching an error from recover(). If we are,
check that the error caught matches errStop.
* Enable the "errorlint" checker
Configure the checker to avoid checking for errorf wraps. These are
often false positives since the suggestion is to blanket wrap errors
with %w, and that exposes the underlying API which you might not want
to do.
The other warnings are good however, and with the current patch stack,
the code base passes all these checks as well.
* Configure rowserrcheck
The project uses sqlx. Configure rowserrcheck to include said package.
* Mechanically rewrite a large set of errors
Mechanically search for errors that look like
fmt.Errorf("...%s", err.Error())
and rewrite those into
fmt.Errorf("...%v", err)
The `fmt` package is error-aware and knows how to call err.Error()
itself.
The rationale is that this is more idiomatic Go; it paves the
way for using error wrapping later with %w in some sites.
This patch only addresses the entirely mechanical rewriting caught by
a project-side search/replace. There are more individual sites not
addressed by this patch.
Reduce allocations. Don't create intermediary arrays which we then
consume right after. Manually fuse the arrays and decode straight into
the sum instead.
Furthermore, don't invoke a Reader, but carve out the locations via a
loop, directly.
These two changes taken together speeds up oshash computations by a
factor of 10 according to the benchmark tests. The main reason for
this change is a much lowered memory allocation rate which in turn
improves GC pressure.
While here, add a benchmark for oshash computations and use it for
testing the performance.
* Refactor scraper structures
* Move matching code into new package
* Add autotag scraper
* Always check first letter of auto-tag names
* Account for nulls
Co-authored-by: Kermie <kermie@isinthe.house>
* Add security against publicly exposed services
* Add trusted proxies setting, validate proxy chain against internet access
* Validate chain on local proxies too
* Move authentication handler to separate file
* Add startup check and log if tripwire is active
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
* Enable safe linters
Enable the linters dogsled, rowserrcheck, and sqlclosecheck.
These report no errors currently in the code base.
Enable misspell.
Misspell finds two spelling mistakes in comments, which are fixed by the
patch as well.
Add and sort linters which are relatively
safe to add over time. Comment them out for now.
* Close the response body
If we can get a HTTP response, it has a body which ought to be closed.
By doing so, we avoid potentially leaking connections.
* Enable the exportloopref linter
There are two places in the code with these warnings. Fix them while
enabling the linter.
* Remove redundant types in tests
If a slice already determines the type, the inner type declaration is
redundant. Remove the inner declarations.
* Mark autotag test cases as parallel
Autotag test cases is by far the outlier when it comes to test time.
While go test runs test cases in parallel,
it doesn't do so inside a given package, unless one marks the test cases
as parallel.
This change provides a significant speedup on a 8-core machine for test
runs.
* Add API support for filtering tags by parent / children
* Add parent & child tags filters for tags to UI
* Add API support for filtering tags by parent / child count
* Add parent & child count filters for tags to UI
* Update db generator
* Add missing build tag
* Add unit tests
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
* Fix all revive warnings in the code base
All of these are of the form
```
var Identifier Type = Expr
```
where the `Type` is known from the output of `Expr` and can be omitted
as a result.
* Handle unchecked errors
* Remove new-from-rev
Since the project passes all linter checks now, including older
revisions, we can remove new-from-rev. While here, reorder the linter
config file, and move the enabled linters up and settings down.
* Fix failing test cases
Studio & Performer export tests use local time rather than UTC. This
fixes the test cases so integration test
passes.
* Add movies and movie_count properties to Performer type
Extend the GraphQL API to allow getting the movies and movie count by
performer.
* Add movies count to performer card
* Add movies and movie_count properties to Studio type
Extend the GraphQL API to allow getting the movies and movie count by
studio.
* Add movies count to studio card
The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil. This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
* Support getting scenes on movies in the API
* Make movie card visually consistent with scene etc
* Add date
* Add synopsis
* Show scene count with hover listing the scenes
* Move scene index to button
* Move scene number to own section
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
* Log 3 unchecked errors
Rather than ignore errors, log them at
the WARNING log level.
The server has been functioning without these, so assume they are not at
the ERROR level.
* Log errors in concurrency test
If we can't initialize the configuration, treat the test as a failure.
* Undo the errcheck on configurations for now.
* Handle unchecked errors in pkg/manager
* Resolve unchecked errors
* Handle DLNA/DMS unchecked errors
* Handle error checking in concurrency test
Generalize config initialization, so we can initialize a configuration
without writing it to disk.
Use this in the test case, since otherwise the test fails to write.
* Handle the remaining unchecked errors
* Heed gosimple in update test
* Use one-line if-initializer statements
While here, fix a wrong variable capture error.
* testing.T doesn't support %w
use %v instead which is supported.
* Remove unused query builder functions
The Int/String criterion handler functions are now generalized.
Thus, there's no need to keep these functions around anymore.
* Mark filterBuilder.addRecursiveWith nolint
The function is useful in the future and no other refactors are looking
nice.
Keep the function around, but tell the linter to ignore it.
* Remove utils.Btoi
There are no users of this utility function
* Return error on scan failure
If we fail to scan the row when looking for the
unique checksum index, then report the error upwards.
* Fix comments on exported functions
* Fix typos
* Fix startup error
* Improve image scanning performance and thumbnail generation
* Add vips-tools to build image
* Add option to write generated thumbnails to disk
* Fallback to image if thumbnail generation fails
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
Rather than passing a pointer to a waitgroup into task.Start(..)
functions, handle the waitgroup.Done() at the callsite.
This makes waitgroup handling local to its definition rather than it
being spread out over multiple files. Tasks now simply execute, and
the policy of waiting on them is handled by the caller.
* Support subpaths when serving stash through a reverse proxy
* Add README documentation
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
* Add indexes for path and checksum to images
The scenes table has unique indexes/constraints on path and checksum
colums. The images table doesn't, which doesn't really make sense, as
scanning uses these colums extensively which warrents an index, and both
should be unique as well.
Adding these indexes thus heavily improves the scanning tasks
performance. On a database containing 4700 images a (re)scan of those
4700 files, which thus shouldn't do anything, took 1.2 seconds, with the
indexes added this only takes 0.4 seconds. Taking the same test on a
generated database containing 4M images + the actual 4700 images took 26
minutes for a rescan, and with the index existing also only takes 0.4
seconds.
* Add images.checksum unique constraint in code with fallback
Work around the issue where in some cases duplicate images (/checksums
on images) might exist. This as discussed in #1740 by creating the index
on startup and in case of an error logging the duplicates. This so the
users where this scenario exists can correct the database (by searching
on the logged checksum(s) and removing the duplicates) and after a
restart the unique index / constraint will still be created. In case
when creating the unique index fails a "normal" / non-unique index is
created as surrogate so the user will still get the performance benefit
(for example during scanning) without being forced to remove the
duplicates and restart beforehand. This surrogate is also automatically
cleaned up after the unique index is succesfully created.
NORMAL sync is safe when using WAL journaliing.
It cuts the amount of sync calls to the disk, resulting in faster write
operation. On power loss, the database will perhaps lose some ongoing
commits, but that is all.
The expectation is that if the database lives on the same disk as the
stash, this could help performance quite a bit under heavier operation.
* Avoid redundant logging in migrations
Return the error and let the caller handle the logging of the error if
needed.
While here, defer m.Close() to the function boundary.
* Treat errors as values
Use %v rather than %s and pass the errors directly.
* Generate a wrapped error on stat-failure
* Log 3 unchecked errors
Rather than ignore errors, log them at
the WARNING log level.
The server has been functioning without these, so assume they are not at
the ERROR level.
* Propagate errors upward
Failure in path generation was ignored. Propagate the errors upward the
call stack, so it can be handled at the level of orchestration.
* Warn on errors
Log errors rather than quenching them.
Errors are logged at the Warn-level for now.
* Check error when creating test databases
Use the builtin log package and stop the program fatally on error.
* Add warnings to uncheck task errors
Focus on the task system in a single commit, logging unchecked
errors as warnings.
* Warn-on-error in API routes
Look through the API routes, and make sure errors are being logged if
they occur. Prefer the Warn-log-level because none of these has proven
to be fatal in the system up until now.
* Propagate error when adding Util API
* Propagate error on adding util API
* Return unhandled error
* JS log API: propagate and log errors
* JS Plugins: log GQL addition failures.
* Warn on failure to write to stdin
* Warn on failure to stop task
* Wrap viper.BindEnv
The current viper code only errors if no name is provided, so it should
never fail. Rewrite the code flow to factor through a panic-function.
This removes error warnings from this part of the code.
* Log errors in concurrency test
If we can't initialize the configuration, treat the test as a failure.
* Warn on errors in configuration code
* Plug an unchecked error in gallery zip walking
* Warn on screenshot serving failure
* Warn on encoder screenshot failure
* Warn on errors in path-handling code
* Undo the errcheck on configurations for now.
* Use one-line initializers where applicable
rather than using
err := f()
if err!= nil { ..
prefer the shorter
if err := f(); err != nil { ..
If f() isn't too long of a name, or wraps a function with a body.
* Execute Gallery.Create.Post plugin hook during scan
Fix issue where Gallery.Create.Post hook is not executed when a new
gallery is created during scan, when the gallery is created based on the
folder.
* Fix Gallery.Create.Post hook being invoked in transaction
Invoke the Gallery.Create.Post hook during zip scan after the
transaction has been committed. This is necessary to allow the plugin to
access the gallery (using GraphQL API). Otherwise the API obviously uses
a different database transaction which can't find the gallery as it
isn't committed yet.
* Fix logs from scraper and plugins not being shown in UI
Using `logger.` in the logger package to write logs is "incorrect". This
as the package contains a variable named `logger` which contains the
logrus instance. So instead of the log line being handled by the custom
log implementation / wrapper which makes sure the lines are shown in the
UI as well, it's written to logrus directly meaning the wrapper is
skipped.
This "issue" is obviously triggered because in any other place
`logger.X` can be used and it will used the custom logger package /
wrapper which works fine.
* Add plugin / scraper name to logging output
Indicate which plugin / scraper wrote a log message by including its
name to the `[Scrape]` prefix.
* Add missing addLogItem call
* Generate screenshot images for markers
In some scenarios it might not be possible to use the preview video or
image of markers, i.e. when only static images are supported like in
Kodi. So generate a static screenshot as well.
* Make generating animated and static image optional for markers
* Use screenshot for markers when preview type is set to static image
* Add migration to create studio aliases table
* Refactor studioQueryBuilder.Query to use filterBuilder
* Expand GraphQL API with aliases support for studio
* Add aliases support for studios to the UI
* List aliases in details panel
* Allow editing aliases in edit panel
* Add 'aliases' filter when searching
* Find studios by alias in filter / select
* Add auto-tagging based on studio aliases
* Support studio aliases for filename parsing
* Support importing and exporting of studio aliases
* Search for studio alias as well during scraping
* Add migration script for tag relations table
* Expand hierarchical filter features
Expand the features of the hierarchical multi input filter with support
for using a relations table, which only has parent_id and child_id
columns, and support adding an additional intermediate table to join on,
for example for scenes and tags which are linked by the scenes_tags
table as well.
* Add hierarchical filtering for tags
* Add hierarchical tags support to scene markers
Refactor filtering of scene markers to filterBuilder and in the process
add support for hierarchical tags as well.
* List parent and child tags on tag details page
* Support setting parent and child tags
Add support for setting parent and child tags during tag creation and
tag updates.
* Validate no loops are created in tags hierarchy
* Update tag merging to support tag hierarcy
* Add unit tests for tags.EnsureUniqueHierarchy
* Fix applying recursive to with clause
The SQL `RECURSIVE` of a `WITH` clause only needs to be applied once,
imediately after the `WITH`. So this fixes the query building to do just
that, automatically applying the `RECURSIVE` keyword when any added with
clause is added as recursive.
* Rename hierarchical root id column
* Rewrite hierarchical filtering for performance
Completely rewrite the hierarchical filtering to optimize for
performance. Doing the recursive query in combination with a complex
query seems to break SQLite optimizing some things which means that the
recursive part might be 2,5 second slower than adding a static
`VALUES()` list. This is mostly noticable in case of the tag hierarchy
where setting an exclusion with any depth (or depth: all) being applied
has this performance impact of 2,5 second. "Include" also suffered this
issue, but some rewritten query by joining in the *_tags table in one
pass and applying a `WHERE x IS NOT NULL` filter did seem to optimize
that case. But that optimization isn't applied to the `IS NULL` filter
of "exclude". Running a simple query beforehand to get all (recursive)
items and then applying them to the query doesn't have this performance
penalty.
* Remove UI references to child studios and tags
* Add parents to tag export
* Support importing of parent relationship for tags
* Assign stable ids to parent / child badges
* Silence Apollo warning on parents/children fields on tags
Silence warning triggered by Apollo GraphQL by explicitly instructing it
to use the incoming parents/children values. By default it already does
this, but it triggers a warning as it might be unintended that it uses
the incoming values (instead of for example merging both arrays).
Setting merge to false still applies the same behaviour (use only
incoming values) but silences the warning as it's explicitly configured
to work like this.
* Rework detecting unique tag hierarchy
Completely rework the unique tag hierarchy to detect invalid hierarchies
for which a tag is "added in the middle". So when there are tags A <- B
and A <- C, you could previously edit tag B and add tag C as a sub tag
without it being noticed as parent A being applied twice (to tag C).
While afterwards saving tag C would fail as tag A was applied as parent
twice. The updated code correctly detects this scenario as well.
Furthermore the error messaging has been reworked a bit and the message
now mentions both the direct parent / sub tag as well as the tag which
would results in the error. So in aboves example it would now show the
message that tag C can't be applied because tag A already is a parent.
* Update relations on cached tags when needed
Update the relations on cached tags when a tag is created / updated /
deleted so these always reflect the correct state. Otherwise (re)opening
a tag might still show the old relations untill the page is fully
reloaded or the list is navigated. But this obviously is strange when
you for example have tag A, create or update tag B to have a relation to
tag A, and from tags B page click through to tag A and it doesn't show
that it is linked to tag B.
The strings.Replace function counts the number of replacements. If 0,
the original string is returned. Hence, there is no need to check if a
replacement will happen before doing the work.
* Remove stuff which isn't being used
Some fields, functions and structs aren't in use by the project. Remove
them for janitorial reasons.
* Remove more unused code
All of these functions are currently not in use. Clean up the code by
removal, since the version control has the code if need be.
* Remove unused functions
There's a large set of unused functions and variables in the code base.
Remove these, so it clearer what code to support going forward.
Dead code has been eliminated.
Where applicable, comment const-sections in tests, so reserved
identifiers are still known.
* Fix use-def of tsURL
The first def of tsURL doesn't matter because there's no use before
we hit the 2nd def.
* Remove dead code assignment
Setting logFile = "" is effectively dead code, because there's no use
of it later.
* Comment out found
The variable 'found' is dead in the function (because no post-process
action is following it). Comment it for now.
* Comment dead code in tests
These might provide hints as to what isn't covered at the moment.
* Dead code removal
In the case of constants where iota is involved, move the iota so it
matches the current key values.
This avoids problems with persistently stored key IDs.
* Bump Go to 1.17, refactor build/x86_64 Dockerfile to make better use of multi-stage
* Bump to 1.17 from 1.16
* Bump packr version, provide needed legacy env var
* Add apple silicon support, fix macos build chain
* Update unused travis ci
* Fix error string capitalization
Error strings often follow another string. Hence, they should not be
capitalized, unless referencing a name.
* Uncapitalize more error strings
While here, use %v on the error directly, which makes it easier to wrap
the error later with %w if need be.
* Uncapitalize more error strings
While here, rename Url to URL as a nitpick.
* When stopping, close the database
This patch is likely to cause errornous behavior in the application.
This is due to the fact that the application doesn't gracefully shut
down, but is forcefully terminated.
However, the purpose is to uncover what needs to be done, to make
it a more graceful shutdown.
The call to p.ExecuteTask happens in a separate go-routine. It writes,
under m.mutex, into the job's details. However, the test goroutine
itself reads j.Details which is a read race.
Protect the reads in the test by the lock.
This makes `package job` pass `go test -race`
* Unify scraped types
* Make name fields optional
* Unify single scrape queries
* Change UI to use new interfaces
* Add multi scrape interfaces
* Use images instead of image
* Add script offset / delay to Handy support.
Further work on #1376.
Offsets are added to the current video position, so a positive value leads to earlier motion. (The most common setting.)
This is needed because most script times have a consistent delay when compared to the video. (Delay from the API calls to the server should be handled by the server offset calculation.)
* Rename scriptOffset to funscriptOffset
* Correct localisation keys
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
* Add sorting on image and gallery count to performers
* Make performer table headers translatable
* Add image and gallery count to performers table
* Make sure list table container fits the table
Make the table container minimally as wide as the table. This fixes the
table "overflowing" to the left and right and the left not being
accessible.
* Remove unnecessary truncation in tables
IMO there is no need to truncate the title in the scenes table and the
name in the performers table. This because both tables also contain an
image which means that multiple lines should be possible without really
extending the height of the row. Furthermore both tables contain other
values which might be way longer and also aren't wrapped (like tags and
performers for scenes, and aliases for performers).
* Fix: config race conditions with RWMutex
Added RWMutex to config.Instance which read or write locks
all instances where viper is used.
Refactored checksum manager to only use config and not
viper directly anymore.
All stash viper operations are now "behind" the config.Instance
and thus mutex "protected".
* Fix hierarchical criteria performance issue
Don't apply recursive clause to hierarchical criteria when the depth is
set to 0 (i.e.: no recursion is needed).
This undoes the current performance penalty on for example the studios
page. This as reported in #1519, using a database of 4M images, 30K
scenes and 500 studios. Without this fix loading the studios overview,
with the default of 40 items per page, takes 6 to 7 seconds. With this
fix it only takes 0,07 seconds reverting the performance back to the
pre-hierarchical filtering performance (tested against 508f7b84 which
was the last commit before #1397 was merged).
* Add (not) between modifiers for number criterion
* Extract list filters into dedicated components
Extract the filters from the AddFiltersDialog into custom components.
This allows for further refactorring where components will be bound to
criterions.
* Add placeholders to number and duration criterions
* Add backwards compatibility for saved filters
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
* Add config option for scraper tag exclusion patterns
Add a config option for exclusing tags / tag patterns from the scraper
results.
* Handle tag exclusion patterns during scraping
* Actually implement TagFilter.marker_count
The marker_count filter/criterion as defined in TagFilterType isn't
actually implemented. This adds an implementation for it.
Do note this implementation _might_ have performance issues because of
using OR (in the join). Another implentation would be to remove both
joins and use:
```SQL
COUNT(
SELECT id FROM scene_markers WHERE primary_tag_id = tags.id
UNION
SELECT scene_marker_id FROM scene_markers_tags WHERE tag_id = tags.id
)
```
Note this doesn't require a DISTINCT as UNION already removes any
duplicate records.
* Restore marker count filter and sorting
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
* Change scrapers overview into collapsible per type
* Move scraping configuration to (renamed) scrapers tab
Rename the Scrapers tab to Scraping and move the scraping configuration
to this tab.
* Fix width in database test setup
* Added more filters on resolution field
* added test to verify resolution range is defined for every resolution
* Refactor UI code
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
* Add API to merge tags
Add new API endpoint, `tagsMerge(source, destination)` to merge multiple
tags into a single one. The "sources" must be provided as a list of ids
and the destination as a single id. All usages of the source tags
(scenes, markers (primary and additional), images, galleries and
performers) will be updated to the destination tag, all aliases of the
source tags will be updated to the destination, and the name of the
source will be added as alias to the destination as well.
* Add merge tag UI
* Add unit tests
* Update test mocks
* Update internationalisation
* Add changelog entry
Co-authored-by: gitgiggety <gitgiggety@outlook.com>
* Add basic support for hierarchical filters
Add a new `hierarchicalMultiCriterionHandlerBuilder` filter type which
can / will be used for filtering hierarchical things like the
parent/child relation of the studios.
On the frontend side a new IHierarchicalLabeledIdCriterion criterion
type has been added to accompany this new filter type.
* Refactor movieQueryBuilder to use filterBuilder
Refactor the movieQueryBuilder to use the filterBuilder just as scene,
image and gallery as well.
* Support specifying depth for studios filter
Add an optional depth field to the studios filter for scenes, images,
galleries and movies. When specified that number of included
(grant)children are shown as well. In other words: this adds support for
showing scenes set to child studios when searching on the parent studio.
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
* Add Tag Update/UpdateFull
* Tag alias implementation
* Refactor tag page
* Add aliases in UI
* Include tag aliases in q filter
* Include aliases in tag select
* Add aliases to auto-tagger
* Use aliases in scraper
* Add tag aliases for filename parser
* utils: vtt: add tests
In lieu of documentation.
* utils: vtt: rewrite for correctness and simplicity
Now handles fractional seconds and negative values correctly.
* cleanup: remove dead code
removing some code that does nothing
* cleanup: fixing usage of deprecated gqlgen/graphql api in api/changeset_translator
* cleanup: changing to recommended comparison methods
Changing byte and case-insensitive string comparison to the recommended methods.
* cleanup: making staticcheck happy
* Add funscript route to scenes
Adds a /scene/:id/funscript route which serves a funscript file, if present.
Current convention is that these are files stored with the same path, but with the extension ".funscript".
* Look for funscript during scan
This is stored in the Scene record and used to drive UI changes for funscript support.
Currently, that's limited to a funscript link in the Scene's file info.
* Add filtering and sorting for interactive
* Add Handy connection key to interface config
* Add Handy client and placeholder component.
Uses defucilis/thehandy, but not thehandy-react as I had difficulty integrating the context with the existing components.
Instead, the expensive calculation for the server time offset is put in localStorage for reuse.
A debounce was added when scrubbing the video, as otherwise it spammed the Handy API with updates to the current offset.
* Apply all post processors to performer
Scraping a performer by fragment doesn't correctly work with tags.
When tags are returned to the scraper then all are recognized as new.
This is due to the post process method not being applied while it should
be, as is done when scraping a performer by URL.
* Add tools settings page
* Add tools and move dupe checker
* Make negative number get all
* Show missing phashes
* Add multi-edit button
* Show scene details
* Make config instance-based
* Remove config dependency in paths
* Refactor config init
* Allow startup without database
* Get system status at UI initialise
* Add setup wizard
* Cache and Metadata optional. Database mandatory
* Handle metadata not set during full import/export
* Add links
* Remove config check middleware
* Stash not mandatory
* Panic on missing mandatory config fields
* Redirect setup to main page if setup not required
* Add migration UI
* Remove unused stuff
* Move UI initialisation into App
* Don't create metadata paths on RefreshConfig
* Add folder selector for generated in setup
* Env variable to set and create config file.
Make docker images use a fixed config file.
* Set config file during setup
* Add scraping support for performer tags
* Add performer count to tag cards
* Refactor sqlite test setup
* Add performer tag filtering in gallery and image
* Add bulk update performer
* Add Performers tab to tag page
* Add count filters and sort bys for tags
* Move scene count to icon in performer card #1148
* find correct python executable
For script scrapers using python, both python and python3 are valid depending on the OS and running environment. To save users from having any issues, this change will find the correct executable for them.
Co-authored-by: bnkai <bnkai@users.noreply.github.com>
* Add resolution enum extension
* Add filter builder
* Use filterBuilder for scene query
* Optimise joins
* Add binary operators to scene query
* Use Query for auto-tag
* Expose url for URLReplace in JSON scrapeByURL and scrapeByFragment
* Apply queryURLReplace to xpath scrapers
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
