Commit graph

829 commits

Author SHA1 Message Date
WithoutPants
f39fa416a9
Fix database locked errors (#3153)
* Make read-only operations use WithReadTxn
* Allow one database write thread
* Add unit test for concurrent transactions
* Perform some actions after commit to release txn
* Suppress some errors from cancelled context
2022-11-21 06:49:10 +11:00
stg-annon
3660bf2d1a
Support filtering by StashID endpoint (#3005)
* Add endpoint to stash_id filter in UI

Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
2022-11-17 10:08:15 +11:00
WithoutPants
3a63f1f9b7
Fire hook when gallery created from image (#3134) 2022-11-17 08:23:08 +11:00
skier233
7eae751d1c
Stash rating system (#2830)
* add rating100 fields to represent rating range 1-100
* deprecate existing (1-5) rating fields
* add half- and quarter-star options for rating system
* add decimal rating system option

Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
2022-11-16 09:31:44 +11:00
gitgiggety
f66333bac9
Add date filters (#2834)
* graphql: support date and timestamp filter types
* sql: add support for date & timestamp criterions
* ui: add support for date and timestamp criterions
* scenes: add support for filtering by date, created at and updated at
* image: support filtering by created at and updated at
* gallery: support filtering by date, created at and updated at
* movie: support filtering by date, created at and updated at
* studio: support filtering by date, created at and updated at
* tag: support filtering by date, created at and updated at
* performer: support filtering by bitrh & death date and created & updated at
* marker: support filtering by created & updated at and scene date, created & updated at
2022-11-15 11:52:05 +11:00
WithoutPants
ce17230c13
Refactor autotag to use individual transactions (#3106)
* Add id filtering to scenes, images, and galleries
* Perform tagging in batches
* One transaction per object tagged
2022-11-14 17:07:24 +11:00
WithoutPants
4a054ab081
Support file-less scenes. Add scene split, merge and reassign file (#3006)
* Reassign scene file functionality
* Implement scene create
* Add scene create UI
* Add sceneMerge backend support
* Add merge scene to UI
* Populate split create with scene details
* Add merge button to duplicate checker
* Handle file-less scenes in marker preview generate
* Make unique file name for file-less scene exports
* Add o-counter to scene update input
* Hide rescan for file-less scenes
* Generate heatmap if no speed set on file
* Fix count in scene/image queries
2022-11-14 16:35:09 +11:00
WithoutPants
9df66024d1
Make title optional for non-user created galleries (#3110) 2022-11-10 14:19:13 +11:00
WithoutPants
c83ebf7c1c
Order performers by name in All (#3111) 2022-11-10 14:18:46 +11:00
WithoutPants
30a7482ddf
Order saved filters by name (#3101) 2022-11-09 11:06:11 +11:00
WithoutPants
d2743cf5fb
Change performer height to be numeric (#3060)
* Make height an int. Add height_cm field
* Change UI to use height_cm
* Use number fields for height/weight
* Add migration note
2022-11-08 14:09:03 +11:00
WithoutPants
962bc7df4e
Fix panic when fileSize is negative (#3089) 2022-11-08 10:01:32 +11:00
HappyAxolotl
eff86bf2f8
[Feature] Add fields director and (studio) code to scenes (#3051)
* added schema migration and updated data models
* added code and director to UI
* new fields are exported and imported
* added filters
* Add changelog entry
2022-11-07 18:16:52 +11:00
WithoutPants
270bc317cb
Performer refactor (#3057)
* Separate performer model from sqlite model
* Use GenderEnum for gender
2022-10-31 14:58:01 +11:00
InfiniteTF
7b7d6758ef
Change performer country value to be ISO code (#1922)
* Change performer country value to be ISO code
* Localize country names
* Use country select for filter

Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
2022-10-28 16:37:57 +11:00
DingDongSoLong4
1c0042c4c2
Fix path filters (#3041)
* Fix path filters
* Replace getPathSearchClause
* Remove incorrect tests
2022-10-26 18:48:13 +11:00
stg-annon
a60afc162f
Add Tag description filter (#3011)
* init tag description filter

issue with rendering UI filter dialog

* move description to string block
* Add 0.18.0 changelog page
2022-10-26 18:06:52 +11:00
WithoutPants
479ebfc88d
Reimplement case-insensitivity move bug fix (#3047)
* Use eq for FindByPath for case sensitivity
* Handle case sensitive moves
2022-10-25 11:37:54 +11:00
WithoutPants
1c92336798
Fix symlink size calculation (#3046) 2022-10-25 10:57:37 +11:00
WithoutPants
47395ce13f
Use basename as title if empty when scraping by fragment (#3040)
* Fallback to file basename if title empty in scrape
* Populate dialog from basename if title empty
2022-10-24 14:36:22 +11:00
WithoutPants
091950615e
Ignore non-existing scenes in fingerprint submits (#3039) 2022-10-24 10:26:21 +11:00
DingDongSoLong4
4db0e48f73
Fix zip gallery renaming (#3036) 2022-10-24 09:38:02 +11:00
WithoutPants
33de28ce5d Warn when failed to migrate from placeholder 2022-10-20 01:18:32 +00:00
WithoutPants
0cf06728d4 Ignore NULL values during migration 2022-10-20 01:01:18 +00:00
WithoutPants
7104bb67ca
Handle null video durations in migration (#3021) 2022-10-20 10:58:42 +11:00
WithoutPants
4c286d7ab5
Fix dataloaders not loading in js plugins (#3014)
* Include dataloaders in js plugin gql calls
* Handle gql errors correctly in js plugins
2022-10-18 11:09:54 +11:00
WithoutPants
6b5d5cc628
Create missing covers during scan (#2995)
* Create missing covers during scan
* Update changelog and release notes
2022-10-11 14:22:23 +11:00
WithoutPants
68a1547e8b
Include primary tag in tag marker count sorting (#2993) 2022-10-11 14:21:28 +11:00
WithoutPants
9bff498c28
Fix tag/studio alias and caption null filtering (#2990)
* Fix null filter for alias/captions
* Fix error when selecting is null for captions
2022-10-11 14:21:07 +11:00
WithoutPants
6ce409cd56
Prevent errors from stopping scan 2022-10-11 14:20:24 +11:00
JackDawson94
b1193227d0
Fixes gender-mapping with StashBox (#2992) 2022-10-11 09:40:14 +11:00
WithoutPants
0e0d201ff3
Ignore other conflicts in 35 migration 2022-10-06 14:54:14 +11:00
WithoutPants
ef9e138a2d
[Files Refactor] Object file management (#2790)
* Add Make Primary file function
* Add delete file functionality
2022-10-06 14:50:06 +11:00
WithoutPants
b160c3bb97
Associate funscript files on scan (#2978) 2022-10-06 11:17:01 +11:00
WithoutPants
55001ddcf1
Ignore conflicts on performers_galleries 2022-10-06 10:05:43 +11:00
pickleahead
4c73f2f845
Add descriptions to tags and display tag cards on hover (#2708)
* add descriptions to tags
* display tag description and tag image on hover

Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
2022-10-06 10:01:06 +11:00
WithoutPants
6ba9f55df0
Add default thumbnails for scenes and images (#2949)
* Use default thumbnail for scene covers
* Use defautl thumbnail for image thumbnails
2022-10-03 13:01:35 +11:00
WithoutPants
88bfda1980
Ignore duplicates for movies_scenes migration 2022-10-03 08:20:04 +11:00
WithoutPants
060ac00fc7
Take out _stash_ids from 35 migration 2022-10-01 08:45:16 +10:00
WithoutPants
7b83d81820
Fix backup issues (#2966)
Make backup directory setting optional. Use the basename of the database file for the naming.
2022-09-30 20:57:28 +10:00
WithoutPants
9e44e13f6d
Fix update duplicate ids (#2965) 2022-09-30 18:44:37 +10:00
WithoutPants
51f4dd8a59
Delete identical studio stash id rows 2022-09-30 18:10:55 +10:00
WithoutPants
30f7a05ebf
Delete identical rows from performer_stash_ids 2022-09-30 17:25:40 +10:00
WithoutPants
8f594e7fed
Fix migration 2022-09-30 16:23:50 +10:00
WithoutPants
c8cbb36fd5
Ignore multiple identical rows in performers_scenes 2022-09-30 15:59:29 +10:00
JackDawson94
554448594c
Add unix timestamp parsing to scrapers parseDate (#2817)
* Add unix timestamp parsing to scrapers parseDate
* Add documentation
* Update ScraperDevelopment.md
* Add unit test

Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
2022-09-30 15:35:56 +10:00
DingDongSoLong4
25bc750295
Performance improvements (#2925)
* Add sqlite_stat4 build tag
* Simplify studio filter criterion queries
* Prevent useList loading data before filter initialized
2022-09-30 10:49:51 +10:00
7dJx1qP
d274f86390
Add backup directory path setting (#2953)
* add backup directory path setting
* Don't default backup path
* handle migration backup path input when given filename or path

Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
2022-09-30 10:00:50 +10:00
WithoutPants
ad7fbce5f7
Rebuild association tables, ensure file-system-based galleries cannot be changed (#2955)
* Re-create tables to include primary keys
* Filesystem-based galleries cannot change images
2022-09-30 09:18:58 +10:00
WithoutPants
dce90a3ed9
Handle file rescan (#2951)
* Fire handlers when file updated or moved
* Create galleries as needed
* Clean empty galleries
* Handle cleaning zip folders when path changed
* Fix gallery association on duplicate images
* Re-create missing folder-based galleries
2022-09-28 16:08:00 +10:00
WithoutPants
d4e706daef
Only set video algorithm after migration 12 (#2946) 2022-09-26 11:27:53 +10:00
WithoutPants
0848b02e93
Various bug fixes (#2945)
* Only update fingerprints if changed
* Fix panic when loading primary file fails
* Fix gallery/scene association
* Fix display of scene gallery in card
* Use natural_cs collation with paths for title sorting
2022-09-25 12:07:55 +10:00
WithoutPants
b74428cb42
Various bug fixes (#2938)
* Don't recalculate MD5 if not enabled

Remove MD5 if oshash has changed and MD5 was not calculated.

* Fix panic in paged DLNA
* Prevent identical hashes in stash-box drafts
2022-09-21 15:39:41 +10:00
WithoutPants
cffcd9f4b8
Various bug fixes (#2935)
* Sort scene/image/gallery tags by name
* Calculate md5 if missing
* Prevent multiple folder create logs
2022-09-20 17:02:14 +10:00
WithoutPants
3fa7b470e7
More timestamp corrections (#2933)
* Fix incorrect timestamp updates
* Correct folder time fields
* Add migration with new indexes
* Correct mod_time format
* Add mod_time to data massage
2022-09-20 13:52:37 +10:00
WithoutPants
98e3610ade
Fix gallery checksum resolver error (#2929) 2022-09-19 16:50:04 +10:00
DingDongSoLong4
8efbcc1c4d
Suppress benign broken pipe and context closed warnings (#2927) 2022-09-19 15:01:40 +10:00
DingDongSoLong4
5e97ecd260
Set explicit SameSite=Lax on session cookie (#2926) 2022-09-19 14:56:05 +10:00
WithoutPants
1207629a76
Fix incorrectly formatted timestamps (#2918)
* Update updated_at when adding file to object
* Use models.SQLTimestamp for timestamps
* Add data massage to fix incorrect timestamps
2022-09-19 14:53:46 +10:00
WithoutPants
2564351265
Use post commit hook for post-create plugin hooks (#2920) 2022-09-19 14:53:06 +10:00
WithoutPants
0359ce2ed8
Fix bulk add movie to scenes and autotag transaction error (#2928)
* Fix bulk add movie to scene
* Fix already in transaction error for autotag
2022-09-19 14:52:40 +10:00
WithoutPants
648247aa00
Split by whitespace for path includes/excludes (#2919) 2022-09-16 14:45:08 +10:00
WithoutPants
8b79eaca67
[Files Refactor] Use batching for pre/post-migration (#2906)
* Use batching for pre/post-migration
* Clarify release notes
2022-09-14 10:57:00 +10:00
DingDongSoLong4
5c383da5ec
Optimize database after migration (#2904) 2022-09-14 09:15:36 +10:00
WithoutPants
cfc8222b9a
[Files Refactor] Cleanup (#2893)
* Clean up notes for develop merge
* Remove commented code
* Lint
2022-09-07 16:50:15 +10:00
WithoutPants
9e08edc76f
[Files Refactor] Don't require fingerprint calculation post-migrate (#2892) 2022-09-07 14:21:10 +10:00
WithoutPants
13bdba5b24 Fix json filename generation (#2887) 2022-09-06 07:04:52 +00:00
WithoutPants
0c513a604d Fix stack overflow creating windows network folders (#2886) 2022-09-06 07:04:52 +00:00
WithoutPants
276f14cdcb Fix filename generation in export (#2883) 2022-09-06 07:04:52 +00:00
WithoutPants
6b0bcdea88 [Files Refactor] Set primary flag when cleaning (#2880)
* Ensure single primary per object
* Set primary file during clean
* Only show migration notes for actual migrations
2022-09-06 07:04:52 +00:00
WithoutPants
94d39da706 [Files Refactor] Bug fixes (#2868)
* Return error if multiple rows returned for id
* Add missing LoadFiles calls
* Show id if path is empty
2022-09-06 07:04:52 +00:00
WithoutPants
273cf0383d [Files Refactor] Performance tuning (#2865)
* Don't load image files by default
* Don't load gallery files by default
* Don't load scene files by default
* Retry locked transactions forever
* Don't show release notes if config not loaded
* Don't translate path slashes in export
2022-09-06 07:04:52 +00:00
WithoutPants
0b534d89c6 [Files Refactor] Import export fixup (#2763)
* Adjust json schema
* Remove mappings file from export
* Import file/folder support
* Update documentation
* Make gallery filenames unique
2022-09-06 07:04:52 +00:00
WithoutPants
1222b7b87b Fix files not being cleaned correctly (#2862) 2022-09-06 07:04:52 +00:00
WithoutPants
7b439556c0 [Files Refactor] Rollback platform-agnostic paths (#2852)
* Rollback platform agnostic path storage
* Add release note for database change
2022-09-06 07:04:52 +00:00
WithoutPants
f4825fadf4 [Files refactor] Bug fixes (#2849)
* Fix scene sorting
* Fix folder-based gallery path sorting
* Fix gallery path filter
* Fix stash-box performer submission
* Fix identify logging
* Remove govet from linter
2022-09-06 07:03:43 +00:00
WithoutPants
00608c167a [Files Refactor] Performance tuning (#2819)
* Load scene relationships on demand
* Load image relationships on demand
* Load gallery relationships on demand
* Add dataloaden
* Use dataloaders
* Use where in for other find many functions
2022-09-06 07:03:42 +00:00
WithoutPants
9b31b20fed [Files Refactor] Performance tuning (#2813)
* Do database txn in same thread. Retry on locked db
* Remove captions from slimscenedata
* Fix tracing
* Use where in instead of individual selects
* Remove scenes_query view
* Remove image query view
* Remove gallery query view
* Use where in for FindMany
* Don't interrupt scanning zip files
* Fix image filesize sort
2022-09-06 07:03:42 +00:00
WithoutPants
87167736f6 [Files Refactor] bug fixes (#2811)
* Fix scan options not saving
* Fix duration stat calculation
2022-09-06 07:03:42 +00:00
WithoutPants
569c3a872a [Files Refactor] Performance tuning (#2809)
* Use cache during migration
* Avoid use of query views
* Use FindMany to find related objects
* Log slow queries
* Add folders to generated files
* Use SlimScene for scene queries
* Include filename in migration error message
2022-09-06 07:03:42 +00:00
WithoutPants
c825cf5d09 Correctly delete files when specified (#2804) 2022-09-06 07:03:42 +00:00
WithoutPants
5843fdcecc [Files Refactor] Migration fix (#2796)
* Fix large wal file during migration
* Fix migration dropping / from network share paths
2022-09-06 07:03:42 +00:00
WithoutPants
bc47932343 [Files Refactor] Performance tuning (#2784)
* Improve image query performance
* Tune queries
* Fix db generator
* Don't show release notes in setup
* Further tune indexes
* Log when creating screenshot
2022-09-06 07:03:42 +00:00
WithoutPants
abb574205a Files refactor fixes (#2743)
* Fix destroy gallery not destroying file
* Re-add minModTime functionality
* Deprecate useFileMetadata and stripFileExtension
* Optimise files post migration
* Decorate moved files. Use first missing file in move
* Include path in thumbnail generation error log
* Fix stash-box draft submission
* Don't destroy files unless deleting
* Call handler for files with no associated objects
* Fix moved zips causing error on scan
2022-09-06 07:03:42 +00:00
WithoutPants
461068462c [Files Refactor] Filter and sort by file count (#2744)
* Add filtering on file count
* Add sorting by file count
2022-09-06 07:03:42 +00:00
WithoutPants
5495d72849 File storage rewrite (#2676)
* Restructure data layer part 2 (#2599)
* Refactor and separate image model
* Refactor image query builder
* Handle relationships in image query builder
* Remove relationship management methods
* Refactor gallery model/query builder
* Add scenes to gallery model
* Convert scene model
* Refactor scene models
* Remove unused methods
* Add unit tests for gallery
* Add image tests
* Add scene tests
* Convert unnecessary scene value pointers to values
* Convert unnecessary pointer values to values
* Refactor scene partial
* Add scene partial tests
* Refactor ImagePartial
* Add image partial tests
* Refactor gallery partial update
* Add partial gallery update tests
* Use zero/null package for null values
* Add files and scan system
* Add sqlite implementation for files/folders
* Add unit tests for files/folders
* Image refactors
* Update image data layer
* Refactor gallery model and creation
* Refactor scene model
* Refactor scenes
* Don't set title from filename
* Allow galleries to freely add/remove images
* Add multiple scene file support to graphql and UI
* Add multiple file support for images in graphql/UI
* Add multiple file for galleries in graphql/UI
* Remove use of some deprecated fields
* Remove scene path usage
* Remove gallery path usage
* Remove path from image
* Move funscript to video file
* Refactor caption detection
* Migrate existing data
* Add post commit/rollback hook system
* Lint. Comment out import/export tests
* Add WithDatabase read only wrapper
* Prepend tasks to list
* Add 32 pre-migration
* Add warnings in release and migration notes
2022-09-06 07:03:42 +00:00
WithoutPants
964b559309 Restructure data layer (#2532)
* Add new txn manager interface
* Add txn management to sqlite
* Rename get to getByID
* Add contexts to repository methods
* Update query builders
* Add context to reader writer interfaces
* Use repository in resolver
* Tighten interfaces
* Tighten interfaces in dlna
* Tighten interfaces in match package
* Tighten interfaces in scraper package
* Tighten interfaces in scan code
* Tighten interfaces on autotag package
* Remove ReaderWriter usage
* Merge database package into sqlite
2022-09-06 07:03:40 +00:00
WithoutPants
7b5bd80515 Separate graphql API from rest of the system (#2503)
* Move graphql generated files to api
* Refactor identify options
* Remove models.StashBoxes
* Move ScraperSource to scraper package
* Rename field strategy enums
* Rename identify.TaskOptions to Options
2022-09-06 07:03:40 +00:00
kermieisinthehouse
30879389ec
Fix golangci OOM (#2889)
* Fix golangci OOM

* Fix all lints
2022-09-05 22:12:59 -07:00
peolic
b8262f5641
Fix non-default video stream from ffprobe result (#2752)
* Fix non-default video stream from ffprobe result
2022-07-22 17:21:39 +10:00
bnkai
6cfb7fe79d
Fix Synopsis json string in movie jsonschema (#2664)
* Fix Synopsis json string in movie jsonschema
* backwards compatible movie synopsis import
2022-06-22 10:59:39 +10:00
TgSeed
abd76f7e58
Fix/ffprobe unmarshalling error (#2685)
Fix/ffprobe unmarshalling error
2022-06-22 10:49:14 +10:00
CJ
9264c15540
Customize recommendations (#2592)
* refactored common code in recommendation row
* Implement front page options in config
* Allow customisation from front page
* Rename recommendations to front page
* Add generic UI settings
* Support adding premade filters

Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
2022-06-14 10:34:04 +10:00
WithoutPants
ff724d82cc
Don't trim extension for folders when auto-tagging galleries (#2658) 2022-06-08 09:02:11 +10:00
Emilo2
8a1c349976
Fix scraping more than 40 scenes from stash-box (#2638) 2022-06-03 09:37:24 +10:00
InfiniteTF
d68d022893
Add support for submitting stash-box scene updates by draft (#2577) 2022-06-01 14:53:31 +10:00
InfiniteTF
e51083c26d
Update stash-box fingerprint query to fully support distance matching (#2509) 2022-06-01 12:59:06 +10:00
DingDongSoLong4
49f579e08e
Fix gallery updating (#2611) 2022-06-01 11:58:44 +10:00
cj
c1a096a1a6
Caption support (#2462)
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
2022-05-06 11:59:28 +10:00
WithoutPants
ce175dcfc6
Fix audio not in video previews (#2547) 2022-05-05 11:04:01 +10:00