Commit graph

220 commits

Author SHA1 Message Date
WithoutPants
84e24eb612
Refactor scraping to include related object fields (#6266)
* Refactor scraper post-processing and process related objects consistently
* Refactor image processing
* Scrape related studio fields consistently
* Don't set image on related objects
2025-12-02 12:49:44 +11:00
Gykes
c99825a453
Feature: Tag StashID support (#6255) 2025-11-13 14:24:09 +11:00
Gykes
678b3de7c8
Feature: Support inputURL and inputHostname in scrapers (#6250) 2025-11-10 15:00:47 +11:00
Gykes
6cace4ff88
Update parser to accept groups (#6228) 2025-11-06 09:53:43 +11:00
WithoutPants
1b2b4c5221
Fix panic when scraping with unknown field (#6220)
* Fix URL in group scraper causing panic
* Return error instead of panicking on unknown field
2025-10-31 19:54:35 +11:00
feederbox826
019fe81de9
Update Freeones scraper from CommunityScrapers (#5956)
1b103ad2d5

Co-authored-by: feederbox826 <feederbox826@users.noreply.github.com>
2025-06-23 14:13:01 +10:00
WithoutPants
62d7076ff3
Add missing group scraper fields (#5820) 2025-04-16 08:55:27 +10:00
its-josh4
db06eae7cb
Sort tags by name while scraping or merging scenes (#5752)
* Sort tags by name while scraping scenes
* TagStore.All should sort by sort_name first
* Sort tag by sort name/name in TagIDSelect
---------
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
2025-04-02 14:26:14 +11:00
WithoutPants
2ec264ed62 Fix merge error 2025-03-25 11:19:14 +11:00
WithoutPants
db7d45792e
Refactor stashbox package (#5699)
* Move stashbox package under pkg
* Remove StashBox from method names
* Add fingerprint conversion methods to Fingerprint

Refactor Fingerprints methods

* Make FindSceneByFingerprints accept fingerprints not scene ids
* Refactor SubmitSceneDraft to not require readers
* Have SubmitFingerprints accept scenes

Remove SceneReader dependency

* Move ScrapedScene to models package
* Move ScrapedImage into models package
* Move ScrapedGallery into models package
* Move Scene relationship matching out of stashbox package

This is now expected to be done in the client code

* Remove TagFinder dependency from stashbox.Client
* Make stashbox scene find full hierarchy of studios
* Move studio resolution into separate method
* Move studio matching out of stashbox package

This is now client code responsibility

* Move performer matching out of FindPerformerByID and FindPerformerByName
* Refactor performer querying logic and remove unused stashbox models

Renames FindStashBoxPerformersByPerformerNames to QueryPerformers and accepts names instead of performer ids

* Refactor SubmitPerformerDraft to not load relationships

This will be the responsibility of the calling code

* Remove repository references
2025-03-25 10:30:51 +11:00
WithoutPants
bc923929bb
Stash scraper scene query (#5722)
* Enable scene querying in stash scraper
* Update docs
2025-03-17 10:20:08 +11:00
DogmaDragon
a391fa4345
Fix code comment in config.go --skip-ci (#5691) 2025-03-01 16:40:25 +11:00
fume8866
4d447c3340
update merged performer upon batch update (#5664)
* update merged performer upon batch update
* Handle aliases and name for merged performer
* Refactor merge performer code

Log when merging performers
---------
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
2025-02-27 14:23:48 +11:00
WithoutPants
ccf79d077f
Use tag exclusions when identifying scenes (#5686)
* Move tag exclusion code back into scraper package

Reverts #2391

* Rearrange stash box client code
* Filter excluded tags in stashbox queries

Re-application of fix for #2379
2025-02-27 09:07:02 +11:00
WithoutPants
1e05766571
Fix scraping multiple URLs (#5677)
* Hack fix for scraping URLs field
* Rewrite apply function using known value types
2025-02-26 08:03:08 +11:00
WeedLordVegeta420
e97f647a43
Add Image Scraping (#5562)
Co-authored-by: keenbed <155155956+keenbed@users.noreply.github.com>
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
2025-02-24 16:38:14 +11:00
InfiniteStash
638398808b
Add death date fetching from stash-box (#5653) 2025-02-11 15:09:50 +11:00
81hvac1k02
dd40c07a6d
removed un-necesarry dependancy (#5641) 2025-02-06 13:35:08 +11:00
WithoutPants
5f690d96bd
Fix stash scraper errors and add apikey field (#5474)
* Use hasura/go-graphql-client instead of shurcooL version
* Fix graphql query errors
* Support setting api key for stash server
2024-11-13 10:14:55 +11:00
InfiniteStash
b1d5dc2a0e
Fix scraping stash-box performers with null birthdates (#5428) 2024-10-31 11:55:58 +11:00
WithoutPants
89f539ee24
Upgrade gqlgenc and regenerate stash-box client (#5391)
* Upgrade gqlgenc and regenerate stash-box client
* Fix go version
* Don't generate resolvers
* Bump go version in compiler image. Bump freebsd version
2024-10-29 17:35:17 +11:00
WithoutPants
b7799df2a6
Add package docs and project vision/goals (#5169)
* Add goals/design vision to contributing doc
* Add barebones package documentation
2024-08-28 09:01:39 +10:00
WithoutPants
10341fba58
Update builtin freeones scraper (#5171) 2024-08-28 09:00:14 +10:00
Gykes
7788a6fd07
PatchComponentRedo (#5136)
* PatchComponent update specifically for SettingsInterfacePanel
* Fix unrelated lint issues
---------
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
2024-08-20 14:25:06 +10:00
WithoutPants
15a7b8a859
Movie group renames (#5039)
* Rename Movie and MoviePartial to Group/GroupPartial
* Rename Movie interfaces
* Update movie url builders to use group
* Rename movieRoutes to groupRoutes
* Update dataloader
* Update names in sqlite package
* Rename in resolvers
* Add GroupByURL to scraper config
* Scraper backward compatibility hacks
2024-07-04 09:10:26 +10:00
WithoutPants
2739696813
Add group graphql interfaces (#5017)
* Deprecate movie and add group interfaces
* UI changes
2024-07-03 13:59:40 +10:00
WithoutPants
f26766033e
Performer urls (#4958)
* Populate URLs from legacy fields
* Return nil properly in xpath/json scrapers
* Improve migration logging
2024-06-18 13:41:05 +10:00
WithoutPants
fda4776d30
Movie/Group tags (#4969)
* Combine common tag control code into hook
* Combine common scraped tag row code into hook
2024-06-18 11:24:15 +10:00
well
4be60310c3
In performer scrapers, forward non-http single performer images (#4947)
* Forward non-http single performer images
* Don't set if Images already set
---------
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
2024-06-11 13:41:43 +10:00
Maista
e18c050fb1
Add log and utils modules to the Javascript VM used in scrapers (#4937) 2024-06-11 13:21:39 +10:00
WithoutPants
ed057c971f
Correct Stash box endpoint inputs (#4924)
* Use stashbox endpoint instead of index
* Update UI to not use deprecated fields
2024-06-11 13:14:12 +10:00
WithoutPants
94a978d063
Scraper inputs (#4922)
* Pass more details in scene/gallery scrape
2024-06-11 13:12:45 +10:00
WithoutPants
62bdff351d
Movie URLs (#4900)
* Fix exclude behaviour for stringListCriterionHandlerBuilder
2024-06-11 13:08:49 +10:00
bob123491234
3e3e8b95e2
Add scenes filter to galleries (#4840) 2024-05-20 13:04:45 +10:00
Dankonite
29859fa4ad
Tag Favoriting (#4728)
* Add missing key unbind
---------
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
2024-05-09 12:04:58 +10:00
WithoutPants
c5abe28375
Fix alias issue when tagging performer from stash-box (#4820) 2024-05-08 12:47:18 +10:00
WithoutPants
9ceea952b6
Replace javascript module otto with goja (#4631)
* Move plugin javascript to own package with goja
* Use javascript package in scraper

Remove otto
2024-03-14 11:03:40 +11:00
Flashy78
2ca9e0f43a
Adding disambiguation to performer draft (#4122) 2024-02-29 12:47:20 +11:00
WithoutPants
6271f18979
Fix error when creating/updating performer with alias == name (#4443)
* Filter out performer aliases that match the name
* Validate when creating/updating performer in stash-box task
2024-01-09 14:57:49 +11:00
Maista
e8af3c8e98
Set PYTHONPATH environment variable for Python script scrapers (#4372)
* Set PYTHONPATH environment variable for Python script scrapers
* Convert PYTHONPATH to absolute
* Generalise and apply to plugins

---------

Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
2023-12-20 13:32:19 +11:00
bob123491234
d1018b4c5d
Add Studio Code and Photographer to Galleries. (#4195)
* Added Studio Code and Photographer to Galleries
* Fix gallery display on mobile
* Fixed potential panic when scraping with a bad configuration
---------
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
2023-11-28 15:05:33 +11:00
DingDongSoLong4
b78771dbcd
Manager refactor, part 1 (#4298)
* Move BackupDatabase and AnonymiseDatabase to internal/manager
* Rename config.Instance to config.Config
* Rename FFMPEG
* Rework manager and initialization process
* Fix Makefile
* Tweak phasher
* Fix config races
* Fix setup error not clearing
2023-11-28 13:56:46 +11:00
Maista
0dcd58763f
Log more when resolving Python (#4185)
* Log more when resolving Python

Users often have problems configuring their Python installations

* Convert if-else ladder to switch statement
* Consolidate Python resolution

Adds additional logging to plugin tasks to
align with the logging that scrapers output.
2023-11-22 10:04:15 +11:00
DingDongSoLong4
72779e618d
Fix batch performer tag panic (#4281) 2023-11-12 03:50:09 +02:00
DingDongSoLong4
9621213424
Genericise sliceutil functions (#4253)
* Genericise sliceutil.SliceSame
* Genericise intslice functions
* Genericise stringutil functions
2023-11-02 08:58:32 +11:00
its-josh4
2b8c2534dd
Update a number of dependencies (incl. CVE fixes) (#4107)
* Update a number of dependencies (incl. CVE fixes)

Includes some dependencies that were upgraded in #4106 as well as a few more dependencies.

Some deps that have been upgraded had CVEs.

Notably, upgrades deprecated dependencies such as:
- `github.com/go-chi/chi` (replaced with `/v5`)
- `github.com/gofrs/uuid` (replaced with `/v5`)
- `github.com/hashicorp/golang-lru` (replaced with `/v2` which uses generics)

* Upgraded a few more deps

* lint

* reverted yaml library to v2

* remove unnecessary mod replace

* Update chromedp

Fixes #3733
2023-10-26 16:24:32 +11:00
WithoutPants
2fd7141f0f
Javascript scraper postprocess (#4200)
* Add javascript post-process action
* Add documentation
2023-10-16 17:17:36 +11:00
DingDongSoLong4
33f2ebf2a3
Model refactor, part 3 (#4152)
* Remove manager.Repository
* Refactor other repositories
* Fix tests and add database mock
* Add AssertExpectations method
* Refactor routes
* Move default movie image to internal/static and add convenience methods
* Refactor default performer image boxes
2023-10-16 14:26:34 +11:00
Emilo2
40bcb4baa5
Fix url's from gallery scrapers (#4206)
* Fill in urls array when gallery scraper provides only single url
2023-10-16 13:27:42 +11:00
DingDongSoLong4
bc5df7cfaf
Fix gallery scraper url loading (#4157) 2023-09-30 10:43:57 +10:00