Commit graph

222 commits

Author SHA1 Message Date
CJ
c0260781a5
fix(scraper): handle base64 data URIs in processImageField (#6480)
Add check to skip HTTP fetch for non-HTTP URLs in processImageField(),
matching the existing behavior in setPerformerImage() and setStudioImage().

This allows scrapers to return base64 data URIs (e.g.,
`data:image/jpeg;base64,...`) directly without triggering an HTTP fetch
error. Previously, processImageField() would attempt to create an HTTP
request with the data URI as the URL, causing "Could not set image using
URL" warnings.
2026-01-06 11:47:32 +11:00
WithoutPants
1580cf9bd9 Provide more information when scraper loadURL fails 2025-12-12 07:17:41 +11:00
WithoutPants
84e24eb612
Refactor scraping to include related object fields (#6266)
* Refactor scraper post-processing and process related objects consistently
* Refactor image processing
* Scrape related studio fields consistently
* Don't set image on related objects
2025-12-02 12:49:44 +11:00
Gykes
c99825a453
Feature: Tag StashID support (#6255) 2025-11-13 14:24:09 +11:00
Gykes
678b3de7c8
Feature: Support inputURL and inputHostname in scrapers (#6250) 2025-11-10 15:00:47 +11:00
Gykes
6cace4ff88
Update parser to accept groups (#6228) 2025-11-06 09:53:43 +11:00
WithoutPants
1b2b4c5221
Fix panic when scraping with unknown field (#6220)
* Fix URL in group scraper causing panic
* Return error instead of panicking on unknown field
2025-10-31 19:54:35 +11:00
feederbox826
019fe81de9
Update Freeones scraper from CommunityScrapers (#5956)
1b103ad2d5

Co-authored-by: feederbox826 <feederbox826@users.noreply.github.com>
2025-06-23 14:13:01 +10:00
WithoutPants
62d7076ff3
Add missing group scraper fields (#5820) 2025-04-16 08:55:27 +10:00
its-josh4
db06eae7cb
Sort tags by name while scraping or merging scenes (#5752)
* Sort tags by name while scraping scenes
* TagStore.All should sort by sort_name first
* Sort tag by sort name/name in TagIDSelect
---------
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
2025-04-02 14:26:14 +11:00
WithoutPants
2ec264ed62 Fix merge error 2025-03-25 11:19:14 +11:00
WithoutPants
db7d45792e
Refactor stashbox package (#5699)
* Move stashbox package under pkg
* Remove StashBox from method names
* Add fingerprint conversion methods to Fingerprint

Refactor Fingerprints methods

* Make FindSceneByFingerprints accept fingerprints not scene ids
* Refactor SubmitSceneDraft to not require readers
* Have SubmitFingerprints accept scenes

Remove SceneReader dependency

* Move ScrapedScene to models package
* Move ScrapedImage into models package
* Move ScrapedGallery into models package
* Move Scene relationship matching out of stashbox package

This is now expected to be done in the client code

* Remove TagFinder dependency from stashbox.Client
* Make stashbox scene find full hierarchy of studios
* Move studio resolution into separate method
* Move studio matching out of stashbox package

This is now client code responsibility

* Move performer matching out of FindPerformerByID and FindPerformerByName
* Refactor performer querying logic and remove unused stashbox models

Renames FindStashBoxPerformersByPerformerNames to QueryPerformers and accepts names instead of performer ids

* Refactor SubmitPerformerDraft to not load relationships

This will be the responsibility of the calling code

* Remove repository references
2025-03-25 10:30:51 +11:00
WithoutPants
bc923929bb
Stash scraper scene query (#5722)
* Enable scene querying in stash scraper
* Update docs
2025-03-17 10:20:08 +11:00
DogmaDragon
a391fa4345
Fix code comment in config.go --skip-ci (#5691) 2025-03-01 16:40:25 +11:00
fume8866
4d447c3340
update merged performer upon batch update (#5664)
* update merged performer upon batch update
* Handle aliases and name for merged performer
* Refactor merge performer code

Log when merging performers
---------
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
2025-02-27 14:23:48 +11:00
WithoutPants
ccf79d077f
Use tag exclusions when identifying scenes (#5686)
* Move tag exclusion code back into scraper package

Reverts #2391

* Rearrange stash box client code
* Filter excluded tags in stashbox queries

Re-application of fix for #2379
2025-02-27 09:07:02 +11:00
WithoutPants
1e05766571
Fix scraping multiple URLs (#5677)
* Hack fix for scraping URLs field
* Rewrite apply function using known value types
2025-02-26 08:03:08 +11:00
WeedLordVegeta420
e97f647a43
Add Image Scraping (#5562)
Co-authored-by: keenbed <155155956+keenbed@users.noreply.github.com>
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
2025-02-24 16:38:14 +11:00
InfiniteStash
638398808b
Add death date fetching from stash-box (#5653) 2025-02-11 15:09:50 +11:00
81hvac1k02
dd40c07a6d
removed un-necesarry dependancy (#5641) 2025-02-06 13:35:08 +11:00
WithoutPants
5f690d96bd
Fix stash scraper errors and add apikey field (#5474)
* Use hasura/go-graphql-client instead of shurcooL version
* Fix graphql query errors
* Support setting api key for stash server
2024-11-13 10:14:55 +11:00
InfiniteStash
b1d5dc2a0e
Fix scraping stash-box performers with null birthdates (#5428) 2024-10-31 11:55:58 +11:00
WithoutPants
89f539ee24
Upgrade gqlgenc and regenerate stash-box client (#5391)
* Upgrade gqlgenc and regenerate stash-box client
* Fix go version
* Don't generate resolvers
* Bump go version in compiler image. Bump freebsd version
2024-10-29 17:35:17 +11:00
WithoutPants
b7799df2a6
Add package docs and project vision/goals (#5169)
* Add goals/design vision to contributing doc
* Add barebones package documentation
2024-08-28 09:01:39 +10:00
WithoutPants
10341fba58
Update builtin freeones scraper (#5171) 2024-08-28 09:00:14 +10:00
Gykes
7788a6fd07
PatchComponentRedo (#5136)
* PatchComponent update specifically for SettingsInterfacePanel
* Fix unrelated lint issues
---------
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
2024-08-20 14:25:06 +10:00
WithoutPants
15a7b8a859
Movie group renames (#5039)
* Rename Movie and MoviePartial to Group/GroupPartial
* Rename Movie interfaces
* Update movie url builders to use group
* Rename movieRoutes to groupRoutes
* Update dataloader
* Update names in sqlite package
* Rename in resolvers
* Add GroupByURL to scraper config
* Scraper backward compatibility hacks
2024-07-04 09:10:26 +10:00
WithoutPants
2739696813
Add group graphql interfaces (#5017)
* Deprecate movie and add group interfaces
* UI changes
2024-07-03 13:59:40 +10:00
WithoutPants
f26766033e
Performer urls (#4958)
* Populate URLs from legacy fields
* Return nil properly in xpath/json scrapers
* Improve migration logging
2024-06-18 13:41:05 +10:00
WithoutPants
fda4776d30
Movie/Group tags (#4969)
* Combine common tag control code into hook
* Combine common scraped tag row code into hook
2024-06-18 11:24:15 +10:00
well
4be60310c3
In performer scrapers, forward non-http single performer images (#4947)
* Forward non-http single performer images
* Don't set if Images already set
---------
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
2024-06-11 13:41:43 +10:00
Maista
e18c050fb1
Add log and utils modules to the Javascript VM used in scrapers (#4937) 2024-06-11 13:21:39 +10:00
WithoutPants
ed057c971f
Correct Stash box endpoint inputs (#4924)
* Use stashbox endpoint instead of index
* Update UI to not use deprecated fields
2024-06-11 13:14:12 +10:00
WithoutPants
94a978d063
Scraper inputs (#4922)
* Pass more details in scene/gallery scrape
2024-06-11 13:12:45 +10:00
WithoutPants
62bdff351d
Movie URLs (#4900)
* Fix exclude behaviour for stringListCriterionHandlerBuilder
2024-06-11 13:08:49 +10:00
bob123491234
3e3e8b95e2
Add scenes filter to galleries (#4840) 2024-05-20 13:04:45 +10:00
Dankonite
29859fa4ad
Tag Favoriting (#4728)
* Add missing key unbind
---------
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
2024-05-09 12:04:58 +10:00
WithoutPants
c5abe28375
Fix alias issue when tagging performer from stash-box (#4820) 2024-05-08 12:47:18 +10:00
WithoutPants
9ceea952b6
Replace javascript module otto with goja (#4631)
* Move plugin javascript to own package with goja
* Use javascript package in scraper

Remove otto
2024-03-14 11:03:40 +11:00
Flashy78
2ca9e0f43a
Adding disambiguation to performer draft (#4122) 2024-02-29 12:47:20 +11:00
WithoutPants
6271f18979
Fix error when creating/updating performer with alias == name (#4443)
* Filter out performer aliases that match the name
* Validate when creating/updating performer in stash-box task
2024-01-09 14:57:49 +11:00
Maista
e8af3c8e98
Set PYTHONPATH environment variable for Python script scrapers (#4372)
* Set PYTHONPATH environment variable for Python script scrapers
* Convert PYTHONPATH to absolute
* Generalise and apply to plugins

---------

Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
2023-12-20 13:32:19 +11:00
bob123491234
d1018b4c5d
Add Studio Code and Photographer to Galleries. (#4195)
* Added Studio Code and Photographer to Galleries
* Fix gallery display on mobile
* Fixed potential panic when scraping with a bad configuration
---------
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
2023-11-28 15:05:33 +11:00
DingDongSoLong4
b78771dbcd
Manager refactor, part 1 (#4298)
* Move BackupDatabase and AnonymiseDatabase to internal/manager
* Rename config.Instance to config.Config
* Rename FFMPEG
* Rework manager and initialization process
* Fix Makefile
* Tweak phasher
* Fix config races
* Fix setup error not clearing
2023-11-28 13:56:46 +11:00
Maista
0dcd58763f
Log more when resolving Python (#4185)
* Log more when resolving Python

Users often have problems configuring their Python installations

* Convert if-else ladder to switch statement
* Consolidate Python resolution

Adds additional logging to plugin tasks to
align with the logging that scrapers output.
2023-11-22 10:04:15 +11:00
DingDongSoLong4
72779e618d
Fix batch performer tag panic (#4281) 2023-11-12 03:50:09 +02:00
DingDongSoLong4
9621213424
Genericise sliceutil functions (#4253)
* Genericise sliceutil.SliceSame
* Genericise intslice functions
* Genericise stringutil functions
2023-11-02 08:58:32 +11:00
its-josh4
2b8c2534dd
Update a number of dependencies (incl. CVE fixes) (#4107)
* Update a number of dependencies (incl. CVE fixes)

Includes some dependencies that were upgraded in #4106 as well as a few more dependencies.

Some deps that have been upgraded had CVEs.

Notably, upgrades deprecated dependencies such as:
- `github.com/go-chi/chi` (replaced with `/v5`)
- `github.com/gofrs/uuid` (replaced with `/v5`)
- `github.com/hashicorp/golang-lru` (replaced with `/v2` which uses generics)

* Upgraded a few more deps

* lint

* reverted yaml library to v2

* remove unnecessary mod replace

* Update chromedp

Fixes #3733
2023-10-26 16:24:32 +11:00
WithoutPants
2fd7141f0f
Javascript scraper postprocess (#4200)
* Add javascript post-process action
* Add documentation
2023-10-16 17:17:36 +11:00
DingDongSoLong4
33f2ebf2a3
Model refactor, part 3 (#4152)
* Remove manager.Repository
* Refactor other repositories
* Fix tests and add database mock
* Add AssertExpectations method
* Refactor routes
* Move default movie image to internal/static and add convenience methods
* Refactor default performer image boxes
2023-10-16 14:26:34 +11:00