Commit graph

16 commits

Author SHA1 Message Date
WithoutPants
e9d48683f8
Autotag scraper (#1817)
* Refactor scraper structures
* Move matching code into new package
* Add autotag scraper
* Always check first letter of auto-tag names
* Account for nulls

Co-authored-by: Kermie <kermie@isinthe.house>
2021-10-11 23:06:06 +11:00
SmallCoccinelle
a5ca8fc678
Enable safe linters (#1786)
* Enable safe linters

Enable the linters dogsled, rowserrcheck, and sqlclosecheck.

These report no errors currently in the code base.

Enable misspell.

Misspell finds two spelling mistakes in comments, which are fixed by the
patch as well.

Add and sort linters which are relatively
safe to add over time. Comment them out for now.

* Close the response body

If we can get a HTTP response, it has a body which ought to be closed.

By doing so, we avoid potentially leaking connections.

* Enable the exportloopref linter

There are two places in the code with these warnings. Fix them while
enabling the linter.

* Remove redundant types in tests

If a slice already determines the type, the inner type declaration is
redundant. Remove the inner declarations.

* Mark autotag test cases as parallel

Autotag test cases is by far the outlier when it comes to test time.
While go test runs test cases in parallel,
it doesn't do so inside a given package, unless one marks the test cases
as parallel.

This change provides a significant speedup on a 8-core machine for test
runs.
2021-10-03 11:48:03 +11:00
WithoutPants
4625e1f955
Unify scrape refactor (#1630)
* Unify scraped types
* Make name fields optional
* Unify single scrape queries
* Change UI to use new interfaces
* Add multi scrape interfaces
* Use images instead of image
2021-09-07 11:54:22 +10:00
bnkai
597576f5e6
Get distinct values from scraper (#1338)
Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
2021-04-29 11:38:55 +10:00
bnkai
aedadc3857
Add lbToKg pp action to the scraper (#1337) 2021-04-26 13:31:25 +10:00
julien0221
d673c4ce03
added details, deathdate, hair color, weight to performers and added details to studios (#1274)
* added details to performers and studios
* added deathdate, hair_color and weight to performers
* Simplify performer/studio create mutations
* Add changelog and recategorised

Co-authored-by: WithoutPants <53250216+WithoutPants@users.noreply.github.com>
2021-04-16 16:06:35 +10:00
WithoutPants
f6ffda7504
Setup and migration UI refactor (#1190)
* Make config instance-based
* Remove config dependency in paths
* Refactor config init
* Allow startup without database
* Get system status at UI initialise
* Add setup wizard
* Cache and Metadata optional. Database mandatory
* Handle metadata not set during full import/export
* Add links
* Remove config check middleware
* Stash not mandatory
* Panic on missing mandatory config fields
* Redirect setup to main page if setup not required
* Add migration UI
* Remove unused stuff
* Move UI initialisation into App
* Don't create metadata paths on RefreshConfig
* Add folder selector for generated in setup
* Env variable to set and create config file.
Make docker images use a fixed config file.
* Set config file during setup
2021-04-12 09:31:33 +10:00
WithoutPants
a0676d5c30
Performer tags (#1132)
* Add scraping support for performer tags
* Add performer count to tag cards
* Refactor sqlite test setup
* Add performer tag filtering in gallery and image
* Add bulk update performer
* Add Performers tab to tag page
* Add count filters and sort bys for tags
* Move scene count to icon in performer card #1148
2021-03-10 12:25:51 +11:00
WithoutPants
1e04deb3d4
Data layer restructuring (#997)
* Move query builders to sqlite package
* Add transaction system
* Wrap model resolvers in transaction
* Add error return value for StringSliceToIntSlice
* Update/refactor mutation resolvers
* Convert query builders
* Remove unused join types
* Add stash id unit tests
* Use WAL journal mode
2021-01-18 12:23:20 +11:00
WithoutPants
9a84726128
Fix xpath comment element parsing (#759) 2020-08-23 17:39:15 +10:00
woodgen
e3ea3ea85e
scraper/mapped: Add feetToCm post process. (#711)
This patch adds a feetToCm post process that converts imperial feet and
inches to centimeters.
2020-08-12 11:17:43 +10:00
WithoutPants
2b9215702e
Refactor xpath scraper code. Add fixed and map (#616)
* Refactor xpath scraper code
* Make post-process a list
* Add map post-process action
* Add fixed xpath values
* Refactor scrapers into cache
* Refactor into mapped config
* Trim test html
2020-07-21 14:06:25 +10:00
bnkai
9d0522f62d
Add "split" xpath in post-processing , newlines in replace support (#579) 2020-06-18 10:47:10 +10:00
WithoutPants
03c07a429d
Add Xpath post processing and performer name query (#333)
* Extend xpath configuration. Support concatenation

* Add parseDate parsing option

* Add regex replacements

* Add xpath query performer by name

* Fix loading spinner on scrape performer

* Change ReplaceAll to Replace
2020-01-31 17:17:40 -05:00
WithoutPants
78eb527ec4 Scraper fixes (#332)
* Fix panic on invalid xpath

* Add missing attrs to scraped performer fragment
2020-01-24 22:36:24 -05:00
WithoutPants
7fdaccf669 Xpath scraping from URL (#285)
* Add xpath performer and scene scraping

* Add studio scraping

* Refactor code

* Fix compile error

* Don't overwrite performer URL during a scrape
2020-01-04 11:39:33 -05:00