stash/pkg/utils
notsafeforgit acc5438af2 perf: massive optimization for image and scene duplicate detection
This update provides significant performance improvements for both image and scene duplicate searching:

1. Optimized the core Hamming distance algorithm in pkg/utils/phash.go:
   - Uses native CPU popcount instructions (math/bits) for bit counting.
   - Pre-calculates hash values to eliminate object allocations in the hot loop.
   - Halves the number of comparisons by leveraging the symmetry of the Hamming distance.
   - The loop is now several orders of magnitude faster and allocation-free.

2. Solved the N+1 database query bottleneck:
   - Replaced individual database lookups for each duplicate group with a single batched query for all duplicate IDs.
   - This optimization was applied to both Image and Scene repositories.

3. Simplified the SQL fast path for exact image matches to remove redundant table joins.
2026-03-20 15:06:05 -07:00
..
boolean.go Lint checks phase 2 (#1747) 2021-09-23 17:15:50 +10:00
date.go FR: Change Career Length to Career Start and Career End (#6449) 2026-02-17 13:44:03 +11:00
date_test.go FR: Change Career Length to Career Start and Career End (#6449) 2026-02-17 13:44:03 +11:00
doc.go Add package docs and project vision/goals (#5169) 2024-08-28 09:01:39 +10:00
func.go Add anonymise database task (#3186) 2022-12-23 09:15:27 +11:00
http.go Use gallery updated at for cover mod time (#5225) 2024-09-05 16:45:15 +10:00
image.go Use proxy in utils.ReadImageFromURL (#4637) 2024-02-29 11:28:30 +11:00
map.go Performer custom fields (#5487) 2024-12-03 13:49:55 +11:00
map_test.go Performer custom fields (#5487) 2024-12-03 13:49:55 +11:00
mutex.go Fix caption handling during scan and check before correcting path (#6634) 2026-03-02 14:44:20 +11:00
mutex_test.go Scan refactor (#1816) 2021-10-15 10:39:48 +11:00
phash.go perf: massive optimization for image and scene duplicate detection 2026-03-20 15:06:05 -07:00
reflect.go Identify task (#1839) 2021-10-28 14:25:17 +11:00
reflect_test.go Identify task (#1839) 2021-10-28 14:25:17 +11:00
resources.go Change thumbnail default size and resize algorithm (#336) 2020-01-31 17:19:15 -05:00
strings.go Add penis length and circumcision stats to performers. (#3627) 2023-05-24 13:19:35 +10:00
strings_test.go [Files Refactor] Cleanup (#2893) 2022-09-07 16:50:15 +10:00
time.go Fix various generate issues (#1322) 2021-04-22 13:51:51 +10:00
url.go Performer urls (#4958) 2024-06-18 13:41:05 +10:00
url_test.go Performer urls (#4958) 2024-06-18 13:41:05 +10:00
urlmap.go Plugin assets, external scripts and CSP overrides (#4260) 2023-11-19 10:41:16 +11:00
urlmap_test.go Plugin assets, external scripts and CSP overrides (#4260) 2023-11-19 10:41:16 +11:00
user_agent.go Restructure go project (#2356) 2022-03-17 11:33:59 +11:00
vtt.go Lint fixes (#5476) 2024-11-13 09:47:29 +11:00
vtt_test.go utils: vtt: test + rewrite (#1284) 2021-05-25 11:25:26 +10:00