stash/pkg/utils
notsafeforgit 6de7195ed6 perf: further optimize image duplicate detection
This update provides additional performance improvements specifically targeted at large image libraries (e.g. 300k+ images):

1. Optimized the exact match SQL query for images:
   - Added filtering for zero/empty fingerprints to avoid massive false-positive groups.
   - Added a LIMIT of 1000 duplicate groups to prevent excessive memory consumption and serialization overhead.
   - Simplified the join structure to ensure better use of the database index.

2. Parallelized the Go comparison loop in pkg/utils/phash.go:
   - Utilizes all available CPU cores to perform Hamming distance calculations.
   - Uses a lock-free design to minimize synchronization overhead.
   - This makes non-zero distance searches significantly faster on multi-core systems.
2026-03-20 15:26:18 -07:00
..
boolean.go Lint checks phase 2 (#1747) 2021-09-23 17:15:50 +10:00
date.go FR: Change Career Length to Career Start and Career End (#6449) 2026-02-17 13:44:03 +11:00
date_test.go FR: Change Career Length to Career Start and Career End (#6449) 2026-02-17 13:44:03 +11:00
doc.go Add package docs and project vision/goals (#5169) 2024-08-28 09:01:39 +10:00
func.go Add anonymise database task (#3186) 2022-12-23 09:15:27 +11:00
http.go Use gallery updated at for cover mod time (#5225) 2024-09-05 16:45:15 +10:00
image.go Use proxy in utils.ReadImageFromURL (#4637) 2024-02-29 11:28:30 +11:00
map.go Performer custom fields (#5487) 2024-12-03 13:49:55 +11:00
map_test.go Performer custom fields (#5487) 2024-12-03 13:49:55 +11:00
mutex.go Fix caption handling during scan and check before correcting path (#6634) 2026-03-02 14:44:20 +11:00
mutex_test.go Scan refactor (#1816) 2021-10-15 10:39:48 +11:00
phash.go perf: further optimize image duplicate detection 2026-03-20 15:26:18 -07:00
reflect.go Identify task (#1839) 2021-10-28 14:25:17 +11:00
reflect_test.go Identify task (#1839) 2021-10-28 14:25:17 +11:00
resources.go Change thumbnail default size and resize algorithm (#336) 2020-01-31 17:19:15 -05:00
strings.go Add penis length and circumcision stats to performers. (#3627) 2023-05-24 13:19:35 +10:00
strings_test.go [Files Refactor] Cleanup (#2893) 2022-09-07 16:50:15 +10:00
time.go Fix various generate issues (#1322) 2021-04-22 13:51:51 +10:00
url.go Performer urls (#4958) 2024-06-18 13:41:05 +10:00
url_test.go Performer urls (#4958) 2024-06-18 13:41:05 +10:00
urlmap.go Plugin assets, external scripts and CSP overrides (#4260) 2023-11-19 10:41:16 +11:00
urlmap_test.go Plugin assets, external scripts and CSP overrides (#4260) 2023-11-19 10:41:16 +11:00
user_agent.go Restructure go project (#2356) 2022-03-17 11:33:59 +11:00
vtt.go Lint fixes (#5476) 2024-11-13 09:47:29 +11:00
vtt_test.go utils: vtt: test + rewrite (#1284) 2021-05-25 11:25:26 +10:00