Two unrelated images ended up 4 bits apart in dHash space…
0 reactions
Two unrelated images ended up 4 bits apart in dHash space — how conservative should duplicate clustering be?
Two completely unrelated images ended up being treated as near-duplicates:
Image A: *beach landscape* photographed through a car window;
Image B: a lifted-up page of a *document.*
The measurements are:
|*Metric*|*Image pair*|*Threshold*|*Result*|
|:-|:-|:-|:-|
|Aspect-ratio Δ|0.000865|≤ 0.02|Pass|
|dHash distance|4|≤ 8|pass|
|pHash distance|30|≤ 10|fail|
|wHash distance|15|≤ 10|fail|
|Color-hash distance|6|informational|—|
The matcher accepted the pair because the *aspect ratio was nearly identical* and the *dHash Hamming distance was only 4*, significantly below the threshold of 8.
The other perceptual hashes strongly disagreed (pHash was 30 against a threshold of 10, and wHash was 15 against a threshold of 10) but were never consulted because the *dHash test did not seem to present a borderline case* and thus was accepted as *proof*.
Interestingly this isn't really a random dHash collision. Both images apparently collapsed into a *highly similar low-frequency brightness-gradient pattern* after compression and downsampling.
dHash is good at surviving compression, in particular because it ignores fine detail and *records coarse local brightness directions*. But that same *usefulness* can be a *weakness* that can make unrelated low-detail images collision-prone.
The more interesting problem in my case is what happens next. Hardening is especially important because the tool uses *union-find* to form duplicate clusters. A *single false-positive pair* can become a *bridge* that attaches an unrelated image to a whole valid duplicate component.
Instead of a binary True/False decision, the matcher now returns the full *evidence*: for each metric (aspect-ratio, dHash, pHash, wHash) delta versus limit and the optional SSIM score are returned, as is the decision and, when rejected, the rejection reason.
This fix itself isn't particularly sophisticated. What I found more interesting is the design question it raised. Should the acceptance threshold for a perceptual-similarity edge depend on what you're going to do with that edge?
For *image retrieval*, a false positive may just mean one irrelevant result. For union-find clustering, a false-positive edge can change an entire connected component.
I'm curious how others approach this. Would you put most of the conservatism in the pair matcher itself, or enforce stronger intra-cluster consistency after constructing candidate relationships?
And for near-duplicate images specifically: would you prefer multiple perceptual hashes, SSIM/local features, embeddings, hierarchical clustering, or another approach?
#technology
source
Two completely unrelated images ended up being treated as near-duplicates:
Image A: *beach landscape* photographed through a car window;
Image B: a lifted-up page of a *document.*
The measurements are:
|*Metric*|*Image pair*|*Threshold*|*Result*|
|:-|:-|:-|:-|
|Aspect-ratio Δ|0.000865|≤ 0.02|Pass|
|dHash distance|4|≤ 8|pass|
|pHash distance|30|≤ 10|fail|
|wHash distance|15|≤ 10|fail|
|Color-hash distance|6|informational|—|
The matcher accepted the pair because the *aspect ratio was nearly identical* and the *dHash Hamming distance was only 4*, significantly below the threshold of 8.
The other perceptual hashes strongly disagreed (pHash was 30 against a threshold of 10, and wHash was 15 against a threshold of 10) but were never consulted because the *dHash test did not seem to present a borderline case* and thus was accepted as *proof*.
Interestingly this isn't really a random dHash collision. Both images apparently collapsed into a *highly similar low-frequency brightness-gradient pattern* after compression and downsampling.
dHash is good at surviving compression, in particular because it ignores fine detail and *records coarse local brightness directions*. But that same *usefulness* can be a *weakness* that can make unrelated low-detail images collision-prone.
The more interesting problem in my case is what happens next. Hardening is especially important because the tool uses *union-find* to form duplicate clusters. A *single false-positive pair* can become a *bridge* that attaches an unrelated image to a whole valid duplicate component.
Instead of a binary True/False decision, the matcher now returns the full *evidence*: for each metric (aspect-ratio, dHash, pHash, wHash) delta versus limit and the optional SSIM score are returned, as is the decision and, when rejected, the rejection reason.
This fix itself isn't particularly sophisticated. What I found more interesting is the design question it raised. Should the acceptance threshold for a perceptual-similarity edge depend on what you're going to do with that edge?
For *image retrieval*, a false positive may just mean one irrelevant result. For union-find clustering, a false-positive edge can change an entire connected component.
I'm curious how others approach this. Would you put most of the conservatism in the pair matcher itself, or enforce stronger intra-cluster consistency after constructing candidate relationships?
And for near-duplicate images specifically: would you prefer multiple perceptual hashes, SSIM/local features, embeddings, hierarchical clustering, or another approach?
#technology
source
| 3,000 mlx | |
| 3 | |
| 0 |