pHash – An open source perceptual hash library (2013) | Hacker News Reader

pHash – An open source perceptual hash library (2013)

phash.org

60 pointstosh24 comments

It's interesting to note that conceptually stuff like pHash (which is very useful) is similar to dimensionality reduction as we see it in machine-learning/data-science scenarios.

It's not just similar, learning a classifier is 100% identical to finding a perceptual hash onto your classes that aligns with human sensibilities

Kind of. The only point of phash is to make comparing images fast and cheap... meaning from a practical standpoint there isn't a ton of useful information in the hash. All you really care about is the final hamming distance between two hashes.

I think classifiers tend to encode actual interesting information.

They can, but neural architectures range in granularity from basic autoencoders with softmax which can be equivalent to a hash function, to convolutional networks which explicitly encode fine detail. Neural networks are a great tool that when used well can operate with explicit functions like phash- they don't have to be a black box mystery. Pick the right architecture and tolerance and you might eliminate a lot of development, maintenance, and troubleshooting.

Glad it's on Github here: https://github.com/aetilius/pHash

Github repos tend to outlive the homepage of some projects. I got worried when I read this:

I've been using this in a Rails app to detect duplicate image uploads for years. It works really well.

https://github.com/westonplatter/phashion

at a high level, how did you implement this? Are you storing the phash of each image in the database and then when new images are uploaded, do you query the database to see if any matches return? If so, how does that query actually work?

Or are you using this to ensure there’s no duplicates in a batch of images being uploaded?

Yep, I store each hash in the DB. The ruby gem has a mechanism to test similarity to your existing hashes with a threshold so small edits still match. It's surprisingly fast too, checking against every existing hash doesn't noticeably slow down the upload. I think it's a bitwise compare, but admittedly I set that up a long time ago so I'm foggy on the details.

Originally I added it because I wanted a way to automatically find the full quality image from my archive and replace the aggressively scaled down versions I used to use. Back in 2003 100KB was considered pretty big for a photo, but now it looks like garbage. I never found time to finish that part of it.

Yea, i've used phashs before to find duplicates in image datasets for training neural nets. Mainly as a high level method for cleaning up large image sets.

Neural nets are just locality-sensitive hashing functions, after all.

It's funny, but as a naive reader, I was reading through with the question of why you would use DCT whether it's significantly different from python's cosine_similarity, and why you would use that instead of say, pythagorean distance, edit distance, conditional entropy, and related methods. Is it a speed/accuracy trade off? Obviously I don't have depth on these as the answer would be trivial to someone who did, but I'm wondering if there a simple breakdown of what these and other information similarity methods optimize for.

? Any references?

Sparse neural networks can be optimized by implementing them using Locality Sensitive Hashing (2019 paper): https://arxiv.org/pdf/1903.03129.pdf

Fun fact, last week one of the main authors of the above 2019 paper spun off a company for the above approach: https://techcrunch.com/2021/08/13/thirdai-raises-6m-to-democ...

When they convert a large number of inputs into a small number of outputs, they're acting like a hash function. (By this metric, lossy compression algorithms are all hash functions.)

One of the most fun times I ever had coding was implementing a simple but quite fast and reasonably efficient image hash.

Loosely based on this algorithm:

https://benhoyt.com/writings/duplicate-image-detection/

Highly recommended fun exercise.

I use this a lot for image hashing. How does it compare to Apple's NeuralHash?

I would expect that Apple probably did a very good job at their implementation, and pHash is quite old tech.

I've been looking for something like this to write a custom solution for finding near duplicates in image libraries. Will give it a spin!

Note that pHash is GPL (if you're thinking of including it in a library or product).

I evaluated a ton of different images hashing algorithms for PhotoStructure, and saw similar accuracy from pHash, dhash, and mhash. I found that an mhash triplet rendered against a L*a*b* space (rather than just looking at a brightness channel) was best.

I also found too many false negatives _and false positives_ when aggregating duplicates until I also included file metadata heuristics. Pesky camera manufacturers are surprisingly inconsistent in how they encode the same metadata across, say, RAW and JPG images. I'm up to a gig of exemplar pairs that CI tests use to verify a litany of weirdnesses, like inconsistently encoding exposure information (in looking at you, Google Pixels), or GPS (iDevices), or even make, model, and serial number (several dSLR and other flagship phone manufacturers).

I wrote this up with some more details if you're interested: https://photostructure.com/faq/what-do-you-mean-by-deduplica...

I don't need to release the software, just using for my personal internet archive server with a few TB of images. Specifically interested in finding images that are a subset of others, or recompressed with JPG, or slightly discolored or rotated, or with text overlaid.

Interesting software!

You'll probably be interested in these links which describe algorithms in simple words: http://hackerfactor.com/blog/index.php?/archives/432-Looks-L... and http://hackerfactor.com/blog/index.php?/archives/529-Kind-of...

Does it pronounce 'P-Hash' or 'Fash'?

Reply on news.ycombinator.com