Image Compression with Neural Networks
research.googleblog.com
research.googleblog.com
When zoomed in, the JPEG artifacts are quite apparent and the RNN produces a much smoother image. However, to my eye when zoomed out the high frequency "noise", particularly in the snout area, looks better in JPEG. The RNN produces a somewhat blurrier image that reminds me of the soft focus effect.
Part of the reason JPEG looks better in this case is that they have chosen to feature a detail- and noise-rich portion of the image, which favors JPEG... High JPEG compression has a sharpening effect around edges, including "ringing" which hides well in noise. However, if you view the full-sized comparison of the dog images, you can see that JPEG has essentially no detail in the darker portions of the image. Presumably both the MS-SIM metric and humans would judge RNN better for these portions.
Ringing is the opposite of a sharpening effect, no? JPEG essentially acts as a low-pass filter over the DCT tiles of the image, so you get ringing because you eliminate the high-frequency components that make the edge sharp and you're left with a wavelet that extends past the sharp edge and isn't cancelled out the way it used to be.
JPEG looses information by quantization, not low-passing. It is true that small high frequency coefficients will get rounded to 0 during quantization, and this does save quite a few bits. But the ringing is just from being imprecise about each DCT coefficient, coding a value that is either too large or too small.
wyager accurately described why in terms of the DCT though -- cosines naturally "ring" and the only way a non-sinusoid can be represented is through the combination of many other terms, so any time a coefficient contributing to this shape is eliminated, the structure "decays" so to speak and the underlying sinusoids become part of the reconstruction.
It appears to be a severely flawed metric.
> Presumably both the MS-SIM metric and humans would judge RNN better for these portions.
doesn't remotely hold for detailed/high contrast. If I can see a huge difference, and vastly prefer one over the other, the metric is not useful by its definition. Please don't get emotional about figures of merit.
I'm guessing a genuinely useful figure of merit would have put both file sizes a bit closer together, and the NN would have shined, especially since it works so well in low contrast areas.
You can, of course, recognize (and remove) textures using the convolution nodes of a convolutional NN, encode the texturing parameters as part of the output of the network, and then reverse the process with a deconvolutional NN, with the textures reappearing at the end. The deconvolution nodes are in effect equivalent to that "noise texture sample library."
But this is, in essence, two algorithms: one that compresses textures, and then another that compresses "everything else" once you've subtracted the texture. Thinking about the second algorithm, I wonder whether you need to go to the trouble of an ML approach at all: a non-textured image would effectively be a smooth color gradient, perfectly amenable to wavelet compression.
1. JPEG 2000 is Patent encumbered.
2. Neural Network methods have good chance of getting significantly better.
3. The way similar architecture / ops have been found to be very effective across variety of tasks, its not unlikely that in future, all machines will have a dedicated neural inference chip, e.g. NVidia Jetson TK1 or Google TPU. In this scenario the more mileage you can get out of such dedicated hardware the better.
It's a proof of concept, not to be used in real life for anything. It would be too expensive. Also, it's not a refined product, just an attempt to see if neural networks can do better compression.
It's also good to maintain some perspective on what it's actually worth to do better than JPEG. We have had actual usable products better than JPEG for a long time (I mentioned J2K but there's been so many others like WebP and BPG etc), and no one cares. People use GIF as a short-form video format--that's how much no one cares about this stuff. So I think it's mostly worthwhile as research, to increase our ability with neural nets in general, rather than as an eventual product. But who knows?
I made a website where one could compare various formats:
"The next challenge will be besting compression methods derived from video compression codecs, such as WebP (which was derived from VP8 video codec), on large images since they employ tricks such as reusing patches that were already decoded."
Beating block based JPEG with a global algorithm doesn't seem that exciting.
Neural networks know that objects move. See for example https://arxiv.org/abs/1511.05440 http://web.mit.edu/vondrick/tinyvideo/
The image labeling neural networks are a good proof of concept of this possibility. After all, what is a text label other than a concise (ie highly compressed) representation of the pixel data in the image. Obviously, being able to represent that a cat is in an image is quite lossy as compared to being able to represent that a particular cat is in an image (and where it is, and how it's scaled, etc). However, it's easy to imagine (in principle) layering on a hierarchy of other features, each explaining more and more of the statistics in the scene, until the original image is reproduced to arbitrary precision.
Could this outperform a hardwired video compressor? In terms of file size/bitrate, my intuition is yes and probably by a lot. In terms of computational efficiency, no idea.
> Think of the classic stereotype that all asian faces look alike
> to europeans: that's ok for still labeling a human face, and useful.
> But for image compression to have different quality based on the
> subject would be useless!
An extreme case of this not happening can be found among persons with prosopagnosia (face blindness). These people have non- or poorly functioning facial processing circuitry and exhibit great difficulty distinguishing faces. Presumably to the extent they can distinguish people at all, they must use specific features in a very cognitive way ("the person has a large nose and brown hair") rather than being able to apprehend the face "all at once".
Incidentally, I think there are a myriad of other examples of this phenomena, especially among specialists (wine tasters, music enthusiasts, etc) who have highly trained processing circuitry for their specialties and can make discriminations that the casual observer simply cannot. Another example that just came to mind is that of distinguishing phonemes that are not typical in ones native tongue. One reason that these sounds are so difficult for non-native speakers to produce is because they are difficult for non-native speakers to distinguish, and it simply takes time to learn to "hear" them.
All this is to say that your perceptual experience is not as stable as you think it is. Any sort of AI compression need only to be good enough for you or some "typical person". If the compressor was trained on asian faces (and others, besides) then it should be able to "understand" and compress them, perhaps even better than a naive white person. I could even imagine the AI being smart enough to "tune" its encoding to the viewers preferences and abilities.
https://my.mixtape.moe/klvzip.png
Static mirror: https://archive.fo/yyozl
So instead of implementing a DCT on my client I need to implement a neural network? Or are these encoder/decoder steps merely used for the iterative "encoding" process? It seems like the representation of a "GRU" file is different from any other.
Super simple to implement the feed-forward scenario for decoding.
Not entirely sure what the residual memory aspect of these networks add in terms of complexity, but it's probably just another vector add-multiply, or something to that effect.
http://cs.stackexchange.com/questions/22317/does-there-exist...
They basically ripped me a new one said it was a stupid idea and that I shouldnt make suggestions in a question. Then I took the suggestions and details out (but left the basic concept in there) and they gave me a lecture on basics of image compression.
Made me really not want to try to discuss anything with anyone after that.
It's obvious that on a theoretical standpoint a RNN can help with compression...
A while ago I thought about a RNN or a NN for encryption i.e. cracking it (I had did something like post on that topic.. but saw no replies whatsoever)..
gonna try to find that if someone's interested
Not a huge fan of the negativity on StackOverflow, but until August 2016 (when the paper this was based on was published - or maybe 2015, with DRAW[2]) people actively working in this area didn't think it was possible.
Also, the single answer there certainly didn't say anything like it was a stupid idea. I don't think the author of that answer knew much about autoencoders.
Also^2, your question isn't really anything like what this addresses. Your question concentrates on the idea of compressing a large set of images, and sharing some kind of representation.
That certainly is possible without using a ML approach. And yes, autoencoders have been around for a long time.
But hoverboards are a great idea, too.
[1] This paper has 7 authors, Gregor et al has 5 authors, DRAW has 6.
It seems to me like the data driven approach could greatly outperform hand tuned codecs in terms of compression ratio by using a far more expressive model of the input data. Computational cost and model size is likely to be a lot higher though, unless that's also factored into the optimization problem as a regularization term: if you don't ask for simplicity, you're unlikely to get it!
Lossy codecs like jpeg are optimized to permit the kinds of errors that humans don't find objectionable. However, it's easy to imagine that this is not the right kind of lossyness for some use cases. With a data driven approach, one could imagine optimizing for compression which only looses information irrelevant to a (potentially nonhuman) process consuming the data.
The idea of using NNs for compression has been around for at least 2 decades. The real issue is that it's ridiculously slow. Performance is a big deal for most applications.
It's also not clear how to handle different resolutions or ratios.
She Simplest is a network with inputs of [X,Y] and outputs of {R,G,B] Where the image is encoded into the network weights. You have to per-image train the network. My guess is it would need large complex images before you could get compression rates comparable to simpler techniques. An example of this can be seen at http://cs.stanford.edu/people/karpathy/convnetjs/demo/image_...
In the same vein, you could encode video as a network of [X,Y,T] --> [R, G, B]. I suspect that would be getting into lifetime of the universe scales of training time to get high quality.
The other way to go is a neural net decoder. The network is trained to generate images from input data, You could theoretically train a network to do a IDCT, so it is also within the bounds of possibility that you could train a better transform that has better quality/compressibility characteristics. This is one network for all possible images.
You can also do hybrids of the above techniques where you train a decoder to handle a class of image and then provide a input bundle.
I think the place where Neural Networks would excel would be as a predictive+delta compression method. Neural networks should be able to predict based upon the context of the parts of the image that have already been decoded.
Imagine a neural network image upscaler that doubled the size of a lower resolution image. If you store a delta map to correct any areas that the upscaler guesses excessively wrong then you have a method to store arbitrary images. Ideally you can roll the delta encoding into the network as well. Rather than just correcting poor guesses, the network could rank possible outputs by likelyhood. The delta map then just picks the correct guess, which if the predictor is good, should result in an extremely compressible delta map.
The principle is broadly similar to the approach to wavelet compression, only with a neural network the network can potentially go "That's an eye/frog/egg/box, I know how this is going to look scaled up"
What if you use uniqueness and eigenface look up table for compression?