Comment by alister | Hacker News Reader

Comment by alister | Hacker News Reader

I have an observation about scanning documents that results in good quality and smaller files, but I can't satisfactorily explain why it works. Consider these two cases:

(1) Scan document at very high resolution as a JPG and then use a third-party program (like Photoshop or whatever) to re-encode the JPG at your preferred low resolution.

(2) Scan document at your preferred low resolution as a JPG straight away. Don't re-encode afterward.

Intuition says that the results of #1 vs #2 should be identical, or that #1 should be worse because you're doing two passes on source material. But I always get better results with case #1 (i.e., high-res scan and re-encoding afterward) regardless of the type or model of scanner, or whether the scanner does the JPG encoding on-board the device itself or through a Windows/Linux/Mac driver bundled with the scanner.

My theory is that scanner manufacturers are deliberately choosing the JPG encoding profile that gets them the fastest result. They want to brag about pages per minute which is an easily measured metric. Quality of JPG encoding and file size take effort to compare, but everyone understands pages per minute.

If anyone has contrary experience I'd like to hear it. I've been seeing this for years with different document scanners and flatbed scanners -- regardless of how I tweak the scanner's settings, I can always get good quality in a small file by re-encoding afterward.

In addition to some other points, the downscaling step in #1 may also smooth out some noise in the source image. Less noise yields more compressible data.

In my own scanning workflow, I scan at 600 or 1200dpi to png, deskew, downscale, and apply a black/white threshold. This is all done with imagemagick: mogrify -deskew 40% -scale 25% -threshold 50%

If I want a PDF after that I'll use img2pdf.

If you're scanning at a lower resolution, the scanner has fewer samples to work with when trying to make a visual representation of your document. If you scan at a higher resolution, the algorithm could at the very least average together nearby samples. It could also detect sharp lines vs fuzzy borders and decide whether the low-res version has a sharp transition or an averaged color between areas.

Low-resolution scans are faster than high-resolution scans for this exact reason: the sampling is worse. There might be the same number of samples per line captured by the linear CCD (and then downsampled), but the distance between scanlines varies by DPI.

> My theory is that scanner manufacturers are deliberately choosing the JPG encoding profile that gets them the fastest result.

This is more-or-less correct. The chips in the printers have a lot less power than your CPU, and the algorithms are a lot worse than those in Photoshop.

I'm surprised there aren't any high-quality JPEG encoder ASICs.

Would scanners benefit from just using some of the plentiful, cheap, excellent-quality H.264 encoder ASICs—and treating the output as HEIF?

My guess slightly different: manufacturers are deliberately choosing the JPG encoding profile that gets them good quality in worst cases. Which also happens to get fast encoding and bigger files. Their motivation is simple. One case of negative user experience outweigh a thousand positive ones and hurts their reputation hard.

While the JPEG encoder in Photoshop is likely a lot better than what's in your scanner, I am, similarly to others here, fairly convinced that the majority of the difference is in the sample rate by the scanner.

I did some vanilla JS that simulates scaling from high DPI vs sampling directly at a specific DPI and the result does resemble the result of a low resolution scan.

http://chrisbenjaminsen.com/shared/samplerate/

I am aware optics does not work exactly work like this, but it's a reasonable approximation.

Of course you get better results with 1.

Here is why: if you go with (2), then (1) is still done: by the crap firmware in your printer or its driver. It scans at high resolution and then downsamples in some way over which you have no control. It might not even be done with floating-point math.

1. is a somewhat like getting the raw image from a camera: a higher quality source for your own processing.

Reply on news.ycombinator.com