Hacker News Reader

Comment by dekhn | Hacker News Reader

The OME-zarr creators are pretty experienced in this field and have already done the work you're describing. Let's not create another standard after OME-zarr.

View on HN

Hey thanks, I tend to agree. Although if I read this right, OME-zarr punted on lossy compression (which is all that really matters) so IMHO it doesn't have much use in the real world yet even though people evidently put a lot of work into it:

https://github.com/ome/ome-zarr-py

https://ngff.openmicroscopy.org/latest/

https://zarr.readthedocs.io/en/v2.1.0/api/codecs.html

https://www.researchgate.net/publication/356622607_OME-NGFF_...

https://www.biorxiv.org/content/10.1101/2023.02.17.528834v1....

  i.munro (Ian Munro) August 15, 2023, 8:54am
  
  On the original point about size I suspect your answer is compression.
  I have found that a lot of image formats use lossy compression.
  I’m not sure about CZI specifically.
  AFAIK ome-zarr (sensibly IMO) uses only lossless compression.
  
  Best
  
  Ian

and:

  sebi06 (Sebi06) August 15, 2023, 1:42pm
  
  Hi all,
  
  for CZI we offer currentyl two compression methods:
  
  JPG-XR
  ZSTD
  
  So we have lossless and lossy options depending on your needs. All of them are implemented in our C++, #NET or Python APIs and also supported from BioFormats.

Looks like CZI is a competitor:

https://www.zeiss.com/microscopy/en/products/software/zeiss-...

https://github.com/cgohlke/czifile

http://www.physics.hmc.edu/~gerbode/wppriv/wp-content/upload...

CZI is basically JPEG compression and is probably missing the multidimensional compressor which is the secret sauce missing from the various formats. It also mentions licensing and legal so is maybe lost in the weeds and not open source.

Without more info, it feels like this is still an open problem.

My interest in this is for video games, since so many are multiple GB in size but are also missing this multidimensional compressor. If we had it, I'd guess a 10:1 to 100:1 reduction in size over what we have now, with no perceptual loss in quality.

Lossy compression is a bad idea for scientific images. For instance, we often need to understand the statistics of photon detection events in background regions. That’s one of the first things to get tossed.

Please don't use lossy compression for scientific data.

Disk space is cheap relative to people-time and data-collection-time.

"Perceptual" is misleading- we're not just humans looking at jpegs. Many people are doing hard-code quantitative analysis for machine learning. better to downscale the images (2X, 4X, 8X or more) than to use lossy compression if you are truly low on disk budget.

It sounds like there should be a standard for compression of scientific data, probably as a noise ceiling instead of a compression ratio, if the noise is normally distributed (a big if). But if I had to choose, I would side with you and vote to use lossless in order to avoid a whole class of errors that might come from lossy compression. Especially when we can get a 1 TB SSD for under $50.

To play devil's advocate, I think there might be an opportunity for lossy compression of images below the Nyquist Rate of the microscope:

https://svi.nl/Nyquist-Calculator

I only bring this up because I worked with industrial cameras at a previous job and even the best lenses only got us to maybe 1/2 or 1/4 of the resolution that the sensors were capable of. Microscopes should probably include an effective resolution in each image file's metadata. If they don't, then we have no data on how noisy the image is, so this is all a moot point.

I understand you're trying to help, but everything you are suggesting is basically not helpful.

Microscopists are savvy enough to match sensors and lenses, it's typically called "diffraction limited" (https://en.wikipedia.org/wiki/Diffraction-limited_system) based on the Rayleigh criterion (https://en.wikipedia.org/wiki/Angular_resolution#The_Rayleig...). I rarely if ever hear microscopists refer to nyquist limits. They would not overimage something at a higher magnification, or more correctly, they would be aware it was happening and either downscale or not worry.

Next, when you make a point like "we can get a 1TB SSD for under $50". It shows you're not understanding the domain microscopists work in. Their scopes typically cost $100K (with a camera for another $50K, and a stage that costs $10K). The cost of the storage system (or parking the data in S3) for 10 years is usually far less than the capital cost of the scope, or the operational expense of maintaining staff on site.

I believe the effective resolution can already be computed from the captured metadata, which includes the objective's numerical aperture, as well as the frequency used for imaging (many microscopes today use lasers to image with a very specific frequency to cause fluorenscence, which itself has a fairly tight output spectrum).

Thanks those are better terms. I remember doing a test where the camera photographs a ceramic plate with an almost perfectly black diagonal line to calculate the effective resolution, but I can't remember what it's called, maybe the diffraction limit. It's basically a measure of how blurry the image is, where a matched lens and sensor would look like a Bresenham line with no antialiasing.

TBH, I think that the scientific community has huge blind spots, mostly due to its own gatekeeping. Some of the things like this that seem to be a struggle remind me of the design-by-committee descent of web development. The brilliant academic work of the 1990s has been mostly replaced by nondeterministic async soup and build processes of such complexity that we have to be full stack developers just to get anything done. All the fault of private industry hoarding the wealth and avoiding any R&D that might risk its meal ticket. Starving the public of grants, much less reforms that are actively blocked like UBI. Now nobody ever seems to step back and examine problems from first principles anymore. That's all I was trying to do.

Edit: I found the term for calculating the effective resolution of a camera with a high-contrast diagonal line, it's "slanted edge modulation transfer function (MTF)":

https://harvestimaging.com/blog/?p=1328

https://www.strollswithmydog.com/the-slanted-edge-method/

https://www.eckop.com/resources/optical-testing/resolution-t...

>which is all that really matters

Except for the domains where nobody uses it because it ruins the data.

Imagine that in the Medicine World with DICOM images.

DICOM technically supports lossy compression, but I think it's regarded as something of a historical mistake. It's pretty much never used and some viewers will plaster big warnings on lossily compressed series and others will outright refuse to load them.

> lossy compression (which is all that really matters)

No, not in this case. This might be great for game files or whatever you are familiar with but lossy compression would basically mean to be mangling the data after going through all the effort of collecting it (I suspect you have only little experience with lab work?).

When doing experiments, everything is documented in full detail. Settings on machines and equipment, preparation steps, version numbers of software used for processing, …

You really don‘t want to lose information on your experiment.

Reply on news.ycombinator.com