Comment by sillysaurusx | Hacker News Reader

Comment by sillysaurusx | Hacker News Reader

Parent Full thread

Which file formats did you use?

One of my frustrations with ML is that almost no one seems to be training on HDR data. It's all 8-bit RGB. There is a big opportunity here to train GANs on raw sensor data.

But that raises the question of which file formats to focus on. Since you have lots of image processing experience, I'm very interested in what the data formats were like at your job. Was it proprietary formats? If so, were there any advantages over e.g. the RAW file format? Is RAW sufficient for standardizing "everything necessary to turn photon readings into a photograph that appeals to the human eye"? E.g. if I gather thousands of RAW photos, do I need any other information not present in the RAW file?

Mainly I am interested in any insights you gained on the job: you might know of some unique advantage that certain data formats have over others. I'm extremely interested in training ML models to synthesize what comes into a camera (photon readings), as opposed to what the world looks like to the human eye (RGB values).

(I apologize for not phrasing my questions more carefully. It's late. The essence is that right now, most ML training operate in terms of RGB, yet there seems to be many advantages to operating in "photon space". The trouble is, it's effortless to gather millions of RGB photos -- but it seems very difficult to gather a million RAW photos. I'm hoping someone knows of a massive trove of "some kind of raw data that can be turned into RGB values" – maybe that's RAW, maybe it's something else. Whatever it is, once I have it, I can feed it through an ML model.)

RAW from most cameras isn't "real RAW". Firstly it generally uses lossy compression secondly a lot of cameras apply some minor corrections to "RAW" data as well (e.g. noise reduction). Some would argue that most cameras can't give you "real" RAW since most sensors perform noise reduction one way or the other internally (e.g. most Sony sensors use correlated double sampling). Most sensors use analog PGAs to amplify the pixel signals before A/D conversion (that's the ISO setting)[2], which changes the characteristics of the sensor quite drastically (not just dynamic range and noise, a lot of older sensors had severe issues with tinting and banding at high gains). RAW formats are generally linear (those used for moving pictures are sometimes log-encoded, which makes more sense [1], but isn't "raw").

A library like libraw reads most RAW files and can give you an appropiately (for the sensor used) debayered RGB image. These images aren't in a defined color space; they're specific to every camera/sensor.

In the moving pictures space OpenEXR is a common format for "arbitrary" images, since it allows any number of channels each with different formats (e.g. you can have a file with 27 channels per pixel, some of which are float and some of which are integer).

[1] E.g. 12 bit log encoding gives even resolution across more of the dynamic range; a linear encoding gives half of each range to the brighter stop.

[2] Sensors that don't have PGAs simply have a fixed sensitivity. If the camera offers an ISO setting, it is emulated in the digital domain by pulling the image.

I’m assuming PGA is Programmable Gain Amplifier? It wasn’t immediately obvious from a web search.

Yes. The basic structure of a CMOS sensor looks a lot like a DRAM array, but instead of a capacitor you have a photo diode and the "bit lines" / column lines go into a programmable gain amplifier and then into a 12 or 14 bit ADC with some DSP [1] and a parallel-to-serial interface. The exposure is determined by the readout (row for row), that's what gives you rolling shutter.

Some sensors have global shutter instead, which makes it even more like DRAM: Conceptually each pixel has a capacitor and a photodiode, and a transistor connecting the two, all of these transistors are connected in parallel and form the "global shutter signal".

Some sensors have faster ADCs and use one ADC for a bunch of columns, which has been claimed as the source of column-banding in some sensors (unclear if correct).

[1] Sensors that are intended for both photo and video typically support things like pixel binning, where the sensor itself averages e.g. 2x2 blocks of pixels internally. A lower quality alternative is line skipping, where the sensor is told to only read out every nth line, thus reducing resolution considerably. The higher quality alternative is "full-sensor readout", i.e. the camera reads all pixels and downsamples the image to the video resolution. I believe some (announced?) sensors can do this in the sensor itself now.

Note: A lot of this is "somewhat informed speculation" on my part, because image sensor manufacturers tend to be very secretive of their sensor's details.

Annnnd comments like this are why I'm on HN daily. Thanks for sharing your knowledge!

TIFF is kind of standard for most scientific imaging, from astronomy to biomedical.

12- or 16-bit greyscale is common, with multiple channels for hyperspectral data. But since it can also store 32-bit data, floats, and complex numbers as well as rationals, it can be used as a container for pretty much anything you want to throw at it.

Regarding RAW. It's often used with colour CCDs with various types of CFA, like Bayer or CMY. It might have a higher bit depth, but it's still basically RGB but spatially separated and stored as greyscale. You could maybe feed it directly to your ML, but it's more common to filter it to create RGB data retaining the bit depth but at the expense of low pass filtering to remove aliasing artefacts. Or keep the separation to keep the high frequency components; really depends upon what information you want to extract.

No one in astronomy uses TIFFs. Most scientific images are stored in FITS format, either in integer form for some raw data or in floating-point form.

FITS is very domain-specific for astronomy. It's not at all common in the wider scientific community.

I used to work on on of the larger imaging tools, supporting over 150 scientific file formats. The vast majority, over 80%, were using TIFF as a container. The next most popular was HDF5. FITS was not something I saw commonly, and we were getting images from researchers all over the world.

If you want a terabyte or two of .CR2 (Canon’s “RAW” file format) files of astrophotography to train on, let me know. They were taken on a camera sensor that had a full spectrum IR conversion performed on it.

All I ask is that you share your findings with me and don’t redistribute the image data set without confirming with me.

Yes, absolutely! That sounds wonderful, and we certainly won't redistribute the data without asking.

We have a server with a 24TB NAS. If you send me your ~/.ssh/id_rsa.pub key, I can add you to it and you can upload straight to the NAS. Would that work?

I have an event to go to today, but if you hop into our ML discord server (https://github.com/shawwn/tpunicorn#ml-community) and DM me (I'm shawn), we can figure out the details. Or email me (shawnpresser@gmail.com).

Thanks so much! This is exactly what I had in mind when I was writing my comment. I think it would be straightforward to extend StyleGAN to synthesize full spectrum data. And the results would look pretty darn cool.

(Not astronomy) When I worked in particle physics it was all TIFF, and often forks of libtiff with non-standard extensions, with nothing under version control. In hindsight it was a miracle the images were readable from one month to the next.

What problem you try to solve with GANs? Your RAW HDR photos are really taken by a sensor with RGB mask filtering out whatever is not visible spectrum. Scientific or indistrial imaging mostly use monochromatic information which is later colorized for human eyes.

Even if you get millions of RAW photos they will come from different sensors with different properties (noise, sensitivity, mosaic, orientation...) and you need to to normalize them all. sRGB is most common normalization target for which all prosumer cameras are optimized for.

The format is not really important here as it is usually array of pixels. You need more bits and resolution? Fine, but you also get more useless noise.

Color and dynamic range of most sensors extends greatly beyond sRGB, it's the camera's JPEG processing engine which maps this larger range to the smaller sRGB (or AdobeRGB) range.

You can attach custom ICC profile to JPEG, also to TIFF or PNG. But still you need to normalize data from different kind of sensors before you feed any ML engine with it. sRGB or AdobeRGB are most common color space profiles you can get with varied content.

Astronomy is done using FITS files[1]. Dedicated astronomical imaging instruments natively produce FITS files.

[1] https://en.wikipedia.org/wiki/FITS

Useful: https://arxiv.org/abs/1905.13189

Well, in remote sensing it was mostly 8-bit data per frequency range, then each satellite or instrument by name (LANDSAT, GOES, SPOT, etc.) has its own file format.

Sometimes a CCD bit was stuck, so we would process around that, etc.

The terms "HDR" and "photo" are not commonly used.

The Perkin-Elmer military system has fortran source, so all the file formats are documented there (if you can get access), as well as a minimal Scottish university system.

Reply on news.ycombinator.com