It's funny how every time I read an astronomy headline mention the word "visible", my brain immediately interprets it as a conventional photograph and is slightly disappointed when there isn't one.
It clarifies the motivation for extension/augmentation of human perception through technology, from microscopes to telescopes - and spectacles/glasses for that matter.
It also implies a continuation of the trend, where more and more of human perception will be mediated by technology, to benefit from advantages like accuracy, long-term memory/storage for data collection and statistics.
I guess augmented/virtual reality is part of it, where we can now perceive and experience a fully technology-mediated world, both real and imaginary. I picture sci-fi possibilities, like being able to "zoom in" to see things at magnifications beyond human biological limits - all controlled by thoughts.
They're multi-layer stacks of CCD observations filtered by frequency range, with false colors then assigned, not three RGB layers. IIRC, typically there's 7 - 11 ranges, depending on the instrument.
So one of the layers included in the depiction might be infrared, then false-colored into something the eye can handle.
Source: I worked in remote sensing.
One of my frustrations with ML is that almost no one seems to be training on HDR data. It's all 8-bit RGB. There is a big opportunity here to train GANs on raw sensor data.
But that raises the question of which file formats to focus on. Since you have lots of image processing experience, I'm very interested in what the data formats were like at your job. Was it proprietary formats? If so, were there any advantages over e.g. the RAW file format? Is RAW sufficient for standardizing "everything necessary to turn photon readings into a photograph that appeals to the human eye"? E.g. if I gather thousands of RAW photos, do I need any other information not present in the RAW file?
Mainly I am interested in any insights you gained on the job: you might know of some unique advantage that certain data formats have over others. I'm extremely interested in training ML models to synthesize what comes into a camera (photon readings), as opposed to what the world looks like to the human eye (RGB values).
(I apologize for not phrasing my questions more carefully. It's late. The essence is that right now, most ML training operate in terms of RGB, yet there seems to be many advantages to operating in "photon space". The trouble is, it's effortless to gather millions of RGB photos -- but it seems very difficult to gather a million RAW photos. I'm hoping someone knows of a massive trove of "some kind of raw data that can be turned into RGB values" – maybe that's RAW, maybe it's something else. Whatever it is, once I have it, I can feed it through an ML model.)
A library like libraw reads most RAW files and can give you an appropiately (for the sensor used) debayered RGB image. These images aren't in a defined color space; they're specific to every camera/sensor.
In the moving pictures space OpenEXR is a common format for "arbitrary" images, since it allows any number of channels each with different formats (e.g. you can have a file with 27 channels per pixel, some of which are float and some of which are integer).
[1] E.g. 12 bit log encoding gives even resolution across more of the dynamic range; a linear encoding gives half of each range to the brighter stop.
[2] Sensors that don't have PGAs simply have a fixed sensitivity. If the camera offers an ISO setting, it is emulated in the digital domain by pulling the image.
Some sensors have global shutter instead, which makes it even more like DRAM: Conceptually each pixel has a capacitor and a photodiode, and a transistor connecting the two, all of these transistors are connected in parallel and form the "global shutter signal".
Some sensors have faster ADCs and use one ADC for a bunch of columns, which has been claimed as the source of column-banding in some sensors (unclear if correct).
[1] Sensors that are intended for both photo and video typically support things like pixel binning, where the sensor itself averages e.g. 2x2 blocks of pixels internally. A lower quality alternative is line skipping, where the sensor is told to only read out every nth line, thus reducing resolution considerably. The higher quality alternative is "full-sensor readout", i.e. the camera reads all pixels and downsamples the image to the video resolution. I believe some (announced?) sensors can do this in the sensor itself now.
Note: A lot of this is "somewhat informed speculation" on my part, because image sensor manufacturers tend to be very secretive of their sensor's details.
12- or 16-bit greyscale is common, with multiple channels for hyperspectral data. But since it can also store 32-bit data, floats, and complex numbers as well as rationals, it can be used as a container for pretty much anything you want to throw at it.
Regarding RAW. It's often used with colour CCDs with various types of CFA, like Bayer or CMY. It might have a higher bit depth, but it's still basically RGB but spatially separated and stored as greyscale. You could maybe feed it directly to your ML, but it's more common to filter it to create RGB data retaining the bit depth but at the expense of low pass filtering to remove aliasing artefacts. Or keep the separation to keep the high frequency components; really depends upon what information you want to extract.
I used to work on on of the larger imaging tools, supporting over 150 scientific file formats. The vast majority, over 80%, were using TIFF as a container. The next most popular was HDF5. FITS was not something I saw commonly, and we were getting images from researchers all over the world.
All I ask is that you share your findings with me and don’t redistribute the image data set without confirming with me.
We have a server with a 24TB NAS. If you send me your ~/.ssh/id_rsa.pub key, I can add you to it and you can upload straight to the NAS. Would that work?
I have an event to go to today, but if you hop into our ML discord server (https://github.com/shawwn/tpunicorn#ml-community) and DM me (I'm shawn), we can figure out the details. Or email me (shawnpresser@gmail.com).
Thanks so much! This is exactly what I had in mind when I was writing my comment. I think it would be straightforward to extend StyleGAN to synthesize full spectrum data. And the results would look pretty darn cool.
Even if you get millions of RAW photos they will come from different sensors with different properties (noise, sensitivity, mosaic, orientation...) and you need to to normalize them all. sRGB is most common normalization target for which all prosumer cameras are optimized for.
The format is not really important here as it is usually array of pixels. You need more bits and resolution? Fine, but you also get more useless noise.
Sometimes a CCD bit was stuck, so we would process around that, etc.
The terms "HDR" and "photo" are not commonly used.
The Perkin-Elmer military system has fortran source, so all the file formats are documented there (if you can get access), as well as a minimal Scottish university system.
That’s one and a half billion astronomical units. The star in question is twice the sun’s mass, and, I guess, not too dissimilar in size. If so, if you had a camera that could take a 1,000,000,000 by 1,000,000,000 pixel photo of our sun from the earth and pointed it at this star, that star would fill a single pixel (Likely a few more more due to imaging imperfections, but those would mean the light gets smeared out)
It typically takes multiple ‘photos’ to even get a single of those CD observations in a frequency range (https://en.wikipedia.org/wiki/S2_(star)#S0–102: “At one-sixteenth the brightness of S0–2, S0–102 was not initially recognized because it required many more years of observations to distinguish it from its local infrared background”
It’s not as though conventional cameras operate on the exact same principles as our eyes, either. And that’s without even getting into the amount of “computational photography” performed on phones.
No. Very likely not.
The photos may be of wavelengths of light that the human eye can't see, so scientists shift (and stretch/shrink) those into the visible spectrum.
That means the photos could contain something you can't see with the naked eye.
If you were to teleport there, you would (probably) see something, but the colors likely wouldn't be like in the photo.
This video has a pretty good explanation of what's going on: https://www.youtube.com/watch?v=WSG0MnmUsEY
The Orion Nebula is a good target for grokking the difference. A long exposure photo with a DLSR (which will be blurry if you don’t have tracking) you’ll see lots of color. Through a pair of binoculars, you won’t although you will see the nebula.