See also:
A Pixel Is Not a Little Square (1995) [pdf] – http://alvyray.com/Memos/CG/Microsoft/6_pixel.pdf
A Pixel Is Not a Little Square (1995) [pdf] – http://alvyray.com/Memos/CG/Microsoft/6_pixel.pdf
They are just bits in a computer. But there is a correct way of to interpret them in a particular context. For example 32 bits can be meaningless - or it can have an interpretation as a twos complement integer which is well defined.
If you are looking to understand how an operating system will display images, or how graphics drivers work, or how photoshop will edit them, or what digital cameras produce, then it’s the point sample definition.
And for pixel art, the intent is usually far from points on a smooth color territory.
Multiple interpretations matter within different contexts inside the computer context.
They use a physical process to attempt to determine light at a single point. That’s their model they try to approximate.
> And for pixel art, the intent is usually far from points on a smooth color territory.
And notice that to display pixel art you need to tell it to interpret the image data differently.
Also it has a vastly different appearance on a CRT where it was designed which is less like a rectangle.
According to who?
A naked camera sensor with lens sure doesn't do that, it collects squares of light, usually in a mosaic of different colors. Any point approximation would have to be in software.
> Any point approximation would have to be in software.
Circuits can process signals too.
And outputs what? Just because the input is an area does not mean the output is an area.
What it if it outputs the peak of the distribution across the area?
> that are pretty square.
If we look at a camera sensor and do not see a uniform grid of packed area elements would that convince you?
I notice you haven’t shared any criticism of the point model - widely understood by the field.
> What it if it outputs the peak of the distribution across the area?
It outputs a voltage proportional to the (filtered) photon count across the entire area.
> If we look at a camera sensor and do not see a uniform grid of packed area elements would that convince you?
Non-uniformity won't convince me points are a better fit, but if the median camera doesn't use a grid I'll be interested in what you have to show.
> I notice you haven’t shared any criticism of the point model - widely understood by the field.
This whole comment line is a criticism of the input being modeled as points, and my criticism of the output is implied by my pixel art comment above (because point-like upscaling causes a giant blur) and also exists in other comments like this one: https://news.ycombinator.com/item?id=43777957
This is not true. And it’s even debunked in the original article.
No, it's not. That article does not mention digital cameras anywhere. It briefly says that scanners give a gaussian, and I don't want to do enough research to see how accurate that is, but that's the only input device that gets detailed.
It also gives the impression that computer rendering uses boxes, when usually it's the opposite and rendering uses points.
Truth is, a pixel is both a sample and a transducer. And in transduction, a pixel is both an integrator and an emitter.
> If you are looking to understand how your operating system will display images, or how your graphics drivers work, or how photoshop will edit them, or what digital cameras aim to produce, then it’s the point sample definition.
They also look more like a square when I back away. And the mismatch of the square model doesn't mean the point model is good.
So your intuition for why squares makes sense is wrong, but you’re still holding on to it.
> doesn't mean the point model is good.
What does show it’s a good model is all the theory of image processing and the implementation of this theory in camera display systems.
You’re welcome to propose an alternative theory, and if that is consistent, try to get manufacturers to adopt it.
I said subpixels are rectangles. Because they are.
If the point model was all you need, then objects small enough to slip between points would be invisible. Which is not the case.
In particular a shot of the night sky would look pure black.
So if being wrong means we should abandon the model, then we can't use squares or points.
https://upload.wikimedia.org/wikipedia/commons/4/4d/Pixel_ge...
I look forward to your paper about a superior digital image representation.
Or are you arguing that the slightly rounded corners on the rectangles make a significant difference in how the filtering math works out? It doesn't. On a scale between "gaussian" and "perfect rectangles", the filtering for this shape is 95% toward the latter.
Even some VGA modes have non-square pixels, and these were used by many games and... Windows 9x splash screen.
In signal processing you have a finite number of samples of an infinitely precise contiguous signal, but in image processing you have a discrete representation mapped to a discrete output. It's contiguous only when you choose to model it that way. Discrete → contiguous → discrete conversion is a useful tool in some cases, but it's not the whole story.
There are images designed for very specific hardware, like sprites for CRT monitors, or font glyphs rendered for LCD subpixels. More generally, nearly all bitmap graphics assumes that pixel alignment is meaningful (and that has been true even in the CRT era before the pixel grid could be aligned with the display's subpixels). Boxes and line widths, especially in GUIs, tend to be designed for integer multiples of pixels. Fonts have/had hinting for aligning to the pixel grid.
Lack of grid alignment, an equivalent of a phase shift that wouldn't matter in pure signal processing, is visually quite noticeable at resolutions where the hardware pixels are little squares to the naked eye.
Since the pixels never were a waveform, never were sampled from such signal (even light in camera sensors isn't sampled along these axis), and don't get displayed as a 2D waveform, the pixels-as-points model from the article at the top of this thread is just an arbitrary abstract model, but it's not an accurate representation of what pixels are.
https://en.wikipedia.org/wiki/Subpixel_rendering
Edit: For the record, I'm on Win 10 with a 1440p monitor and disabling font smoothing makes a very noticeable difference.
People are acting like this is some issue that no longer exists, and you don't have to be concerned with subpixel rendering anymore. That's not true, and highlights a bias that's very prevalent here on HN. Just because I have a fancy retina display doesn't mean the average user does. If you pretend like subpixel rendering is no longer a concern, you can run into situations where fonts look great on your end, but an ugly jagged mess for your average user.
And you can tell who the Apple users are because they believe all this went away years ago.
The reason is mostly that it is too hard to make it work under transformations and compositing, while higher resolution screens are a better solution for anyone who cares enough.
This is a little misleading, as the new versions of Edge and Windows Terminal do use subpixel antialiasing.
What Microsoft did was remove the feature on a system level, and leave implementation up to individual apps.
Laptop screen is 4k with 200% scaling.
Seriously the font rendering in certain areas (i.e. windows notification panel) is actually dogshit. If I turn off the 200% scaling on the laptop screen then reboot it looks correct again.
Anti-aliasing can be and is done on squares routinely. It’s called ‘greyscale antialiasing’ to differentiate from LCD subpixel antialiasing, but the name is confusing since it works and is most often used on colors.
The problem Alvy-Ray is talking about is far more subtle. You can do anti-aliasing with little squares, but the result isn’t 100% correct and is not the best result possible no matter how many samples you take. What he’s really referring to is what signal processing people call a box filter, versus something better like a sinc or Gaussian or Mitchell filter.
Regarding your edit, on a high DPI display there’s very little practical difference bewteen LCD subpixel antialiasing and ‘greyscale’ (color) antialiasing. You don’t need LCD subpixels to get effective antialiasing, and you can get mostly effective antialiasing with square shaped pixels.
And I guess I should have explicitly stated that I'm not talking about high-DPI displays, subpixel rendering obviously doesn't do much good there!
My point is simply this, if you don't treat pixel like discrete little boxes that display a single color, you can use subpixels to effectively increase the resolution on low-DPI monitors. Yes, you can use greyscale antialiasing instead, you will even get better performance, but the visual quality will suffer on your standard desktop PC monitor.
If you don't treat a logical pixel like a discrete little square, you can take advantage of subpixel geometry to effectively increase the resolution. It's not the same as antialiasing, even though it can be used for antialiasing.
Arguably, instead of treating pixels as squares, you're treating them as three times as many rectangles, but that should still contradict the mental model of pixels as little squares.
No, it doesn’t. You seem to be missing this point. Using subpixel rectangles does not in any way correct the misconception, it perpetuates it.
Having worked on both game engines and CG films, I think that’s misleading. Point sample is kind of an overloaded term in practice, but I still don’t think your statement is accurate. Many games, especially modern games are modeling pixels with area; explicitly integrating over regions for both visibility and shading calculations. In fact I would say games are generally treating pixels as squares not point samples. That’s what DirectX and Vulkan and OpenGL do. That’s what people typically do with ray tracing APIs in games as well. Even a point sample can still have an associated area, and games always display pixels that have area. The fact that you can’t display a point sample without using area should be reason enough to avoid describing pixels that way.
I'm not sure how raytracing as done by video games can be construed a more area-based than point sample based. It seems to me like ray-hit testing is done in a point-sample manner, and games aren't doing multiple rays per pixel. (They're usually doing less than one ray per pixel, and averaging over several frames that sampled from different locations within a square pixel, then running it all through a blurring denoiser filter and upscaling, but the result is so bad for anything other than a static scene that I don't think it should be used to support any argument.)
Ignoring the denoiser, games are quite commonly using Box Filter for pixels. That’s the square in square pixels. The point sampling is in service of an integral whose shape is a square, and that’s the problem Alvy Ray is talking about.
That point sampling is distinctly different from the “point sample” that represents the pixel value itself, so let’s not conflate them. The averaging over an area that is square shaped, whether it’s multiple samples or over time, is the reason that the pixel shape is summarized as square, not as a point.
A 100x100 thumbnail that was reduced from a 1000x1000 image might have pixels which are derived from 100 samples of the original image (e.g. a simple average of a 10x10 pixel block). Or other possibilities.
As an abstraction, a pixel definitely doesn't represent a point sample, let alone an infinitely small one. (There could be some special context in which it does but not as a generality.)
And if a downsampling algorithm tries to approximate a point sample, it'll give you a massively increased chance of ugly moire patterns.
The audio equivalent is that you drop 3/4 of your samples and it reflects the higher frequencies down into the lower ones and hurts the quality. You need to do a low-pass filter first. And "point samples from a source where no frequencies exist above X, also you need to change X before doing certain operations" is very different and significantly more complicated than "point samples". Point samples are one leaky abstraction among many leaky abstractions, not the truth. Especially when an image has a hard edge with frequencies approaching infinity.
Someone with just the small version wouldn't know if it's supposed to look like that, but we're not asking them.
Unless they can infer it's a picture of normal objects, in which case they can declare it's moire and incorrect.