Pigment Mixing into Digital Painting

It’s ancient history now but in 2012, the team behind the Paper drawing app for iPad shared interesting details about their color mixing: https://www.fastcompany.com/3002676/magical-tech-behind-pape...

In short, they first implemented realistic pigment mixing behavior. But in user testing it turned out non-painters are not good at getting the color they want by mixing; you usually end up with brown. So they had to make an unrealistic mixing algorithm that felt more intuitive.

That's very interesting!

My first thought, looking at the webpage: "Huh, that's neat. I didn't know that painting software didn't even attempt to do color mixing beyond naive interpolation, though I guess it figures; the physics behind all the light stuff must be fairly gnarly, and there's a lot of information lost in RGB that probably can't be just reconstructed."

Scrolling down a bit: "Huh, there's some snippets for using it as a library. Wait, it does operations in RGB? What's going on here?"

Finally, clicking the paper link, I found the interesting bit: "We achieve this by establishing a latent color space, where RGB colors are represented as mixtures of primary pigments together with additive residuals. The latents can be manipulated with linear operations, leading to expected, plausible results."

That's very clever, and seems like a great use for modern machine learning techniques outside the fashionable realm of language models. It uses perceptual color spaces internally too, and physics based priors. All around very technically impressive and beautiful piece of work.

It rhymes with an idea that's been floating in my head for a bit - would generative image models, or image encoder models, work better if rather than rgb, we fed them with wavelength data, or at least a perceptually uniform color space? Seems it'd be closer to truth than arbitrarily using the wavelengths our cone cells happen to respond to (and roughly, at that).

This has been around for a few years now, but it is doomed by a non-commerical/restrictive license.

https://krita-artists.org/t/implementing-mixbox-mixing-for-k...

I remember that in Windows 3.0 time there was "Fractal Designer Painter" software, which had different engines to precisely emulate different types of painting materials: pastel, oil, water color, gouache. It imitaded mixing, stroke, pressure, etc. It looks incredible realistic in these times when 800x600x24bit screen was a luxury. And it was very slow on i486 computer.

As I learned several minutes ago now it is Corel Painter.

I wonder, is it still good and how it (both old and new algorithms, I allow that it was simplified down the road!) compares to this new library.

Unfortunately, authors doesn't compare themselves with Corel Painter.

I'd be curious to learn why Secret Weapons (the makers of this tool) didn't raise $100M in venture funding two or three years ago, before the start of the AI video boom.

They were first on the scene with EbSynth [1], which was the first widely used AI-powered video-to-video style transfer and keyframe interpolation system. EbSynth gave rise to comedians like Joel Haver [2] and lots of others like him.

Several other companies [3] used EbSynth to base their technology on and wound up raising large seed rounds. Major musicians used EbSynth either directly or through these other companies to make music videos [4,5].

Secret Weapons was in a great position to conquer AI video and AI creative tools, but they seemingly slept on it.

[1] https://ebsynth.com/

[2] https://www.youtube.com/watch?v=SY3y6zNTiLs&list=PLKtIcOP0Wv... (playlist with all of his animations)

[3] https://kaiber.ai/ and others

[4] Linkin Park https://www.youtube.com/watch?v=7NK_JOkuSVY

[5] Magdalena Bay https://www.youtube.com/watch?v=dXLCHvRsgRQ

Personally, I don't care about the library, and just want the pure math implementation for shaders and such. For that, they have the algorithm available here in GLSL, vec3 mixbox_lerp is the function you want:

https://github.com/scrtwpns/mixbox/blob/master/shaders/mixbo...

I don't have time to whip up a demo, but it seems like it's pretty well optimized and it looks really nice in the examples. Great work from these folks.

Previous discussions, from 3 years ago:

https://news.ycombinator.com/item?id=30116316 (71 points, 15 comments)

https://news.ycombinator.com/item?id=30033611 (60 points, 13 comments)

Has anyone tried Rebelle, the commercial painting app that uses this method?

Looks awesome, sadly it's All Rights Reserved + CC-BY-NC, so it won't be possible to implement it in Krita, GIMP or Blender natively.

This is great. I paint miniatures and once you get past the basics and start using glazes for filters/blends software isn't great for prototyping your paint schemes.

While this is really cool and should be an option in all painting programs, a part of me wonders if this complicates the learning process of painting?

After all, mixing paint colors is something you have to learn because it's not very intuitive.

Almost everywhere else in life, when you mix A and B, you expect a linear transition to happen. By adding salt to water, you are increasing its saltiness, it's not expected that at some point, the water becomes sour before it becomes salty again.

I use Rebelle (which uses this) for all of my digital painting. It’s excellent. They do need a new UI framework and they say they’re working on it.

As someone who teaches both oil painting and digital painting, I find this project fascinating. They have certainly identified two of the key difference between digital and real paint:

- RYB mixing (e.g. yellow painting plus blue paint = green). I tested it on a mix of alizarin crimson plus hansa yellow to produce a cold orange (e.g. blood orange).

- Relative behavior of specific pigments. For example, a Prussian blue is almost black when applied thickly, but very chromatic when applied as thinly over white. A cerulean blue is pretty much the same in those two states. I tested this on a Prussian and the app performed well.

The app seems to make the assumption that the artist will always want the paint to mix. In a glaze, a thin transparent layer of paint is applied over dry paint. Effectively the result is a filter (the result will always be darker). To a degree, this can be faked using paint applied in a layer the blend mode of which has been set to multiply. It might be nice to add a drying brush to the tool set which selectively dries the paint to prevent mixing. Maybe also a mode switch to selectively different modes of physical application such as glazing, scumbling and dry brushing. While we are at it, why not add a Tonk mode using which thick paint can be lifted off the surface without smudging.

When I paint digitally, I mix the incredible nuance of real paint... even its smell. When I paint with oils I miss the incredible flexibility of the digital process (blend modes, compositing, even the humble undo). Never the twain...

I've an oddball interest in exploring what science education content might look like if someday crafted with implausibly extreme expertise and effort. Given that first-tier physical-sciences graduate students not uncommonly have foundational misconceptions around color, and color is commonly taught in K-2, I was exploring how one might emphasize spectra when teaching color, in preK to whatever. With interactives like mousing over hyperspectral images, a light/image manipulation workbench, and apropos topic, physically realistic Paint (with fluorescence, controllable lighting, glitter, blackbody, fire, make your own pigment, collage). Now community around that oddball interest has been hard to come by, especially no longer being around MIT. So in support of a "find your people - build your village" 2025 New Year's theme, if anyone knows of potentially overlapping interests...

This is the feature that sold me on Rebelle 7 (which is hands down the best digital painting software I've used). Being able to translate oil painting mixing techniques onto a digital workflow was a game changer for me.

I was already impressed but at the bottom of the page they show they wanna make an addon to color blender fluids accurately. Paint me impressed. Paint me impressed but with pigment blended accurately

I think that the colors here (in this screen-world) are 99% mental. Like everything else here. Which is to say, it is not the configuration of the dots or their color that significate, it is the meanings and memories that they evoke. 1% of the game happens on the screen and the other 99% happens inside your head.

Which makes the whole thing a bit solipsistic when you think about it.

Cool, could see this being useful as a palette with a combination of the normal mode. I don’t see a clear way of controlling the amount of paint in the demo, if i want 3:1 red to white how would i do it?

Wow, this is really cool color mixing! It’s strange how come they haven’t been bought (yet) by a big company like Adobe.

Rebelle does this.

Here it is in 2021:

https://www.escapemotions.com/blog/rebelle-5-meet-color-pigm...

And here it is currently:

https://escapemotions.com/products/rebelle/manual/starting-p...

I have been reading the license, it is CC by-nc then it is not free software.

not an artist but this is so cool. Way better than the comparisons.

Doesn’t Krita do this?

That's great!

That seems like something that Adobe would want to borg.

"Colors in painting software do not act like real pigments. Why? Because there was no practical way to implement true pigment mixing into digital painting." - not sure that is accurate - digital tools were developed such that they colour mix not colour blend. I'm not sure it's so much as there was no framework for it, so much as if I want to blend 2 colours I'd mix them then use a blend mode on a layer. Traditionally most people working with pro imaging tools had a background in colour theory, I can't imagine working on imaging and not knowing blue and yellow is green, so if I want green on my mixed yellow and blue, I'd either draw on with green, or draw with a blend mode if I want the exact green. Doesn't make this any less cool, just saying'

Why isn't this more widely used? First, we have to understand that digital painting is a fundamentally different beast than traditional, in how you can use the tool, what you need to know, and how you learn it.

In traditional, you tend to have some rules that make it easy to get good colors. One method is to restrict your paints to a few primary colors and burnt sienna (or another low saturation brown). Then by mixing these colors you get the colors you want, in a way where these colors have a good relationship with the others. So a simple rule is for instance that you get your green by putting lemon yellow with some black and a bit of sienna to lower the saturation. The relationship is almost 'free' from the properties of the physical pigments if you follow such simple rules.

What's the deal about 'relationship'? Colors are not treated individually, they are perceived in context. In real life, lightning will unify a scene by changing colors. However, there is a semantic superposition that affects it. Grass is green in your mind, so if the color you see is a light blue/purple, the brain will adjust its understanding to match it. This is where many illusions come from (most famously the white and gold/blue and black dress meme come from).

This is a big deal to learn painting, but can be almost completely side-stepped by learning how to mix colors.

In digital however, artists got used to the fact that color is generally treated as HSV and adjusted their approach. So the most commonly taught approach to manage colors is to work in black and white, only dealing with value, which is the most essential component of color. Values are the primary issue because this is fundamentally what implies to our brain what are the normal vectors of light, and so carry almost all 'volume information'. Then, you can apply colors via some layers set to multiply, color and overlay mode. Note that it has some antecedent in the traditional world, called 'grisaille' technique.

So, you can color pick anywhere you want on the color wheel without mixing. The impact of color mixing is actually pretty small. The other technique is called gamut masking; you're only allowed to pick colors on a subset of the color wheel that is in a good HSV relationship with the chosen color scheme. The thing is, even though this is not perceptually accurate, this is good enough.

The main impact of color mixing is that if working with low opacity brushes, the colors will look 'muddy'. This problem is mostly side-stepped by avoiding it altogether with high-opacity brushes, and using other layer blending modes.

Because now most digital artists have learnt through these techniques, bringing techniques that work best coming from the traditional world is not very useful, to say the least. In my own practice, traditional and digital are indeed almost entirely separate activities, at every level. The fundamental, theoretical understanding of all aspects is the same, but it doesn't really matter very much; almost all other artists I ever discuss with do not have such an understanding, and build their practice from a more 'human'-centric foundation (technique and artistic concepts).

I'm of course interested in the space, and it is a bit similar to asking why would people would use bash or plain text; it's not necessarily that these are the best tools, but there is much more to it.

What is the core of the problem?

Is it that when you have two colors like RGB(0,0,255) and RGB(255,255,0) there is no easy formula to calculate what a mix of these two colors would look like in the real world?

And therefore one needs a lookup table? And since 16Mx16M is a large number, a complete lookup table is impractical?