synapticpaint

The camera motions do not need to be the same. Gaussian splatting reconstructs the scene in 3d, and you can then render the scene from arbitrary angles, so they just gave it a random camera motion to show you the 3dness of it.

This technology is already close to making animation. Check out some of my experiments with text to video here:

https://www.youtube.com/watch?v=CgKNTAjQpkk

https://youtu.be/X0AhqMhEe-c

Pretty close. Not the same kind of animation, but here are some of my experiments with text to video for narrative content:

https://www.youtube.com/watch?v=CgKNTAjQpkk

https://youtu.be/X0AhqMhEe-c

You'd have to convert to vector, or tweak your model architecture to work with vector format.

"The flickering comes from the fundamental nature of the de-noising mechanism involved in the diffusion model." -- agreed

"Keeping the same seed wouldn't be helpful if you want the image to move." -- No, I'm using the same seed (and prompt). The image moves because ControlNet opens up another channel of input, in this case the pose data.

Are you using canny mode? One of the other modes (HED, segmentation, or depth) may give you more consistency. Lmk how it goes if you try this.

Not really, but I think there are other things you can do to reduce flickering that I'm looking into.

Yes, it's asking SD for an image with some set of characteristics, and SD has some notion of what is plausible from what it saw during training.

No, ControlNet wasn't made to solve temporal consistency, it was made to add more control (hence the name) to image models. I am using it in a way that the authors may not have thought of, because the paper doesn't mention video editing.

So, the video was generated by applying ControlNet to the input video frame by frame. Every inference setting is the same for every frame -- seed, prompt, CFG, steps, and sampler. The only thing that changes frame to frame is that the pose changes slightly. So actually, if SD was well behaved, you would expect the difference between adjacent frames to be small, because the change in the input is small. But SD is somewhat schizophrenic so you get this amount of flicker from even small changes in input.

I also had to specify what the outfit should be (I got a lot more discrepancies when I didn't do this from the outfit changing frame to frame). You can see that the outfit changes color in the second version, I bet you can get that to be even more consistent if you specify the color in the prompt too.

If you create a dreambooth model of a character, you can probably also get consistency of the face that way. In this case I didn't need to do this because I didn't care who I got, I just asked for an "average woman".

Correct, it's generated by SD.

Here is a preliminary test for video editing using ControlNet I made: https://www.youtube.com/watch?v=u52MOA4YaGk

As you can see, there is still quite a bit of flicker, I'm working to reduce that. But the consistency is much better compared to, say, img2img.

I'm hoping to ship a prototype this week.

Proof of concept for video to video editing. An input video is transformed to an output that matches the pose but with an AI generated character via a prompt. There is some flicker, but the result is much more consistent than existing methods (e.g., image2image).

I wouldn't say that "the opportunity all seems to be on the front end". Specifically for stable diffusion, there are a lot of different ways to use the model. I think we're just starting to scratch the surface of what SD can do, so there is some value in tinkering with different ways to use and apply the model.

Example 1: have a look through here: http://synapticpaint.com/dreambooth/browse/ for some examples of dreambooth models people have created

Example 2: you can merge different dreambooth models together to varying degrees of success (the idea being, you train model A on subject A, model B on subject B, and now you want to generate pictures of A and B together). My understanding is that this doesn't work too well at the moment, but it's possible that a different interpolation algorithm can yield better results.

I do agree with the general sentiment that you wouldn't necessarily be training your own models or creating your own architecture, just want to provide the perspective that understanding the AI side is valuable because it can lead to different capabilities and products.

Hi, I run http://synapticpaint.com/ and using AI image generation for graphic novels/comics is one of the directions I'm exploring. If you're interested in collaborating to make your graphic novel a reality (I'll provide the tooling in return for product feedback), please email me at the email address in my profile! (I poked around on your site but couldn't find an email address.) Thanks!

On AWS, a g5 instance costs $1/hr. I can generate roughly 10 images per minute (should be able to get this down with some optimization), so 600 images per hour, so the cost per image is 1/6 of a cent, before adding overhead (idle time, start up/shut down).

Source: I run a similar project: https://synapticpaint.com/

I also offer dreambooth model training for around $2-$4 / model as well as inference on custom dreambooth models. Inference on custom models is where things get a little tricky because if users are using different models and you're loading up new models all the time just to generate 6 images, then that quickly becomes the majority of the work load, drastically pushing up the inference cost. I haven't solved this problem yet. If you have any great ideas, feel free to email me (email in profile)!

"Finally, a vital question is, how will this affect today’s working artists? Here the answer is not so optimistic."

I have a different take on this. I think this technology will allow more people, not less, to make money as a living (so, professionally) in a visual arts related industry. So I'm broadening the field to include not just "artists" but "commercial art" as well (designers, commercial illustrators, video/film post-production, etc.).

The reason is that it changes and lowers the bar to entry for these fields, automates away a lot of the labor intensive work, thereby lowering the cost of production.

Whenever something becomes cheaper (in this case, labor for art), its consumption increases. So in the future, because producing commercial art is so much cheaper, it will be consumed a lot more.

At the same time, we're not at the point where we can actually remove humans entirely from the process. AI generated art is a different process and requires a different skillset, but it still requires skill and learning to do well.

The analogy would be something like a word processor reducing the number of secretaries needed in the workforce, but increasing the number of office workers. People no longer need someone to take notes / dictation, but all kinds of new workflows emerged on top of the technology, and almost all office workers need to know how to use something like a word processor.

Therefore, the opportunity here to do is to build tooling that make it easier and more accessible for more people to work with AI image generation.

Disclaimer: I'm doing exactly that (building tooling to make content generation easier and more accessible) with https://synapticpaint.com/

Cheaper services are here now: https://synapticpaint.com/ I charge $2-$4 per model training.

I don't think that the profile pic/avatar space is going to be very profitable. As you can see from this post, there is quite a bit of competition already. Because this is not that hard to set up, there is nothing stopping someone from charging the cost of production for this, which is essentially what I'm doing (I added in a small amount of overhead as a buffer for non-GPU compute and in case I messed up my math, math is hard). That's why I'm focused on building for other use cases instead (making content creation more accessible).

I don't think that AI image generation in its current form replaces human creativity. You still need a human to guide the process, compose the image, judge the output, etc. What's being automated is the manual, traditional process of painting and drawing.

I see this technology as more analogous to hand writing books -> printing. I predict that more people, not less, will be involved in creative industries related to visual arts (design, film, illustration, animation, etc.) as a result of this technology becoming more accessible.

That's true. I just wanted to provide some context for OP and others reading since they mentioned an interest in AI generation, and I think not everyone knows that dreambooth + stable diffusion has this kind of capability.

It sounds like your app turns photos into a watercolor style image? You can definitely do this faster with Stable Diffusion (~6s per image before optimization).

Here's how I would approach it: train a dreambooth model on watercolor style images, then run image-to-image using that model.

For examples of what dreambooth models can do see: https://synapticpaint.com/dreambooth/info/ (sample images here generated using a "modern disney" style model).

If you need help getting this set up feel free to email me! This stuff is probably not harder than getting gimp to run in a container.

Agree that dreambooth is the future. I'm building an app that lets users with no technical background train their own dreambooth models for $2-$4: https://synapticpaint.com/dreambooth/info/ They can also share their trained models for others to use.

Here are some potential use cases:

- for fun (giving yourself a makeover, inserting yourself into famous movies)

- cheaper way to get studio photos (wedding photos, professional headshots for actors/models)

- easy way to create marketing assets. Like if you own an etsy store and don't want to engage a marketing studio, instead just create a dreambooth model of your necklace or whatever and create high quality product photos

There are probably a bunch of other use cases. I think making this easy (no figuring out how to do a git pull or rent a gpu) plus the community sharing aspects will make this technology a lot more accessible to artists and general users, and then the users will be doing all kinds of cool things with it organically.

I'm building an app that lets users with no technical background train their own dreambooth models for $2-$4: https://synapticpaint.com/dreambooth/info/ They can also share their trained models for others to use. I think making this easy (no figuring out how to do a git pull or rent a gpu) plus the community sharing aspects will make this technology a lot more accessible to artists and general users.