Did you try purging the dkms version and building the one in the post?

I also spent an inordinate amount of time messing with it, but in the end the actual steps that worked were simple. I couldn't get it to work with my flatpak version, and Debian stable is not fun trying to compile OBS (need more recent version of cmake).

But Ubuntu should be more straightforward... Did you get the option in the Tool menu yet? The other thing I got reading through all the GitHub issues was the need to sometimes change the YUV setting in that dialogue to YU2 (from memory).

Hope you get it cracked without much grief, and I can share what I can remember fwiw.