You show admirable qualities that good programmers have. For example -- boldness to take on a challenging project, tenacity to pursue a rewrite, and drive to pursue improvements such as parallelism. After all, if you only pursue easy projects then you are unlikely to improve your skills!

There are definitely challenges involved with multimedia apps. They demand good performance, and they require you to rely on third-party libraries and drivers.

I assume you are decent at debugging to already have gotten as far as you have. So, I would continue to take guesses at what the problem might be then test whether you are correct. Based on your description, perhaps the CPU cannot always keep up with the framerate. You could test this (for example) by seeing if rendering a lower-resolution output would alleviate some of the bugs.

If none of your guesses seem to work, pare your code down to make it as simple as possible (e.g. a basic webcam app without overlays). Once you get it reliably working, then slowly re-add your features. Worst case, you can always post-process the video after streaming it to disk, or have a tool such as FFmpeg add the overlay.

Good luck!