Multimedia is tricky but, from the way you describe it, you've come across the more general problem that writing concurrent software in general, and parallel software in particular, can be a real sod.

What you're likely to be experiencing are bugs due to race conditions. These are notoriously hard to debug.

It's one of the reasons why there's so much fuss around languages like Rust and Go. They have builtin features that try to make these types of issue less prevalent. I appreciate this doesn't help you directly but should make you feel a bit better; there are whole engineering teams working to avoid the types of issues that you're experiencing.

In terms of how to help you move forward, that's quite hard in this format. Sorry.