In my experience only NVENC is supported by most video recording tools, and if you are on AMD, it's using the CPU, thus slowing everything to a crawl.

AMD should be using VA-API but it's not a very good system as it fails or breaks if you look at it the wrong way. I use a tool to stream my desktop to my TV (sunshine) which uses it, and every month VA-API fails with a new error so it has to resort to CPU encoding.