It will always have some impact because you're thrashing the caches.

A couple more things to try:

Reduce swappiness. Linux tries to swap out idle pages to free memory for block cache, even when there's only light memory pressure. If you have a dozen processes working hard and your UI is untouched for several minutes, it can get swapped out and lag.

  sudo sysctl vm.swappiness=1

Set the CPU affinity for your build processes to exclude CPU 0, ensuring one is always free for UI:

  taskset 0xFFFFFFFE nice your_build_command_here

Neither is perfect but they can help.