Gentoo recommends -jN+1 where N is the number of physical and virtual cores.
The kernel can multi-task processes, but each process still gets exclusive use of the CPU when it runs. So if it doesn't need an adder, that adder sits idle.
With hyperthreading you can run two processes at once and the CPU merges them at the instruction level making maximum use of the components on the CPU.