The description of the scheduler[1] explains why:

“The RSP executes both graphics and audio processes. A graphics process sometimes extends over one frame, but an audio process must be provided in each frame to prevent inappropriate pausing. Therefore, in each frame, if the graphics task is executing, the Scheduler suspends it and saves the task state. Then it starts the audio task executing and prepares the RSP for the restart of the graphics task execution when the audio task finishes.”

[1] http://n64.icequake.net/doc/n64intro/kantan/step2/1-7.html