I understand that grandparents' suggestion that the audio is actually "played back" during the process appears to be a misunderstanding, hence "Yes, but..."

My point was that the "natural speed" of video (i.e. the frame rate - clear slow motion requires more frames, while fast blurred motion will generally have fewer) does actually impact how much data must be processed. It's true of audio too (sample rate) and downsampling is a common optimization technique.