Yes, for two reasons. The VIC-20 used 6522 VIAs as I/O chips, and these had a notorious bug in their shift registers requiring a bitbanged IEC serial I/O routine instead. This wasn't a big deal on a system with 5K of RAM, but it was a real problem on the 64. The 64 has 6526 CIAs for I/O, where faster IEC serial communications should have been possible, but the wiring got screwed up during board design and VIC-II DMA ("badlines") caused timing interruptions at inconvenient times, which demanded slowing it down even more. (A remnant of this is UI- and UI+ to speed up and slow down loading when used with the VIC-20, but that only works on the 64 if the screen is off.)

Commodore tried to solve this twice. The first time was for the Plus/4 series with TCBM drives which connected through the expansion port. The only drive of this type was the 1551, which was very fast, but only worked with the Plus/4 family. Commodore was going to design a TCBM interface for the 64 as well, but it wasn't really necessary as pretty much everyone had a fastloader by then. The 128 has a fixed serial bus and burst mode when combined with the 1571 disk drive, which is also very fast and doesn't require using the expansion port, but by then it was 1985 and the 8-bits were on their way out.

The 1571 is way better than the 1541, IME. It's faster with the 128, it's 100% compatible, it's more reliable and less prone to alignment problems, and it can also read MFM formats. But Bob worked on the 1541, so he loves it. :)