It generally does cache built crates for future builds on the same system, so it is usually only the initial build on a given system and the build after dependencies are updated that requires rebuilding serde_derive. I honestly don't know why they thought it was worth doing.