Yes, but AFAIK, the Saturn V was not full of fuel. Given that, the fact that the module was on top of the rocket didn't change the risk profile much. The fire could have happened, even if the command module was in a garage.
Going to space involves strapping yourself into a tiny capsule on top of something that's filled with explosives. There's nothing safe about it.
As a sidenote, it wasn't a Saturn V.