I believe you are incorrect about the defections as well. One source supporting this notion is Adrian Goldsworthy's "The Fall of Carthage: The Punic Wars 265–146 BC", which, by way of Wikipedia, claims:
https://en.wikipedia.org/wiki/Second_Punic_War#Hannibal_cros...
However, the majority of Rome's allies remained loyal, including many in southern Italy.