I've never used a Chromecast Audio, but their website explicitly mentions a password and a guest mode w/ 4-digit PIN to let other people play music on it. Sounds like yours isn't working as intended, could it be something else causing the pauses?
https://support.google.com/chromecast/answer/6279388?hl=en
EDIT: jordanthoms points out that "guest mode" is for letting people stream audio without giving them your wifi password. Once they're on the network everyone has access.