I would use Pytorch as your starting point. Its metal backend is pretty quick on Apple Silicon, and it's the most widely used library for everyone from hackers to foundation model builders.