After messing around with this, is there any reason to use any of the other sizes besides xs? With xs it seems like every device is covered, from the phone up to the large desktop.
Then using another size as xs let's you define the grid for bigger sreens while on small screens the content is just stacked vertically.
90% of my designs use col-sm-*, and then if stuff needs fiddling at md or lg I do it there. For xs I just usually let it default to stacked 100% cols.