Without severely paranoid steps being taken, if someone has your physical device they are going to be be able to gain access to the files on that device. This isn't hard math to do.