For small cluster you can just create service account for a user, create token for it and write it in the kubeconfig. Then assign role to this service account and that's about it.
The main issue with this approach is that you can't organize those "users" into a groups. But for a small number of users you can just create all rolebindings and be done with it.