I’m also ignorant of this, and to add a question on top of yours: is it worth worrying about robots.txt for personal portfolio websites built from scratch?