Sitting down and not actively thinking is actually very beneficial and healthy. It will give your brain time to re-order and reset, sometimes even makes you come up with great ideas. Just letting your brain to its thing for a couple minutes is also part of meditation practice for this reason. So I try to use these moments to just check in with myself and let any feelings come to life without trying to judge or think about anything particular.
I am very compelled to start any new task which “feels productive” even in the shortest amount of breaks (also during calls when other people talk etc), it is really tiring. If you also have this strong urge to do so maybe you should check if you have ADHD.