And the things that scare me are just the few things that we can easily reason about already, not even the things that we do not yet know being possible with it. Surveillance states turned up to eleven, psychological violence up to eleven, interrogation, torture, ethically challenged tech bros going full-Zck on people's minds. Could maybe one mind invade another brain and use it to enhance its capabilities (suppressing the mind that resided there so far)? A few brains interacting so closely that they create an a lot more intelligent thing - outpacing everyone who is not willing to subject themself to that (what would that do to society?)? What happens when the interface fails biologically (for invasive methods of interfacing), like developing inflammation, bleedings, scar tissue? Technically failures like the interface device dying, broken or unreliable connections or even just batteries running low? Before you dismiss the latter, imagine how you feel when you involuntarily cannot use your phone and scale that up by a lot - as you would not be losing a tool but a part of yourself.
Enough anxiety about the future for now - have a nice day instead ;)