I typically have to sleep with the TV on. I put on some boring documentary with a droning narrator. Otherwise my thoughts just keep me awake until I finally fall asleep from exhaustion.
The echo of our internal thoughts darting around as we approach sleep is quite normal. In the absence of TV background noise, the trick would be to not focus or chase or indulge the thoughts but "allow" them to simmer without your involvement, or allow them to pass by without looking at each one in depth; be indifferent from a distance. That's a technique not difficult to master, but is basically how to still your mind and go to sleep.
I decided I couldn't live that way, so I worked on breathing techniques like this. After a couple of weeks, I could finally focus on breathing long enough to fall asleep.
I still have nights where it just doesn't work, but it's made a big difference in my life.
I hope this post doesn't sound patronizing as that is not my intention at all. I can definitely relate to the frustration you've described, but for other issues. Based on my own anecdotal experience, it's quite unlikely that you've exhausted all the possible solutions. At the very least, you don't have much to lose in trying one new thing.
I found a surprising source of motivation in books about the Eastern Front of WWII. I figure that if Soviet soldiers managed to survive in those circumstances, I can stay calm and collected in much more favorable conditions.