For using GPU's, there's CUDA, or offloading with OpenMP or OpenACC.
Some supercomputer vendors have specialized software running, but others also support open source distributed computing frameworks like Apache Spark.
If you are wondering why we call them supercomputers, it’s because it’s not as easy to just buy thousands of computers. Even easy things like shipping, installation and power consumption becomes a problem on its own. So rather than buying thousands of machines it pays off to buy them through a supercomputer vendor that will optimize space, power consumption, install cluster management software, etc.
So if you want to get started, I would just get a small n-node cluster from any cloud provider and get better at writing distributed code.
I could be completely 100% wrong.