Samsung is currently working on Linux for Galaxy which is exactly this.
Also, the processors in your phone are closer to desktop performance that you might think. If you put any weight in GeekBench scores, single core scores for the new A11 are about the same as the single core scores for the Ryzen 1800x.