You could add the whole albums to the playlist, one by one, and then play the playlist w/ shuffle disabled. This last bit makes it a limited workaround, however.
On the other hand, Spotify API could allow to implement the custom solution to shuffle these, similar to https://spotifyshuffler.com/, if you have enough skills, time, and determination to do that.