SD cards have the very great advantage that you can have multiple ones with different OSes or versions and swap them in seconds.
It's fine for a normal user doing a few builds a day. Most files never get written to the physical device, being in tempfs or disk cache and deleted before being flushed to disk.
Because of wear-levelling it's not how many times the busiest file is rewritten, but how many GB are written in total.
512GB cards currently cost $32, $35, or $40 depending on whether you want 100, 150, or 200 MB/s read speeds.
A GCC src tree is 2.5GB, the build tree is 11GB. It's going to take ~40 clean builds to put one write cycle on the whole card, so you'll probably get 100,000 clean builds off a card before you wear it out.
That's around 30 builds per cent.
It's fine if you're using tmpfs, but I would expect written files on anything else to make it to disk even if they also spend some time in the write buffer. Especially on a build box where memory pressure is higher.