You can use SDFs (signed distance functions) to express your objects in a mathematical form. It has its pros and cons. One of the bigger cons is that most tools use the polygonal approach, so you'd have to build a lot from scratch.

Lots of SDF examples in ShaderToy[2], and many introductory videos on YouTube (I specially recommend Inigo Quilez's channel[2] --co-creator of [1]--, and The Art of Code [3])

[1] https://www.shadertoy.com/

[2] https://www.youtube.com/channel/UCdmAhiG8HQDlz8uyekw4ENw

[3] https://youtu.be/PGtv-dBi2wE

It's completely doable, but most modeling software is built to produce triagonal meshes, and most graphics cards are optimized to render triagonal meshes.