There is both. Some phones, e.g. OnePlus, have analog audio output on USB-C and can use a passive adapter. If the phone lacks analog output, e.g. Samsung, it needs an active adapter with a USB controller and DAC like you mentioned.
My current phone is a pixel 6. I got a USB-C adapter with a DAC built into it, and it actually sounds pretty good. The volume levels are reasonable, and there's no noticable noise other than perhaps an unnoticeable hiss.
Years ago, I had a desktop case front IO panel with a really bad ground loop. I ended up cutting the molded over plastic open and cutting the PCB traces between the audio ports and the USB ports, and the problem went away.