The problem is exacerbated by TLS encryption. It requires six packets, or three round trip times, to establish a TLS over TCP connection. That is hundreds of milliseconds before a single byte of payload can be transferred.
The QUIC protocol combines the three way handshake and the three way key exchange to half the time I takes to set up a connection and provides a fast start (0rtt) for resuming a previously configured connection.
Additionally TCP connections do not survive a changed IP address like when changing from wifi to mobile data.
The stream model of TCP with head of the line blocking is conceptually easy but ultimately the wrong model for transmitting blobs of data. Rejecting a valid packet because the order got shuffled is wasteful, when we could just as easily reassemble the packets on the receiver side (even outside the sliding window).
There are a lot of times when the stream model is a good choice (dynamically generated data) but I would argue that sending sized blobs (like images) is a larger volume of traffic in terms of bytes transferred.
But the OP is right, writing custom UDP protocol is fraught with peril. DDoS amplification is just one way to screw up.