But then there's QUIC, which is a better TCP built on top of UDP. It's new, but there's already some adoption.
> good luck getting end-to-end transit
This isn't a TCP/UDP problem, you need to go down a level. This is because of NATs, which exist because the IPv4 address space can't possibly fit all those networked devices people use these days. IPv6 solves that. Why is its adoption so slow despite widespread support? I have no idea.
Packets on the internet are transferred from host to host until they reach their destination. Your laptop on your local network is probably sending packets to your consumer router, which sends the packet to the ISP's router, and to other routers, until the packet reaches its destination. At each step of the way, routers can choose whether they wish to drop the packet or not. Due to various historically complicated reasons, many routers will drop packets they don't recognize.
To give some examples, a small transit provider may have some really old routers that don't recognize modern versions of TLS, and these older routers usually did packet analysis on the hardware, so changes to the protocol need to be reflected in a hardware upgrade, which the transit provider will not do unless absolutely necessary for cost reasons. The provider may then choose to drop the packet because they don't recognize what kind of packet it is.
Why should a middlebox care about what kind of packet is being sent? Many complicated reasons. CGNAT (Carrier-Grade NAT) is one of the iconic examples. Organizations do not want to learn IPv6, administer IPv6, or upgrade their hardware for IPv6, so they instead put layers of NAT in and block end-to-end connectivity on the internet. Another is that some netops will try to blanket drop packets that they don't recognize (think only allowing DNS queries to their specific DNS server, HTTP/TCP traffic, and maybe UDP traffic on a blessed VOIP port) to both defend from malicious users (instead of putting in the work to detect malicious traffic) and to better utilize the network they have and defer any network upgrades until absolutely necessary. Many middleboxes just drop non TCP/UDP traffic altogether, so protocols like SCTP which were designed for media streaming have a really hard time getting off the ground.
It's complicated. Combinations of organizational culture, low margins, and lack of consumer understanding of their networks creates situations where middleboxes are aggressive without reason upon their traffic. This phenomenon is called protocol ossification, or just ossification in networking speak.
What we're seeing today is that routing hardware is also parsing the session transport layer (and sometimes even more!). TLS 1.3 lies about its TLS version (pretends to be a TLS 1.2 client hello) because some big companies can't deal with arbitrary version numbers. To "fix" this problem in the future, the real version numbers in the TLS protocol are now pretty much randomized to ensure no new middleboxes look at them again.
Also take HTTP/2: most of the fixes HTTP/2 provides (basically forming a tunnel) are already embedded into smarter transport protocols like SCTP. Because of lack of middlebox support, using such an alternative transport wasn't even an option. Alternatives like SST also had no chance, leaving everything to the TCP-packets-within-TCP HTTP/2 approach because of compatibility.
It really doesn't care what sort of wire it comes to router and what sort of wire it leaves over. And then not really even what is inside, it supports up to 252 different protocols inside.
Now gathering and transferring information where to transfer is somewhat more tricky, but really IP is pair of addresses and some data.