These days, professional optical equipment, aka expensive lasers+supporting hardware, can do 10-Gbit over multiple kilometers through the air, so you're right that optical transmission through the air should be able to support higher data rates.

The problem with Irda is that it's old. Technology has significantly advanced since the 90's, when Irda was popular on cellphones, so a modern implementation could do better data rates even accounting for the significant interference from the environment. We barely had wifi back then, and now it'll do a few hundred megabytes per second without breaking a sweat (your ISP might though). All the technology required to do that didn't exist in the 90's. We have Bluetooth now though, so there's that same bootstrapping problem, where you'd just use Bluetooth, and not spend a bunch of money building a system very few people are asking for, so then there's little demand for a modern high performance Irda system in any devices.

It was harder to extract a clean signal due to ambient environmental conditions.

You could probably solve those issues with modern tech though. Things have advanced significantly since IR was popular. For example, back then Bluetooth was slow too.

Not through a tiny photodiode + amp on a spare UART RX, if not repurposed TV remote phototransistor. They can be slow.

I'm frankly baffled at all these reports of IR being unreliable and slow. It... wasn't. Not for the file sizes of the day. I exchanged plenty of files back in the day, even at 115200bps a picture would be 2-3 seconds tops (pictures were small!). And when devices started supporting 4Mbps, even a large-ish MP3 would go in 5-6 seconds. All without setup or pairing, beautiful. Huge files (like full resolution pictures from an SLR camera) would take a while - but frankly they took almost the same time with a cable! You'd just have to plug their memory card directly into your computer if you were in a hurry.

The only really clunky use case for me was internet access - keeping phone and laptop positioned and aligned for 30 minutes was limiting.

And yes there IS plenty of bandwidth at those frequencies. In fact latest IR standards reach 1Gbps, but it's pretty much extinct. There was an attempt called Li-Fi to use it for as a wireless networking but I don't think it went far.

What I really miss is OBEX (Object Exchange), which worked also over Bluetooth, and which Apple sadly chose not to implement: simplest protocol to just ship a file or a contact vCard over, no setup, just worked - and it's been a standard for 20+ years. Early Android had it too, it was since dropped I think. Sigh.

Mostly just SNR issues.