- CAN is very slow despite high baudrates. As used in automotive max 40% of all bits are actual databits. That value is for frames with 8 bytes. For 4 bytes that drops to 25% so people avoid these frames. That means in practice one gets only 1500 frames/s at a typical 250kbps.
- a roundtrip takes 2 frames because the CAN request frame is broken to the point that standards like J1939 do not use it.
- CAN-B error detection mechanism has holes in it due to bit-stuffing. Effectively the CRC is circumvented, so error detection rates are nowhere near the projections made in the design documents.
- CAN-FD increases the payload, stuffing 5x (typical) more databits in the same frame. That means bits are transmitted at only 22% of the datarate, worse for higher datarates, much worse for frames with only 8 bytes. CAN-FD hardly improves latency or the frame-rate, but it does much improve error-detection rates.
So the take-away here is, yes, CAN sucks, quite badly. CAN-FD improvements really only suit niche applications.
It is possible to do much better and achieve roughly 4x the CAN-B performance (on metrics such as throughput, latency and robustness (all of them)), on the same bus.
But not for the self-driving car generation, where the sensors are higher-bandwidth.
As I recall, Mobileye worked around this by just putting the detection intelligence with the camera and sending back synthetic data.
There's a relatively recent extension called CAN FD that can go faster, as can the FlexRay protocol, which is sometimes used instead of CAN.