It's been done. There's XNS, and there's QNX networking over raw Ethernet. Both worked well. Look them up. You probably don't want to go that route, but it is technically possible on a LAN.
I believe there are other broadcast protocols that runs on Ethernet for the same reason.
Fibre Channel-over-Ethernet used to be a thing too, but I haven't seen it for a while. Perhaps latency wasn't as much of an issue as people thought and it lost to iSCSI.
I don't think it necessarily is bad idea to run protocols directly on the data link layer. The fewer parts the better. It's just that somewhere someone probably wants to route it, and the more general usage tends to win. So it's always going to be a niche market where latency is really important.