Speaking of Geerling testing Pi 5 things, is it just me or is it super weird that the uPCity PCIe breakout board hasn't been put on sale yet? I think it's almost half a year since he got it working in a pre-release state.
But so far there's no straight PCIe expansion board available to purchase yet. It is a slimmer market than 'NVMe on top' or other more standard use cases, but it's one I think could expand as people do weird things with the Pi 5.
Looks way more compact and well thought out in terms of mounting so definitely seems like the better option once it's released.
Anyways, I ended up making my own and it works great. No fuss, no wait, no complications.
As for bandwidth, well, it's one lane of PCIe gen 2. This won't win any races but can be useful to access exotic hardware not available in usb or if you don't care about bandwidth. (e.g. HBA with many drives for mass storage without speed requirement).
What SmartNIC are you using? Most SmartNICs that I'm aware of suck a decent amount of power, many more require significant external airflow. Are you using the Mikrotik active cooled one? https://mikrotik.com/product/ccr2004_1g_2xs_pcie
The 12V comes from an external power supply through a barrel jack, from which I also derive the 3.3V rail. The Pi provides no power whatsoever. I should publish the design files somewhere.
As for the NIC, it's a Netronome Agilio-CX which is fully programmable using eBPF and such.