A large system, so much higher chance of something breaking.
What happens if it loses a node or a link? Or some memory becomes unreliable? This thing needs some sophisticated fault tolerance.
What happens if it loses a node or a link? Or some memory becomes unreliable? This thing needs some sophisticated fault tolerance.
Define "fail".
> I didn't have any insight on whether it was a hardware or software failure.
Have scripts check nvidia-smi for ECC errors and dmesg for devices dropping of the PCI bus.
For the former, replace the card. For the later, just perform a device reset (a power toggle of the device and a rescan of the bus is often enough to be back online within 5 seconds)