I did not develop the sim itself but did develop the hardware-in-the-loop portion of it along with things like real-time debugging, and output to the hitl. We had the sim rendering cameras which we output from the workstation to custom HDMI bridges over MIPI that we could treat as real cameras on the NVIDIA Xavier AGX. There was a data channel over Ethernet for IMU data.
I made a custom version of Eclipse that interfaced w GDB for debugging, which also was modified to stay in sync w the sim using PTP, w rewind capability.
As for sailboat modeling, yes it’s more complicated because of the effects of both wind and fluid dynamics. If I were approaching this, I’d probably try and find a physics simulator to start with. Getting ground truth will be difficult, but I imagine you would start w the IMU and GPS data off the boat, but having time synced ground truth for the waves and wind will probably be the hardest part.