That's going to be interesting. At those speeds (1.5GB/s, 850k IOPS) you must bump[1] into all kinds of DRBD bottlenecks?
Also I'm very curious about your approach to redundancy/multipath. Dual-primary or failover?
My current feeling on load balancing is to keep it simple, have half the clients treat node A as their active storage and the other half using node B. If node A becomes unavailable failover will occur and node B will serve all clients.
As far as multipathing goes - my approach is that I honestly have no idea. That's something that's going to be learnt along the way and I'm in uncertain if it'll make it into the final build or not.
Edit: I can't get that link to load on my connection at present but I've bookmarked to look at tomorrow. If you have any experience / advice I'm always very open to assistance!
The biggest frustration that we met was using LACP in trying to get a better data rate out of DRBD. Regardless of the LACP mode, we were not able to get DRBD to scale to full bandwidth (20gbps) and had to settle for using it for redundancy only.
I imagine with your setup (read: much faster peak write capability), you might actually be able to bottleneck DRBD pretty well over a single 10GBE pipe if you were writing at peak capacity. I'll be interested to see if you happen upon a work-around.
Load balancing as you've suggested is an intriguing compromise. Have you tried doing a two-way heartbeat setup where each both boxes have their own active IP and can fail over to each other?
MPTCP establishes multiple subflows across individual IP paths and can load balance or failover across all subflows. Applications do not need to be rewritten to take advantage of it. I'm not sure if that includes kernel modules like DRBD though. I suppose someone needs to find out :)