TBH most people I know who regularly drive there still take the Millau valley route, since the viaduct toll is quite expensive at 13€ in the summer (just to cross the bridge)
You thing during particular holidays the single lane somehow has even less lanes, less grade separation and such? That would be quite a phenomenon.
In particular most intersections on the now D809 are roundabouts, continuing on the D809 often requires making a turn on the roundabout, and roundabouts are notorious for gridlocking with high turn volumes. Let that gridlock cascade across multiple intersections and you now have rapidly deteriorating travel times.
At other times, traffic is less high so this gridlocking is less likely to occur.