How does "redo the call with a smaller size" possibly help you? If you could get by with less memory, you should have asked for less memory in the first place.
I have seen applications react by pausing and trying the allocation again a few seconds later --- this can be shockingly effective! https://hacks.mozilla.org/2022/11/improving-firefox-stabilit...