It does fail if there is not enough entropy.
> An error occurs if the PRNG has not been seeded with enough randomness to ensure an unpredictable byte sequence.
https://www.openssl.org/docs/crypto/RAND_bytes.html
But there is also this:
> Pseudo-random byte sequences generated by RAND_pseudo_bytes() will be unique if they are of sufficient length, but are not necessarily unpredictable.