It's worse than that: once we start refilling it we bring back all of the demand that releasing oil from the SPR kicked into the future.
It follows that if either 1.) demand drops or 2.) supply increases the refilling will create exactly the same problem in the future that releasing oil from the SPR was supposed to solve.