I think you can do better when a buffer runs dry. Instead of outputting silence, you could output the same frequency spectrum as you were right before the event. That way you will not hear any cracks or pops.
And of course you can fade out the effect when the buffer stays dry for more than a few seconds.
Obviously, you'd have to do some filtering when the audio continues because that too can introduce cracks.