I don’t know if could use capacitor or need battery. If need lots of battery for load, might end up with UPS. It would need to be cheap.
It would be possible to build this kind of on-line conversion with a very very small battery that could only provide power for a few seconds, and that kind of device would serve the goal you mentioned. I suspect the cost for this device would quickly approach a UPS in cost, so it might not be the 'cheap' solution.
One way is small, inexpensive standby UPS: Something with fast switchover, but without the capacity to survive anything longer than necessary to avoid some flickering and to conduct an orderly shutdown if the badness persists. This is the cheapest way. (And for small low-voltage loads like Raspberry Pis or consumer routers, there are kits available using things like 18650 cells.)
Another way is a small-ish online UPS. These have no switchover time, but they're more expensive because the parts are bigger, and they're less efficient than a standby UPS.
Another way is a ferroresonant transformer. This isn't cheap, but it is lasting (no moving parts or chemistry to degrade) and can work through some flickering.
And finally: Motor-generators, with a flywheel to pick up the slack. These work fine, but they're big, alarmingly expensive, require maintenance sometimes, and there are good reasons to be scared of spinning flywheels.