The biggest downside is that if there is a big slump, the gov't doesn't have spending money for a stimulus. We could then get stuck in a protracted rut or deflationary spiral. In the longer term, GDP growth can make the debt moot (if more is not added), but if we have a big slump sooner, we are screwed.
Calling the Federal Reserve an arm of the government is misleading at best.
Does that erode trust in currency and cause hyper inflation? I suspect no.