Some of my time was spent being fussy about audio quality, but I still think it's important - I've unsubscribed from a lot of podcasts because the audio was unbearable. At the very least, learn a little about dynamics/compression, so that your podcast is a steady volume and not too loud or soft compared to others. Look at something like The Levelator [1] as a bare minimum.
Personally, I used iZotope RX [2] for noise reduction when necessary, and iZotope Ozone [3] for controlling dynamics and applying EQ. Ozone has a feature where you can analyze the EQ of a recording - say, Leo Laporte's TWIT podcasts - and apply that EQ curve to your own recording. That together with Ozone's dynamics compression helped get a much more professional sound.
Lastly: you might want to look at Christopher Hawkins book Record & Release [4] about every step of the podcasting process.
[1] http://www.conversationsnetwork.org/levelator
[2] https://www.izotope.com/en/products/repair-and-edit/rx.html
[3] https://www.izotope.com/en/products/master-and-deliver/ozone...