Interestingly, you don't need to force quit in Activity Monitor. After cmd-q in Podcasts, sending a normal quit signal via Activity Monitor works.
I consider this a bug in Podcasts then since a refresh can be very CPU intensive and the user would naturally quit the app to reduce CPU usage but doing that surprisingly doesn't work.