The problem is a bit more nuanced than that. The judge can call contempt even if someone answers the summons, but can not afford the fines or follow court orders due to financial hardship - solely at the Judge's discretion.

The most ridiculous yet iconic example: Man is jailed because he can not afford to sod lawn. Didn't break any laws, just crossed a HOA.

http://www.tampabay.com/news/humaninterest/brown-lawn-means-...