A Board is often an elected body in the US and can’t technically be “fired”.
So who's responsible for holding them accountable for their actions normally?
Well, they can either resign under public pressure, be voted out in the next election, or (maybe, depends on the jurisdiction in question--don't know if school boards usually fall under this option though) be subject to a recall election.
Correct - some places may be able to do a recall; and most of these “low” political offices are lost upon conviction of a major crime, too (think: taking bribes).