The interview process takes resources. They spent money but did not get the commodity (you) they had hoped for. They know that going into the "game".

The fundamental question of every position has three parts: can you do the job, will you do the job, and how well do you fit the position/team/organization -- the fit. This means that the acceptable way to say no is in those terms, or really the third one. Whether the culture is offensive, or there isn't good professional trajectory - it always boils down to not being the best fit. Speaking in those terms allows them to be glad that you declined without you compromising future opportunity.

Note: don't ever burn a bridge. You don't know today what you will consider to be a bridge tomorrow. Don't close and lock a door, just don't walk through it.