"Moderate" views are just the status quo, so of course that's the case. In an oversimplified 1-dimensional view of politics ("both sides") where you have a left, center, and right, you will always be criticized by the other two.
Perhaps more bluntly, are you yourself a Nazi, a Communist, or happy with where the US is politically? If you're none of the above (as you likely aren't), then you must be able to see that moderate views can still be left or right-leaning.
Maybe the kindest reading of your comment is that the average person that's overly critical on Facebook only sees in black and white, left vs right. And if so, we completely agree :)