I believe it's meant to be ironic, in other words taking a poke at the so called "facts" that people generally toss around (although that doesn't make it less annoying).

You are right, I hate that phrase too. Unfortunately it's pretty widespread. From my Collins Cobuild 5 dictionary:

"When you refer to something as a fact or as fact, you mean that you think it is true or correct."

See also: http://www.thefreedictionary.com/fact

Although, the Wikipedia disagrees: http://en.wikipedia.org/wiki/Fact#Etymology_and_usage

Example from football (soccer):

In the opinion of the referee, the ball did not cross the goal line between the posts and below the crossbar despite the ball actually doing so.

However the fact is in accord with the referee's opinion, not the actual path of the ball.

In logic and philosophy, a fact is a proposition that can be true or false.