What problem does the concept of an empirical fact solve that isn't solved better with consensus?
Because : consensus of what? of sheer numbers of human adherents? of other statistic? historical evidence? of widest agreement by those with higher education in that field? of those with the most money to afford advertising? other? a consensus of multiple consensus(es)?
The dynamic of myriad possible vectors through language to express a given 'truth' remains, whether we define truth as empirical or consensus.
Your founding premise, that observed data (that is one definition of 'empirical fact') and "consensus" are somehow opposing ideologies seems flawed however.
If you would prefer to read the term consensus in my original post, rather than empirical fact, then please do - it does not change my point.