Maybe because you led in with “to be fair”? Perhaps “as an aside” may have been more appropriate, to indicate an intentional context switch?

Your point is valid as a tangent, but it doesn’t read as a tangent to me - it reads as attempting to address the point you are responding to.

Just my two cents here.