I used to feel this way too. It turns out the mirror test is not a good measure. Hard metrics are much better. Weight alone isn't the best, but it is a good place to start.
How so?
Of course if you're exercising regularly and strenuously, there are some athletic metrics: how long you can run until exhaustion, how much weight you can lift, etc.