That's more or less like saying dollars aren't fungible because you can check the serial number on a bill to see if it matches one that was reported stolen.
A stranger across the world can't track your physical currency in real time. Whenever serial numbers are checked (if it ever is), the tracker just knows that bill was at that spot at that time, they don't know how many times it was exchanged or where it was since it was last tracked.
With BTC, anyone can track it in real time, anywhere. The same cannot be said about physical currency.
I suppose it's not a stretch to imagine banks have a machine that scans serial numbers (eg to catch DB Cooper's cash), but I've never heard of any evidence of such machines.