Perhaps more controversially, I also don't think statistics in general is a particularly rigorous or well-founded field. Where there is any analysis at all in the field - that is, very rarely - it provides guarantees only with hefty assumptions and in the limit. I'm fine with that from a theoretical perspective, but practitioners of all stripes seem to assume that with real world phenomena, the data either satisfy the assumptions or, more defensibly, don't violate the assumptions by enough to invalidate the results.
I don't know about others, but I see real-world datasets all the time that wildly invalidate the assumptions of common regression analysis, to take just one example.
To reign this little rant back into a well-formed comment, perhaps my summary is "everything and everyone is wrong, almost all the time, and I'm not quite sure how to fix that."
* In my personal experience.