To me the most annoying thing about Japanese counters is not so much that there are so many, but that you also have to remember when the numbers that go with them will need on-yomi or kun-yomi (ichi, ni... vs. hito, futa...) (well, okay, that essentially depends on the counter itself being on or kun. Makes me wonder if there are jūbako-yomi or yutō-yomi cases, never looked it up). And if that wasn't enough you also need to remember when e.g. いち becomes いっ and when a h becomes a b or a p (ippon, nihon, sanbon...)

Edit: and there are the fun cases where it's not always on or kun: ichinichi, futsuka... or other fun exceptions.