This might actually be partly generational, if you, like me, grew up using a calculator. I must've done that test with probably about a hundred patients by now, and I have to tell you, the average halfway mentally competent seventy year old can do it faster than me. It's particularly embarrassing when they make a mistake early on, and then I can't just look down at the test sheet in my hand for the correct answers to see if the rest of the numbers are correct.
That said, like other commenters have said, it's supposed to be a little bit hard. This exercise is part of the mini mental state examination [1], which is intended to be a screening test for the early stages of dementia, so you want to challenge people a little. The math part is actually the second-hardest task - the most difficult part for people with even very mild cognitive problems seems to be drawing the two interlocking pentagons.
1: http://en.wikipedia.org/wiki/Mini%E2%80%93mental_state_exami...