Special characters don't make as much of a difference as password length: http://xkcd.com/936/ (a comic that must appear on any password-related thread). According to the XKCD analysis, an 8 character password based mostly on letters with some numbers and symbols has ~28 bits of entropy, while a random four word password has 44 bits. [Edit to add: http://news.ycombinator.com/item?id=4083381 claims over 3 billion MD5 hashes a second for $1000, meaning your 28-bit password could be cracked effectively instantaneously by a high school kid buying phishing gear with stolen credit cards or a summer job].
If you're an international web site, it might make sense to disallow symbols, as your customers may not be able to enter their chosen symbols when traveling internationally.