If you want to build a bleeding edge, highly polished app or game that can take advantage of the latest Android API's, and turn that into important features of your app, it's probably alright to go just with Android 4.0. If you do it like this, make sure to take advantage of the "fragments" in Android 4.0, to make the app look good on tablets, too.
One of the cool things about Android is that you can target so many devices. So I see no good reasons not to support the version that runs on most of the devices out there.
http://developer.android.com/resources/dashboard/platform-ve...
In any case I recommend to set the targetSdkVersion to level 11 or higher. Otherwise, your app won't have the holo theme or Actionbar if it's running on honeycomb or above.