One of the biggest problems Java faces for real adoption amongst tech people is still the perception (and I am not sure it is entirely flawed either) that it is insecure. Yes Flash is just as bad if not worse, but we are stuck with them because they are so ingrained within the Internet. Java never caught on, and I know that when I see a Java application I let out a huge sigh and go download and install Java again.
I've gone so far, and so have many of my friends, to remove Java completely and it is not like I am missing out on anything important! I just don't trust Java. I don't trust Flash either, but that is nicely sandboxed in Chrome, sure exploits may exist there, but at least it will be a lot less likely, and Click to Flash helps as well!