WebOS was far ahead of its time, and the shitty hardware all but killed it. FFOS seemed to be what WebOS was, but that didn't go anywhere, either.
I think the big problem a new OS will have is compatibility with existing iOS/Android software stacks. You need to be able to run one or the other until you get market share for your own applications, because the majority of the populace wants their bread-and-butter Facebook, Snapchat, et al and these companies will not give your upstart OS any love until you have users.