This has been on my mind recently as someone who was in the front office for a few years (but not making trades) but is now on the outside.

    I mean each day 100's of Phd's start with clean market
    data, more data sources than you could possibly think of
    and statistical back testing systems that have 1000's of
    man hours put into them, trying to find a way to make money.

It seems to me that the Tiger Rule applies here. You don't have to outrun the tiger, you just have to outrun your buddy.

Is the choice really that one can be in the cohort you mention, or you can buy an index fund (or whatever the equivalent is in the market you're interested in)... and that's it?[1] Is the market so efficient that there is no middle ground where a smart and methodical person can make more money than the index plodders without being obliterated by the big players?

That seems really unlikely to me.

[1] (I don't think that's what you are saying... but you've given me a chance to try and express something I've been thinking about. Thank you for that)