The problem is that strict file system sandboxing in particular also breaks a substantial number of workflows that can't be modelled as 'only ever open the exact file the user explicitly' picked. (Any multi-file file formats are particularly affected, as well as any UI workflows that don't integrate well with strictly having to use the OS file picker.)
So you need some escape hatch for optionally allowing access to larger swathes of the file system, or even really everything as before, but that in turn then risks being abused again by malicious actors. And then…?
Plus things like Android's implementation initially using an API completely incompatible with classical file APIs, as well as causing some noticeable performance overhead even today if you need more than simply accessing the occasional single file here and there.