The problem is trust. It's helpful to remember that all the participants in an Ethereum network are adversarial. It doesn't matter how many bad actors there are as long as >50% of participants agree on the outcome. (This is a simplified explanation -- the real system is transitioning to proof of stake rather than proof of work.)
If you allow solidity to access the internet or the system compilers, you can't guarantee determinism anymore. For example, if you increment a counter by the result of accessing https://some-value.com (which returns a JSON value), you could program it to return a different result for everyone who accesses it. None of the nodes would agree what the true value of the counter should be. It'd also result in a central point of failure.
These same problems exist if you let Solidity access the system software. You could program it to give different results, so no one would agree.
The innovation of Solidity is that it's a set of operations where no matter what you do, the majority of nodes will converge to the same answer.
Does the EVM prevent this in some way?
I did some searching but I'm not sure I know what to look for either.
Moreover, a point that was not made clear to me when I first read about Ethereum, the programs are inherently limited in the number of steps that can be run, and the VM monitors this and terminates the program prematurely if it exceeds the limit. Moreover, longer programs cost the caller (the person or other contract asking the network to run this program) more money.
If the connection to the third party service is cryptographically secure, the Oracle can even provide proof the results were not tampered with, and if multiple data sources are used it can provide a consensus answer based on them.
Otherwise, the smart contract only has access to what is in the blockchain, but Oracles are pretty easy to write and use.