I stayed in the Al Bat'ha district when I was there, where there were virtually no short-term visitors (can't say "tourists" since Saudi Arabia didn't have tourist visas; I was speaking at a conference, but stayed for an extra week). It never felt dangerous, but there was a massive gap between ritzy central Riyadh and the market district. Close to half of the population of Riyad are guest workers with limited rights and low salaries; I can't remember if that seemed to be the reason for the gap. This was also 20 years ago, so I don't have any sense for how it's developed since, but I'd be surprised if it has changed.