I think this is the ideal spot. I wouldn’t trust an agent doing remediation but I would gladly take input from them regarding services that are acting up or have a trend to disaster. This would improve using my time in what matters instead of cleaning data to determine what matters. In a perfect world I’d like to provide my threshold or what success mean to the agent and expect information without me tweaking dashboards, configuring alerts, etc.

That already exists, though? Log anomalies is a thing.

For all the grief people give Datadog, it is an incredibly good product. Unfortunately, they know it, and charge accordingly.

Splunk has had this for close to two decades.

And I’ve worked on some of the world’s largest systems and in most cases simply looking for the words: error, exception etc is enough for parsing through the logs.

For everything else you need systems like Datadog to visually show you issues e.g. service connection failures.

FWIW this is part of the promise of AIOps of the 2010s and it's still only barely starting to happen. I'm glad it has, but there's so, so far to go here.