You may want to read Google's recently released SRE book, watch the many conferences(monitorama, @scale, and many more) videos on YouTube and/or hunt down slides, check various engineering blogs (e.g Twitter posts about observability, Uber posts about their Atlas outliers detection system etc) and talk to people who are already dealing with those kind of problems.

That said, I think what's more important is to log everything ( use Kafka ), even if you don't know in advance how to make use of them. You may need them later. And you need to break this down into 3 different problems. Data exploration, data visualization, and alerts. And you probably shouldn't or won't be able to find one product that will help you address all those needs - or maybe will be too expensive. You should instead use multiple and all should consume from your events feed/stream. If you want to do it yourself, and you don't have the hardware resources to handle lots of data points, then use two differ sets of services. One that will retain say upto 24 hours worth of data so that you will get predictable, consistently fast query, processing and visualization time and another that may hold data for a longer period. Chances are you will want to understand issues and that happened recently so optimize for that. Good luck :)