sergey-obukhov

you're absolutely right!

While the research and prototyping were done in R the service itself is in python. The implementation is so simple that it doesn't require any specific ML libs. More complex tasks will probably require using scikit (if desired to stay on python ground) or plugging in R in one way or another, etc.

There are many approaches https://en.wikipedia.org/wiki/Cluster_analysis. In the research k-means clustering was used. It's probably not the best clustering algo for this task and a much better results would be achieved if a different one was used or if some data manipulation was applied prior to the clustering. Here's k-means clustering explanation if you're interested https://youtu.be/_aWzGGNrcic

What part? :)

Re-pasting my comment here where it belongs as a reply:

It could be a great idea - to plot the data on 2 axes (x - html length or html size, y - processing time, if I understand this correctly). It's simple and elegant. I'll try that. It could be though that the chart will get messy with all this data points. One of the reasons I like the percentiles approach is that it makes it clear that there is a trade-off between message processing time and the number / percentage of messages we can process.

Html size and html tags count was a natural choice. If it didn't work out the next step would be to try something else. You're right that it's a very naive example and that in a way it was solved before any ml was applied. The surprising part for me was that both features turned out to be interchangeable i.e. any of them could be used. I would expect html tags count to be much more accurate / reliable, etc. Another interesting part for me was the threshold. It's somewhat clear that it should be somewhere between 20 and 1 sec probably but where exactly?

It could be a great idea - to plot the data on 2 axes (x - html length or html size, y - processing time, if I understand this correctly). It's simple and elegant. I'll try that. It could be though that the chart will get messy with all this data points. One of the reasons I like the percentiles approach is that it makes it clear that there is a trade-off between message processing time and the number / percentage of messages we can process.