I ended up reducing the number of clients. In my case I had a thousand servers, and I was able to change the application structure and merge them into a dozen big application servers. Now with connection pooling each server has less than 100 connections.
As a more permanent solution for scaling, I'm moving out of mysql into something more distributed.
Or you can have move to mariadb, using configuration "thread_handling=pool-of-threads" to enable threadpool that is exactly the solution.