There is an open source Im working on, that will be able to achieve all this just without dashboard https://github.com/LazarKap/buzzbench.io. How did you analyze resource consumption with your tool??
We just piggyback on the Prometheus metrics provided by cadvisor for the container level metrics. Then for the jvm we collect metrics such as memory allocation etc.
This all goes into a big table and the user can select the right version. Next iteration will be where they can configure whether to aim for latency, throughput, or efficiency, and which weights so we can select the winner automatically.