I'm not convinced.

Your problem was not that your CSS was 340 KBytes, but the simple fact you were using 5 CSS frameworks. And if you start removing unused styles at random, any time you want to work on the site and add features you may encounter edge cases where the style you want to use has been removed. And, if your CSS is used on multiple pages with different layouts, it looks like you will need to manually browse each one and merge the results.

However, this is not to say that merging and compressing stylesheets is bad. If you do so then you can embed the style in the page to remove a render blocking resource and speed up your site. Then you can repeat the same process with Javascript.