Pruning is a great idea to reduce memory usage. One thing to be careful with is that pruned matrices use irregular memory access and they might be slower as we don't have SIMD support for sparse matrix multiplication on generic CPU's (e.g. ARM) yet.
It is done using SVD and improves speed without sacrificing memory access patterns.