Those minified models are still equal or bigger compared to the initial "attention is all you need" transformer.