Is it difficult/desirable to squeeze/compress an open-sourced 200B parameter model to fit into 40GB?
Are these techniques for specific architectures or can they be made generic ?
Are these techniques for specific architectures or can they be made generic ?
See the paper here, Figure A28: https://kstatic.googleusercontent.com/files/b068c6c0e64d6f93...
But if your downstream task is simple, like sequence classification, then it may be possible to compress the model without losing much quality.