Given a large enough context window the need for fine tuning diminishes a little bit....

The cost of fine tuning these massive models is crazy right now. Fine tuning a smaller model can get you better performance for specific tasks in many cases. A lot of models are out there and totally free to adapt as needed via hugging face etc.