The code needed create, train, and perform inference on a Transformer is quite short. How short depends on how you count the `import` statements in https://github.com/openai/gpt-2/blob/master/src/model.py and https://github.com/openai/gpt-2/blob/master/src/sample.py etc.
Spreadsheets performing linear regression etc., — Do they learn? Sure!
If you accept that Transformers adapt and learn then you must accept that a spreadsheet also does, because someone implemented GPT-2 in Excel: https://github.com/ianand/spreadsheets-are-all-you-need
Do polymorphic computer viruses adapt? Border Gateway Protocol? Exponential backoff? Autocomplete? And that's aside from any algorithmic search results or "social" feeds, which are nothing but that.