The limitation on sequence length in these architectures is important. For example if you ask someone about a book they just read they can probably give you a summary fairly easily. This is beyond Xformer models today. Note this is distinct from training on a book or series of books which networks handle fairly well.

More practically lets say I want to do an English language -> SQL translation task. If I feed a schema to GPT-3 along with the English language this actually works amazingly well. Unfortunately with only a couple thousand tokens to work with as input budget my schemas have to be short. I can't feed in a ton of schemas and meta data and have it work out complicated joins.