Let's say you have 100 intersections, and you want to predict the traffic on each in cars/sec. You sample every hour, and you keep 24 hours of context, and try to predict the next 4.
First, you'd make 100 "tokens" (really stretching the meaning of token here), one for each stoplight, and loading 24 samples (the history of that stoplight) into each token, and normalize.
Next, you run each token through a Multi-Layer Perceptron (vanilla, old-school neural network) to make a vector of dim D.
Next, for each layer of the transformer, you: 1. Perform "cross-attention," i.e. the query/key/value dance. This is how the different time series (erm, tokens) get to share information. 2. Normalize across all. 3. Run another bog-standard MLP independently on each token. This is the opportunity to examine the history of each time series. 4. Normalize again across all.
Then, you map each "token" (ugh) from being D-dimensional to 4-dimensional, so for each stoplight it predicts the traffic ahead for the next 4 hours. This is also a regular MLP.
So specifically, if you're only predicting a single time series (one stoplight), this method is equivalent to running a regular neural network.
It also, interestingly enough, skips the cool sinusoidal position embedding that transformers use to embed token position. Fair enough, since here the time dimension is fixed and the index of the feed-forward neurons in each MLP layer corresponds (roughly) to the time index of the sample.
The architecture looks weird to me, but apparently it works so that's cool! But I'm not sure how well it works, and my unscientific gut feel is that there's a better and simpler architecture crying out to be found, because this looks a bit tortured. Like, nothing in it explicitly models the time dimension - that task is left to the MLPs - and that seems weird.