Thanks for the explanation. Your description superficially reminds me of a Markov chain (https://en.wikipedia.org/wiki/Markov_chain). Is this related or is it totally different?
RNNs are just fancy parametric functions that take a (state, input)-pair and return a new (state', output)-pair.