This is a great usecase for diff tracking.
1) Store all actions by a user as as an array of add/remove commands (where a change is remove+add).
2) Whenever you need to re-render take the cached version from the last render and perform the new add/remove commands that happened since the last render.
3) Cache the result for the next render and repeat.