Any sufficiently rich data structure is probably enough on its own. Lists certainly are. For example, you don't need arithmetic separately, since you can simulate it with lists in any number of ways; e.g., the von Neumann-esque `0 = ()` and `succ n = n : n`, according to which a number is represented by a canonical list of that length. Once you can define `succ`, you can define all other arithmetic. If you're willing to give up canonical representatives, and represent a number by
any list of that length, then you get an easy definition of addition by concatenation.
If you'll forgive my giving an answer that clearly wasn't what you intended, rather famously, lambdas are sufficient: https://en.wikipedia.org/wiki/Lambda_calculus. See also https://en.wikipedia.org/wiki/Combinatory_logic.