I wish I hadn't thought about scaling when my [last] startup was small.
There are definitely things that can be done, not necessarily requiring any extra effort, that can alleviate scaling problems down the road though.
What sort of things specifically scaling related did you do that you now wish you hadn't?