Another good technique is to pair metrics. Find a second metric for each metric you care about that you expect to see negatively impacted if you overfit for the first metric. For example, you might pair user growth rates with per user acquisition cost.