Big-O is a way of categorizing the growth of mathematical functions. Those functions can represent anything. It is wrong to talk about the big-O of an algorithm without specifying what you are measuring. Be it average operations, worst case operations, average memory, worst case memory, amortized average time, amortized average memory, and so on.
It happens to be that when we talk informally, we're usually talking about the worst case we are willing to think about. Quicksort's worst case is a sorted set, so we think of that as O(n^2). But then we turn around and cheerfully call a hash lookup O(1) because its O(n) worst case is incredibly rare in practice.