For instance we might say:
sin(x) = x -x^3/6 + x^5/120 + O(x^7)
To indicate that the terms we did not write are in O(x^7). Also note that, in this case, we are actually looking at big-O as x->0.For instance we might say:
sin(x) = x -x^3/6 + x^5/120 + O(x^7)
To indicate that the terms we did not write are in O(x^7). Also note that, in this case, we are actually looking at big-O as x->0.However, mathematicians do indeed use similar notation in this context, that is, little-o notation. It is in fact true that
sin(x) = x -x^3/6 + x^5/120 + o(x^5), x -> 0.
https://www.wolframalpha.com/input/?i=taylor+series+sin+x
Notice that in your example, you have o(x^5) and an explicit x^5 term. In my example I have O(x^7), but no explicit x^7 term. It is true that I cannot think of a circumstance where you want to do this abuse of notation and would care if you were forced to use little-o or big-O instead of the other.
In my experience, it happens to be more common to use big-O.
The parent comment was right and you are wrong, around zero x^5 absolutely dominates x^7 and the big-O notation is used. See for example here [1]
[1] https://en.wikipedia.org/wiki/Taylor_series#First_example