But how can that hold when you're approaching infinity? It would only apply to sets finite in both directions.
The "..." signifies that there are infinitely many such numbers, so there is no largest natural number but there is a smallest (1 is smallest). In that sense it is infinite in only one direction - written here it is infinite to the right but has an end at the left.
Nothing special happens as you "approach infinity" and in fact that's not a thing you can do. No matter how high you count, you have only have counted finitely many numbers and there are an infinite set more that you have not counted.