Be wary of saying things like "from an infinitely dense point" because you might mistakenly think the universe is expanding from some distinguished point in our space. Instead, the infinitely dense position was in time, and the universe has always been (we think) infinitely big.