That on top of the unwarranted distribution assumption is very dangerous. How dangerous? If the true distribution has heavier tails than assumed, then the true variance can be many times what is assumed. If the true variance is, say, 4x what is assumed then an average standard deviation variation will show up as passing a 95% confidence interval in a random direction. Getting to 99% confidence by accident becomes quite easy, with bad results.
However the difference between the t-test and the normal to which it is an approximation is that the t-test takes into account corrections the rate of convergence to the estimated normal. The details of that convergence is dependent upon the distribution of individual samples, and the t-test is predicated on the assumption that each and every sample is from a normal distribution.