He probably didn't say that. As http://quoteinvestigator.com/2010/05/04/universe-einstein/ points out, the original saying was attributed to a "great astronomer", then followed by "To-day we know that this statement is not quite correct. Einstein has proved that the universe is limited." These later got merged.

In any case, you are making a fallacious argument from authority. Quoting https://en.wikipedia.org/wiki/Argument_from_authority#Appeal... :

> A common example of the fallacy is appealing to an authority in one subject to pontificate on another - for example citing Albert Einstein as an authority on religion when his expertise was in physics.