Kind of. What carried on was not the Roman Empire in toto, but the Eastern Empire. Which was placed on the ancient Greek mainland (wider than current Greece, e.g. including southern Italy and Asia minor) and the hellenistic (Greek-speaking, as a result of Alexander's conquests) areas, like Egypt, Syria, Lebanon, and so on.
They called themselves Romaioi (Romans) because (aside from the peaceful state-capital transition to the east instigated by a legitimate Roman emperor himself) the first emperors and many early higher-ups were latin-speaking romans, and because, even those who weren't wanted the state influence associated with being successors to Roman empire.
As for the population, they weren't romans in the large, either by ethnicity (italic), language, or culture (hence "Cato’s corrupt Greeks"). They were "roman citizens" but this wasn't much different from "roman subjects + civic rights". Heck, the barbarians that followed Rome (Franks, Germanic tribes, etc) also called themselves "Holy Roman Empire", while having little to do with Romans, and even Mehmed the Conqueror when the ottomans conquered the eastern empire called himself "emperor of the Romans".
And while the Eastern Roman empire went on for over a millenium (with ups and downs, attacked by Persians, Arabs, and so on), Rome and the actual Roman empire did fell, and there was a very real and measurable loss of population, state function, quality of life, culture, infrastructure, and so on that lasted for many centuries.
So to say "Roman empire didn't decline, it moved east" is not really accurate.