Say you have the necklace of some Pharaoh. OK, who is the rightful owner? The present day Arabs inhabiting Egypt, who conquered it? Or the Greeks who conquered it before then? Or the Romans? Or the "ancient Egyptians" (this refers to many people, as Egypt was conquered several times by many different groups).
So then your fallback position is, "we don't know which 'people' the necklace should belong to, but we know which government it should belong to -- the present day government of Egypt, which is the current government occupying the geographic area where the necklace was made. It belongs to that government."
We can call this the "occupying government" (OG) theory of ownership. It doesn't matter that that the present day Arabs were invaders number 50, but because they presently occupy the land, they own the necklace. Let's explore the consequences of this rule:
When Britain controlled that territory, that meant that it was the rightful owner -- since by the OG logic the rightful owner is whatever government was in charge at the time -- and it moved the necklace to a museum in London, which the owner has a right to do with their property.
But then you say "no, that doesn't apply to colonial governments - not to white people. Only legitimate governments can benefit from the OG rule", to which I would answer
1) "How were the Arab invasions legitimate? How were the Egyptian dynasties legitimate? They also ruled by force and enslaved their populations".
2) If your ownership rule is the OG but it's nullified if you believe (somehow) that a government can be illegitimate, then you create a situation in which the antiquities have no owner -- since the previous government doesn't exist, the original racial group is gone, so who is the owner if you are not going to follow the ownership laws of the government in charge?
So really at the end of the day, these types of arguments boil down to not liking the behavior of the UK in Egypt, which is fine by me, but they are not objective arguments if you try to apply the same standards of property ownership across thousands of years to all governments that would remove antiquities -- which is basically every government in history, whenever they found something beautiful and shiny, from the Romans carrying back booty to the Pharaohs carrying back booty, some of which was turned into beautiful jewelry.