As an analogy, consider doing this to a person and trying that same excuse.
Why? A robot is not a person. It is a robot.
Unless the delivery robot has a legal permit to conduct its commercial business in the public space that is the sidewalk, then there is no reason that a human being should care about it.
Just like when businesses put up bandit signs in the median. People have the legal right to remove them because the signs are not there legally.
I have seen nothing that indicates these robots have any special-use permits for the sidewalk the way, say, a street vendor would.