Well, they're wrong. What exactly do they mean when they say that the physical world exists? When you say that the moon exists, you basically mean that when you look up at night, you expect to see moon. It's a statement about your perceptions / consciousness.