And the point is that you don't need the "world" aka reality to make that change. It can come from within, for example a faulty sensor creating an image of a cloud that does not exist.
And the implication that follows is that just because you percieve something does not mean that it is "real".
This can be made more clear if you understand that every "real" object is made up of pixe dust aka fields. When you see a particle at some point, say an electron, there is actually nothing there...but the space at that location behaves, for some reason, as if there is an electron there...
And that is another problem with the physical idea. What happens if you continuously split an object? If it is really physical, then it should remain physical no matter how many times it is split. But we see that it is not the case.