It's a best effort though, you're relying on the package maintainer to have tracked all the installed files and clean up anything they might have modified, which hopefully you or something else hasn't modified after the package was installed.

The packages authored by distributions tend to be pretty good, but I've seen RPMs with nothing more than tar zxf in them and no tracking of the files installed.

The major package managers expect the package to manage the artifacts installed, this works if everyone plays nice and does their job...