But they can see that you are not following them, right? I can see everyone who has me in their circles (following). So therefore I can see who has removed me, even if no notification is given.
Knowing this beforehand I would have made a new email address to manage the account, and could freely allow circling. I guess I'll go find out now if I can change the owner of the page.
In other words, if you have person B in your circles, you can hide whether person C (or the general public) can find out that person B is in your circles. You can't prevent person B from knowing that they are in your circles (thought they won't know which circles they are in).
I'm pretty sure this is the model. I can confirm it if you'd like.