You're right, that is annoying. Thanks for the suggestion. Here's what we're going to do. On hover on the Circles list, we'll swap out the member count for an Edit link:
https://img.skitch.com/20120517-gg1dd9n3ecy5882gd8335tap44.j...
https://img.skitch.com/20120517-gg1dd9n3ecy5882gd8335tap44.j...