I did that too, and used to worry about it. Then I saw a video of a guy messing around with a giant tub of liquid mercury, talking about how it's not all that dangerous in that form.
This was all prominently on my mind as I broke a mercury manometer during an unofficial experiment in school and was trying to find all the little drops that exploded across the floor and hid themselves in the corners. However, I can say that it's a very efficient way to collect dust bunnies... ;-)
http://en.wikipedia.org/wiki/Compact_fluorescent_lamp#Enviro...