The only freedom you give up by using the GPL is the freedom to screw your users over by denying them access to the source code. All you are demanded is to let others enjoy the freedom you enjoy. That, surely, isn't too much to ask.
Meh, hardware vendors routinely ignore GPL, and for the money it takes to sue them in court you could in most cases reverse engineer their drivers and write them for scratch.