Edit: ./firefox/<profile_name>/chrome/userContent.css
a[rel] { background-color: #ff00ff !important; }
The above will change any anchor with any 'rel' attribute, but if you want to get fancy by only modifying anchors with 'rel' attribute having 'nofollow' then you'd need to realize rel is a space separated list.
a[rel~="nofollow"] {background-color: #ff00ff !important; }
Given that some idiots incorrectly use semi-colon or comma delimited lists for rel, you'd be better off with using the "contains" idiom.
a[rel*="nofollow"] {background-color: #ff00ff !important; }
http://www.w3.org/Style/CSS/Test/CSS3/Selectors/current/html...
If you're really have your heart set on using in-browser javascript, look into userChrome.js --You can do many amazing things with userChrome.js due to when it runs, and it's access to the browser.
http://userchromejs.mozdev.org/