I would prefer to see some kind of CLI / shell only build system (Makefile or something else) be given a bit more attention by the official releases. As it is, it seems like a bit of an afterthought.
Personally, I use Atmel's development boards and AVR Studio under VMware.
http://www.arduino.cc/en/Hacking/BuildProcess
That alone should be enough to get you going -- it's a pretty straightforward compile+link+upload process, with only a funky preprocessor that makes their language easier to deal with than C or C++ for beginners (doesn't require includes or care about order of function declaration | prototyes ).
Note that the Arduino IDE does ship with Makefiles for doing builds and installations via the CLI.
In fact, on debian, there's an "arduino-core" core package that contains all the parts you need without the GUI (or java for that matter). I run it on an ARM-based Linux box I've got.
Try CrossPack if you're on OSX. It's exactly what you ask for - just wipe the Adrduino stuff off the chip and use it like a regular AVR development board. You can just write regular C code.