As I understand it the Arduino libraries are big, but I have never tried. What are the advantages to not just using the AVR libraries if you are embedding the chip in the circuit and using the programmer?
That said, I've done plenty of projects with bare avr-gcc (especially for chips not supported by Arduino at the time) and it's not terribly onerous, but often you don't need that level of control for every little thing.