This seems like how I read some old unix guide on compiling software. I put it all in ~/local/$software/$version with $software/latest symlink. I really prefer it as I can wipe software just with removing the directory.
I hate software like MySQL though which seems to ignore me setting --prefix in configure and still insisting to be default look in random directories.
With software with a good makefile and pkgconfig file, I've not had major trouble compiling all the software I use from source.