I found this to be a great introduction on how to write a compiler:
http://www.cs.indiana.edu/~aghuloum/compilers-tutorial-2006-...It starts with a compiler for a language that only accepts integers and gradually extends it to support complex expressions, functions, closures, conditionals etc. Since the compiler is written in scheme and the target language has a lispy syntax, it completely skips the parsing step (using scheme's builtin reader). I really enjoyed this fact, since I find parsing to be the most boring part of a compiler. The fun comes after ;)
Another interesting read is http://repository.readscheme.org/ftp/papers/ai-lab-pubs/AIM-...
My first language implementation was an interpreter (interpreters are easier to write, although compilers are more satisfing when you start to benchmark the speed of the generated programs) written in C for a dialect of lisp. I chose lisp because it was easier to parse. The program was converted in an AST (abstract syntax tree) that was then directly evaluated. At first, there was no GC, no support for direct recursion, no closures etc. I started with something small and easy to implement and then incrementally extended it.
Golden rule when writing your first language implementation: keep your language simple. Keep the implementation straightforward to understand, even if it means that it is slow. Don't bother adding support for debugging or meaningful error messages. You can add those things later after you've grasped better how an interpreter/compiler works.