Rubinius specifies a bytecode (see http://rubini.us/doc/en/virtual-machine/instructions/) and implements a VM that executes this bytecode. The VM is written in C++. The main difference between Rubinius and other Ruby implementations is that nearly everything else is written in Ruby. The lexer, parser, and parts that compile the AST to bytecode is all written in pure Ruby. Further, large parts of the standard library and Ruby language that are implemented in lower-level languages in other implementations are instead written as runtime-level Ruby. However, large and important parts of the infrastructure (including GC and memory allocation) are written in C++.
PyPy takes an altogether different approach. It specifies a restricted subset of Python known as "RPython". Basically, it's statically-typed Python. There is also a "compiler" (also called a "JIT generator"), written in pure Python, that compiles an arbitrary interpreter written in RPython into a fast JIT. (See http://morepypy.blogspot.com/2011/04/tutorial-writing-interp... for an example of writing a very simple, small interpreter in RPython.) A cool feature of RPython is that while there must a statically RPython AST to compile, during the "eval" stage you can leverage the full dynamic capabilities of Python.
The second part of PyPy is a full Python interpreter written in RPython. Since RPython is a subset of Python, you can actually run this interpreter sandboxed inside any other Python implementation, albeit very slowly. The PyPy toolchain compiles this down to a very fast Python interpreter that features JIT compilation, among other cool features.
The neat part of PyPy (in contrast to Rubinius) is that the toolchain part is largely decoupled and generalized away from Python-the-language. Conceivably, one could write ANY language in RPython (or what's compiled down to RPython) and have a fast JIT interpreter. Experimental language features can be added and changed (see the source code for the different garbage collectors in PyPy) with little mucking with the low-level VM.
Rubinius is a neat, pragmatic project that could turn into one of the best Ruby implementations around. However, it's not breaking new ground like PyPy is.