One absolutely phenomenal implementation of a state machine is in Intel's Thread Building Blocks library. They (somewhat[1]) recently added a feature called flow graphs, which is an abstraction over tasks (themselves an abstraction over threads). I've yet to find a better control structure for concurrent scheduling problems.

TBB isn't NUMA-aware though, which limits its use in HPC.

[1] http://software.intel.com/en-us/blogs/2011/09/08/the-intel-t...