(Author here.) That looks like a great library --- I will have to try it out!
In case it helps anyone, I also wrote a much smaller MDP library in ruby to support this 2048 project (and several others): https://github.com/jdleesmiller/finite_mdp . It basically just does value iteration and policy iteration.