Why? Vim is a well tested codebase. It has many features. Starting over would produce better code, but in what timeframe? I think the neovim contributors made the right decision to refactor.
The current UI tests are hard to write and based on the ASCII output [2] of the commands of the test [1]. By using the Neovim API we will be able to rewrite these tests to look more like automated Sellenium based web app tests and make more fine grained assertions.
[1] A test: https://github.com/neovim/neovim/blob/39cdb5f214bcb1e4a918f6... [2] The expected test output spec: https://github.com/neovim/neovim/blob/39cdb5f214bcb1e4a918f6...