GNU Readline already provides a lot of VI's (so almost VIM) functionality as a library. Maybe that could be used as a starting point? It probably still is a lot of work though.