First, there's just a ridiculous amount of plugins and such already made. Either they'd need to be ported to the new interface (an insane amount of work) or the old interface (Vimscript) would need to be maintained alongside the new. Neither option seems very practical.
Second, the main advantage of Vimscript is that it uses the same commands you use every day while working with Vim. `edit foo.py` in a Vim plugin does the same thing as `:edit foo.py` in your daily routine. This makes it really easy to get your feet wet with Vim scripting -- in fact it makes it all but unavoidable.
Users learn tidbits of Vimscript while working, then start hacking stuff into their .vimrc files, then move on to learning more and writing plugins. It's a fairly natural progression.
If you're going to change Vimscript to something else you have a couple of options, all of which suck:
* Replace Vimscript in the daily routines with Lua/whatever. Instead of typing `:e foo.py<cr>` I now type `:edit("foo.py")<cr>`. No way this would ever work with Vim's community.
* Keep using Vimscript for daily commands, but use Lua when reading from files. This loses the main advantage of Vimscript (which I think is a really important one), plus you now need to maintain two interfaces.