How about representing Lua tables by arrays with an object in position 0? Then { 11, ["11']=11} would map to [{"11"=11}, 11]. The extra layer of depth is annoying, but the semantics is closer to what you want and you avoid collisions between strings and array indices.
We'll keep thinking about this, but I don't think it's sensible to try to encode {11, ["11"]=11} at all. That's neither an array nor a dictionary, and so it doesn't have a natural JSON representation. I'm inclined to make it an error, but right now we try to be generous in attempting to encode.
"A JSON text is a serialized object or array."
Now I don't know why I thought this wasn't allowed.
Flagging errors in cases with seems to be a good way to handle this kind of problematic input. It's surely better than having applications have nonsensical data if they are given peculiar input.