I would generally recommend Luau over Lua in most cases. If you don't need the simplicity that Lua provides (C is a lot easier to compile than C++), there's no contest to me, especially for game development.
LuaJIT is great, but if you plan to ship to mobile or console, you can't use JIT. You could use its FFI library, but that's the only real advantage.
Luau, however, has a bunch. Its interpreter as often as fast as LuaJIT's interpreter (without JIT), it's improved weekly with a variety of optimizations, and you have a killer feature: native vectors. For game development or heavy math code, that's a really big deal because it means vectors live directly on the stack and aren't garbage collected, making them very fast. Luau also has some very nice QOL and of course type checking.
There's some disadvantages, mind you. The most notable one is that as far as I'm aware all tooling for Luau comes from Roblox users. The Luau LSP extension someone linked earlier, as an example, is developed by a Roblox user. Luau also doesn't implement tailcalls, which might matter to you. I forget the reasoning but it was mostly a UX thing.
There's also no native integers and there probably won't ever be. LuaJIT and Lua 5.3+ have support for 64-bit integers but Luau doesn't, since they have complicated semantics if you want them to be first class citizens and a lot of Luau users are children.