There’s a few papers in this space but it’s still early days, both using Formal Verification to test outputs of LLMs and using LLMs in formal verification itself

Examples of both:

https://sumitkumarjha.com/papers/2023_ICAA_LLM_Dehallucinati...

https://mathai2023.github.io/papers/28.pdf