We gave up and switched to vitest.
In my case it was literally a drop-in-replacement for jest, done in two minutes and without a config file. A simple config file only was required a bit later when I added test- and coverage-reporting as junit.xml and cobertura format for Gitlab CI.
Didn't even have to change a single line of testing code (I didn't use Jest's mocking magic in that project though).