The linter/typechecker should add in the inferred types explicitly. If you want to refactor, it's up to you to delete the types. Then a re-run of the linter/typechecker can re-check and add them back in.
For bonus credit, we should also automate the removal of unnecessary explicit types. A de-expliciter program should be able to go delete all explicit types which are inferable.
Personally I really like the spot readability of explicit types. I'd rather it be in the code, see the ripple out of changes to the codebase.