- Build middleware (REST API endpoint) in front of the mainframe that was 1:1 mainframe functions. It initially does nothing except auth and routing.
- Re-write/migrate all other software from using the mainframe directly to using the new middleware.
- Work to further restrict anything directly connecting to the mainframe aside from the middleware, including staff's terminal access by building management solutions (e.g. new UIs, new management middleware).
- Once the mainframe is completely isolated, start building drop-in replacements for business segments and then switch the middleware(s) to point at the new solution from the mainframe. This should be invisible to all external consumers. You can "hot test" it by having the API execute on the mainframe + new solution, and checking for 1:1 responses.
The hardest part about the above isn't the tech, it is the internal norms that you're fighting against (e.g. staff have had terminal access for tens of years, and know the commands to get their work done by memory). So you have to create very compelling alternatives using the new API middleware/web/apps, "export to Microsoft Excel," graphing, mobile support, and similar that a terminal cannot directly offer is clutch here.
There was a lot more complexity that that, but that's the gist of it. By the time I left, my team had carved out the bulk of the mainframe's services and had stood up over 50 microservices.
The hardest part about the above isn't the tech, it is the internal norms that you're fighting against
This is so true it's almost painful.
I have sympathy for those people. Not a fan of AS/400, but working in those terminals is incredibly fast once you know all shortcuts. Modern GUIs, especially using all those fancy frameworks, are much slower. If the alternative to the terminal is a new web application, it's extremely likely that it'll be harder to work with.
But one of the easiest way to retire anything, is to stop doing new work on the older systems. Any new project should go on the new tech.
Soon enough you will see the older systems gone. It won't happen in weeks or months, but 1 - 2 years is enough.
At one point the whole banking industry used to de-facto run on Java. Eventually people just started doing new work on Python. These days Java is just another tech there.
Just because you don't invest in a system doesn't mean it becomes obsolete.