Comment by greenyoda | Hacker News Reader

Comment by greenyoda | Hacker News Reader

"There are no easy fixes for the fact that a 30 year old mainframe can not handle thousands of simultaneous queries. And upgrading all the back-end systems is a bigger job than the web site itself. Some of those systems are still there because attempts to upgrade them failed in the past. Too much legacy software, too many other co-reliant systems, etc."

30 year old (1983) mainframes and databases were designed to handle large transaction loads. For example, airline reservation systems and banking systems were built on them.

And upgrading a mainframe (at least an IBM mainframe) to a faster mainframe isn't such a daunting task, since all the code from 30 years ago (or even from the 1960s) is still object-code compatible with the new machines - you can make it run even if you've lost your source code. There's still lots of 30 year old (and older) Cobol code running on mainframes today.

I agree that re-writing the 30 year old software would be hard, but simply getting it to run faster could probably be done just by spending money on the latest mainframes and disk drives. But if nobody ever did a load test on the site, they wouldn't have known that they had to do this. They probably just thought: "Oh, we have to write a web site that talks to a bunch of databases, how hard could that be?" (By the way, they could have written test code to do a load test on those legacy systems without even having a web site running. In retrospect, that's the first thing they should have done, and it would have shown them that their critical path wasn't the user interface.)

In theory you could at least improve the hardware the legacy systems is running on. In practice that may not have been anywhere near enough to ensure success.

1. In this case you would've had to start benchmarking the performance of the legacy systems early, far earlier than when they started development in March. If you determine that hardware upgrades are needed, then you'd need to initiate procurement and upgrade projects at one or more of these other agencies. Projects like that may not necessarily be quick to implement.

Maybe the physical space can't accommodate new hardware. Maybe there isn't enough budget to do an upgrade like that. Maybe there aren't enough personnel resources to plan and implement an upgrade of that scale quickly. Maybe those organizations are just barely keeping their heads above water with the way those systems are currently functioning. Maybe they don't even have a handle on what their hardware configuration is. I know someone who worked at an agency where they had to start unplugging stuff to figure out what server did what.

2. This is all making the assumption that the data in those systems is correct and well-formed, and the business logic in those systems is free of bugs. Maybe you get the database schema and find out that A. It's out of date, B. There's no data dictionary, and C. There's 250k lines of business logic tied up in undocumented triggers. Good luck.

Load testing might just be the tip of the iceberg in situations like this. But bottom line is, if the people leading your project don't even think to start looking into this kind of stuff very early on, you might be screwed before you even started.

Having integrated with these kind of systems, they aren't up to internet-scale traffic. Apart from raw capacity they are usually architected for transactional consistency, not large indeterminate numbers of concurrent users. This is why we have things like ESB's and async enterprise integration patterns.

I'd be surprised if those weren't already in place on this project but bet it's really poorly done. My money is on a totally manual test process which means a deployment misses loads of cases and takes weeks and where a load test is 10 guys in India hitting f5.

You make very good points about early load testing. Unfortunately, as detailed in so many places including the OP, the integrator, the government's CMS, clearly didn't have the expertise to realize this. E.g. any vaguely competent at software development organization knows you have to freeze the requirements well before the week before the launch, and not make major changes less than 2 months before (the heavy registration process instead of allowing window shopping).

Reply on news.ycombinator.com