Comment by crazygringo | Hacker News Reader

Comment by crazygringo | Hacker News Reader

Parent Full thread

I imagine the conversation between the programmer(s) and management went exactly like this:

Management: So, what can we do about the loading times?

Programmer(s): That's just how long it takes to load JSON. After all, the algorithm/function couldn't be more straightforward. Most of the complaints are probably coming from older hardware. And with new PC's and next-gen consoles it probably won't be noticeable at all.

Management: OK, guess that's that then. Sucks but nothing we can do.

Management had no idea of knowing whether this is true or not -- they have to trust what their devs tell them. And every time over the years someone asked "hey why is loading so slow?" they get told "yeah they looked into it when it was built, turns out there was no way to speed it up, so not worth looking into again."

And I'm guessing that while Rockstar's best devs are put on the really complex in-game performance stuff... their least experienced ones are put on stuff like... loading a game's JSON config from servers.

I've seen it personally in the past where the supposedly "easy" dev tasks are given to a separate team entirely, accountable to management directly, instead of accountable to the highly capable tech lead in charge of all the rest. I've got to assume that was basically the root cause here.

But I agree, this is incredibly embarrassing and unforgiveable. Whatever chain of accountability allowed this to happen... goddamn there's got to be one hell of an internal postmortem on this one.

I can pretty much guarantee that there was no discussion with management like that. From experience, live ops games are essentially a perpetually broken code base that was rushed into production, then a breakneck release schedule for new features and monetization. I've personally had this conversation a few times:

Programmer: Loading times are really slow, I want to look into it next sprint.

Management: Feature X is higher priority, put it in the backlog and we'll get to it.

Former gamedev here. I can vouch for this conversation. The idea that management would ask about slow loading times doesn't seem to be too realistic in my experience.

Normally I'd agree with you but this particularly problem is SO visible, and experienced by everyone, that I have to think management must have looked into it. I mean, it's the loading screen. They couldn't not encounter it themselves, every time.

But I could be wrong. I've only worked with sites and apps not gaming.

Do you really think management at Rockstar is using their product?

Their children are, at least.

Do you really think they're not?

I don't think you choose to become a manager at one of the world's top video game companies if you don't love video games.

Hell, I don't think one of the world's top video game companies would hire you as a manager if you didn't convince them, in interviews, that you understand and love their products by using them yourself.

Using your own company's products whenever possible is generally a pretty important part of being a manager.

I think someone in QA likely might have the scenario you describe, but I think management might only be playing the game 2-3 times a day at most. They're likely trying to keep the studio organized enough to have the game ship decently.

Ughh Every time I want to fix anything I have to sneak it in with something else I'm working on, or wait so long to get approval that I forgot all the details.

I stopped doing this after realizing that all the extra work was just taking time away from my family. The last time I did anything out of work hours was when I told my manager that I was doing some optimization in my free time and he responded "I don't want you working on that in your free time, you could be working on this instead!". I don't plan on doing anything out of work hours again. He left, and now we're hiring more people, instead, since I can't get everything done. Win for everyone.

Oh i never work outside of work hours. I meant that I'll just include fixes to things that are tangentially related to my current user story, or even sometimes not related at all. It helps that the team implicitly trusts my code, so anytime I say "oh and I also fixed this", it's fine. The problem is that if I say "oh I want to fix this" it gets put in the system, and ranked, and just kills my momentum.

At my last job I had that conversation several times :( Our website would regularly take 30s+ for a page to load, and we had an hour of scheduled downtime each week, because that’s how long it took the webapp to restart each time code was pushed. “Scheduled downtime doesn’t count as downtime, we still have the three 9’s that meets our SLA, and there’s nothing in the SLA about page load times. Now get back to building that feature which Sales promised a client was already finished”...

Aside from being generally shameful, the real kicker was that this was a "website performance & reliability" company x__x

Reminds me of working for a company in the 1990's that ran constant television ads convincing everyone their experts could fix everyone's networking problems. OTOH, as an engineer working at the company the file share used for editing code/building/etc, died for an hour or two seemingly every day when the network took its daily vacation.

Many of us, after having run out of other crap to do, would sit around and wonder if the "B" grade network engineers were assigned to run the company LAN, or the ones we sent onsite were as incompetent.

> Many of us, after having run out of other crap to do, would sit around and wonder if the "B" grade network engineers were assigned to run the company LAN

Internal IT is almost invariably a cost center, the technicians providing the service you are selling to customers are working in a profit center. So, yeah, probably that plus be managed in a way which focussed on minimizing cost not maximizing internal customer satisfaction or other quality metrics.

Just because you are minimizing costs doesn't mean you have to minimize costs until you no longer get the quality you need.

I once did get to look an optimsing a bad load time... only because at that point it was crashing due to running out of memory. (32-bit process)

In this case it was a JSON, but using .NET so Newtonsoft is at lease efficient. The issues were many cases of: * Converting string to lowercase to compare them as the keys were case insensitive. (I replaced with a StringComparison.OrderinalCaseInsensitve) * Reduntant Dictionary's * Using Dictionary<Key, Key> as a Set. replaced with HashSet.

The data wasn't meant to be that big, but when a big client was migrated it ended up with 200MB of JSON. (If there data was organised differently it would've been split accross many json blobs instead)

It would also be nice to handle it alls as UTF8 like System.Text.Json does. That would half all the strings saving a fair bit. (I mean the JSON blob it starts with is converted to UTF-16 because .NET)

I had that exact conversation at my old job. They only started listening to me when some requests started timing out because of Heroku's 30 second request duration limit.

Same thing happened with memory consumption. The problem was ignored until we regularly had background jobs using > 4 GB of memory, causing cascading failures of our EC2 instances and a whole bunch of strange behaviour as the OOM killer sniped random processes.

I experienced this to incredible extremes. It was taking our clients two hours to load their data into an external tool that they use, daily, to view that data. This was caused by our use of a data store that was only half supported by that tool. I showed that if we took two weeks to switch to the other data store, their load times would be 30 seconds. It took two years, and a new manager, to get that implemented. Unfortunately, most of the clients are still using the old data store. They haven't had time to switch to the new version...

Alternatively (from my experience):

Programmer(s): Can we set aside some time to fix the long loading times?

Management: No, that won't earn us any money, focus on adding features

That's not how it's been in any of my past gamedev jobs.

I work LiveOps and usually long loading times are something that we would take seriously, as it negatively impacts the reputation of the game.

Reply on news.ycombinator.com