Good on them for acknowledging something a little embarrassing and responding with gratitude instead ignoring to try to save some face.
This bug has existed for years and has lead to hundreds, if not thousands of people quitting the game citing this exact reason yet Rockstar did nothing.
I think "shameful" is really more appropriate. It's not like they simply didn't know this existed. I think they have known all along it existed and didn't give 2 shits to fix it.
The game probably has tens of millions of lines of code.
I'm glad they shipped. If they can enhance the startup time, great, bonus for them. Additional credit for thanking the third party that uncovered the perf issue.
I also think it's relevant to note that GTA V has sold 140 million copies, 20 million last year alone, and Rockstar makes hundreds of millions of dollars a year on the title. For a multi billion dollar product to have a known, relatively simple, problem is shameful. I don't know what went wrong, but I don't think it was the competency of engineers or the complexity of the problem that allowed this to fester.
[0] https://nee.lv/2021/02/28/How-I-cut-GTA-Online-loading-times...
Not to take away from the feat and initiative of doing so without access to source code.
There are a lot of excuses for why this wasn’t fixed but no great reasons.
Bugs happen. Serious bugs happen. They will always happen. Shaming people for bugs undermines institutional robustness against bugs because it incentivizes people to hide mistakes. Furthermore, companies with blame-n-shame culture typically fault users when those users make inevitable "mistakes" statistically guaranteed by bad UX design.
In contrast, transparency is good, acknowledgment is good. Rockstar acknowledged fault and rewarded the reporter, which is difficult. That's a positive institutional arc I'd like to encourage, at Rockstar and in the wider industry.
If you stop at "embarrassing", I'm with you. But when you proceed to "shameful", that's quite different and we part ways.
Not fixing a serious, visible, known bug for six years is what’s shameful. There is nothing positive about neglect like that. It’s kind of admirable that they actually rewarded the guy but it shouldn’t have even got to that point.
Totally agree you should not shame for bugs. The shame is for ignoring this bug until one of your customers finally had enough and fixed it by patching the binary. If it was my company I would feel ashamed.
Managerial problems that allow high-impact, low-fix-time, high-prevalence bugs to exist for 7 years are shameful and are in fact poison to an organization themselves.
Don't mistake the shaming for really being about the bug itself here. The problem isn't the bug, it's the organizational culture that allowed such a massive, easily fixed, widely encountered bug to persist for 7 years in a game that has made them $6 billion dollars over the last few years.
Yes, they should be ashamed of this.
> I think "shameful" is really more appropriate.
When you make an inadvertent mistake in the future, karma is coming for you: people will speculate that you "didn't give 2 shits".
This bug existed for a very long time and was the #1 outcry from gamers. It is well documented all over the internet.
Every company I’ve ever worked at and every code base I’ve ever worked on would have made this a #1 priority. If it were a technical limitation of the devs I would agree it is “embarrassing” but this was not outside of their ability.
I simply don’t blame the devs, I blame management. Management should have allocated the resources to fix this truly game breaking bug but they didn’t because money was still flowing. Is it any coincidence that ads played during the 6+ minutes you would wait to get into a game?
> This one escaped, because stuff happens.
No, they mostly just don't listen to their players when compared with other game companies.
And if they didn't care at all given what you say then why would they care to implement a fix for it all these years later?
Sure, ideally there would be someone internal advocating for a fix like this on behalf of customers. But Shameful isn't the right word. It's a miss like any other.
I personally feel like the magnitude of this -- especially compared to how simple the solution is -- makes it a miss that's really much more extreme than most others.
EDIT: Though I agree that "shameful" isn't the right word to use. Rockstar wronged themselves much more so than any other party.
If the fix had been some super application specific super obscure thing, we would all understand.
Instead... the bug is easily describable to anyone who knows what JSON is and the fix is “don’t parse JSON the slow way”.
So I think some of the leniency afforded to the devs may be misplaced. Then again, hindsight and all that.
Would we?
Even if I don't understand what it's doing, devs doing nothing about an O(n^2) algorithm that takes multiple minutes every load is clearly a big mistake.
Even if it's something that has to be that slow, cache the result.
I’m sure GTA has unlimited bugs to look into, but a bug that turns people away before they even start should have probably been given higher priority.
That said, they're all things like "refactor the frobnicator code so it's more obvious what it does" and "make templating system log changes to server". Nothing on my list says "Look in to that major performance issue that literally thousands of customers complain about every single day". That stuff gets sorted immediately.
You risk making discourse difficult when you continue punishing people who are attempting to do the right after doing wrong for a while.
This example will be ammunition in internal debates about how much to weigh product pressure vs engineering suggestions that improvement is technically possible; the next time an eng says "let's look at this slowness one more time, it may be fixable more easily than you think", they might be listened to.
lol without context you'd think something tragic actually happened here.
I can't believe you're missing this huge point.
On the other hand, a few days of doing the absolute bare minimum of quality control doesn't undo a decade of actively hostile practices. They're still very very deeply in the red when it comes to their relations with the playerbase and for good reason.
Paying $10k to the person who documented this bug is a good start, but you need to do a whole lot of work to make up for the impressions caused by that kind of prolonged systemic negligence.
Have you heard the one about the country with the death penalty for all crime? It just leads criminals to maximally try not to get caught, without regard to damage done [i.e. removing all witnesses to even the smallest crimes]. This isn't what you want; even within "negative" relationships both sides lose when you don't have a gradient of responses.
A criminal prosecuted for murder and petty theft simultaneously can't really go around complaining that they got 25 years in gaol for stealing an apple, but if everyone had already known about the murder and dealt with that accordingly it'd be pretty harsh.
(not that fixing a bug is exactly a criminal act, but the general principle remains)
Since switching the file format is probably some work, maybe involving many differnt teams and network protocols, the bug was closed because it too much efford.
And then, whenever somebody complained, the old bug was referred which unfortunately never got a proper analysis.
Sad, but I saw this happening many times in the past in some projects. Since the old and wrong analysis will be in many powerpoint slides and heads, it is incredible hard to argue against or get even time to work on such a problem. Usually, you need someone who just takes some time form other projects without telling anyone and ninja fixes the problem to solve it.
Just as I'm certain that this unsolicited 3rd party fix was used to settle some old scores.
Corporate politics is a blood sport.
This can result in severe consequences for the developers who worked on this particular part of the codebase.
I'm quite surprised they are even going to patch the game because the level of "we dont give a shit" I thought they had about this issue certainly would have prevented them from doing anything now.
A game video company employees artists, sound effect engineers, and etc. They aren't full of programmers combing over the source code for quadratic footguns.
The issue is that slow loading times was a contentious issue with many, many customer complaints (IMHO it was the #1 complaint) and during many years not once did they spend even trivial effort on trying to look into that complaint, and that's shameful. As the tostercx's article shows, if they had ever assigned any engineer a couple hours to look into why loading is slow with a profiler, that issue would have been easily found; the fix as described was a bit more tricky because the author did not have access to the source code, but within a company that would have literally be done within an afternoon by a single non-exceptional person. But they never cared enough to have someone look into it.
The thing is that this loading time issue probably didn't exist at release as it was part of something that got expanded on over time and only got bad worse by the years.
Assuming somewhat linear growth of the data the loading time would have been at least 4x shorter only half the games lifetime ago (3 years) and 16x shorter 1.5 years after release. After that period they would probably not consider there to be issues in those places in the code so nobody really looked or realized anything was amiss there.
What are you talking about? Nobody knew about it? Is that a joke?
This long loading issue was posted EVERYWHERE and was the reason so many people gave up on GTA v. It’s simply impossible that they didn’t know the issue existed.
That they chose to ignore it is the reality.
But I still think people are demonizing too much Rockstar and are trivializing the bug when it's far from obvious. In my opinion, there is a lot of hyperbole statements about this issue when it's more useful to think if your organization could or would catch bugs like this.
I don't work at Rockstar so it's true I have no idea how this bug was handled.
I just have a hard time believing they didn't know about the bug, and I also have a hard time believing they didn't understand that it was one of the chief complaints from their end users (a simple check of any forum or discussion group is all thats needed).
So, I an only assume short of an actual explanation, that they intentionally sidelined a fix for this bug. Why, is anyone's guess. If you had to ask me, management put all devs onto money making tasks like new items and store support, and fixing quality of life bugs even ones as egregious as this one, was a low priority.
Imagine now that 160 Million people use my codebase and have been suffering from this for 7 years.
The problem is that it’s existed for 6 years, affected anyone and everyone, and would have been caught and resolved by basically anyone who attempted to look into the issue with fairly low effort/experience-needed. That is, the problem is that no one bothered to look into it, despite good reason to do so.
I don't have the numbers but would have imagined that still takes the cake.
1: Some dev gets tasked with implementing the system, at the time there probably wasn't that many ingame purchases so that 10mb json maybe was a 500k file tops? So going by the O(N2) factor it would've buzzed by in a second (probably faster since being smaller there would've been more cache hits giving another factor or 2). Being at about a second or less they don't even consider optimizing it since it's not noticed among all other assets being loaded.
2: The content/ops teams adds more IAP over the years (it's a datafile so that devs don't have to implement everything), since they're adding big models,etc they don't notice too big changes and probably just chalk it up to bigger models,textures,etc.
3: Engine devs (those most often tasked or obsessive about optimizations) might be running a cut-down server for most testing (not noticing the huge loading time) and probably presume it's other assets if testing in a full env.
So in reality it's probably just an organizational oversight rather than "bad developers", now considering this public "shaming" they received they might actually place some people in an production optimization role for a while and optimize other parts if it's possible (the question is.. will they find it worthwhile to keep up in a longer period?).
“I want you to improve loading times whatever it takes”
Sometimes you’ve got to go hunting instead of fishing.
Edit: an anecdote from the professor of my college's startup program (a hundred-millionaire former CEO/founder)- "never say sorry."
I think the real advice is “don’t say sorry unless you actually mean it.” People see right through that, because your actions and follow through won’t match your words. But people seem to appreciate sincere apologies that come with substantive corrective action.
Yeah, it's always a tradeoff.
I worked at a non-tech fortune 500 for a few years and was able to see the inside of repair/warranty/disputes. These were very high ticket items and the brand was/is very highly regarded for after sales support.
In the employee handbook for repair support staff (the people communicating with clients about the status of their repair/claim) there was a passage similar to the below:
"Apologies MUST NOT acknowledge anything other than the customer's explicit feelings without written acknowledgment of fault from $MANAGER".
I asked a member of that team to decipher WTF that meant, and they showed me an example email. They were only allowed to apologise in the below way:
"Hi bob, we're sorry you feel our product is faulty".
I always found these weasel apologies infuriating in my personal life.
It’s still awful though. I would hesitate to do business with a company that doesn’t take any responsibility for their own products.
That's more offensive and insulting and time-wasting than saying nothing.
Not apologised at all comes off better than a backhanded apology IMO.
But the below isn't correct:
> Right, so the company is saying "I will never acknowledge our mistakes because if I do we might have to fix them."
I probably didn't word it perfectly in my original comment, but the company did acknowledge mistakes quite often after assessments of the claimed faulty/broken/whatever product (roughly 30% of faulty claims were approved and either repaired or reimbursed from memory). The rejected claims were often taken to court (like 15%) due to the value of products.
I was mostly commenting on the deception of only allowing pseudo-apologies.
Well, this HN hate fest of a discussion certainly bears out that saying! The internet is a terrible place. You will get dragged mercilessly no matter what you do. The incentives are all towards sweeping mistakes under the rug.
Which is BS tbh. Everyone makes mistakes.
I think with politicians and political appointees, it's a little different because that's very much a public trust. Nobody deserves to hold political office, they should instead be deserving of it by being ethical and having integrity.
Amen. And avoiding a culture of blame (taking inspiration from the NTSB) makes it much more straightforward to put systemic measures in place which reduce the impact of those mistakes. People still make mistakes at the same rate, but more mistakes get trapped and more negative outcomes can be avoided.
The vitriol of this discussion shows just how difficult it is to establish such a culture. People really want to see mistakes as evidence of moral inferiority.
And I guarantee anyone who's written code long enough has written plenty of footguns of their own.
I'd say that by talking about mistakes without blame, I think we actually avoid people making mistakes because folks can learn from other's mistakes, and see them as something they could just as easily do, and so internalize the lesson.
as-if that's the worst they could do. Suing the guy, pressing for charges with CFAA or DMCA anti-circumvention. There were far worse ways this could have gone.
Only the prosecuting attorney (someone employed by the government) gets to choose when and how someone is charged with a crime. The victim has no say whatsoever in the matter, and indeed the DA can charge someone even when the victim doesn't want them to, or decline to prosecute even when the victim requests it.
Most of the time when people simply use "press charges" this is what they mean. I specifically used "press for charges" to try to avoid such pedantry. I guess I failed.
[0]: https://www.gigaspaces.com/blog/amazon-found-every-100ms-of-...