There’s an implicit story here which I find quite sad that not a single developer at Rockstar in the prior eight years had the autonomy or bandwidth to discover and fix this themselves.
Every company talks about wanting to have driven curious employees to solve complex issues or challenges but you just get stomped with a boot to your face if you do anything outside your lane or simply ask why.
It’s also worse because I’m starting to realize how lucky I was to be raised as a curious individual from my parents who never went to college but at least were curious themselves about different things (pottery, soldering, gardening).
The quote “Curiosity is insubordination in its purest form” suddenly has a different context when working for a company.
People might start off curious but eventually they keep running into political roadblocks and other nonsense, which slowly beats the interest out of them.
If those are the kinds of people that continue surviving then those are the kinds of people which your environment cultivates.
The funny thing is, all the people who made that decision left the company/project and now we have the option to change it. Can't wait to see what happens next, if the new leadership changes that or not (my money is on "not").
The other funny thing is, I can think of at least two workarounds that only need a day of work and a couple for testing it.
I have this at my current job, I seem to be the only one (not really, but seems like it sometimes) interested in finding out why things are slow/broken and investing the time to fix it. If you're "busy" all the time, but put up with things that slow you down, you're hamstringing yourself by not fixing the brokenness. When only a small number of people take the time to fix things (typically other's issues), that becomes very demotivating.
That would be like being on an all-day ski tour, and feeling too rushed to bother getting rid of the slush on your skis. Yes, you feel rushed because you're moving slower that usual. Taking 4 minutes to kick the slush off will make you much faster on the overall trip. (Sharpening the saw analogy, etc).
Sometimes "busy" is a state of mind, more than it is a reflection of reality. I know each situation is unique, but I've seen this enough times to understand that it's often a form of dysfunction rather than a reasonable reaction to the work situation. This is like when people used to say "we're too busy to write tests". Well, maybe you need to get better at communicating the trade-offs to the broader business, and be a bit more assertive about non-functional aspects that are still important to address.
> I have this at my current job, I seem to be the only one (not really, but seems like it sometimes) interested in finding out why things are slow/broken and investing the time to fix it.
You will quickly shed yourself of this philosophy if you become a business owner. You can test those waters by releasing a non-trivial open source project that gains a non-trivial amount of users.
There will always be at least 100 things that are broken in some way but not impacting the core business. You'll have to pick only 1 to work on. And, if you're unlucky, some frustrated customer may write a blog post about issue #65. An issue that you're fully aware of but haven't prioritized.
Guess what you'll do as a business owner? You'll prioritize that issue since it's now very visible and maybe even throw $10,000 at the customer who took time to do their research; as is what happened here.
> This is like when people used to say "we're too busy to write tests". Well, maybe you need to get better at communicating the trade-offs to the broader business, and be a bit more assertive about non-functional aspects that are still important to address.
Or perhaps the need for tests was communicated and the business made the decision to not prioritize it.
You may not like that decision.
Mind that the decision to "not write tests" is never explicitly stated so simply. No one wants to look bad. Instead that decision is expressed through minimizing its level of priority compared to other things.
Still don't like the decision? Well you have three options:
1) Ignore the stated goals of the company and do what you think is right instead. In which case you're acting against the interests of the company that's paying you.
2) Sacrifice your own free time in order to provide your idea of value to the company (that may actually be valuable! but that the company doesn't value enough to prioritize)
3) Do what the company asks with plausible deniability. Also known as kicking the can. Also known as not my problem.
Sometimes doing #1 or #2 could greatly benefit you. Perhaps that thing you're doing (eg: writing tests) will somehow measurably improve the business in the future WITH PROOF so that you'll get get recognized. Or perhaps there's management shake-up and the new leadership perfectly aligns with your philosophy, etc, etc.
But that's rare.
More often than not your attempt at fixing the company's problems behind the scenes will go unnoticed and unrewarded. And if you do get rewarded, often times it's a pittance compared to the amount of time you invested.
My personal philosophy is I simply do not work for companies that don't align with what I value. For example, to me, it's exhausting and unrewarding to work at a company that doesn't appreciate a GREAT user experience.
Dysfunctional... possibly true but isn't that (unfortunately) more or less the norm in IT? Not to say it shouldn't be fixed but at the end of the day you have 'x' time to ship 'y' features in order get paid 'z' dollars so that the company can continue to survive... combined with "management" often under-estimating with respect to time, requirements and (un)foreseen hurdles.
I guess it's possible that the engineers who were talented enough to create this extremely impressive game in the first place somehow threw up their hands and said "guess it can never take less than 7 minutes to load, best we can do". But it seems much more likely that they were well aware, but never allowed by poor management to go back and fix it. The game was released and printing money, why bother. They're on to the next title.
On the other side, a really good developer has the knowledge and experience to frame the problem in an understandable business context when communicating the issue and options to management. Some of the best work experiences I've ever had were in orgs with such experienced developers and managers. Working together we were often able to solve apparently intractable problems with creative options which would never have been conceived absent the clarity of thought and communication on both sides.
I feel like the "20% time" paradigm where one day out of the week is left up to the developer to schedule (within reason) is the right way to balance the conflicts of interest and still leave room for people to be curious.
Maybe a long load time just wasn't that important for the success of GTA.
In my experience it's the opposite. Most people are born innately curious. It takes intense systemic conditioning to suppress their natural curiosity. For many of us that conditioning begins in the traditional education system and is completed in the workplace.
Yep, the public school system in America excels at squelching curiosity and any desire to learn.
You better tell Congress, the last thing they want is for America to fall into totalitarianism.
Not to be personal or anything, but I went to public school.
I can remember reading "Deschooling Society" by Ivan Illich whilst in the UK's comprehensive school system.
On retrospect, I don't think I realised the importance of schools facilitating socialisation, rather than just learning.
Lots of incredibly talented people worked on GTA Online, it's worth trying to think through why a team of smart developers might not have prioritized this, rather than assuming something negative about them.
Talented people are found everywhere, but hopefully someone with authority over them doesn’t slowly poison the drive out of them.
Companies are successful not because they delight every customer consistently, but because they do just enough to make the customer spend money with them.
After all, I will still buy the next version.
That said given the quality of what they have put out, I'd surmise the issue here is morelikely they don't feel the same pain on load times?
Bandwidth too good, rigs too powerful, dev envs overpowered, etc. to enable developer velocity can often also mask real world problems.
Hell, even hot reload can mean you don't see load as often and are able to easily rationalize as "not that big a deal" - I wouldn't necessarily take it as a sign they are not deeply invested in their craft.
One of the things that I've instituted at my work is our methodology for acquiring test phones for our product (WhatsApp for hospitals, basically). We go to a supermarket and buy the most eye-catching box of the budget phones available in the phone aisle.
We've squashed so many bugs and performance problems this way. It's cheaper, it instils empathy in product decisions, and our customers love the user experience as a result.
Similarly, one of my first big decisions after joining was to wean ourselves from our AWS addiction using Kubernetes. This also allowed us to design a dev env that fits on a modern machine comfortably, but can scale to entire Hospital Trusts in production. Once again: focusing on portability now means we can sell our product on-prem and hybrid, because we designed our developer UX this way.
And given it was data-size dependent, I wouldn't be surprised this was not an issue at the start while someone who's job it was to look for this kind of issue was looking. And the issue appeared after they got retasked to something else.
In fact maybe this is the only kind of issue that you end up with after all is said and done. Just like in WW2 they reinforced the parts of the fuselage on returning fighter craft without bullet holes, because if a bullet hit there, the craft would not return.
I’ve worked around too many startups ruled by developers. They’re being run into the ground because those developers think technical NFRs as defined by them are the be all and end all. They’re not. Those same developers will happily spend months fiddling with the system without making any actual improvements and call it “paying down technical debt”.
Give me a PM (or good engineer) that understands the bigger picture every time.
This is the definition of a flippant comment. Why should society require we become the lord of our own domain or a political chameleon just so that we can advocate for decent standards? Should people who live on flood plains "just move" when the tide comes in? Ridiculous.
Understanding the bigger picture includes understanding the developer's point of view.
> I’ve worked around too many startups ruled by developers. They’re being run into the ground because those developers think technical NFRs as defined by them are the be all and end all. They’re not. Those same developers will happily spend months fiddling with the system without making any actual improvements and call it “paying down technical debt”.
Sounds like those super-clever managers didn't understand what the hell was going on to me. Perhaps they should've held their nose and tried to learn about their product so they can understand and communicate that bigger picture they speak of instead of abdicating their adversarial role and leaving dev teams rudderless.
I was referring to this sentence: you're now telling us that the ideal good engineer understands the developer's point of view but doesn't agree with it. It sounds like you just want to be surrounded by yes men to me. The perfect recipe for a failed startup.
As a senior programmer I respect business priorities and I don't go fixing technical debt for months.
That being said, we the people suck at moderation. See, the problem I faced was exactly the opposite of what you encountered: I've been too accommodating and allowed myself to be micro-managed to the point of a manager telling me how should I do my job; that's when it finally clicked with me on what a toxic workplace have I landed back then.
So neither of both extremes is good. I agree business priorities have to be respected; absolutely. At the same time we shouldn't allow the businessmen to tell us we can't ever fix technical debt.
What makes you think this isn't the case? Business metrics holding an adversarial position over other interests obviously doesn't preclude business metrics defeating other interests.
It's the #1 complaint. The thing that makes this egregious isn't so much that they didn't fix the #1 complain, the thing that makes it egregious is that it's a simple enough flaw that a mediocre developer could have just run WPR/WPA against it and see that 90% of those 5+ minutes was spent in strlen. Maybe it's not their area, maybe they don't know how to fix it, but they will recognize that it's a problem with a likely easy fix.
So even if we completely ignore the fact that it's the #1 complaint, it's still egregious that it's been allowed to go on for this long.
I don't disagree that the "environment" itself could do more to further making improvements outside of the assigned work - initiatives such as 20% time, cafe days, etc. exist in other companies. But I don't put the blame on the curiosity of the people themselves.
If SWEs can only work on features and do nothing else that is still a sign of poor project management. Just accepting that fate is worse for your growth I’d wager.
But idk, I never worked in an environment that encouraged curious people either. I suppose the closest for me was working at Comcast but that was only one manager with a small team of 4 people and no real deadlines.
I feel like two separate thoughts came from replies to my comment. One advocating for engineers to push back and care about their crafts, the other absolving the choices that were clearly made by people as some mechanized process that could never be changed.
The only thing I’m wondering is what the replies would be like 10 years ago on HN.
How much time did perfectly qualified engineers, who could have attached a profiler at any moment, spend sitting at a loading screen spending most of its time in unnecessary strlen() calls? I understand we see this through 20/20 hindsight, serendipity can work against you, etc., but that's a crazy irony.
The players who are still there are used to it so the value is likely near zero. It's the sort of thing that's 100% worth fixing before launch or soon after so you don't lose players in the first place. But the fix loses value every day it goes unfixed because you're already lost those load time-sensitive players. Keep in mind that years after release it still makes Rockstar/Take-Two millions per day (https://www.thegamer.com/gta-5-rockstar-2-5-million-per-day/) so I guess a lot of players just don't care.
For the record: I'm surprised anyone is willing to tolerate that shit.
That does sounds like awfully like confirming that indeed no-one was curious, because they were too busy chasing KPIs or whatever other metrics management were using.
Understandable given the heavy crunch it sounds like Rock Star works in. If you’re a developer spending 60+ hours a week working on it, you’re not going to play that in the few personal hours left over.
I feel like history is repeating itself here (at least in regards to death marches):
https://en.wikipedia.org/wiki/Erin_Hoffman#%22EA_Spouse%22_b...
That's because "curiosity" translates to "more work" at #BigCo. And people work at #BigCo so that they can comfortably do as little work as necessary to maintain their lifestyles. This is a hard rule for any organization. So if you want what you're describing, it will only be found at small startups where everyone has skin in the game.
The project managers could put load times as a priority but they genuinely dont see it as a problem or dont care at all. It happens all the time. There is a lot of unoptimized software.
Perhaps the issue comes from the top: the project managers are supposed to only do things that increase revenue and all other things are not rewarded at all? Perhaps executives only care if they hit quartely results? (Short term gain traded for long term loss - the game is made for whales who are so invested that they will play even when it is technically bad - so milk them now with new content for whales and ignore everththing else, who cares that new players will be discouraged by bad load times).
In my opinion if nobody plays own game / eats own dogfood, they dont even see its problems. Blizzard is at this stage now - some of their games barely work. You buy Starcraft 1 through the launcher... and it doesnt work(I think it works now, but it took them 1 month to notice it and another to fix it). Heroes of the storm - downloaded a 137mb patch every run - it took them half a year to notice it and do something with it. It is like nobody cares about the brand at all. Game company where your games dont work.
IMHO a problem with people - that comes from the top. They want only story points for things that bring money. The project managers either dont care, or cannot deal with other problems at all. I bet they dont care. This happens in many companies - there were few people who tried, they got burned for that: terminated or quit -> so now nobody cares. "The game is good enough to still sell some skins to the whales, so why bother".
Look at the results: they got an external comment what to fix. Did they launch their own internal project to optimize the game? Did they check for other things that make it load slow? It looka that someone else did a partial fix that is "good enough" so everything is back to usual and nobody cares about load times.
Designers' unnecessary desire to delight users staves off the more Kafkaesque user experiences, and we all live in the gap of cops' unnecessary (occasional) sympathy for (some) nominal outlaws (we're all nominal outlaws).[0]
When you mentioned curiosity I thought of a firm where colleagues scoffed at me as I rummaged through conference room drawers, saying, "you're not going to find anything interesting in there." I'm curious about cupboards as well as log traces -- an almost ubiquitously "eyes-glaze-over" boring artifact of modernity that, alongside popularly mind-numbing "code", I am equipped to find interesting.
Is it any wonder, then, in this workplace where such innocent, cheap, curiosity were scoffed at, I was not too long after threatened with the extraordinarily archaic (in our at-will state, at least) legal notion of "insubordination"?
It's a shame, because insofar as "all ambiguity is resolved by actions of practitioners at the sharp end of the system,"[1] the liberated curiosity of individual contributors is often the only "crack in everything" that's "how the light gets in."[2]
0. "With two lines of a man's handwriting, an accusation could be made against the most innocent." (Françoise Bertaut de Motteville)
1. https://how.complexsystems.fail/
2. "Anthem" - Leonard Cohen
In short, every job needs to be able to have autonomy to enact on the ground solutions for what their immediate team needs. You shouldn't be penalized for going off the reservation and coming back with a creative solution. The problem is corporate structure for the past few decades has been about rooting out this autonomy and anything opaque about job responsibilities. There is a lot that's been done that would need undoing to change those environments, and imo you are better off finding another workplace than bringing about the monumental change required. A mature corporation is like a boulder tumbling down a mountainside; you can hardly change its course much less stop it short of where its heading.
I ALWAYS ask questions like "How did this come to be, How much did this cost, who designed this and why" etc.
My brother gets annoyed that A) I ask and question everything, and B) Laments he doesnt know as much as me.
However, this is spurred by my ADHD... and I at times wish to be able to have a more precise control of my focus.
Perhaps this might not be appropriate in the gaming industry... and who knows if Rockstar don't already have this.
However, the GTA loading time is horrendous (and I had a spectrum sinclair when I was a kid) - surely there must have been a work item, ticket, bug reports, or whatever, for this.
For 19 year we just looked at it in defeat. For the last year, bit on and off. Starting last monday it crashed every morning for a user sitting next to me, a big nasty german boss (Im French).
I sat down, read his logs, saw it failed on ImageList creation again for some fucking reason, looked how many of these things we used and what they did. 2 hours later this 20 yo mystery bug dozens of devs had noped out of was fixed. We fully get it now. Wasnt even a matter of priority, it took 2 hours of looking at it. A lunch break basically.
I dont have a sort of morale of the story or whatnot, but well, Rockstar would have figured it out eventually. Just need a nasty fucker next to you whining abt it everyday.
But seriously, look at tier-2 investment banks: we're nicer and less stressed than the Goldmans of the world but still make more money than most. And problems are interesting I find. But you need to like handling money and not "changing the world" in any shape or form.
All of the Source games (HL2, Portal, Alyx, etc...) have arguably bad load times. They all sold great! GTA sold great even with it's previous load times.
It's just not a priority for most teams. In Rockstar's case it happened to be easy to fix but in most cases it's not so easy because the engineering team didn't want to deal with it.
AFAIK, Source, and maybe Unreal/Unity are built in such a way that by default, if you need to start the level over you have to re-load it entirely. They have mutable state spread all over the place and have no way to mark what's important (needs to be saved, what's not (can be zeroed), and/or they haven't bother to put all mutable state in a single places so either it can be restored from memory or loaded by just loading a tiny bit.
There's zero reason HL:Alyx should take 15-30 seconds to restart a level you're already in when clearly less than 1k of state has changed since you started the level. But, fixing that would require a major re-design of the Source engine. Slow load times don't affect sales. So almost no one does it. Easy just to let each gameplay programmer do whatever they want and to restart just reload the level.
I put up with it for some games (the ones mentioned above for example). But I've also quit a few recently. Games where you die a lot and to add to the frustration it takes 1-2 minutes to load again only to get killed in 5-20 seconds. That frustration means trial and erroring to figure out what to do gets a punch in the face and so I stop playing what might have been a good game otherwise.
All I can do is prioritize and hope nobody gets too unhappy that their pet features is never going to happen.
1) It has to be alright to make miscellaneous changes like this, and
2) It has to contribute to getting promoted.
The second point is IMO why random QoL changes like this are rare in large corporations.
I've also seen a single-minded focus on pushing cards across a board, which seems to typically correlate with higher levels of a dogmatic SCRUM process.
Pushing cards is great, but not when it prevents the various little things that have to happen to cultivate quality software. Especially not when there's a gatekeeper prioritizing these things into oblivion.
When I see a pain point that looks soluble, I take a crack at it. Sometimes it works, sometimes it doesn't. Sometimes I get recognized for it, sometimes I don't. One, relatively minor (to me) thing, ended up being a big business win and my manager made sure I got a fat bonus for it. Other things have seemed more like a win for me that nobody but 2 of my peers ended up caring about.
#1 is true to the nth power though. I've seen places where if you do something even one millimeter outside of your silo, it will come up as a negative in your performance review. As in, if you write a 5 line perl script that makes you significantly more productive, do not share it as you will be asked time and time again why you wasted your time on something outside of your job description.
I think something missing for the context of TFA is that the gaming industry tends to be much more deadline focused than other industries. In theory you might be free to "make miscellaneous changes like this" but only if things aren't already behind schedule, and things are always behind schedule.
This sounds like something I'd see in a Dilbert comic. Do you mind sharing what industry you work in? Perhaps you saw this in old-fashioned industries like banking or insurance? Have you experienced this in tech?
That sort of thing also varies with perceived team performance. If your team is perceived by upper mid-management (or higher) as being "good" then individuals on the team have more freedom. If your team is perceived as being "not good" then every deviation from standard practices is clearly the cause of the poor team performance.
It took someone who didn't know what that code looked like being curious about what was actually happening.
I imagine that even if you were on the GTA team, you'd most likely think the long loading time was unavoidable due to some complexity you were not aware of.
Great work to solve it from outside of the company.
Maybe they saw sscanf and assumed it is normal that it takes time, but more likely there were so much stuff to do and somehow no one checked