Hacker News Reader

Comment by jstx1 | Hacker News Reader

> One former Tesla engineer, who spoke on the condition of anonymity to candidly describe the matter but was not involved, said Tesla engineers would have trouble capably assessing Twitter’s code. Distributed systems, the large-scale and spread-out network that Twitter is composed of, are not the automaker’s specialty, the person said.

> The “idea of Elon being flanked by his Tesla engineers reviewing Twitter code is laughable,” the person said.

It is laughable, and it's just another signal that Elon has no clue what he's doing. Imagine how farcical the whole thing felt to anyone involved who isn't him.

View on HN

> Distributed systems, the large-scale and spread-out network that Twitter is composed of, are not the automaker’s specialty, the person said.

You can giggle along with the anonymous source who wasn't involved if you wish but Tesla collects a rather large amount of telemetry and has all manner of micro services crap internally just like Twitter.

I imagine the Twitter codebase itself is a farcical Rube Goldberg machine. Insert your own how many engineers does it take to change lightbulb jokes.

  As part of that process, some engineers received a calendar invite Friday, telling them: “Stop printing, please be ready to show your recent code,” a reference to engineers being asked to show the code they had written in the last 30 to 60 days on their computers.

Sounds like he knows exactly what he is doing. I'd do just that. For an unfathomable amount of the people working at twitter the answer will be zero lines of code.

> For an unfathomable amount of the people working at twitter the answer will be zero lines of code.

True, but an accusatory calendar invite still isn't a good way to do this. He could find out himself from the commit history/JIRA tickets, with the additional benefit of not putting people on the defensive and making a poor first impression. Then again, relationship building isn't really his style, so this doesn't surprise me.

Heaven forbid the people who show up with blank printed pages be made to feel defensive. Relationship might suffer so much they might just have to break up permanently.

The problem of course is not annoying the underperformers, but you want to avoid annoying the valuable engineers who get things done and keep the company running. There's plenty of demand for engineers -- if they want, they could find another job in a week where they don't have to deal with hostile management. Treat people well. Assume the best and let them prove you right, rather than assuming the worst and making them prove you wrong.

Trust me, the absolute last thing a valuable engineer is annoyed about is to finally have some time to show off some of the things he did.

Musk clearly dislikes fragile corporate cultures. He's going to destroy and rebuild Twitter's culture, that is very clearly his intent. I don't know how much more obvious it could be. He'll rebuild it in a way that he prefers companies to operate, staffed with the kind of people he prefers (whether anyone else likes that or not).

I mean, you're not wrong, but I was more replying to the "I'd do just that" sentence which seems to imply the scorched earth approach is a good thing. We don't need people going around emulating Musk.

Of course you don't. ;)

Exactly this.

> You can giggle along with the anonymous source who wasn't involved if you wish but Tesla collects a rather large amount of telemetry and has all manner of micro services crap internally just like Twitter.

Yes, collecting telemetry is quite equivalent to making a globally-distributed, many-to-many messaging platform available to everyone in real time.

> For an unfathomable amount of the people working at twitter the answer will be zero lines of code.

One of our most productive engineers left the company with negative several hundred thousand lines of code impact over his career. What would he print out?

Is the globally-distributed many-to-many messaging platform web scale?

> One of our most productive engineers left the company with negative several hundred thousand lines of code impact over his career. What would he print out?

Certainly deleting twitter code is even better than writing twitter code. Zen like nothingness is really neither here nor there, now is it?

Depends. I haven’t written a line of code in a month either. That’s because I’m planning what code needs to be written in the next five years. I’ve found a lot of good engineers spend more of their time in Google Docs than they might in their IDE.

The really bad ones spend all their time there.

> What would he print out?

Diffs.

The old and the new.

> For an unfathomable amount of the people working at twitter the answer will be zero lines of code.

And what’s the average salary of these people? $15k/mo? $20k/mo?

> the answer will be zero lines of code.

Ugh, I bet you're exactly right..

> I imagine the Twitter codebase itself is a farcical Rube Goldberg machine.

I'd presume Tesla's codebase is exactly the same.

Absolutely correct. :)

Tesla engineers don't just work on in-vehicle code though. The servers and systems needed to implement Tesla's telemetry and software updates would be in the same ballpark of expertise, albeit with much less scale and different requirements than Twitter.

But Musk got started by confounding x.com/PayPal, which does have very similar engineering needs as Twitter. It's not like Musk is a newbie to this stuff, people pretending that he's clueless in this domain just want to feel schadenfreude.

All stories from the x.com/Paypal days say that he was entirely clueless during that time. The board removed him.

We've gotten quite a few public comments from Musk on what he plans to do with Twitter, and they indicate total cluelessness from a product and engineering point of view. There's nothing to indicate an appreciation for the unique challenges Twitter has in balancing free expression with appeal for advertisers (or even that these needs might be in conflict), or even on the subtleties in what free speech even means for Twitter. The acquisition seems motivated by politics and impulsiveness rather than any coherent vision for the product.

A rocketing share heals a lot of wounds. His abusive management practices might fly at Tesla, or his vision oriented startups. There's ample room for doubt his tactics will get results on a turnaround job like Twitter.

I’m taking a “let’s wait and see” approach to Elon’s Twitter.

It’s much more comfortable than all the public kvetching

> and they indicate total cluelessness from a product and engineering point of view

Gotta say, this sounds exactly what they said about Tesla, and SpaceX, and Starlink.

> But Musk got started by confounding x.com/PayPal, which does have very similar engineering needs as Twitter.

But PayPal was running Oracle on IBM bigiron machines even after Musk left them.

> But PayPal was running Oracle on IBM bigiron machines even after Musk left them.

Oracle on IBM big iron is a pragmatic (but expensive) way to achieve a highly vertically scalable database platform. Not trendy at all but also not a bad technology position to be in, even now.

What telemetry do cars send Tesla? The EDR system which stores all telemetry data is not suppose to leave the car.

https://edr.tesla.com/

They definitely send data for evaluating safety score and training FSD.

From Musk’s POV, a bunch of solid, well-rounded, and experienced engineers you trust might be enough to tell you “this guy is full of shit” when talking to new people.

Maybe it’s less about code quality and more about eng quality.

I think that's all besides the point. Does anyone think any of Twitter's problems are related to code quality? Literally go read the unit test suite and you'll what it does and if it works. If he just wants to cut heads, cut them from verticals you want to exit (whatever they might be). What the hell could he possibly achieve with this stunt? It feels more like the old cliche about your first day at prison where you should pick the toughest guy and beat him up to establish dominance.

I guarantee Elon is surrounded by yes-men.

That's not how you build a spaceship.

Eessh, I hope not.

I guarantee you he isn’t.

Then maybe he’s always right. As chief engineer at Space X for example, if he was an idiot or surrounded by idiots do you really think he would be succeeding?

> As chief engineer at Space X

If that line itself isn't enough to tell you he's full of shit you have hopelessly fallen for the myth of his Tony Stark-like brilliance. Do you honestly think he contributes any meaningful engineering work at SpaceX? If he didn't own SpaceX, what about his history suggests he could even be an engineering intern there?

SpaceX is successfully solving engineering problems. If Elon is surrounded by yes men, that means he's always right. That he probably isn't always right implies he doesn't surround himself solely with sycophants.

> SpaceX is successfully solving engineering problems. If Elon is surrounded by yes men, that means he's always right.

Alternatively, it means the organization is resilient enough to handle it.

The US government has successfully accomplished all sorts of things while having various shitty people at the helm.

It's possible to be a sycophant who manages the politics of appeasing a narcissist while also solving problems on your own and letting said narcissist take the credit.

If you’re the kind of narcissist who can attract that kind of sycophant you’ll have no problem getting investors. And no problem getting employees either.

I'm glad somebody got my point, lol.

>If that line itself isn't enough to tell you he's full of shit you have hopelessly fallen for the myth of his Tony Stark-like brilliance. Do you honestly think he contributes any meaningful engineering work at SpaceX? If he didn't own SpaceX, what about his history suggests he could even be an engineering intern there?

Every day there are a lot of software developers on this very website (in this very thread) and elsewhere, with no professional engineering background, nor uphold similar standards and professionalism, who refer to themself as engineer.

"Computer engineer" and "mechanical engineer" have very different meanings when it comes to liability, certifications, etc.

Mike Griffin was the real chief engineer at SpaceX...

https://en.m.wikipedia.org/wiki/Michael_D._Griffin#Career

They guy didn't even work at SpaceX.

Please explain how it's possible to be chief engineer at SpaceX without even working there.

Read his career history and do a little digging. Mike was the government side of Elon's circus.

Yes? Many, many people who don’t work for musk or have a financial interest in him being competent have said in interviews that he has a deep engineering understanding of the rockets and especially their engines.

Most people have shit understanding of rockets. There's plenty of otherwise very intelligent people that do not have any grasp of rocketry. A competent amateur rocket enthusiast can easily sound like a "rocket scientist" to someone otherwise uninvolved with rocketry.

Additionally, anyone being interviewed about Musk can't be said not to have a financial interest in him. Musk doesn't hang out with nerdy rocket scientists in his spare time. He rubs elbows with fellow rich people.

It doesn't take much actual knowledge of rocketry to sound super knowledgeable about rocketry to someone with no knowledge of rocketry. Musk isn't a nincompoop but I've yet to hear anything out of his mouth about rocketry that suggests he could intern at his own company, let alone be seriously considered their chief engineer.

The is such a common thing people say, that there are lists of reasons why you are wrong: https://www.reddit.com/r/SpaceXLounge/comments/k1e0ta/eviden...

Watch elons multiple hour long interviews with The Everyday Astronaut about rocket design and tell me with a straight face that he has little knowledge about rockets.

Or don't because it will destroy your narrative and you would rather live in your own bubble of delusion.

May want to read this if you're interested in having your opinion changed by actual testimonials rather than just going with the tribal Spaceman bad line. https://www.reddit.com/r/SpaceXLounge/comments/k1e0ta/eviden...

The downvote answers that lol

lol that’s what Gwynne Shotwell is for

who is a yes-man according the the post we're all replying to. lol

Indeed, Elon knows how to hire good people, the kind who can assess others in their area of expertise.

What a weird thing to say. I can evaluate any code. If it’s code I’m unfamiliar with it takes longer, and the result is more variable, but it’s far from impossible.

It’s certainly more reliable than the evaluation of the person whose job depends on it being evaluated as correct.

Given 24-48 hours in a gigantic tech company’s code base made up of dozens of teams and hundreds (thousands?) of logical services? No, I couldn’t tell you the “quality” of the code.

Someone who doesn’t understand engineering at that scale would presume it’s the structure of the if statements. And sure, if that’s what you’re looking for, print out some lines and let a person lacking all context take a look.

> Given 24-48 hours in a gigantic tech company’s code base made up of dozens of teams and hundreds (thousands?) of logical services? No, I couldn’t tell you the “quality” of the code.

You could probably call out the bullshitters, though. They aren't refactoring the code base.

I would guess anyone calling someone out 48 hours into a huge codebase is themselves the bullshitter. Maybe if that 48 hours is work time and presentations/walkthroughs and the person catching up is a seasoned director (they tend to have to do this quick ramp of huge projects more often than an engineer). But no one is going to look at Twitter's code base and make any meaningful conclusions in 48 hours.

Agreed. I had the opportunity to audit a FAANG codebase (many many repositories) a while back and while I'm a lowly engineer, I was able to start understanding quite a bit after a few weeks. Obviously there's no way I could contribute or make recommendations in that short time, but obvious smells start working their way out after exposure. The clarity only increases with the amount of time spent immersed in it.

Off course you can have an opinion about any code. It will just not always be correct.

I'd like your opinion on the MGLRU code they just merged in the kernel. It's a self contained piece of code, you can look at it and print it on paper.

Or the folio patchset. People had a variety of reactions to it, whole lot of energy was spent evaluating that code (which in the end also made it in)

The person whose job it was to make the patch and who has interest in it being correct is not necessarily less reliable than a random HN commenter

You would probably think otherwise if it was your code and some other random commenter with no domain knowledge evaluating it, wouldn't you?

> If it’s code I’m unfamiliar with it takes longer, and the result is more variable, but it’s far from impossible.

Yeah, I'm sure it's of value to have Tesla engineers come in and spend the 3 months to a year getting enough context with the code base to be be able to produce useful and informed critiques.

What hubris to think that your system is so complex that other engineers couldn't understand it.

Also what hubris of the Twitter engineers, who seem to be in this thread, thinking anything they built was that special.

Your video player somehow managed to be worse than Reddit, the perennial joke of web dev.

Not to mention “something went wrong” again and again, maybe by the 4th reload would a linked tweet actually show up for me.

I’m on 25 gig symmetrical fiber with <10ms to my local AWS hub, FWIW

> Not to mention “something went wrong” again and again, maybe by the 4th reload would a linked tweet actually show up for me.

That seems like a purpose-built feature to punish non-Twitter users to induce them to sign up for Twitter and/or use the Twitter app instead of the web UI. It's not a bug but a misfeature.

No, that happens on the Twitter app too.

Video player works fine for me, and I'm on 20 gig symmetrical fiber with 8ms to my local Azure hub. Maybe check your firewall settings?

Oh please, that's a ridiculous assessment of what was said. Coding giant distributed web apps is a very different domain than coding for cars. I wouldn't put much stock in Twitter engineers assessing self driving car code either.

"Tesla engineers don't just work on in-vehicle code though. The servers and systems needed to implement Tesla's telemetry and software updates would be in the same ballpark of expertise, albeit with much less scale and different requirements than Twitter."

-- hn_throwaway_99 https://news.ycombinator.com/item?id=33389128

Yes, both were me. I'll respond since you seem to be implying my responses are incongruent.

Here I was responding to a comment "What hubris to think that your system is so complex that other engineers couldn't understand it." Which is still a bullshit assessment of the argument (which the original comment was about) that you need expertise in specific domains to be able to evaluate code, which I agree with.

My comment you quoted above is that I think that there would be a subset of Tesla engineers (just not the ones that, for example, solely focus on driver systems) that would overlap with Twitter's domain.

>Oh please, that's a ridiculous assessment of what was said. Coding giant distributed web apps is a very different domain than coding for cars. I wouldn't put much stock in Twitter engineers assessing self driving car code either.

You don't need to be a part of any particular domain to effectively audit code.

This is true to an extent. At the extremities, a software system can be audited without any previous domain exposure: architecture, and implementation means (language, deployment, etc.)

It can be difficult to critique choices, specially in system architecture, when domain requirements are presented or falsely asserted as 'dictating the choice', but this can be addressed by directing conversations with stakeholders to nail down the conceptual model that has been implemented in architecture style x, and then the conversation is on more equal footing.

Auditing language usage, libraries, development process, etc. are much less domain specific.

Really the only way I've been able to make sense of it is if he brought people in to produce an audit of changes made in the last couple months. IE he knows he doesn't know how to get that information, but he knows he has people at Tesla who do.

They wouldn't have to understand any of it, just be able to identify where and when changes occurred, for later review for any last minute shenanigans.

I'm not personally convinced this whole incident even happened, but if it did, this is what I think would make sense.

What hubris to think you can wander into a system as complex as a $44bn tech company and draw any meaningful conclusions about it at all in a weekend. I mean, in my world we have some regular "tech reviews" with our Sr. Directors/VPs at a pretty abstract slides-and-diagrams level, and even that takes weeks of prep leading up like 4+ hour presentations. For something with only a glancing relationship to code level reality. I have occasionally seen my work reflected in these reviews, and the game of telephone that produces them is quite lossy.

Any competent engineer can come to understand most any part of the system, but it's going to take minimum several weeks to understand at a level where you can seriously challenge the domain experts about what's really going on in a little 0.01% corner of the engineering shop.

The hubris is thinking Tesla engineers (who are offered worse pay and wlb relative to Twitter) are coding masters of every domain. I won't even snipe at their own software here.

It takes a long time to onboard to a big mature infrastructure that is serving hundreds of millions of users for people moving from a similar company, even longer if your day job is whatever Tesla engineers are doing.

And that's onboarding, not pretending to be an expert which seems to be the role they're taking on.

>The hubris is thinking Tesla engineers (who are offered worse pay and wlb relative to Twitter) are coding masters of every domain. I won't even snipe at their own software here.

Who is thinking that?

I think you think that the GP is thinking the Tesla engineers are thinking that :D

On one hand, you can usually safely ignore 99% of code in a system that does the plumbing in a system and focus on the few key algorithms. It wouldn't be a stretch at all to take one of the self driving AI guys and have them evaluate say, an ML classifier used to flag spam/abuse content. That would be well inside their domain.

> It wouldn't be a stretch at all to take one of the self driving AI guys and have them evaluate say, an ML classifier used to flag spam/abuse content.

Facebook has a years-old bug with their ML classifier used to flag spam/abuse content in Facebook Groups. Its exact behavior has morphed over time as they flailingly attempt to fix it, but for several straight months that behavior was "posting to a Group via the Graph API fails with 'unknown error' if it includes a dollar sign". This in an API used by tens of thousands of apps and many millions of users, at one of the richest (in both money and programmers) companies on the planet.

ML is often a black box "computer says no" scenario with little meaningful ability to debug.

Key algorithms to do…what, exactly? Why not just read the documentation on that instead of having people bring you a random sample of their code on paper?

Who said they're evaluating the code? Maybe they're evaluating the people by asking them questions about the code they've just written. That's a great way to find out who's full of shit.

Why the printouts then?

So they can have the code on hand while interviewing them, ask about the code, why they made a certain choice, what other options were available, etc. Have you never had an interview with code printed out or otherwise provided for discussion?

Usually we both have copies on our computers.

> That would be well inside their domain.

Would it?

>Would it?

Would it not?

Knowing enough to call bullshit is way lower than knowing enough to contribute meaningfully to a codebase.

Even just having access to code while someone talks I can prove with much higher value questions on what they are saying… because even if I’m asking something dumb it prompts a deeper explanation.

Pay and work life benefits are generally independent of technical competence. But I agree with the rest of your comment.

You don't need to be a "coding master of every domain" to recognize shit code.

Is there any reason to suspect that there is shit code at Twitter? Or that that is in any way related to the problems that Twitter has or the "problems" that Elon said he wanted to fix?

There isn't any reason to expect that there isn't shit code, either.

If you spend $44 billion dollars on something, you're going to want to look it over extremely well.

In fairness there’s plenty of Tesla engineers that used to work across the bay at Twitter.

Did they bring the experts in autonomous vehicles?

It makes sense because twitter is mostly written in scala and Tesla's engineers are scala experts right? ..... right?

At the moment, my default position on all stories out of Twitter is that they didn't happen unless confirmed officially by Twitter. And the more nonsensical the story, the firmer that belief is for me.

Well it's not like Twitter will ever officially state: yeah they just walzted in here and did a bunch of moves that made absolutely zero sense, it was a shit show.

The primary reason enterprise distributed systems software are complicated is that most code is hidden behind teams of teams of teams of hundreds of mediocre people, not inherent complexity.

Heh, I’m still fairly confident that our 100 man software project would proceed much faster if the team was cut down to the best 6 people.

Isn’t twitter a pretty simple thing… db with writes and reads… it was maybe still is rails? I don’t know MySQL , Pgsql have come a long way and most of the hard parts would be scaling these services…. Keeping good caches in memory with redis or memcached… not really sure I see twitter being all that crazy other then it being so many people and so many years of edge case handling they would be hard for an outsider to follow … but it’s not like it was handling a complex UI or enterprise style hacks for a handful of worth it customers…

Would be really interesting to see the code base for sure and definitely exciting to modify a code base handling so much traffic… not sure I can recall a fail whale in many years so maybe they did something right or maybe fearing the whale set the large organization into stone code fear

Contrast that with Tesla and ever car is requesting data from Tesla servers and many many different complex interactions with much more complex engineering to accomplish a task and id imagine the Tesla engineering to be pretty impressive… but who knows it probably does not handle the same amount of read or write traffic…

I think you're massively underselling how complex it is to run something at scale. It's not a matter of just making the database go fast. Just using the database as an example (even though there's way more to it than just the database), you can scale a database a bit by sharding or running it as a cluster or what-have-you, but you can only shard so wide before you run into other problems which are much harder to solve. Running at a massive scale requires rearchitecting almost everything that came before.

Twitter is just a ticket booth. Had over request, get tickets back.

Except the ticket booth is in the middle of Times Square in New York and it needs to serve millions of people per second and be instantly accessible around the globe.

The complexity is in the infrastructure, not in the complexity of what they're serving.

Hahaha. No. They switched to the JVM eons ago. Also, you are severely underestimating how difficult it is to build and run something like twitter at scale.

"It's just a simple matter of programming"

Tesla has millions of cars in service and Twitter has millions of users. Same level of complexity!

14 months ago a lot of people said it was “laughable” that Tesla thought they’d have a working prototype of a humanoid robot in under a year…

I’ll take Tesla engineer’s over 95% of the industry any day.

No. Most of us correctly noted that no matter how much effort Tesla puts into Optimus, it won't be as good as Honda's Asimo demo for years.

Let alone Boston Dynamic's demos. Or Disney's stunt robot.

-------

So we're mostly laughing at the wasted effort. Of course it's possible, other companies made humanoid walking robots like 20 years ago.

The other questions, like how to actually make money from them, remain unanswered.

The same way Musk’s other effort make money: government subsidies.

The differentiator with Tesla’s bot is not in the way it walks, but the way it learns and “thinks” - Tesla engineers are building Artificial General Intelligence into their bot:

https://en.m.wikipedia.org/wiki/Artificial_general_intellige...

In a few years I think even the general public will be able to understand the difference between Asimo and Optimus. It will be remarkable to see Optimus performing tasks independently.

Update: it’s fair to be skeptical, but bear in mind the concept of a desirable EV was an “I’ll believe it when I see it” until Musk’s sheer force of willpower made it happen. He delivers later than he promises, but he does tend to deliver.

That claim has about as much behind it as all of the full self driving claims.

The self driving claims that the full released version would be in the 2010s.

"Full self-driving will come in n+1 one years - I promise"

I'll believe it when I see it. In the meantime his team still can't even deliver on Tesla's self driving promises.

If they actually have an AGI up their sleeve then spending time on hardware seems like a waste of time.

Oh, I missed this. It now makes sense. Thank you!

No, people said they’d never have a working prototype that was in any way useful or demonstrative.

And those people were right.

Is it as working as their FSD?

Reply on news.ycombinator.com