Former Tesla Firmware Engineer Discusses the System
twitter.com
twitter.com
They could have bought something off the shelf. Not sure what was the value in building everything from scratch
What I don't understand is why the UI sucks so very, very much every single time. And why it's so very, very slow. It seems like it has to be on purpose. Can anyone with insight explain it to me?
I also think it is a cultural. See SAP's blog entry "Why users might think their SAP user interface is crumby" https://blogs.sap.com/2015/09/15/why-users-might-think-their...
I find C# and .NET runtime (before it meets windows) quite nice. I'm not a big C#-er myself though.
Maybe my brain just doesn't get it, but the documentation makes me crazy too. I just hate everything about it.
The documentation is actually fairly good in my eyes iff you're only writing applications. As a library and custom control vendor (a position I find myself in at work) it can be atrocious and sourceof.net is hugely helpful (and I still find myself wanting to debug framework source code at times).
Still, if you have specific questions, you can throw me an e-mail if you want.
Once you accept that reactive data models are the 'correct' pattern things become so much simpler.
Throw in JSON.net and QuickType (amazing if you haven't seen it, feed it JSON or JSON Schema and it outputs correct code to serialize to from JSON in about 25 languages pretty much idiomatically (for C# it uses JSON.net for TypeScript interfaces and if you want it runtime validation).
It's a remarkably stable way of hoisting an API.
WPF binds to those objects and when you change the thing via a public property the change notification is fired and the UI updates.
Docs you want are MVVM and particularly INotifyProperyChanged.
https://www.c-sharpcorner.com/article/explain-inotifypropert...
I think a lot of the problem probably depends on the work you're doing. I'm an engineer, usually I just want a GUI that shows me the information I need, I don't care much for design aesthetics beyond making sure it's not hideously ugly. Winforms was good for this, but it definitely looks dated and needed to be replaced or massively overhauled.
I can absolutely see how somebody doing more attractive design work would like WPF. It just makes me grouchy. Somebody else mentioned mixing blend into their workflow, maybe I'll take a look at that.
Expression blend saves a lot of typing. Also if you provide design-time data, saves time because WYSIWYG gives faster feedback than change-build-test cycle.
When I work on XAML-based GUI, I open the project in both VS and Blend, and use them alternately.
What OS kernels are actually used that are written in anything other than C? Plenty of them (INTEGRITY, VxWorks, QNX) written in C are used in secure, mission-critical applications.
A language is a tool, and like any tool it can be badly designed and/or unfit for certain purposes.
My jaw hit the floor the first time I heard this. Why Linux instead of an RTOS?? Apparently Tesla's autopilot also runs Linux, which seems like a huge accident waiting to happen (pun intended).
Regardless. There are many and far better alternatives to Linux for real time applications.
People are using Linux when they need HW and driver support, e.g. gigabit Ethernet, firewire and such. RTOS vendors charging shitload of money on those drivers. I trust the Linux drivers more than the RTLinux scheduler or libc. But well, recently networking went to hell, so even there they start fucking up.
You don't (want to) make huge changes to the kernel and libraries codebase tho, even if the changes are meant to remove code you don't need, because testing a heavily modified OS gets prohibitively expensive.
Especially on "modern" embedded from the last 10 years were RAM and storage are not that limited.
Im asking because I run few instances with very heavy traffic and have no issues whatsoever. Just added nano and fail2ban and it runs with no issues for about 2 years now.
Basically you rip out anything not strictly required for the task at hand.
Running on embedded hardware is quite different from running on server hardware, disk space and memory are measured in megabytes, not gigabytes...
Obviously you can go much slimmer, but a $10 board is surprisingly capable.
EDIT: This is meant to be a bit tongue-in-cheek, but I seriously do prefer Alpine over literally every other Linux distro I've yet seen for minimalism. Also geared towards embedded-type work.
For instance, did you know that Windows XP has a long history of being used in military embedded devices that store user data (on a writeable, obviously, FAT32 file system) where the way you turn them off is to just cut the power? I shit you not. I've seen state-of-the-art Navy-used sonars where the internal computer was running Windows, and you would transfer data off of the internal hard drive by FTP over Ethernet, and it had no on/off switch, just power or no power.
Not to worry, the tight control loops the require determinism all run on Arduino boards.
Out of curiosity, why do traffic light controls need to be that precise?
"Ironically, the biggest concern with red-light camera systems is that they are so precise. They measure a driver’s speed and exact location within a fraction of a second — but do not leave any wiggle room for the errors of traffic signals such as inconsistent yellow light times"[0]
If there's not an accuracy threshold for safety reasons, there's gotta be one when traffic ticketing revenue is on the line (also I guess determining fault at accidents, vehicular manslaughter cases, etc.)
[0] https://www.mercurynews.com/2014/06/06/red-light-cameras-how...
still, revenue driven development actually makes sense as a thing too ...
Why would anyone need "sub millisecond control of Traffic Lights"?
Traffic lights are mission critical systems, of course, but even millisecond precision should be more than enough, and possibly even 0.5x-1 second precision...
When I was a kid there was one light that, when you drove over the pressure sensor, it wouldn't really do much. But if you backed up and drove over it again it must have registered an additional car coming through and the light would almost immediately go through its light cycle to change. It was really interesting to see!
Nowadays I think it's mostly cameras? We have a light near my home and the left signal will literally never trigger unless someone is in one of the left lanes.
Look up inductive coupling. The basic idea is that a changing (AC) current in a conductor generates a changing magnetic field (Ampere's law). This changing magnetic field then induces a voltage in the second conductor (Faraday's law). This is the principle behind how transformers work.
The Wikipedia article https://en.wikipedia.org/wiki/Induction_loop#Vehicle_detecti... has fairly extensive details on modern implementation.
[1] These do exist for weight-in-motion systems, however.
If the connection is lost, the exchange can just fallback to "naive" mode.
Why arent their efforts to create "autonomous only" traffic managment scenarios, where people drive into a given, known area, and the area then takes control of managing the traffic and vehicles. Such that you relinquish control of the vehicle to that area's control system, with your destination stated and then your vehicle is managed accordingly.
For example, a parking lot for a really large venue with an autonomous valet system.
YOu drive up and get out and then the system takes over your car and drives off iwth it and parks it and you recall it when needed...
Or managing traffic in a very heavily trafficed bottle-neck of a grid; such as the baybridge merging egress from SF financial district.
If you put in your destination, and join the group, all the cars could then be managed for getting onto the bridge more rapidiously ...
Autonomous doesnt need to drive me from SF to LA, but it would be great if an autonomous hive mind could get all the cars to up throughput in given situations, no?
I seem to recall that similar ideas go back to the early 1990s, at least, for highways: Drive your car to the entrance ramp, plug in your destination, and the autonomous system takes it from there.
But for many of these things, such as the Bay Bridge or a highway, it seems like there is a simpler solution: Put the cars on a train and take them across by rail. I suspect I'm not the first person to think of it so I wonder why it's never been done (i.e., what problem I'm overlooking).
Not just for cars, but also for cargo... just have a constant gondola-like conveyer that detaches a platform from the line to slow it enough to allow for cargo to get on, then re-zip-it backinto the line and speed it along, de-rail it once it hits its exit/location...
ideally though, in cities, there would be no surface streets and all cars would have their own level below that of bikes pedestrians.
What would SF look like if a superstructure was built above all streets and all pedestrian and bike traffic was moved up there? (sure, SF may be a poor example, so just select [city])
Look at Singapore's vast underground connecting malls between facilities. Those are pretty amazing.
I grade US urban/city planners rather poorly.
Besides, the best integrated transport solution in the world already exists in places like Utrecht, Groningen and Assen thanks to reforms that started decades ago.
The street level would be dark and storefronts would become difficult to access. If the stores moved up to the 2nd floor (a massive transformation of real estate, probably greatly reducing available living space), what would go on the first level? Not many people would want to live in the dark.
The results are mostly "garden apartments" which are damp, dim, and slightly less expensive.
* Full disclosure: I ride bicycles and only plan to ever live in localities where I won't need to purchase an entire car.
More cars, or even car-friendly tech will never be the solution for the issue of too many cars. See also "induced demand": https://en.wikipedia.org/wiki/Induced_demand
https://www.mrmoneymustache.com/2011/10/06/the-true-cost-of-...
https://carbusters.org/2011/09/08/are-cars-really-our-greate...
On tracks they own.
With no other traffic than theirs.
And unlimited funding.
And much less apologetic about wielding power.
Lots of mines start from little companies searching for a possible ore body (the idea or market fit), then raising money to perform a closer survey (seed funding). If the closer geological work is promising they often obtain a lease (patents or other IP).
At this point it goes one of two ways. Either they raise enough money to start and operate the mine themselves (series A, B etc, leading to an IPO) or they sell the prospect to a major company.
Then the newly-minted millionaires, who know a lot about mining, invest in the next crop of junior miners.
So as with tech there are conceptual, exploratory, growth and liquidity phases, followed by a process of reinvestment.
I remember realising this when living in Perth and being frustrated that, with quite literally billions of dollars sloshing around the city looking to invest, you'd be hard-pressed to pitch anything smarter than a brochureware website to the local investment class.
There were other structural problems. Stock options are not A Thing for various legal reasons. Failure in starting a high-risk business is a bit of a black mark. There are VCs but so much of their money came from governments trying to jump-start a market that they were about as risk-taking as a loans officer at a bank (what government wants "10 MILLION WASTED ON PHONE APPS" as a headline?).
Meanwhile the super funds are collectively sitting on trillions of dollars[0] and investing an absurdly dumb fraction of it in the ASX. Putting just 0.5% of their holdings into VC would unlock tens of billions of dollars of potential investments.
For which, hey, VCs who lurk here and want to raise a fund: go talk to the Australian superannuation industry. It is a massive pool of underperforming cash languishing in the same dozen public companies and, because Australian law forces all Australians to set aside at least 9.5% of income for retirement, the industry will never stop having incoming funds. There will always be new money to raise[1] and it will probably the 2nd largest pool of pension investments sometime in the next 10-15 years.
I will accept finder's fees and/or massively remunerative job offers as reward for this insight.
[0] https://www.superannuation.asn.au/ArticleDocuments/269/Super...
[1] https://www.willistowerswatson.com/-/media/WTW/Images/Press/...
I mean, heck. If it were that easy to start a unicorn...
But seriously - I am not saying its easy, I'm just surprised that we haven't made much (publicly known about/announced) efforts along these lines.
I mean, we have TCP down pretty good - if we are simply thinking of cars as packets, a lot of the math should exist to ensure collision-less delivery?
You better hope you don't get any dropped packets.
Neither of these models is really analogous to cars on the road.
But applying collision detection and exponential back off in road traffic is a "fun" thought experiment.
A more apt model would be critical sections and semaphores from concurrent programming. Which is named after a collision avoidance scheme used to control trains. And we all know how difficult concurrent programming can be. I don't want traffic with deadlocks, starvation, busy waiting or live locks.
I don't think there is any possibility of large scale autonomous driving without a shared control infrastructure. Autonomous driving will only work as long as autonomous cars are a small minority.
As soon as they stop being in the minority, some shared control infrastructure is necessary.
Case in point: 4 cars arriving in a no-lights 4 way intersection simultaneously will cause a deadlock. A tie breaking scheme requiring some form of communication is necessary.
Even if manufacturers would somehow manage to agree on an API, it could then be "abused" by competitors or accessory vendors to sell their own customized car assistants, which would instantly work with any car brand - without them having to negotiate with the manufacturers.
I fear we will sooner have a usable open IoT standard than manufacturers giving up that level of control.
0. which infamously occurred on the Mars Pathfinder. https://www.rapitasystems.com/blog/what-really-happened-to-t...
Any decent RTOS should have Priority inheritance that should avoid this.
Pointing to this one things as RTOS issue isn't really an accurate portrayal of current RTOS capabilities.
Priority inversion had been known about since the 70s. Priority inheritance seems to have first been proposed in 1990:
https://www3.nd.edu/~dwang5/courses/spring18/papers/real-tim... (Priority Inheritance Protocols: An Approach to Real-Time Synchronization)
The Pathfinder engineers were apparently unaware of the priority inheritance option available in VxWorks until they had to debug the issue live from a few hundred million km away.
Given all the wtf stuff in TFA, it might very well be...
Also in my experience it might have been easier to reach the ASIL-D requirements, using a smarter combination of a Limiter on RTOS and using more generic code on something like linux for more of the code. This probably also would end up in more used and tested applications reaching more stability. (That's is partly outside ASIL-D).
Functional safety and ISO-26262 is much misunderstood in automotive development and architecture.
Also imho the certifications, well with out the safety case are kind of useless. You still have to make the assessment how you will find the problems with it in your use case. That might ever so slightly differ from what they certified. The automotive industry thou loves to have someone else to blame, e.g. the supplier of the RTOS, Compiler etc. Using Linux makes the blame game hard.
maybe you don't, but genuinely curious how do you validate/guarantee scenarios then?
1) What concrete guarantees do you think you get from a special RTOS?
2) Which of those guarantees are meaningful to the scenario?
3) How (by what mechanisms) do you think the special RTOS guarantees the things that it guarantees?
4) Which of those behaviors are something that only an RTOS can provide?
Here's what's inside of every autonomous vehicle ever made: a message-passing subsystem, sensors, fusers, navigation, dynamic control, actuator device drivers, and thruster device drivers.
Sensors measure things and emit readings. Your most expensive, highest frequency general purpose sensors emit new readings at something too fast for a human but hella slow for a computer, like 100Hz-5KHz. Your common sensors, a video camera for instance, don't get even close to that. These sensors are often connected, even today because milspec companies hate modernity, via RS-232 serial cables. For those younger than 30, RS-232 is what non-Apple computers used for non-keyboard/mouse peripherals prior to the introduction of the first iMac in 1998 because USB didn't really take off until then.
Sensors send their readings via the message-passing subsystem to fusers.
Fusers take the readings from the sensors and, hur hur, "fuse" them together into a description of where the vehicle is and what the environment is like. This usually involves something like a kalman filter. Fusing even your very fastest sensors, the 5KHz IMUs of the world, is just a small bit of math and basically takes no time at all.
Fusers send their fused states via the message-passing subsystem to navigation.
Navigation takes the fused sense of self and the world and decides which direction to head and how fast to go. The objective could be something like hitting route waypoints or it could be something like staying in a lane and not being rear-ended and avoiding obstacles. Car navigation probably doesn't act on new input more frequently than 100Hz, you certainly can't act on new input more frequently than 100Hz, and it takes basically no time at all.
Navigation sends its directives via the message-passing subsystem to dynamic control.
Dynamic control takes navigation's "which way" and "how fast" directives and turns them into more realistic short-term goals accounting for hysteresis and other physical limitations of the system like minimum turn radius. This is just a small bit of math and basically takes no time at all.
Dynamic control sends its directives via the message-passing subsystem to the actuator and thruster drivers.
Actuator drivers convert dynamic control's "go more left" message into trying to go more left.
Thruster drivers convert dynamic control's "go more fast" message into trying to go more fast.
Actuator and thruster drivers send readings (hopefully) from the actuators and thrusters, because those are also sensors, back to dynamic control and fusion.
Sensors feed into fusers, fusers feed into nav, nav feeds into dynamic control, dynamic control feeds into actuation and thrust. When you have new data, you do something new with it which is technically doing the same old thing with it and just producing new output.
Now there aren't that many sensors. There are way fewer fusers. There's only one navigation. There's probably only one dynamic control, though there could be a couple.
Anything else that I haven't already described, like Waymo's machine learning object classifying 4D mustache adding hotdog detectors, are just sensors and fusers sitting on their own computers feeding new lat/lng/heading/speed to navigation at a rate that is hella slow for a computer. And for sure Waymo's convolutional neural network middle-out jaywalking yoga mom detector takes a lot of processing, but it's running on its own computer, not competing for resources, and emitting its fused readings at some hella slow for a computer rate.
This stuff really does get complex. A sensor controller will likely be on multiple cycles internally: one for oversampling the sensor hatdware and one for transmitting the (filtered/corrected/calibrated) results. A "fuser" as you call it (never heard that term before) needs to make sure that it does never act on stale sensor information (sensor malfunction, accumulated communication issues). Transmission errors need to be detected. Random bitflips in values that are stored in volatile memory for long time spans need to be checked and acted upon.
Every independent controller in such a system requires some kind of watchdog that needs to be reset periodically. Too many watchdog resets in a row indicate a failure and the affected system must shut down in a defined way. You need ways to deal with any combinations of controllers going belly up and avoid taking unsafe actions. For many systems transitioning into a totally inert safe mode is sufficient, but not always.
All of the hardware must constantly run self tests. That includes periodic CPU and memory tests (both volatile and non-volatile memory) and also all periphery that us involved. If, for example, a DAC is used to send a signal, the resulting signal must be read back by different hardware to check that the generated voltage is indeed correct.
Manually threading together all these different kinds of cycles and asynchronous events without a RTOS scheduler is hard and becomes error prone. The result is likely less resilient than a preemptively multithreaded firmware.
An operating system, real time or otherwise, handles activating processes, IO, and interprocess messaging. That's it. You don't get magical serial line noise clearing pixies with it, and it doesn't make your actuators less drunk.
And...
> Manually threading together all these different kinds of cycles and asynchronous events without a RTOS scheduler is hard and becomes error prone.
You just said "getting input and sending output is way too hard for software on a Linux kernel." That's a crazy person statement. It turns out that Linux is, and has been for a loooong time, very good at doing operating system things like activating processes and interprocess messaging.
> The result is likely less resilient
Saying "likely" here suggests that you don't know what the result actually is. So what are you arguing? What was my statement?
Whatever assumption you're making that says "This. This right here is the reason why we definitely need an RTOS." Just don't make that assumption. That assumption is wrong.
So to me, "you don't need an RTOS" means that you're running on bare metal. And that would be hard to pull off for the reasons that I outlined above. And I think this is where we ended up misunderstanding each other.
I enjoy the kinds of restricted RTOS environments that we use because their simplicity means that I can get a total understanding of what is going on quite easily.
This does not mean that Linux is completely inappropriate for real time tasks. I am sure that you could analyze and patch the kernel to match pretty high standards (others mentioned patches). Given the relative size and complexity of a Linux system, this is no simple task. But if you run it on appropriate hardware (not your run of the mill x86), I don't see why you couldn't get reliable realtime responses.
But safety essentially means that the software will not fail more often than once every x hours where x ranges between 10^5 and 10^8, depending on the level of safety required. Proving that for a complex system is hard. For example, how do you show that the essentially indeterministic pattern of dynamic memory allocations happening in a Linux system will never lead to memory exhaustion by fragmentation?
I know of no version of the Linux kernel (or GCC, for that matter) that got a functional safety certification. Safety standards are transitioning away from allowing positive track records as sufficient proof that a piece of software meets safety standards. Do-178 now only allows certified software AFAIK and I expect this to be carriedover into ISO 61508 and ISO 26262. This means a regulated development process, pretty strict coding standards, complete test coverage, full documentation, and so on also for all 3rd party software. Not sure how this transition is going to play out in practice.
Do you know any ASIL-C or ASIL-D (or SIL-2/SIL-3) software that is running on Linux? I am curious whether anybody managed to get that certified. I know that Linux is running on some class II medical equipment, but then, standards for these devices are inexplicably lower in practice in my experience.
A stock modern Linux kernel on almost any hardware platform will give millisecond level responses. Much of the old PREEMPT_RT patch set features from the old 2.6.x days for real-time response has been merged to the mainline kernel.
There are lots of problems in software where you are controlling something physical with a control loop from 1 to a couple hundred Hz. Many people assume a hard real-time deadlines are necessary for this sort of system, but through good system design practices it often is not necessary. For example, if something physical must be sampled with very low jitter, let some hardware do the sampling and latch it in a register and then let the software come in with a variance of hundreds of microseconds to get its work done. Once again, write the output to a latched register and let hardware worry about taking the shadow register with very low jitter.
Having worked on bare metal microcontrollers, to various RTOSes, to higher-performance embedded CPUs with Linux, I prefer Linux on higher-performance hardware. Obviously, this isn't always possible, especially in power constrained situations. But with Linux, when you suddenly need to have support for an arbitrary network protocol, a database, a filesystem, graphical output, etc. you can have something together in no time. It is often a monumental effort for such a task when bare metal or with a RTOS. It is often difficult to get the supporting software and libraries to build on an RTOS in the first place.
https://kernel-recipes.org/en/2016/talks/who-needs-a-real-ti...
I haven't messed with an RTOS before but have done some fooling around with scheduling on microcontrollers and I can see why linux is tempting for ease and speed. But we're talking rockets and self-driving cars. These things are expensive as hell, can easily kill people, or both. It seems like the exact sort of place you'd want to take the time and effort to be sure.
I’ve seen the same pattern a few other times. Slow, hand built, rad hard systems CAN be more stable and demonstratively safer... but that is rarely the case and the effort required to get such a system right is orders of magnitude greater than using standard “undeterministic” systems. That engineering effort can be better spent innovating and building fundamentally more advanced solutions.
Just my experience. Just my opinion
These things frighten me everyday. What frightens me even more, is the people who work on autonomy without a real grasp on determinism. It's unfortunate that the people who have the most high tech backgrounds (phds in computer vision AI, etc) applicable to autonomy, have never implemented safety critical autonomous systems outside of a research project that tested some aspect of detection or control in a test environment and had to only work once to get a paper published.
For general robotics linux is great. But there is an enormous difference between a robot roaming around your house bumping off walls, and a vehicle carrying a whole family at 70mph.
Most of the linux based systems I have worked with have some form of redundancy, whether it be other chips running linux, or ideally ECU's running an RTOS that perform monitoring, gating, and/or some level of safety fallback control. The RTOS based redundancies often are what provide ASILD. Trusting a single linux processor is what everyone does to get funding, but when you go out and test on public roads with human lives at stake, or start selling a product, you better have some quantitative guarantees other than "It's been fine so far..." That kind of stuff makes me angry.
But on the other hand, has any Tesla car ever had an accident because of this? At some point, "heavily tested and validated end to end in real-life conditions for years" and "formally proven on a simplified model using reasonable assumptions made by human engineers" become relatively close in terms of how much trust you can put in a system.
But somehow we tend to prefer to later. I am not sure if this paradigm is still relevant these days.
That seems like a more wordy way to state "It's been fine so far".
> "formally proven on a simplified model using reasonable assumptions made by human engineers"
I didn't know about that. What kind of formal proofs did they do? Did they involve the linux scheduler?
I guess you'd need three separate implementations to achive some redundancy when you can't just slam the emergency brakes if the systems disagree.
(... For some reason, I find it higly demotivating that another team is doing the exact same thing. Maybe I just want to be a snowflake...)
Did you know that Linux can handle hard RT, if you use the right hardware (and maybe the right kernel/patches).?
Besides, if you follow Linux kernel development, you see that the effort is virtually never for real-time but for general purpose.
There is no reason per se to attribute any downsides onto these "filters".
Does TFA (which sure, is for Tesla, but same leadership) describe an environment where people use "the right hardware"?
So yes, I'm sure that Linux can work. But it will be difficult to prove to auditors that it will always work.
Same reason SpaceX eschews radiation-hardened processors for redundant off-the-shelf cores: supplier competition. There aren't many RTOS engineers on the market; there are many Linux engineers. Once they got over the cost of hardening the kernel, SpaceX found itself at a scaling advantage versus RTOS-based competitors.
At least with Linux you're getting a system that's been used so much that all major issues like that are ironed out. Nothing beats a few million testers.
edit: I saw the same on a fleet of thousands of JVMs which hung on 100% CPU after 248 days very consistently. Closest thing to an explanation I ever got was perhaps it is storing uptime in hundredths of a second (why not ms???) in signed 32 bit integers, see: https://ma.ttias.be/248-days/ In the end we solved it by restarting with a cronjob between 2am and 4am after 247 days...
A devil-may-care throw-caution-to-the-wind cavalier attitude?
>supplier competition
I stand corrected. And I'm never buying a Tesla or property even remotely close to SpaceX launch sites...
Although I don't know the market. It might screwed by some weird market dynamic.
There aren't many Linux engineers who have experience with resource-constrained systems or real-time programming requirements.
An RTOS has nice guarantees, and I definitely see the appeal, but on the other hand:
- SpaceX machines receive more irradiation that computers on the ground, so computation errors already make the behaviour of the software chaotic,
- The wealth of complex, widespread libraries helps a lot.
The tech and products were complex. The turnover rate was high and training new hires was a lengthy process. The new projects coming down the pipeline never ceased (this was during a period where FH/F9-1.1/Dragon/Crew was all under design/development and constant iteration).
It was fun for a young engineer, but burnout is real.
*WarpDrive was actually pretty impressive given the amount of stuff that it did.
Agree about WarpDrive being pretty amazing for all the stuff it did. Although amazing things tend to just clump up from all the features that you need, and you end up with an app that is hard to manage.
It wasn't all bad. There were other groups at X that provided pretty amazing tools for people to get things done (thanks, @cbanek and friends!)
While you might need to pay more and improve working conditions, would you be more efficient given the individual staff would spend more time being productive?
In my opinion, this is the biggest flaw of the system. For many investors, a company is less about what it makes and more like a process to grow their money. Even when a company becomes profitable, there's always pressure to make it even more profitable quarter after quarter.
Would you keep your retirement funds in a company that doesn't grow?
Not everything needs to grow into the sky.
It would be worse than buying a bond: you'll get the risk of equity with the returns of a bond.
(Then again English is not my first language.)
Well they landed a rocket on a barge in the ocean so something about that model must be working right
To put it another way: If your method of writing novels is to hire an infinite amount of monkeys and put them to work on typewriters, you can't say "Something about this model must be working right, I came out of it with the complete works of Shakespeare!"
They landed a rocket on a barge in the ocean. Maybe with a better process, they could have done that two years faster, for 1/100th the cost, with no burnout. You don't know, and you can't say the model works right just because there's something to show for it.
All you know is that the process is able to eventually land a rocket on a barge. It doesn't tell you whether it's good at it.
Get a lot of monkeys and put them at work. They will produce something. Better have something than have nothing.
While touting the purchase cost or energy benefits, these ideas routinely ignore the overhead cost inherent in a distributed system, let alone the Fallacies [1], which is the GP's and OC's (and possibly your) point, I believe.
[1] https://en.wikipedia.org/wiki/Fallacies_of_distributed_compu...
So while it may not be the most efficient process, the overall process is much better than their competitors since they can launch for so much less money.
I don't think you can answer that question by just looking at Tesla.
The whole company seems to be operating in the "burning the candle at both ends," not just the workers at the bottom. Also, it's not just "saving money" but pushing super hard to accomplish something extraordinary, i.e. generating new revenue, not just reducing costs. Additionally, the workers are partially compensated via stock options, so they share in the success of the company even if not through higher wages alone. So I'm not sure "mistreatment" is the right word to use.
At the end of the day, SpaceX (and Tesla) are not for everyone forever. I am not in a station in life to want to join right now, but may in the future. And maybe this strenuous effort is not especially profitable for SpaceX because of the churn that it creates. But that churn IS helpful for the industry (and thus, in my opinion, society) at large because it has spread SpaceX's know-how throughout the US aerospace community and resulted in alumni founding probably dozens of companies that can leverage the lessons learned from SpaceX. But some people work well in that environment and stay long term (which isn't to say it can't be improved).
So I am glad SpaceX is the way it is, and I hope they're successful in the future. But it also doesn't have to be the model for everyone else to copy. It might not work for everyone else, nor should it be expected to.
I wonder if the constant burnout and churn keeps employees from vesting and thus ever collecting much if anything in stock?
I'm refuting the statement that "saving money by paying peanuts, grinding people down to burnout, and then constantly having to rehire/retrain new people as the old ones leave" is unanswerable in the current context based on 1 company especially because this company seems to be destroying their competition.
If they have a process proven to work, in a world where they are already doing things no one else has been able to do, changes to that process should be introduced very slowly.
"We're paying more for programmers than almost anyone else in the industry!" is an obvious thing a bean counter would notice and point out. Productivity is harder to measure, and all that time lost to training on the job doesn't immediately leap out in spreadsheets because it's blended with actual work.
But no employer tracks that, not even in academia...
I can’t find the reference I’m looking for, but cost associated with turn over rate due to narcissists is apparently as high as taking care is ASD. I would imagine turn over rate associated with burnouts not due to narcissism, but simply poor & short term management & vision to be at least on par.
I can't speak to development specifically, but I looked into one of Tesla's devops offerings and I make slightly more for a mid-level engineer than they offer for a senior position. I also 'only' work 40-hour weeks and my cost of living is about 20-30% lower than there.
If other high end software jobs are paying the same for 45-50 that Tesla is paying for 60, that’s on the low side hourly, but the low side of the high end.
75% (45/60) of 150k is still $112.5k plus presumably good benefits and some sort of equity component. That’s damn fine compensation for someone fresh out of undergrad even in 2018.
I wouldn’t want to work 60 indefinitely even for great pay, but that’s a separate issue.
Maybe I am just lucky but I have worked for 2 of the major big tech companies and came out of college making 100k+ while primarily working 40 hour weeks at both.
> 75% (45/60) of 150k
Tesla new grad software engineer total comp is 150k? Damn, in that case they are pretty close with big tech (amazon is 145k and G/FB is 165k from what I have heard). I assumed it was lower since my brother was a PM with 6 YOE and got payed 130k a year.
> That’s damn fine compensation for someone fresh out of undergrad even in 2018.
Oh totally, my girlfriend is probably going to make like 60k out of grad school. However, while it is much better than anyone besides what my finance friends are making, that does not mean they are paying well relative to the tech industry.
Why would you want to run backoffice on Linux and then re-create all those wheels by hand in-house? Relying on the expertise of other companies for basic backoffice systems is actually recommended practice until you become big enough to actually need custom software (generally, north ten thousand employees).
The former is generally a given for a company of any significant size (employees or business activity). The latter is unheard of for most backoffice functions (other than specialized accounting and finance functions) since it's a waste of money and would place the company at significant legal and regulatory risks--it would require effectively becoming experts in accounting, HR, etc.
Even dynamics isn't .net
It seems as if there were two distinct cultures of engineers. Those working on workstation-grade hardware networked over TCP/IP (whether running proprietary UNIX, open source UNIX, or Windows NT) -- and Java emerged out of this.
The second cultures were developers building mainframe applications; usually they would be ones working on problems related data processing, planning, and automation for businesses (not just enterprises but also many SMBs, government organizations, hospitals, etc...)
Java clearly emerged from the first culture being built by a vendor of networked UNIX workstations. Some of Java's most memorable failures - either exceedingly complex and brittle systems like RMI, JMS, and J2EE (I mean this literally: not modern Java EE like Jersey/CDI/etc... but EJB 2.0) or features that were in retrospect far ahead of its time (JINI or JXTA, compare with consul/etcd/zookeeper and the idea of a service mesh today) came as an attempt to commoditise approaches commonly used by the first group as frameworks for solving the domain specific problems of the second.
This applies even if you need heavy customization. In fact, it applies even more--since that level of customization usually means sufficient complexity of backoffice needs that only the pre-built service providers will have the sufficient depth and scope to cover you.
wow - this is surprising to me and wasnt mentioned in the biography - any ideas why ?
That felt weird to type.
https://www.amazon.com/Founders-Work-Stories-Startups-Early/...
Jackson was the guy who realised that PayPal and eBay had massive synergy and worked super hard to get PayPal in there. Eventually leading to eBay buying them out, and Musk and Thiel going from merely rich to actually billionaires.
Musk has even admitted as much that he prefers game developers. Maybe he see's the parallel for working overtime and uses this "perk" to his advantage? From an article from 2015[0]:
> "We actually hire a lot of our best software engineers out of the gaming industry," said SpaceX CEO Elon Musk, when Fast Company posed this question during the May 29 Dragon V2 unveiling. "In gaming there's a lot of smart engineering talent doing really complex things. [Compared to] a lot of the algorithms involved in massive multiplayer online games…a docking sequence [between spacecraft] is actually relatively straightforward. So I'd encourage people in the gaming industry to think about creating the next generation of spacecraft and rockets."
[0]: https://www.businessinsider.com/why-is-spacex-at-a-video-gam...
Both are reasonably capable of high service uptimes and solid performance. With Server Core and PowerShell, there's a lot more parity than my fellow Linux admins want to admit, but either is a viable choice for general IT services at this point.
Note - I'm excluding licensing entirely from this, as well as infrastructure maintenance and control surfaces. Nobody likes DSC, and there are several superior config management solutions for Linux that don't have meaningful analogs on Windows.
Back at that time of NT4 & Linux 2.2, I'd argue Solaris was the best option.
(1) Stable platform that's backward compatible over long periods of time.
(2) Very good rapid application development tooling, e.g. Visual Studio which is probably still the best IDE overall.
(3) A huge trained developer base making it easy to recruit. Same goes for IT personnel.
(4) A huge pool of software, custom dev firms, etc.
(5) Certification for US DOD and other certification-heavy environments where Windows is used heavily, which may be important for an aerospace company.
(6) Integration with everything in the business and government world is already done.
(7) Windows has a lot of complex user, permission, and policy management stuff. Active Directory is The Standard for UAM in the corporate world.
The cost of Microsoft licensing is chicken feed compared to the cost of building and launching rockets.
Overall I don't think it's a bad decision. Not everything is an Internet startup or hacker project. Right tools for the job.
I've always wondered why there's such a small list of decent IDEs for C anything.
I usually just stick to vim and a handful of plugins.
For instance, try writing an auto completion tool for C++ and Java. The first one take orders of magnitude more work for a mediocre result.
Using vim is symptom of the problem. The available tooling is so bad or non existent that it's comparable to a text editor.
And this is why Torvalds takes kernel API stability seriously...
Soon after he was ousted and Thiel was made CEO. Interesting to see he's still pushing Windows.
Source: Paypal Wars by Eric Jackson
That's still true today and it will probably always be true unless Microsoft ports Visual Studio one day. It's not a super great reason to choose it as an operating platform though. :\
That's good to know. When I saw the inside of the Dragon capsule, with its shiny touch controls, I was already imagining Astronauts having to deal with installing Android updates on their control tablets while mid-flight, or some other crazy stuff along those lines.
Remember the demo of the first iPhone (vs the huge expertise of Nokia). Or how Microsoft won the desktop starting from a single user, cooperative multitasking system (vs all the sophisticated Unix-based systems). Or Facebook running on despised MySQL and PHP.
These hacks are part of the strategy. It's risky but probably doing these 'properly' would increase the risk even more.
The security researchers who were expert witnesses in the Toyota case and that analyzed Toyota's source code testified in court that they managed to reproduce the unintended acceleration with a real car on a dynamometer.
I once had a project with a major lobbying organisation with links throughout the automotive industry. The one that published regular reports on real vs declared emissions discrepancies long before the VW scandal.
They were unanimous when asked about the unintended acceleration bug. It was a case of mass hysteria.
At least according to the summary based on sources used for Wikipedia:
https://en.wikipedia.org/wiki/2009%E2%80%9311_Toyota_vehicle...
The page you refer to has a section "NHTSA and NASA Investigation Verdict" and it says nothing about the software bugs.
https://en.wikipedia.org/wiki/2009%E2%80%9311_Toyota_vehicle...
The engineer was questioned:
" Q. Now, you have not reproduced in vehicle testing your 25 theory that there's a software bug that opens the THIS TRANSCRIPT IS NOT PROOFREAD 30 1 throttle and then the task dies, have you? 2 A. No."
From page 245. Further reproduction questions were all answered with a no further.
To get a successfully task death failure (what the engineer wanted to happen) the engineer had to modify the source code.
Further in the limited testing even with modifications, the failsafe triggered 100% of the time.
Your statement is incorrect.
https://www.caranddriver.com/features/its-all-your-fault-the...
http://www.safetyresearch.net/blog/articles/toyota-unintende...
> Skid marks notwithstanding, two of the plaintiffs’ software experts, Phillip Koopman, and Michael Barr, provided fascinating insights into the myriad problems with Toyota’s software development process and its source code – possible bit flips, task deaths that would disable the failsafes, memory corruption, single-point failures, inadequate protections against stack overflow and buffer overflow, single-fault containment regions, thousands of global variables. The list of deficiencies in process and product was lengthy.
>There are a large number of functions that are overly complex. By the standard industry metrics some of them are untestable, meaning that it is so complicated a recipe that there is no way to develop a reliable test suite or test methodology to test all the possible things that can happen in it. Some of them are even so complex that they are what is called unmaintainable, which means that if you go in to fix a bug or to make a change, you're likely to create a new bug in the process. Just because your car has the latest version of the firmware -- that is what we call embedded software -- doesn't mean it is safer necessarily than the older one….And that conclusion is that the failsafes are inadequate. The failsafes that they have contain defects or gaps. But on the whole, the safety architecture is a house of cards. It is possible for a large percentage of the failsafes to be disabled at the same time that the throttle control is lost.
etc. etc.
Most code is buggy, and the more closely you look, the buggier it is. Most of that same code operates without noticeable error.
That code analysis turns up "inadequate protections against stack overflow and buffer overflow" does not actually suggest that there was any stack overflow or buffer overflow, and people quote this as if it does.
Meanwhile, it is all-but-certain that the overall findings were correct: people hit the wrong pedal regularly, and if enough press attention is given, all of those wrong-pedal-pushers find each other and try to blame the manufacturer instead.
You started with "Regardless of whether the acceleration were caused by the software or not," but I think that's the important thing, and that software is buggy should surprise nobody.
At risk of beating a dead horse, I do find it interesting how despite the myriad of "driver hit the wrong pedal" news stories out there, only the Prius one starts from the "it had a mind of its own" angle.
I think what most people don't realise is that it's so incredibly easy to be fallible with "automatic" familiar everyday activities (i.e., not maths or remembering facts). I recall in driving school approaching a quiet intersection where I once lost proprioception of my foot and had to call the instructor to use her dual brake because I didn't know where my foot was or could no longer reach the brake pedal (I guess I'd have attempted the hand brake if I were alone).
From this single experience along with running other dark hypotheticals through my head, I am so cautious with many things that it bewilders me that activities such as "carpool karaoke" is not an atypical practice.
But otherwise yep, same thing with software only that the stakes are often lower thanks to the sheer number of non-mission critical projects out there. It's only the mission critical bugs that get the most attention and surprise.
That's fascinating. I can say that is something I have never experienced. Does it happen to you with any regularity?
At one point in my life, I was simply an inexperienced learner driver. It isn't and wasn't a medical condition (for me anyway). I literally couldn't find where the brake pedal was because I couldn't really position my foot in the right place as I had clearly lost track of where it was in the footwell area. I also didn't want to take my eyes off the road and knew I was going slowly enough that nothing bad would happen (because the professional driving instructor had a dual brake, and I had a working handbrake too which I never used - we'll call that risk compensation).
Now all this said, it's incredible how many people are completely unaware of their own inabilities when it comes to mission-critical tasks (the Dunning-Kruger effect). This isn't even always limited to "incomptent" people - experienced professionals also make "simple" blunders. See also Air France 447, or various deceased pro/amateur race car drivers.
Speaking of failures though - another time (also while I was learning), I was reversing out of the car shed with my dad supervising. Before leaving, I complained to my mother (who was just outside the car) that my brake was difficult to press. She replied "just press harder", so I shrugged, obliged and went on with the "supervised" driving practice. Her confidence in me had me convinced that everything was fine so off I went.
Damn, the brake was so difficult to press - I needed both feet as well as some bodyweight on it to get the car to stop at each intersection - it hardly budged each time and felt like a dodgem car brake. I figured at the time that if I just kept things slow and got the handbrake ready, things would be okay? Anyway nothing bad happened; it was only a short session before dinner on quiet streets.
The next morning, my mother went to drive the car out of the shed and was alarmed at how broken the brake was. What was I to know - I was pretty new to driving and thought it was just the car being old and figured it was good enough at the time for my two feet. It was probably bad enough to require towing but I think my dad ended up driving it very slowly/carefully to the mechanic.
That was a bit of a long story, but I'm detailing this to highlight how "stupid" cascading failures are "alarmingly" common - it's not unheard of to exist in both the human side, or the electronic/mechanical side. It doesn't really faze me whether the Prius "acceleration bug" was human or machine induced. Both are often as bad as each other. The only thing we should depend on are multiple layers of redundancy and good systems design.
Edit: When I first started driving, I mulled over whether to go barefoot or not (either is legal here so long as the driver has control). These days, I fortunately can handle either confidently. I'm also much better at sport now (and therefore coordination) than when I was a teenager.
And what was particularly negligent about Toyota's software practices that "more likely than not" caused issues with ETCS.
https://betterembsw.blogspot.com/2014/09/a-case-study-of-toy...
Welp, sadly I am not able to read your source.
Happened to my 2001 Clio several years back, although in my case it was when I started the car, not during driving. My hypothesis was that it triggered due to my bad habit of turning the car off in traffic queues then restarting it with the pedal held down.
The success of the first iPhone or Facebook app didn't depend on using it to navigate through life-and-death situations not only for the users but also for everyone around them.
There are places for 'move fast and break things'. But cars move fast already, and they can really break things.
Every organization has these types of things internally. As an engineer, I don't like it, but it's a fact of life.
Source - https://www.businessinsider.com/tesla-hit-model-3-target-by-...
We're at the point in the Tesla story similar to that of 2008 where Dr. Berry et el were watching the housing market collapse around them and the banks wouldn't re-price the swaps. Tesla is already bankrupt, most people don't know this yet. But they will.
The details matter here and given the quality that I see in the field, I'm not convinced that this is such a horrible situation.
I have so many electronic devices, from cheap to expensive, that have some passive component manually bodged on somewhere. They work fine. It just means that paying someone to rework 1000 boards was cheaper than throwing the boards away and spinning up Revision B immediately. It's not a big deal. Waste is worse than a product that is imperfect immediately off the assembly line.
Is Tesla making money?
Tesla cannot get the rest of their booked-sale cash for an undelivered car.
But they must pay all the fixed production costs anyway. In cash, creditors at this point will be getting squirrelly.
Therefore: slowness in production is the fast-track to a possibly fatal cash crunch.
It's not about money. Correcting and refacturing a board takes time, in the order of months even for a simple one. It's too long.
• In July, over 60,000 test drive requests in the US alone
• 5000 Model 3 new net orders in one week in mid-July
• Total deposits greater than total refunds in the twelve months ending April 2018
• Reports of Cancellations Outpacing Orders proved to be False wording
Meaningless negative indicators include Goldman Sachs analyst David Tamberrino saying Model 3 social media activity had lessened and frequent critic Latrilife said Tesla’s Burbank Airport lot is under 24/7 surveillance
Sources include:
• Seekingalpha.com/article/4189303-tesla-short-thesis-model-3-demand-wrong
• Seekingalpha.com/article/4179986-tesla-model-3-reservations-tip-iceberg
And - https://twitter.com/Paul91701736/status/1033136573615730688
All the first hand Twitter intelligence is gathered by people who are short on TSLA. They see the ground level reality.
It's not like those matter. The bank itself guarantees the integrity of your account and can reverse charges. And of course would be insured for such losses.
Being killed by a car, otoh, is more permanent.
Only if it's deliberately isolated in the vehicle. It should be. Aaaand, it isn't: the firmware upgrades to all other car computing elements go through it.
Can you imagine a zero day being found in Windows with Windows Update being down?
IIRC car manufactures are irresponsibly lazy and never properly air-gap infotainment from critical stuff.
Source: Done this a few times to clear bad map data or occasional glitch.
To be sure, it's a bit unsettling, and I wouldn't be thrilled about it deciding to not work for a day or two, but, in a way, it makes me MORE comfortable that the vehicle control systems function as expected.
In hot weather it can become very uncomfortable or even impossible to drive without AC.
Safety equipment must not fail, but essential equipment can fail as much as your customers will tolerate.
Our '83 Vanagon doesn't have AC, it can be uncomfortable but sure isn't the end of the world.
Because drivers can't get distracted and crash because of a failure in the infotainment and IT infrastructure?
Almost everything in a cars front panel and dashboard can be. I've read somewhere that people have been killed even because of something as quaint as the wrong placement of car ashtray.
Anyway this little illustrated anecdote of mine aside, driver distraction is a genuine issue - even for drivers at red lights. The number of times I've seen this type of behaviour transcend into moving off while remaining distracted (whether that's heads down visually or just mentally) is too boringly frequent to detail. Sometimes the distracted drivers even creep forward unconsciously while traffic is flowing across them. Emergency vehicles can't get through, drivers end up splitting their attention, pressure mounts once proper movement starts again, and all the while they don't realise they don't have full attention in a changing environment.
It's almost like a hypoxia.
Tom Scott did a pretty good video on this: https://youtu.be/_-aDHxoblr4
2. Apparently people can ssh into the infotainment system.
1 + 2 => someone hacks into the infotainment system and flashes a safety-critical ECU. I guess/hope there's some protection in place to prevent that.
An iPhone handling an emergency call is an exceptional use case.
A car driving is not an exceptional use case for a car.
A life and death scenario is occurs every few minutes, or continuously for something like a mountain road, if something like brakes, limited throttle, or steering were to fail.
But, I would, naively, assume that the operation of these critical components were in no way related to timing requirements of some process in Linux.
It seems you have to prove that failure of these hack-y systems would lead to the disastrous result you postulate.
Even PC's have now a hidden baseband processor for the mission critical stuff, like backdoors and surveillance.
Clearly when we "verify" Waymo or Tesla auto-pilot, we're going to want to use that stuff, right? Surely they won't just provide insurers with some data about the billions of miles they've driven without accidents and how humans can only drive like a million miles without an accident and try to get the insurers to give them policies...
Just like when we hand out licenses, we always check to make sure the 16 year old took some formal driving classes from professional driving instructors... I wish things were better but we don't care about this stuff as a society until much later usually. When did the car first appear? When did the first seatbelt law appear?
Musk fan or not, you can't deny this has no interest
Tesla has a larger market cap than Ford, GM, or Honda.
For every good success example, there is a good failure example.
My point is that for every good example of a success story (about startups in this case), there's a good story of a failure. And the failures are much more than the successes. If you want to discount the failures because they don't fit your narrative, then you should equally oppose the successes when they don't fit your narrative.
TL;DR my post is a reply to a post and should be seen in that context. You've taken my post out of context; please don't do that.
Moreover, I recently read the book Bad Blood and I found it interesting to get an inside look at a startup who present themselves better than they actually are. I don't believe that part of the Theranos debacle is so uncommon. The severity and unique market though, are. And, that's actually underlined by the Twitter thread (the pictures). Another similarity is the massive quitting and burnout of quality personal, the fear of being fired and standing up, low morale. Those are, IMO, interesting similarities.
Bosch supplies VW (includes Audi, Seat, Skoda, ...) and various other large incumbents, as one of the biggest car manufacturer supplier. Why do people think they are considerably better off? Their practices are often equally ridiculous and dangerous.
As a rat in that cage you do whatever you need to make things work.
First, we're talking about a multi-billion dollar company here (they've raised ~ 15 B), not some small "challenger".
Second, your example is the iPhone and Facebook. Those don't have the potential to kill their owners or run over people.
We can't have them be both challenger in that sense, and accept that challenger in their case means multi-billions.
Challenger or not, $15 B doesn't explain a totally crappy software engineering process and result even if its just for the infotainment system.
>The iPhone could play a section of a song or a video, but it couldn’t play an entire clip reliably without crashing. It worked fine if you sent an e-mail and then surfed the Web. If you did those things in reverse, however, it might not. Hours of trial and error had helped the iPhone team develop what engineers called “the golden path,” a specific set of tasks, performed in a specific way and order, that made the phone look as if it worked.
>They had AT&T, the iPhone’s wireless carrier, bring in a portable cell tower, so they knew reception would be strong. Then, with Jobs’s approval, they preprogrammed the phone’s display to always show five bars of signal strength regardless of its true strength.
>None of these kludges fixed the iPhone’s biggest problem: it often ran out of memory and had to be restarted if made to do more than a handful of tasks at a time. Jobs had a number of demo units onstage with him to manage this problem. If memory ran low on one, he would switch to another while the first was restarted. But given how many demos Jobs planned, Grignon worried that there were far too many potential points of failure.
Steve was a full on sales person and was really, really good at presentations.
a) interesting, because it shows how other companies solve (or fail to solve) specific common problems.
b) surprising, because I thought they'd be much better at it.
Car infotainment is generally not categorized as safety-critical and can be quite similar to regular software development. Having remote access changes things: I always thought their security must be top notch if they had the confidence to launch that. Ssh-ing and deleting files sounds like the opposite of that.
Btw, saying that doing things properly increases risk must be the biggest load of nonsense I have ever seen on HN.
All of the what? The only competition on the desktop when Microsoft won was QDOS.
Having both would be nice, of course.
Do you not have bluetooth, satellite, or LTE in your car?
This poster seems to think well of the QA team they had. However,
It doesn't matter how great your QA team is, if the quality isn't in the system the QA team can't put it there. All they can do is tell you that you're making poor quality products, and attempt to inform management (as QA rarely has real authority) who should then act on that and work to improve the system. If quality isn't part of the corporate ethos, then they (QA) will make little difference in the end.
Infotainment systems are increasingly used as primary UI for almost anything on car that is not controlled by steering wheel and pedals and thus has to be able to communicate with almost everxthing that is in the car. Tesla is pretty extreme example of this, but it works this way for most manufacturers.
If the only interaction between the automated systems and the braking system is to apply more braking power and the cars breaks can significantly overpower the engine then nothing the infotainment system can do would prevent the car from coming to a complete stop.
Similarly, if the power steering was limited to applying say 5lb of force to the steering wheel (as felt by someone holding the outer edge) a driver could overpower any steering adjustments with minimal effort.
With just those two choices the risks associated with hacking the car dramatically decrease. Yes, the car could prevent someone moving, but that’s also an inherent risk from engine failure.
Referring to an event where nothing happened does not support a ‘very dangerous’ argument. If you simply mean it’s possible and after that it’s then possible that something bad would then happen then I agree.
I am simply saying it’s vastly lower risk than a car that’s impossible to steer or slow down without causing massive mechanical failure.
By inform, I don't mean "rat out". I mean, management has to have a learning objective. To understand the system that they're managing (because no one understands it fully, their mental model is different than what's actually happening). QA, along with other sources, inform the model of management who can then work with teams to improve the overall system.
This is the core of the TQM/Six Sigma theory
You will always have some product that needs to be reworked or rejected. The goal is to ensure quality at every stage in order to reduce the amount of rework. You don't want to go the way of the old US car companies who had to suffer major setbacks in the market because they were spending gross amounts of time reworking defective cars before they could ship them. I do not have a copy of it in front of me, but one of Womack's books had some numbers. Double digit percentage of time for a car's production was spent reworking the car after it had been made to make it suitable for sale.
Now, a premise of Agile (and Lean) is to improve the feedback loop. This is where testing and other inspections come into play. By doing them more often (run unit tests on check in, reject if any formerly passing tests start failing, for instance) you can address some quality concerns earlier.
But you still have to address the cause of the rejections. If Stage 10 of production consistently requires rework, it's great that you're catching, addressing, and doing the rework right then. At least it's better than after Stage 20. But management has to work with QA to not just inspect and accept/reject, they have to address the systemic causes of the failure.
can you as an owner delay updates?
I have a Model 3 and so far haven't had any problems with updates. Installed the most recent one (2018.32.2) last night.
"... caused almost the entire fleet to reboot loop ..."
I am not interested in being part of someone's fleet. The fact that they use this language at all to describe an end-user who has purchased an automobile suggests that their expectations and my own - of what it means to purchase and operate a car - are in deeply (possibly dangerously) mis-aligned.
So, if you have a car manufactured any time in the last 25 years, you're running software that hasn't likely been patched in years, has a ton of unknown defects and bugs, and might kill you if you hit an edge case that wasn't tested before it was shipped to you.
I much prefer Tesla's ability to fix software defects remotely than driving a defective piece of software. For example, I had a 2009 Hyundai Genesis that worked great until I had about 75,000 miles on it, then it mysteriously started losing engine power and I had the entire computer reboot a couple times while I was driving. Imagine your entire instrument cluster going dark and losing engine, braking, and steering power while you're traveling 75 mph on the freeway. The Hyundai dealership said the only way I could get a software/firmware upgrade was by purchasing a $500 maps DVD and having them update my system manually, which takes several hours, for which I'd have to pay one of their trained technicians to do it. Fuck them.
I vowed after that to never buy a car again that can't receive OTA updates. Would you buy a smart phone that can never get security fixes or updates? Given the tech in our cars now, why would you buy a car that wouldn't either?
So what do I do instead? I mostly cycle and ride where motorists don't drive and set the largest possible safety margins. I recognise this is not immediately practical for everyone but I'm fortunately set up in the right place with the right knowledge to achieve this.
So far this year, I've been in a car four times, a train twice and a plane twice. Musk is pushing for a world where everyone is dependent upon a form of low-occupancy heavy motorised transportation (including wanting to reinvent the train). Naturally, I recoil at this and so should more. More cars will never save the world.
Source: Father worked in car industry. All of my neighbors, too. Pretty much everyone I knew and then, eventually, me... albeit tangentially.
I am aware of that and I hear that terminology used by rental car companies and equipment dealers, etc.
My objection is to what I hear as a subtle difference - the post-sale automobile to a private, end-user is still referred to as belonging to their fleet - as if one's ownership and use of the car were a minor detail.
I dislike this subtle shift in language and attitude.
It's the language used by the rest of the industry including manufacturers, not just private owners.
If you buy a Tesla, you will not be able to control access to it without limiting its normal set of features.
If you buy a normal bicycle, you always have full direct control of it. No software updates, no data uploading, no tracking.
Normally fleet is reserved to meaning ownership in the management sense, not the micromanagement sense. Even a Navy fleet has autonomy within. Not so with Tesla software by default.
This could be rms territory. Free vs non-free, or even Airbus vs Boeing, etc.
And I think it's fair for the engineers that have responsibility to have an internal sense of partial ownership.
As long as they still have responsibilities, they also have partial ownership.
> ol' musky isn't totally paranoid - we did catch bad actors doing stuff and they were nailed to the wall. finding a real apt in your network can be some next level shit
That looks like confirmation that there really were internal threats found as claimed by Elon.
[1]: https://forums.somethingawful.com/showthread.php?threadid=38...
My personal expectation that it's all true is about 60% right now. Some of the figures could be exaggerated or misremembered, but most of the hackiness described is plausible.
> As a long-ago ex-Googler I find it amusing to watch employees name-drop where they work, always with some underlying subtext implying authority [...] Parent comment doesn't simply imply this but explicitly uses it as the basis for calling OP out. It's amusing because the namedrop is always accompanied by some claim they felt they couldn't support solely through a sound argument.
Also, you also left out the part where he is actually agreeing with the name-dropper person:
"Meanwhile the points made are relatively sound, 100kLOC C firmware updater set off my BS alarm (assuming 98k of that wasn't some static data tables), and the 700TB MySQL DB while possible is extremely unlikely.. even given a huge JBOD setup which nobody in their right mind would plug into a single server, InnoDB tops out at 64TB per table in the best case, which means OP would need a single database with somewhere around 10 64TB tables"
https://www.sitebuilderreport.com/blog/the-story-of-elon-mus...
I am short TSLA, make of that what you will.
Elon likes to push that narrative that there's this epic battle between himself and people with short positions in a good vs. evil way.
In fact, it seems pretty apparent that the company is in trouble and the people saying that have put their money where their mouth is. They also probably aren't obsessing over it every last minute or pushing narratives to try to get the stock to tank, it'll do that on it's own. In fact, I bet the majority of people who are holding tesla shorts aren't sneaky oil execs, but hedge fund guys who want a pickup/hedge for the market as a whole. It's pretty common to short stocks that look weak to protect against general market volatility for when your main portfolio takes a little dip
The conspiracies and the cult of personality surrounding a company in moderate financial trouble with problems delivering their product is pretty strange to me. Startups fail all the time, they also over promise and under deliver all the time.
There haven't been any real claims of people trying to influence the tesla stock price other than Elon. If he has this kind of info, he should send it to the SEC since they should be able to track down the nefarious bastards that hold a short position.
This is one of those situations that is probably exactly what it looks like. Elon is learning that hardware is much tougher to build than software.
My point was only that when there's a lot of money on the line, we should be skeptical of uncorroborated info (good or bad).
USENIX talk from VP of software at SpaceX (2016)
https://www.usenix.org/conference/lisa16/conference-program/...
(Speaker was concurrently interim VP of Autopilot at Tesla for a bit as well)
Few interesting bits:
- Speaker’s background is SRE at Google
- Engineering culture: Authority, autonomy, accountability. Blameless postmortems
- High Integrity C++ & MISRA coding standards
- The spacecraft has multiple onboard computers, all running Linux
- Tripe String Architecture, 3 redundant computers, whose results are cross checked with majority voting before being applied in real time, given radiation tolerant over radiation hardened hardware
Left as a reference, citation. Mods: Wouldn't suggest changing to this URL, UX is pretty ick, easier to read the Twitter screenshots.
I mean, they are held to lower standards than random production VMs that I manage at work, and those can't even kill anyone.
I may trust Tesla not to crash my car or violate my privacy, but I also have to trust them to sufficiently secure the SSH keys from bad actors who would (or sell them on the black market where they almost certainly would arrive in malevolent and capable hands). The latter is what scares me.
This is the most WTF aspect of the model 3 to me and seems to confirm that there isn't going to be some last-minute mitigation. I know it was done because autopilot was expected to be mature by the time model 3 rolls out, but I don't see how it can even be street-legal when you need to look away from the road to determine your speed.
There's an entire generation of traffic safety cameras that detect people looking down away from the road and it looks to me like they'd be tripped by a model 3 driver just reading their speed...
It's also faster and easier to take screenshots than it is to preserve formatting of HTML/CSS and it suits Twitter.
Related topic: Azealia Banks's recent screenshots of messages relating to Elon Musk, his companies, and other things.
The OP user did end up posting a link to the thread, but I never bothered clicking it. The first screenshot grabbed my attention sufficiently that I just continued looking at the next. They almost act in a similar manner to Wikipedia link previews. Twitter for better or worse is where things still go "viral". I had a love-hate relationship with it but no longer spend a lot of time there these days. There is some method to the madness...
There are lots of people with motivation to pull it off. Either for money or for some political reason nothing to do with Tesla.
Tesla is not alone. Hardware companies in general don't think about software security at all. They are security nightmares waiting to happen.
While there is some cringeworthy stuff in there, I've seen way worse in healthcare when people's personal data and medical history was at stake. Nothing made me so alarmed that I wouldn't drive my Tesla.
I really don't think tesla is the 'worse offender' when it comes to the type of horror stories we read in this article.
This is what systems look like in every industry. Look no further than the crappy Java SWT GUIs you see in those leaked abhorrent NSA powerpoints.
I will never sign a NDA that doesn't expire in <10 yrs after cessation of work, as its just too risky to indefinitely hold any information secret to the level required in most NDAs. Management changes, and what the current management is fine with, future management could not, or you do some work that the company doesn't like 10 years down the road and they dig through the filing cabinet looking for something to hit you with.
https://forums.somethingawful.com/showthread.php?threadid=36...
Do software engineers typically have confidentiality agreements that prohibit releasing proprietary information?
How does that invalidate anything of what he said or vindicate Tesla exactly? Do you believe that because some large companies are shitty places to work that everyone should just accept it as the status quo and shut the fuck up about it? Why are you expecting a single employee to try to 'change things' and why do you think that him speaking up about the issues is not an attempt to do just that?
I don't get your attitude at all.
If you say so. As an embedded system engineer, I mentioned this twitter thread when I was shooting the breeze with my boss because it reminds me we hold ourselves to a high standard even though my brain sometimes turns into a perfectionist that can only see the problems with our product.
I don't know how to say this without going into full on humblebrag mode, so I'll give the condensed version. He touched on a handful of points that are common between our companies/products, and where I often worry about quality. In those cases I was reassured by the fact we don't have that severe of problems, and the way our processes are designed it would never be allowed out of the lab.
You would be shocked to find the amount of Perl/Tcl glue in any ASIC company, for example.
Free software cars when?
Here: https://www.reddit.com/r/EnoughMuskSpam/comments/99sbwa/form...
While not even remotely on the same scale, I just spent 3 weeks building an app as fast as possible because my client's old one was causing 75% of their support calls. The code isn't pretty, but it works and it works a lot better than the old one. I know deep down that code is going to stay ugly for a while but that isn't what's important right now.
I guess I was commenting more on the general messiness he was describing. I can empathize with it is all. I can see how quickly and easily it happens in the small apps I build, I can't imagine how hard it would be to reign it all in at Tesla scale at the pace they've been going.
Sounds like every corporate environment I ever encountered, and Tesla it seems. It usually happens becuase they have the money to do it wrong multiple times rather than no choice but to get it right the first time.
Did you ever floor a car and have the engine shut off because there was too much fuel?
No thanks, I don't want to go back to those days!
I certainly miss those days. Working on my current vehicle is a PITA. The fuel rails, coils and plugs take hours to change. On my old car, I could do that in 15 minutes even if the engine was hot. I can't even imagine how proprietary the components on a Tesla must be or how difficult it will be to fix myself.
I've never operated a fuel-injected vehicle that I wished had a carb.
The truck I learned to drive on would cut out if you turned left abruptly, or hit a slight bump while turning left somewhat less abruptly, while off the throttle. We lived on a really steep hill with a ravine at the bottom which meant if you weren't careful to give it gas on the (downhill!) left turn, you would soon find yourself operating a vehicle with no power steering and no power brakes, trying to use one arm to steer the now-manual steering, while using the other to try to restart it, while stomping on the manual brakes, hoping everything came together and restarted before the ever-increasing downward grade ran you across a busy road and into a ravine.
My first motorcycle had a carb, and fuelling issues on motorcycles are even WORSE because they screw up your balance.
Does anyone know what forum they are referring too?
In the few hours after I stopped procrastinating at work today, every post in that thread has went to multiple thousands of likes/share/etc. Guess that's my answer.
thanks but no thanks!
Tesla:Elon Musk:?
Think I will be holding on to my ten year-old car as long as possible.
The latter fits better with their marketing vs. reality gap on “autopilot” and subsequent crashes.
Excuse me, I am a native English speaker and I do not understand this reference
Autocorrect didn’t enter my mind as a possibility.
a) car manufacturing is one of the most competitive industries, b) 1000s of engineers are working on making better cars, c) if it was possible to make profitable, affordable, self-driving electric cars, somebody would have done it. Comes off as a snake oil salesman.