Hacker News Reader

Comment by spookthesunset | Hacker News Reader

What about all the other chat apps? Why wasn't the whole thing dumped into a pile of hot lava the second slack could scale to an org the size of uber?

What what does scale mean anyway? Just employees using it or employees + contractors?

It's easy to armchair quarterback on this stuff, but I find it hilarious how tech companies that employee supposedly smart people justify writing this kind of stuff. I'm sure some junior engineer did a cost/benefit that completely neglected the total cost of ownership for writing a chat app and focused on the sticker shock of having to pay tens of thousands a month for a chat app.

View on HN

another issue was data domiciling and retention policies around 2016 when uber was evaluating slack.

Were these actual concerns that were raised by the actual legal council of the company, or were they issues invented by engineers who are not lawyers but suffer from engineers disease and think that knowing to code means they are experts in everything else? 'Cause I know plenty of those!

I've worked in plenty of companies with engineers who think they are legal, privacy and security experts. If we made business decisions based on these folks, we'd never do anything. Hell, every damn text change we makes invokes a few who worry about the legal ramifications ("should we file a JIRA ticket with legal?").

Sadly without somebody stepping in, people actually listen to them instead of the actual experts employed by the company... thus you wind up wasting tons of engineering bandwidth building a feature / platform / product that has no business existing all (or worse, not building some feature / platform / product) because nobody bothered to check with the legal / privacy / security team that some concern by a engineer was actually a genuine concern.

And even then, in any good organization legal / privacy / security teams exist only to point out all the risks in any business decision... sometimes it is worth the risk despite what legal / privacy / security say. What appetite the company has for the risk depends on a lot of things that are well outside of legal/privacy/security.

it was the privacy/legal/security types driving this strategic decision, I can't really talk about it in more depth. A lot has changed since the decision was made, and if I personally was to reassess the situation now I believe I would go with a 3rd party SAAS solution, but back then building a chat app was on balance the correct decision to make.

> it was the privacy/legal/security types driving this strategic decision

Booo on legal.... like I said earlier, it's easy to armchair quarterback. Sometimes I wonder how the fuck humanity even manages to make progress with so much waste and boneheaded decision making.

> Were these actual concerns that were raised by the actual legal council of the company, or were they issues invented by engineers who are not lawyers but suffer from engineers disease and think that knowing to code means they are experts in everything else? 'Cause I know plenty of those!

the general counsel and thus the entire legal org required for any company approved comms system a way to automatically by default destroy chat logs after 7 days (and to turn that off for folks who were on legal holds). slack in 2016 did not have that capability.

The fact that that number was "7 days" tells you everything you need to know about Uber's internal culture.

Not sure why the parent is down voted. I have worked in financial services and had the same experience. IT typically held a view on the interpretation of rules and regulations that was far more literal and restrictive than the actual compliance officers.

Lyft is no better, spending $8M per month on AWS to process 1M transactions per day.

AWS is the opposite of homebrew. Lyft has no business maintaining its own data centers and operational overhead. There is a huge opportunity cost involved with running your own infrastructure. AWS and the like free your own teams from worrying about how to provision new systems, upgrading DB server versions, etc. Running your own data center is hard.

Running your own DC is the same as building your own chat app. It is almost always more expensive and almost all the people who claim they can do it cheaper inhouse aren't factoring in all the costs--both costs that are easy to measure and those that are almost impossible to measure

Examples of hard-to-quantify costs for running your own infrastructure include:

- What is the cost to the company of being three versions behind on mongo because IT doesn't have the bandwidth to upgrade? How many work-arounds do teams have to do because they couldn't use the latest features of mongo?

- What is the cost to the company when developers cannot quickly spin up new environments like they could using a service like heroku?

- How much innovation and new business (read $$$$) is not happening because the in-house IT simply doesn't have the toolset so a dev can quickly riff on an idea?

- How many engineers have good ideas and don't bother building them because spinning up an environment is too much trouble?

I'd argue by not using AWS you are holding your product teams back and stifle innovation. Unless of course you want to waste company money to build out your own half-assed provisioning tools that are lousy knock-offs of AWS's tools. But then, you might as well just use AWS...

I should have been more clear. AWS is absolutely the right choice.

What I was trying to point out is someone engineered a system that costs 8M/mo to run yet only handles 1M rides per day.

I don't see the analogy here. This is Lyft's core business. Of course they'll go all in. On the other hand chat is not Uber's core business. They didn't have to reinvent the wheel for an internal tool.

Is 8M/mo they outrageous a number here? Assuming Lyft paying 200k/year on average to its employees, 8M is about hiring 500 of them.

It is sizable, not ridiculous. And optimizing your cloud cost is definitely your own responsibility and deserves every bit of attention.

So they're paying 26 cents per ride on all compute resources across their entire company?

That doesn't really sound that bad, considering that rides can easily cost dozens of dollars. Just about any other non-tech business would kill for such low overhead.

> 26 cents per ride That's terrible.

Other non-tech businesses, hell EVERY business that I've ever worked in that used AWS did better per transaction - healthcare, digital advertising, virtual office tooling, gaming. If you can't do better than a credit card processing fee per paid customer interaction across the company on AWS, you might as well be selling retail goods...and depending on age, your company might be soon.

Even if it is terrible, is the business such than a 26 cent overhead on rides is going to make or break it?

Companies like Lyft are doing a lot more than updating a few fields for each ride. They would be processing massive amounts of location data and they are building out models to eventually use for self driving cars.

Thank you, my point exactly. Their infrastructure costs are terrible. Especially considering how easy their data is to shard etc (buyer and seller belong are in the same geo physically).

What? You are crazy. At a certain scale, it is cheaper to run your own DC. But that scale is reached by a very limited few.

Running your own data center is at one extreme. Most companies lease space in a datacenter, and all of the overhead of running the data center is on a different company. You'd have an infrastructure team racking and stacking, and supporting the deployments.

But past a certain point AWS is very expensive compared to renting a rack in a couple of DC's

Reply on news.ycombinator.com