runT1ME

Computation over Petabytes of data sounds fairly expensive, jobs that I was running over close to a PB could cost hundreds of dollars. Am I misremembering, doing it wrong, or underestimating your teams’s cloud budget?

Being able to predict ride demand, optimal routes, and pricing on the fly seems like a complicated enough of a problem that it would both require a lot of engineering and provide enough value over just a 'physical service company'.

I certainly enjoy using Uber and Lyft over Taxis, and when I travel for business I never rent a car these days where as I would always rent a car when confronted with the alternative of dealing with Taxis. Do you think i'm overestimating the advantage the software brings to the experience?

It's important to find the balance between letting your other engineers design and implement the more interesting and challenging problems, and also making sure they don't get in over their head.

Sometimes tech leads (often rightfully) assess that they are the best engineer in the room, so they should tackle the hardest problem. I don't think this is a good idea most of the time!

Giving ownership and autonomy is often one of the easiest ways to increase someone's productivity, so let your team take ownership of the parts of the project they are interested in.

Often a tech lead might end up having to take on the least fun tickets, the drudgery, yak shaving, etc. while the engineers working under you will get assigned work that has more prestige/glory. This won't go unnoticed, and will be appreciated by your team.

My current gig is using Scylla for storing time series data, we're partitioning by day and anywhere from 5-15 million rows per day.

We've mostly been using Apache Beam/Cloud Dataflow for working wtih the data en mass for occasional reports, and we're planning on trying Presto soon as well. I'm interested in what solution you come up with, happy to chat more about this kind of thing on twitter or email.

> but the primary datastore used at both Google and FB is MySQL

I'm almost certain this isn't true, BigTable and Spanner are much more widely used at Google because.... well, MySql doesn't really scale.

I do think Kotlin is a 'better java' while Scala fails to satisfy that niche. However, I wonder if we'll actually see some big Open Source projects/platforms released that are written in Kotlin. For Scala you have Kafka, Spark and Flink which are widely deployed in the enterprise. Will Kotlin have a marquee infrastructure project to tout soon?

I primarily use airpods for making calls, and over the ear headphones for music.

You are both making different points, because I agree with both of you. Some of the best engineers I’ve worked with have been foreign born, and I’ve also seen disasters happen due to outsourcing to foreign talent. These are two vastly different groups we are talking about. The best foreign engineers almost always end up working for well paying product companies. The other end of the spectrum, foreign or US, often end up at contracting companies that compete on price.

off topic, but how did you find your way to HN? Are you also in IT/engineering? A hobbyist? Just like the discussion?

Google is not dumb and realizes this, which is why they're pushing so many tools that prevent vendor lock-in. K8s entire pitch is you can seamlessly go from a cloud offering to your locally managed K8s in a datacenter. Our (small) company has done it, it works very smoothly.

Their 'serverless' batch/stream processing works this way as well, with Apache Beam you can deploy effortlessly to GCP for large compute jobs (i've spun up hundreds of nodes in minutes), and also run it against on prem spark clusters with zero code changes.

So Google realizes being late to the game means they have to offer something more, and what they're offering is a lack of vendor lock in.

>and scalability in databases has no universal solution and the ones that exists come with the load of caveats.

How is this true? Cassandra/MongoDB/Scylla are all very very popular NoSQL DBs that scale very well and work for a good number of common use cases. Sure you have to be a little more clever when writing certain types of applications with them, but seems like an appropriate trade off. If you don't like that trade off, you can spend a bit more money and get Spanner/FaunaDB and you don't have really any tradeoffs (other than $$$).

I had to do a project in C# many years ago, small piece of code that needed lots of concurrency and error handling. We were able to write it in a very clean monadic DSL (ala do notation in haskell or 'for' in Scala) by wrapping Task in lawful monad types, we wrote monad transformers, etc. We ended up being extremely productive and getting good performance.

I would almost never recommned doing something like this given the maintenance issues and how long it takes to bring the average C# programmer up to speed, but this was three fairly advanced FP folks working on the application. It was probably some of the cleanest C# I've gotten to work with.

>The boilerplate often helps when reading unfamiliar code

I don't believe this to be true at all, especially for large code bases. Often the people who support this position have not worked on 100k, 500k, 1m LOC applications I've found.

Boilerplate makes it much more difficult to read and scan large swatehs of code. I think when people say boilerplate 'helps', they often noticing a correlation between the fact that many strongly typed systems happen to have excess boilerplate. I believe it's the typing that helps, not the boilerplate.

Sure, it's unlikely someone will be totally clueless who can talk the ins and outs of a project. Let me put it another way. If we put all the engineers I've managed/lead throughout the years through your interview process, you would probably filter out the one or two who were bad, but you would not always select the strongest engineers. You'd select the strong engineers who were confident, gregarious, and articulate, and leave out the ones who are humble, shy, embarassed by their accents, not used to bragging about themselves or articulating the details of past projects.

Talking about those things is a learned skill just like writing on a whitebaord. Hence why good companeis that do whiteboarding interviews also have traditional rounds. You want to give candidates multiple avenues to shine.

I've been involved in software for almost twenty years, and while I'd agree the a majority of whiteboard interviewing is done poorly, it is something that developed out of necessity to both filter bad candidates and have a more objective interivew process.

When I first got into the industry, it was much more rare to have coding tests vs. just general interviews, and that meant teams often hired the best talkers or the folks who boasted the most about their accomplishments, and those two skills had very little correlation to how well someone could code.

Even worse, 'team fit' was often prioritized above others, so be it on purpose or implicitly, good candidates from diverse backgrounds were much easier to overlook. When you have the majority of engineers solely focusing on 'can this person solve a problem', it is harder to argue against strong candidates just because they look older/younger/darker/lighter/more feminine/masculine etc.

Take home tests are incredibly biased against candidates without free time, and as I've pointed out in other places, can easily be abused by spamming candidates with them which is just not possible with whiteboard coding.

Coding tests are the best way to hire, if they are done right. They aren't perfect, but so much better than take home tests and 'discussion' based interviews.

How so? Presumably someone with an internal 'cloud' needs someone to be responsible for a large kube cluster, to handle a uniform way to do scheduling/resource management, etc. Having an engineer who knows how to do this in AWS doesn't seem more expensive than an engineer who can roll their own internal 'cloud' system...

I can corroborate this, at Verizon we had problems scaling out ELBs for a very similar use case.

If I am doing a phone screen + an all day onsite, sure it sucks but at the very least I know I'm a serious enough candidate that the company is willing to burn one engineer day to evaluate me. With a coding challenge, I have no way of knowing! Maybe they sent the challenge to ten other candidates, maybe twenty, etc? I don't even know if the code will be reviewed. Make me jump through as many in person hoops as you want, I refuse to do take home coding challenges. shrug

I'm in SoCal, lots of companies around here will do low five figure starting bonuses now if you ask. I've seen 5k-15k in my offers at either smaller startups or non FAANG.

Wonderful article that both talks about overall problems and at the same time gives concrete easy to understand ways to test your model.

Of current renters, which often trails “market rate”.

Housing going up 13% means most homeowners made over 60% return on their money.

>we are not even close to achieving insect-level intelligence.

Is this true? Is there an insect turing test?

I believe that chart is using the governemnt's CPI to calculate the 'inflation adjusted' part. You can find all sorts of academic criticism of how CPI doesn't accurately reflect inflation (most notably of housing/rent prices), which is now the lion's share of a family's expense in the United States.

The times i've ever actually needed 'runtime' polymorphism (usually implemented w/ subtype & virtual methods) are utterly dwarfed by the times I'd been able to make use of parametric polymorphism, which I find far easier to reason about and use.

Scala

just a heads up, your resume is a bit...confusing. Are you looking for a PM position or as an engineer? I think 4 pages is too long as well, I'd try to keep it between 1 and 2 pages.

Huh, interesting. Looks pretty similar to some of Fanout's stuff. https://github.com/fanout/flychat

I would imagine part of it is how much comp has risen for developers. If a couple founders are trying to raise 50-100k I would wonder why they couldn't get the cash themsevles by working for a big tech company and being frugal with their RSUs or by working the salt mines of contract development...

Love what you're doing. Think the model could work for other well paying jobs such as the medical field or legal?