erikbern

Spoiler alerts but this exactly the plot twist at the end of the book "Around the World in 80 Days" by Jules Verne

https://en.wikipedia.org/wiki/Around_the_World_in_Eighty_Day...

The following day Fogg apologises to Aouda for bringing her with him since he now has to live in poverty and cannot support her. Aouda confesses that she loves him and asks him to marry her. As Passepartout notifies a minister, he learns that he is mistaken in the date – it is not 22 December, but instead 21 December. Because the party had travelled eastward, their days were shortened by four minutes for every degree of longitude they crossed; thus, although they had experienced the same amount of time abroad as people had experienced in London, they had seen 80 sunrises and sunsets while London had seen only 79. Passepartout informs Fogg of his mistake and Fogg hurries to the Club just in time to meet his deadline and win the wager. Having spent almost £19,000 of his travel money during the journey, he divides the remainder between Passepartout and Fix and marries Aouda.

> Pay yourself and a VC-appointed board member below market rate, hire six good engineers, lease some office space, buy equipment, pay for legal and accounting services… and presto, you’re burning through $5M a year

Would love to see the math behind this. How does this add up to $5M/year??

Founder of Modal here. We've spent a ton of time on this, including building our own distributed file system optimized for low-latency high-througput workloads. We don't use K8s or Docker and built our own custom infrastructure instead.

Cold starting containers quickly is a fascinating problems. We've gotten a long way but there's still a lot more to do. For GPU-based inference, starting containers isn't enough – you also need to initialize the model GPU quickly. We are working on a long list of things that will bring down cold start latency even further.

Annoy author here. What you're describing is Locality Sensitive Hashing (LSH) which is something I spent a lot of time trying back in 2009-2012 but never got it working. It has some elegant theoretical properties, but empirically it always has terrible performance. The reason I don't think it works well is that data often lies near a lower-dimensionality manifold that may have some sort of shape. LSH would "waste" most splits (because it doesn't understand the data distribution) but using trees (and finding splits such that the point set is partitioned well) ends up "discovering" the data distribution better.

(but HNSW is generally much better than this tree paritioning scheme)

I don't really buy this argument that you can rent and pay $1000/month, or you can buy an equivalent home and pay $1000/month, but now you can deduct interest rate and build equity.

Landlords can also deduct their interest expenses and so they benefit from leverage too (meaning they don't want to build up equity).

In a reasonably efficient market, this means that the equivalent cost of homeownership would be higher than rent, to offset these things (the rate at which you're paying down the principal, and the deductability of interest rate expenses). This is obviously a very simplified argument, and there's a lot of other factors going into this.

haha, author here, I do admit that Matt Levine is one of my favorite writers today

I'm one of the authors – there's actually slightly newer benchmarks at http://ann-benchmarks.com/

Reminds me it's probably time to re-run the benchmarks and publish newer numbers

i'm the author – would love to read any existing literature on this if you can provide any pointers!

I wrote the code and you're wrong :)

1. Diagonal moves have a slightly higher weight. 2. I have an edge weight of 100 to go through another person. This makes it possible to go _towards_ the right direction even though there is no path 3. For the perpendicular lines method, the distance is modified to 1e-3 4. A* doesn't work because the heuristic becomes pretty useless when you have moves that are very "cheap": you need a lower bound that's pretty much zero

So for all those reasons, I just went with plain old Dijkstra!

This is probably just a variant of Berkson's paradox, similar to Google's observation that success in programming competitions is negatively correlated with job performance: http://www.catonmat.net/blog/programming-competitions-work-p...

The mechanism would work this way: sales people exhibit multiple features, and they are promoted based on some combination of those. If a sales person has outstanding other credentials, they might be promoted despite poor sales percentile. Those other credentials might actually be better predictors of managerial experience. Conversely, many of the top sales people might have been promoted on the grounds that they were good sales people, without exhibiting any other skills.

Note that there might still be a positive correlations between sales skills and managerial skills, but due to how the promotions are selected, you end up observing a negative correlation in the promoted group.

https://en.wikipedia.org/wiki/Berkson%27s_paradox

They should have used containers.

I've found these tools extremely useful to understand user behavior. We rely on Fullstory all the time to improve the user experience, to identify bugs, and to debug user issues.

The result of these tools is a far better user experience on our site and many other sites.

"recording every keystroke" makes it sound like there's malicious intent, but it's misleading. It should be added that all these tools have a lot of options to avoid tracking sensitive data (like password fields) and we always rely on that (in fact anything else would be a compliance violation for us).

classical music has notoriously bad metadata on spotify – it just doesn't fit into the data model of artist/album/track. i think that causes the system to filter out a lot of the listening data or fail to find patterns.

not sure what's up with jazz but i'm guessing it could be a similar problem

only thing you can do to influence the collaborative filtering system is to make users listen to both you and some other artist.

as i mentioned in another reply, CF is really what powers DW

We had to build some explicit filters for this at Spotify. For instance we blacklisted Christmas music from being part of the listening data – until them people would get a ton of Christmas music recommendations every January. But to your point, there's a long tail of other contextual things.

Spotify re-runs the latent vector models regularly and re-indexes them into Annoy indexes. There is no need to do that in real time, you can be a few weeks delayed and it's usually fine. New music doesn't have much data and need different methods anyway.

I built the foundation of this system while at Spotify. While it's true that we looked at a lot of different signal, at the point when I left (early 2015), it was all based on collaborative filtering.

The reason collaborative filtering works so much better than anything else is that given enough data, it will already encompass everything else. If there are reasons why certain users prefer certain sounds, or certain lyrics, those patterns will emerge in the listening data.

The main reason to use any non-CF method is mainly for new content that Spotify doesn't have much listening data for.

I'm no longer at Spotify, but let me know if you have any questions

I interview 20-30 people per week and send most of them Hackerrank tests. I'm a big proponent of vetting that people can write code, but 99% of the questions on HR are waaaay too "algorithmic". Had to go through almost all of them to find a few tests that are not about graph algos or dynamic programming – finally found a question that was based on regular expressions and a few other ones that I think are a bit more representative of real world challenges.

I really wish Hackerrank could add more problems or let employers add their own question. Are there any good alternatives to HR?

Author here. It's slightly embarrassing that this turns out to be an old idea – it wasn't my intent to rip it off. I'm fairly sure I must have seen it a long time ago and then forgot about its origin. In retrospect I probably should have googled it.

Somewhat speculative, but as a Swedish person living in the US, a huge difference between the gov't spending is that very little is means tested in Sweden. Doesn't matter how much your income is, you still receive child allowance, extremely subsidized child care, schools, etc.

As a consequence, the welfare system is as much of a transfer between rich and poor as it is between different parts of life. And I suspect the willingness to pay higher taxes is that (almost) everyone "benefits" from the it at some point (putting it in citation marks since there's no free lunch obviously).

In contrast, US has far more means tested programs like SNAP etc, and the things that the wealthy benefits from are things like mortgage interest deductions, which are not seen as gov't spending in the same way. Which I think goes a long way explaining the tax adversity in the US.

Right. Both the matrix S and the identity matrix will project the stationary distribution onto itself. So any linear combination of them will project the stationary distribution onto itself. Let me know if I'm saying something really stupid

Yes, this is correct. The second table shows conditional probabilities. Almost everyone moving from ObjC goes to Swift, but conversely out of the people moving from Swift it's more spread out

This is a great point. It's a model that predicts something about the future. I could have backtested it on historical Google stats to figure out if it's a good model :)

Author here. Yes you are reading this incorrectly. Look at the contingency table.

objective c to swift: 5216 swift to objective c: 1639

(sorry about the small font size though. had to squint really hard)

Author here. You are absolutely right. As I mentioned in the notes, I think this matters a bit less than it might seem like (the stationary distribution does not change if you add a diagonal matrix) but clearly some languages will have a higher propensity for people to stay.

I think this flaw is even smaller than the issue of using Google statistics to infer transition probabilities. It's just a shitty proxy, at best.

At the end of the day, there's a lot of assumptions going into this analysis. I hope I didn't make it seem more serious than I meant it to be – it's really just a fun project and kind of a joke not to meant taken seriously.

That being said, I think the conclusions are at least "directionally" correct. They might be off by a factor of 2x or 5x or even 10x, but the stationary distribution exhibits an even bigger spread (multiple orders of magnitude) so I suspect the final ranking is still "roughly" correct (with a very liberal definition of "rough")

I'm one of the original Luigi authors, and used to maintain it for a while, merging PR's daily etc. I left Spotify a couple of years ago. Since then it's been maintained mostly by Arash Rouhani, who is increasingly getting busy with other things.

But it's very much in active development and there are multiple pull requests merged every day.

I haven't had a lot of time to check out Airflow, but it seems great. Data engineering and thinking of data processing as functional pipelines is a great paradigm, think we're going to see a lot of future development in this area. Luigi will probably evolve a lot over the next few years. Eventually I think there will be better frameworks. No idea if Airflow is a step change, I think there are still projects yet to be built that unifies everything beautifully

There's some rationality to people's fear for terrorism. The impact of a terrorist attack is probably exponentially distributed, like an earthquake attack. Saying no one has died in earthquakes in SF in the last 10 years is not relevant.

Or pandemics – people are afraid of the next Ebola for the right reasons. The 1918 influenza killed 20-50M people.

I switched to Dvorak about ~10 years ago. Took about 1 month to get comfortable. After 2-3 months I still wasn't as fast as qwerty, but it already felt more comfortable to type than qwerty for some reason.

Unfortunately I gave up after about 3 months, mostly because I kept changing computers at that point and it was a bit annoying to have to switch back and forth. In retrospect I regret that – maybe it's time to give it another shot

"requests per minute" is such a useless unit of measurement. Please always quote request rates per second (i.e. Hz).

Makes me think of the Abraham Simpson quote: "My car gets 40 rods to the hogshead and that's the way I likes it!"

There's a similar argument for why Goldbach's conjecture* might just be a statistical fluke. I remember realizing this in high school and deriving the probability – alas it's not a novel idea and Wikipedia has a great summary: https://en.wikipedia.org/wiki/Goldbach%27s_conjecture#Heuris...

It's very possible that it's an unprovable true conjecture, which is kind of interesting in itself.

* every even number can be written as a sum of two primes