ntonozzi

One of the critical points here is using pretrained models to generate embeddings that the application level programmer uses. This technique isn't common knowledge among programmers without any ML experience, but is a key to getting good, generalizable that extend to unseen domains.

A 40 minute 10k is very fast, especially for a training run that is done many times a week.

No, an empty array means that the cache is never invalidated.

Yeah I’m not sure the “just” is justified, and it’s not good but it’s certainly different than being able to send tweets as a particular user. IPMI access was typically only given to SREs.

If you read the document "Security Chief's Final Report to Twitter" on the Washington Post article (https://www.washingtonpost.com/technology/interactive/2022/t...), you will see that 'god mode' just means they have IPMI access to servers.

> The other two languages I’ve used mostly in recent decades are Java and Ruby and the profiler situation is for both those languages is kind of shitty. I had to pay real money to get the Java profiler I used at AWS and while it worked, it was klunky, not fun to use.

These days, async profiler (https://github.com/jvm-profiling-tools/async-profiler) is much better than the Go tooling for performance. It is a joy to use and features a top-like view for the hottest methods. It works for locks, allocations and CPU time. It also integrates with JMH.

PgTyped is another high quality alternative: https://github.com/adelsz/pgtyped.

I'm not in the market right now, so don't do it for me :)

That makes sense, I think it's the right call.

This looks awesome, I'm excited to try it out.

It'd be cool to be able to dynamically deploy new configuration options without deploying new device code, or remove bad configurations. Looks like that could be possible but it would depend on your app to supply the configuration.

Is there any support for continuous variables (e.g. find me the best price between $10 and $20)?

They use the term EventBus, but really it's a thin EventBus client library that uses Kafka under the hood. I think all EventBus clients at Twitter have been migrated to use Kafka.

I discussed this method a little more in a blog post (https://blog.twitter.com/engineering/en_us/topics/infrastruc...), if you are trying to get an intuition here:

> we only allocate one pointer for every level of the skip list. In a typical skip list, a node will have a value, a pointer to the next largest element in the list and a pointer to the lower level of the skip list. This means that a new value allocated into the second level will require the space for two values and four pointers. We avoid this by always allocating skip list towers contiguously. Each level K pointer will only point to other level K pointers, so to extract the value associated with a level K pointer P, you read the value at P - K. Once you reach a node with a value greater than the one you are searching for, you go back to the previous node, and descend a level by simply subtracting one from the pointer. This lets us allocate a value into the second level by just consuming the space for the value and two pointers. It also reduces the amount of time we need to spend chasing pointers, because a pointer into a tower is likely to be on the same cache line as the lower pointers in that tower.

A similar idea is finger search (https://en.wikipedia.org/wiki/Finger_search), where you can turn a traversal of a tree from O(log(number of items in the tree)) into O(log(number of items between the finger and the key)).

Great article!

This seems like a fairly tricky ranking function. I wonder if they compared it to combining TF-IDF and the page popularity. This would help with the problem they explained.

It'd be interesting to see more details about how they implemented the query-specific page rank.

Very cool, seems much higher quality than the RMI library. I’m hoping more of these learned data structures make their way into libraries.

A related technique is generating an index using a machine learned model (well explained here: https://blog.acolyer.org/2018/01/08/the-case-for-learned-ind...). In interpolation search, we just use a simple linear model. Kraska and co. use much more complex models to beat BTree indexes. However, the Recursive Model Indexes (RMIs) that they published (https://github.com/learnedsystems/RMI) use Rust to a C++ model that is exported as a library, so it definitely doesn't seem ready for production yet!

Oh, that makes sense. I like the solution you arrived at!

It'd be interesting to see it compared to https://doc.rust-lang.org/std/collections/struct.BTreeSet.ht....

Their skiplist description is a little funny — typically a skiplist has one element per leaf node, and you should never need to move items between buckets. Also skiplists have logarithmic layers in the number of elements, so inserting and searching are always bounded by O(log n).

We left out some detail here to keep the post readable. In reality, we can handle 16 Tweets per millisecond per partition. We have tens of partitions, so that makes the edge case less likely. If we do see a gigantic growth in Tweet rate, we would need to increase the number of partitions anyways, so we decided this edge case was acceptable.

Congrats Quinn and company! I hope there's not too much of my code lurking in there =)

MVCC - Multi version concurrency control.

You can use search to find your tweets that are not in your most recent 3,200.

Ben Thompson has some awesome writing on tech titans, and their role in the economy. Here's some definitions: https://stratechery.com/2017/defining-aggregators/ and here's an some writing on why they are not necessarily bad monopolies: https://stratechery.com/2015/aggregation-theory/.

Verdaccio [https://github.com/verdaccio/verdaccio] looks like a viable open source alternative. Presumably you could use it to download the packages you needed, then disable proxying to the NPM registry.

That's the one I meant. You're right, that one does seem worse. I didn't realize that the previous one had information on so many people.

This one seems especially terrible because it had information on people unrelated to the US government.

Not just employees, but also "subjects, witnesses, and complainants". Yikes. This is way worse than the last one.

I agree, the Chrome dev tools feel much faster.

One essential feature for me is being able to pause the debugger and use the mouse selector on DOM elements, which doesn't work in Firefox right now. This is pretty much the only way I know how to inspect and debug popups.

Oh I see. Yes that does seem prohibitively slow.

The syntax highlighting is done in process using a declarative format, eg: https://github.com/Microsoft/vscode/blob/master/extensions/g....

Thanks for the report! We've fixed the bug now.

That definitely earns a sticker, check out https://sourcegraph.com/about if you want to redeem it :)

Another way to look at it: Sorting is linear in the size of the input in bytes, but not item count. It's only comparison sort that happens to be O(n log n).