1,513 karma · joined August 24, 2015
[ my public key: https://keybase.io/refset; my proof: https://keybase.io/refset/sigs/pyTK0thu8g5O4Zs7EA3HrQYKkYZjeEz_TUr8nZV2hlY ] c0ba44896414421c9009bc8a77a86ceb
meet.hn/city/51.0612766,-1.3131692/Winchester
Socials: - github.com/refset
---
I'm personally rather curious about the intersection of compilers and database query engines, where ~cheap dynamic/incremental compilation across extremely diverse runtime workloads is often a basic requirement.
[0] "Alibaba: Domain Knowledge Augmented AI for Databases (Jian Tan)" - https://www.youtube.com/watch?v=dsgHthzROj4&list=PLSE8ODhjZX...
[1] "CatSQL: Towards Real World Natural Language to SQL Applications" - https://www.vldb.org/pvldb/vol16/p1534-fu.pdf
He has also written many comments (arcfide on HN) about this stuff before, e.g. discussing "semantic density" [1]
> The compiler is designed so that I can see as much as possible with as little indirection as possible, so that when I see a piece of code I not only know how it works in complete detail, but how it connects to the world around it, and every single dependency related to it in basically one single half screen full of code (usually much less than that) without any jumps, paging, scrolling or any movement. [...] The idea of semantic density is critical to this point. The semantic density of the APL code I'm using to solve the problem is at a certain rate. I maintain a consistent density rate by choosing my variable names in such a way that they visually align with the expressivity per character of the built in primitive symbols.
And
> [...] idiomatic programming methods that are so concise, they can begin to be read as we read and chunk English phrases. By doing so, it becomes actually easier to just write out most algorithms, because the normal name for such an algorithm is basically as long as the algorithm itself written out. This means that I start to learn to chunk idioms as phrases and can read code directly, without the cost of name lookup indirection. I can get away with this because I've made reusability and abstraction less important (vastly so) because I can literally see every use case of every idiom on the screen at the same time. It literally would take more time to write the reusable abstraction than it would to just replace the idiomatic code in every place.
The key insight being that in this model "locks are never necessary", which is huge source of efficiency when working with sharded data across multiple cores.
> Thread-per-core architectures are friendly to modern hardware, as their local nature helps the application to take advantage of the fact that processors ship with more and more cores while storage gets faster, with modern NVMe devices having response times in the ballpark of an operating system context switch.
> A purely relational database that jettisons SQL doesn't have to have the limitations the author is poking at.
Agreed. Relational databases can take us a lot further yet.
Undoubtedly the reality of widely available SQL systems today has not lived up to that original relational promise in the context of modern expectations for large-scale reactive applications - maybe (hopefully) that can change - but in the meantime it's good to see Rama here with a fresh take on what can be achieved with a modern 3GL approach.
100% agreed. One of the biggest issues SQL databases have faced is that the scope & scale of "one transactional system" has evolved a lot more quickly than any off-the-shelf database architecture has been able to keep up with, resulting in an explosion of workaround technologies and approaches that really shouldn't need to exist.
We're now firmly in the cloud database era and can look at things like Spanner to judge how far away Postgres remains from satisfying modern availability & scaling expectations. But it will be great to see OSS close the gap (hopefully soon!).
> With the formal commencement of the "Notes" project upon us, it seemed appropriate to set down a few brief notions about the project, its scope, and its strategic importance to Lotus. This material should be regarded as more than highly confidential.
https://web.archive.org/web/20180225100127/http://www.kapor....
Also discussed previously: https://news.ycombinator.com/item?id=13168969
Beyond LLMs and vector search, the scope for applying AI / machine learning _within_ databases is enormous: join planning, learned indexes, compression, workload prediction, configuration tuning etc.
Given the current pace of advances perhaps a dedicated "AI in Databases in 2024" review will be on the cards this time next year.
Could Postgres yet evolve to become a Spanner-like multi-writer system?
Pat Helland has some really good writing on this stuff, e.g. https://pathelland.substack.com/p/i-am-so-glad-im-uncoordina...
Note it only supports "transaction time" (or "system time" per SQL:2011) but not "valid time" (~"event time") which is needed for a bitemporal data model. [1]
[1] https://vvvvalvalval.github.io/posts/2017-07-08-Datomic-this...
> Engineering is about trade-offs, and we’ve made a significant one with WarpStream: latency. The current implementation has a P99 of ~400ms for Produce requests because we never acknowledge data until it has been durably persisted in S3 and committed to our cloud control plane. In addition, our current P99 latency of data end-to-end from producer-to-consumer is around 1s
via https://www.warpstream.com/blog/kafka-is-dead-long-live-kafk...
> Truffle works by taking your interpreter and generating a compiler by partial evaluation. So I believe they have written a JVM bytecode interpreter and Truffle makes the JIT.
> I'm super impressed by how the Truffle/GraalVM team has been able to turn this theoretical concept into a system that yields production grade compilers
I would love to see this approach extended to add some animation too.