HNHacker News
TopNewBestAskShowJobs

jamii

3,462 karma · joined November 17, 2008

http://scattered-thoughts.net/

jamie@scattered-thoughts.net

submissionscomments
jamii··on Assorted Thoughts on Zig and Rust
I believe zig does not intend to implement this eg https://github.com/ziglang/zig/issues/2479#issuecomment-5709...

Naming conventions seem like the way to go.

jamii··on Assorted Thoughts on Zig and Rust
I don't think this is fair. There is definitely some unforced error, but a lot of the complexity is shared with other languages in it's heritage eg the trait system is no more complicated than haskell's typeclasses, and haskell also has macro systems on top of that.

Also the simplicity in zig comes from a major innovation in controlled partial evaluation, which we've yet to really explore the consequences of. Rust had already made a major innovation in the form of the lifetime system. It needed to figure out all the implications of that so it made sense to draw from existing designs where possible for the rest of the language rather than piling on more new and untested ideas.

jamii··on Assorted Thoughts on Zig and Rust
I forgot rust doesn't upcast numbers, but I was mostly thinking about https://doc.rust-lang.org/nomicon/coercions.html
jamii··on Pinephone first steps
Wow, definitely going to try that out. It takes that long for my android phone just to unlock sometimes.
jamii··on SELECT wat FROM sql
ORDER BY actually happens last.

    jamie=# (select a, a+1 as c from nums) order by b,c;
     a | c 
    ---+---
     3 | 4
     0 | 1
     1 | 2
     2 | 3
(4 rows)

It's a separate part of the grammar:

    <query primary> ::=
        <simple table>
      | <left paren> <query expression body> [ <order by clause> ] [ <result offset clause> ] [ <fetch first clause> ] <right paren>
But, if "QE is a <query expression body> that is a <query term> that is a <query primary> that is a <simpletable> that is a <query specification>" then the <order by clause> may select columns from the <table expression> even though those don't really exist any more by the time the <order by clause> runs. It's tricky to implement.
jamii··on SELECT wat FROM sql
There isn't really any math involved. It's just a lot of edge cases written out in lots of detail.

I think a lot of what makes SQL hard to implement comes down to consistently prioritizing local convenience over global complexity, combined with not being able to break backwards compatibility to simplify the design.

jamii··on SELECT wat FROM sql
I never figured out why AS isn't allowed in the GROUP BY clause. It would be much simpler to implement `SELECT a1 FROM nums GROUP BY a+1 AS a1`.
jamii··on SELECT wat FROM sql
Exactly. We're often on both ends of this - some BI tool will generate a SQL expression that a human would never have typed, and we have to handle it the same way that postgres would.
jamii··on SELECT wat FROM sql
I think a lot of these are in the ANSI 2016 spec. I haven't looked at the older specs, maybe they were simpler.

Eg here is the `a+1` vs `1+a` thing explained for ORDER BY - https://i.imgur.com/bklpjkP.png. Here are the conditions under which ORDER BY can reach inside SELECT to pick columns from the FROM clause - https://i.imgur.com/aoeqDGn.png.

Chapter 2 of the spec is 1732 pages of that kind of language, so it's hard to be sure that any particular behaviour is not in there somewhere. But I'm fairly sure that all the examples with set-valued functions are postgres-isms.

jamii··on SELECT wat FROM sql
> PostgreSQL (and maybe other SQL implementations; not sure!) has some smarts about detecting when an expression is used verbatim in a SQL query

I think this one is in the SQL 2016 spec. There's definitely a section about how to put the AST into a normal form to compare expressions for syntactic equality.

EDIT found it - https://news.ycombinator.com/item?id=22991487

jamii··on Materialize: A Streaming Data Warehouse
> would you go with rust again

Absolutely. Even aside from safety, rust has so many quality of life improvements over C++ - algebraic data types, pattern matching, checked errors instead of exceptions, sane standard library, easy packaging, readable error messages, #[must_use], immutable utf8 strings, traits instead of templates, hygienic macros etc.

Other than compile times and editor tooling, which are being actively worked on, the only real pain point I can think of is the poor support for custom allocators.

As for other alternatives, I don't personally have strong opinions on go but I think it's notable that none of the ex-cockroachdb folks at materialize suggested using it instead of rust.

jamii··on Materialize: A Streaming Data Warehouse
We're working on benchmarks, but I expect that at the moment Materialize will be slower for one-off queries but faster for frequently repeated queries.
jamii··on Materialize: A Streaming Data Warehouse
Adding sources is pretty easy. We just need them to provide:

* timestamped changes eg "record foo was deleted at time 42"

* watermarks eg "I've now sent you all the changes up to time 42"

Here's the implementation of the kafka source - https://github.com/MaterializeInc/materialize/blob/master/sr.... (There's also a bit of code in the parser/planner to wire up the CREATE SOURCE syntax, but it's fairly trivial.)

jamii··on Materialize: A Streaming Data Warehouse
The main difference vs KSQL is that we support standard SQL (roughly SQL92 + correlated subqueries + json so far) and provide strong consistency:

KSQL has a distinction between streams and tables, effectively giving you control over how views are materialized but also forcing you to do that work yourself. In Materialize you just write the same SQL that you would for a batch job and the planner figures out how to transform it into a streaming dataflow.

KSQL is also eventually consistent - late arriving records might cause changes to previously emitted answers. Materialize won't give you an answer for a given timestamp until it can guarantee that the answer is correct and won't be retroactively updated later.

Expect to see some performance comparisons soon too.

jamii··on Materialize: A Streaming Data Warehouse
> Instead, you register persistent, stateful enrichment "streaming jobs" (i.e. incrementally-materialized views) into the data warehouse; and then, when data comes into a table upstream of these views, it gets streamed into and through the related job to incrementally populate the matview.

This is correct. There is an example in the docs of dumping json records into a source and then using materialized views to normalize and query them - https://materialize.io/docs/demos/microservice/

jamii··on Programmers Should Plan for Lower Pay?
> old and crippled finally going on the vacation you dreamed of > living off rehydrated beans and eggs

I'm continually baffled by this kind of reaction.

I spend ~30k USD per year - more than the average american and more than 99% of the world. I live in a nice apartment with a garden, a 10 minute commute from the centre of one of the most expensive cities in the world. I buy organic food from the local supermarket and eat out multiple times per week. This year I vacationed in Tenerife, Greece, Italy, San Francisco and Vancouver.

This is, by any reasonable standards, a life of luxury. And on a typical FAANG salary it would take ~7 years to save for retirement - https://networthify.com/calculator/earlyretirement?income=13...

You would think most people would be interested to learn that their level of wealth opens up the option of complete financial independence in a comfortable middle class lifestyle before the age of 30, but instead it's always straight to yelling about dried beans.

jamii··on PinePhone Review
It's running straight-up linux. You can write apps in python with gtk or qt, or anything else that works on a linux desktop.
jamii··on Shenandoah GC in production: experience report
But the per-process gc is pretty slow, and stw pauses are still proportional to the size of the working set in that process. If you have a multi-gb data-set you have to choose between a single process with long pauses, or spreading the data across multiple processes and paying for copying, locking and cache contention. Not to mention the extra code complexity of speading operations on single data-structure across multiple asynchronous sections. In many cases, for large working sets you're still better off with one the JVM GCs.
jamii··on Giving up on wlroots-rs
If you want to see some examples, Neon uses this pattern to solve a similar problem - preventing rust code from holding onto js objects that might get gced between callbacks.

https://github.com/neon-bindings/neon

jamii··on Entrepreneurism Is Overrated
Judging by the comments, a lot of people value entrepreneurship for the personal freedom and autonomy.

The article suggests being able to save 89k per year. Under reasonable assumptions, 7.5 years of saving at this rate will support an income of 31k / year (the median US salary) for the rest of your life, leaving you with ~50 years of runway for whatever else you care to do.

This is especially interesting if the thing you actually want to do is something that is valuable to society but not profitable eg maintaining free software or raising a child.

https://networthify.com/calculator/earlyretirement?income=12...

jamii··on Karl Friston: a neuroscientist who might hold the key to true AI
Notes from the last time I tried to understand this - https://www.lesswrong.com/posts/wpZJvgQ4HvJE2bysy/god-help-u...
jamii··on Small robots may kill the tractor and make farming efficient
Seeing Like A State goes into a fair amount of detail.

From my notes:

* Soviet collectivization. Inspired by huge mechanized farms in the US. Vision of doing for farming what assembly lines did for manufacturing. Both Soviet and US attempts to do so failed badly. Party resorted instead to martial law and grain seizures, creating peasant uprisings. Forcibly relocated peasants into standardized, pre-planned mega-farms which were ruled by agricultural specialists. Ignored local conditions, vastly over-simplified and over-abstracted different areas of land and differently-skilled populations. Pressure from above to deny failures led to plans quickly losing contact with reality. Peasants gained most of their food by farming their own private plots in their little free time. Peasants effectively became indentured slaves, stripped of any cultural institutions that might be a focal point for rebellion, and naturally responded with terrible productivity. More deaths from starvation that WW1 and civil war combined.

* Compulsory villagization in Tanzania. Similar story. Government (with support of the Western world) wanted to modernize the peasantry. Forcibly relocated peasants to standardized, pre-planned villages. (Was supposed to be voluntary, but top-down pressure for results led to initiative-taking). Peasants were moved vast distances, rendering their deep local knowledge worthless. (Peasants don’t come in standardized, fungible units). Forced to apply Western farming techniques which failed badly in the local climate and ecology. (The peasant practices of dense polycropping turn out to be much more effective in climates with high primary productivity than monocropping and ridging as is common in the West). Authorities heard reports of atrocities but insisted they were isolated cases.

* And again in Ethiopia. Lead to widespread famine.

jamii··on Beating hash tables with trees? The ART-ful radix trie
My experience has been that the vast majority of papers on data-structures are at best misleading, and at worst deliberately biased.

For example:

> The hash table used by the authors of ART in their study was a chained hash table, but this kind of hash tables can be suboptimal in terms of space and performance due to their potentially high use of pointers.

> Our experiments strongly indicate that neither ART nor Judy are competitive to the aforementioned hashing schemes in terms of performance, and, in the case of ART, sometimes not even in terms of space.

https://www.victoralvarez.net/papers/A%20Comparison%20of%20A...

jamii··on Ask HN: Freelancer? Seeking freelancer? (November 2018)
SEEKING WORK | London | Remote is great

I specialize in systems engineering and performance optimization. I can help you:

* handle big data without building an expensive distributed system

* replace complex hand-coded data pipelines with simple declarative languages

* improve response times and reduce server expenses for your web service

* use web assembly to speed up hotspots in your web app

* train your team in Rust or Julia

"Jamie is a first principles thinker with a deep understanding of systems and algorithms. He is fearless in the way he approaches problems. Jamie’s research on staged compilation saved our team many person years of effort, and his training and support was crucial to our adoption of Julia. I hope to have the opportunity to work with him again soon."

-- Molham Aref, CEO, RelationalAI

"Jamie brought a degree of rigor and theoretical knowledge that took the company to places we never could've gone otherwise. From exploring the implications of language semantics to building high performance database engines, he didn't just help us get things done, he helped us get them done right. As a result, we were able to explore in weeks what could've easily taken years."

-- Chris Granger, CEO, Eve

"Jamie took over the latexsearch engine, our most technically challenging project to date, and turned it from a foundering prototype into a successful service with over 200k users. His engine required only one small server and ran uninterrupted for over 5 years with no errors."

-- Brian Bishop, VP Platform Development, Springer

jamie@scattered-thoughts.net

http://scattered-thoughts.net/resume

jamii··on Zero-copy deserialization in Julia
Also being able to remove all the dispatch and stack-allocate everything. Haskell would probably do a good job of that too but in python or js, even with codegen, it would be hard to avoid heap allocation. The simple examples here might fall to escape analysis, but in production code the intermediate Blob values typically cross a lot of function boundaries.
jamii··on Zero-copy deserialization in Julia
I used Ancient years ago for https://github.com/jamii/texsearch to stop the GC pauses caused by pointlessly traversing the huge immutable index. I don't know how else I would have met the latency requirements. Thanks for writing it :)
jamii··on Zero-copy deserialization in Julia
Huh, yeah, this is surprisingly similar, even down to needing a custom pointer type to do the offset<->pointer conversion - https://github.com/ipkn/dumpable/blob/master/dptr.h#L64-L69
jamii··on Zero-copy deserialization in Julia
If you allow codegen in advance it looks pretty similar eg https://capnproto.org/cxx.html
jamii··on Zero-copy deserialization in Julia
JuliaDB already does this eg `loadtable("foo.csv", output="foo/")` will store the data in "foo/" in binary form and mmap it.
jamii··on Zero-copy deserialization in Julia
Arrow is basically a dataframe format. It's not really suitable for building data-structures. Eg one of the things Blobs is being used to implement is a Bε-tree (http://supertech.csail.mit.edu/papers/BenderFaJa15.pdf).

Blobs is less a data-format and more a replacement for C's ability to cast arbitrary memory locations to structs.

← PreviousPage 4 of 32Next →