kyllo

Indexes don't solve the same problem as columnar because with an index you still have to dereference a pointer, read the page (performing I/O if on-disk), and locate each tuple you want within the page--the tuples you need to aggregate from may be scattered across many pages, and you have to read the whole page into memory even if you only want one column.

With a column store, the values of the same column are stored contiguously so if you're summing a column for example, you can read one or more pages that could consist entirely of values the column you need to sum, sequentially. It's much more efficient for aggregations due to cache locality--much fewer cache misses and I/O operations needed to summarize a column.

The major trade-off is that updates and deletes are more expensive in a column store. But this is usually fine for OLAP workloads.

Here's a YT video of one of the implementers giving a talk to CMU students about what DuckDB is, why they made it, and how it works: https://www.youtube.com/watch?v=PFUZlNQIndo

In a nutshell, it's a file-per-database embedded DB like SQLite, and it's installable as a Python or R package from PyPI or CRAN, but unlike SQLite, it's a column store with vectorized operations so it's more performant for analytical queries.

I was floored by this comment yesterday from one of their Developer Relations people:

> Did any of you actually read the article? We are passing the Jepsen test suite and it was back in 2017 already. So, no, MongoDB is not losing anything if you know what you are doing.

https://twitter.com/MBeugnet/status/1253622755049734150?s=20

Can you imagine saying the phrase "if you know what you are doing," in public, to your users, as a DevRel person? Unbelievable.

I was wondering the same--I guess because it's fairly new (2018) and it came out of a database research group at a European university, rather than a SV tech firm. Therefore, limited marketing budget.

By the way, here's a YT video of a talk given by one of the DuckDB implementers about why they made it, what it's for, and how it works: https://www.youtube.com/watch?v=PFUZlNQIndo

Basically--it's a struct containing a byte array, weight, scale, and sign, and all the arithmetic operations are implemented in software. So it's really slow, and each RDBMS has to implement this type from scratch, or find a vendor lib that provides a suitable one, because C/C++ standard libraries don't provide one.

If you like SQLite for data analysis, you might want to check out DuckDB https://github.com/cwida/duckdb which is billed as "SQLite for analytics."

SQLite is a row store, which is best for OLTP (point queries, inserting/updating/deleting one or a few rows at a time), while DuckDB is a column store, which means the data layout has values from the same column stored contiguously, making aggregation queries (GROUP BY) perform much better.

Most people don't realize that Excel (like PowerBI) has an in-memory, compressed, column store database inside of it.

Loading hundreds of millions of rows into it takes a while, but given a commensurate amount of RAM and a reasonable data model (single table or star schema), performing aggregations with a pivot table is pretty snappy.

Is there any form of distributed query engine in use today that doesn't fit the definition of the MapReduce pattern? Is describing a distributed query engine as MapReduce still a meaningful distinction from some other, non-MapReduce approach?

By contrast, China is the US' third largest trading partner (after Canada and Mexico) at 10% of US trade. Taiwan is much more intertwined with China economically. Which makes sense given the geographic proximity and linguistic and cultural ties.

Politically yes, Taiwan is de facto independent, but economically they are deeply intertwined; although they have their own currency, stock exchange etc., a lot of major companies operating in China are under Taiwanese ownership (Foxconn and Pegatron for example) and the commerce between China and Taiwan is significant; China is Taiwan's largest trading partner at 30% of Taiwan's total trade.

Good point, but it's worth noting that "go get a job on autopilot" behavior is entirely rational as it's driven by economic necessity--most people go get jobs because that's the only way they can afford food, shelter, healthcare, and other basic necessities. I wish it were optional!

A few people will do that, yes, but there are a couple million software developers in the US, and most of them are not going to suddenly move to sparsely populated interior states. They'll spread out from the SF bay area a bit.

This realistically isn't a major concern because the pool of qualified software engineers in a state like Idaho is far too small to make a noticeable dent in market salary levels.

That's basically what Salesforce did, but they ended up skimming a lot higher % than that

Yes, Filipino spaghetti is definitely intended to be sweet--it uses banana ketchup in the sauce. Bananas were used as a substitute for tomatoes due to the Philippines' relative abundance of the former.

https://en.wikipedia.org/wiki/Banana_ketchup

Yes and programming languages are human interfaces to machine instructions, so context sensitivity can be manageable and even desirable to human users of the language, even if it makes the implementation of the language interpreter more complex or less elegant. Programming language designers make this trade-off all the time.

It's still context-free, the reason is because by the time you hit the '=' symbol you already know whether you're in a <Statement> or a <Compare Exp> production rule, based on the preceding symbols (namely the LET keyword), based on these excerpts from the BASIC EBNF:

    <Statement>   ::= CLOSE '#' Integer
                    | DATA <Constant List> 
                    [...]    
                    | LET Id '=' <Expression> 
                    [...]
                    | Remark

    <Compare Exp> ::= <Add Exp> '='  <Compare Exp> 
                    | <Add Exp> '<>' <Compare Exp> 
                    | <Add Exp> '><' <Compare Exp> 
                    | <Add Exp> '>'  <Compare Exp> 
                    | <Add Exp> '>=' <Compare Exp> 
                    | <Add Exp> '<'  <Compare Exp> 
                    | <Add Exp> '<=' <Compare Exp> 
                    | <Add Exp>

You're right, I meant that each valid RHS matches one and only one LHS--but that's also true of CSGs.

I have learned both and agree with this statement. I think that Rust is harder to learn if you've only worked with high-level, GC languages, and don't have a background doing lower-level programming in C/C++/Obj-C, as well as some experience with functional languages.

Going from something like Java or Python to Rust, one would have a lot to learn.

Context-free has nothing to do with symbol tables, it just means that in the grammar, the left-hand side of a production rule can only have a single non-terminal symbol, which can always be replaced by the expression on the right-hand side, without ambiguity.

Language features are an orthogonal issue--you can implement any language feature with a CFG, but you just can't reuse the same keyword or operator to have different meanings in different grammatical contexts.

A classic example is that in C++ it is impossible to know whether << is brackets around template arguments, or the left bit shift operator, or the stream output operator, without arbitrary lookahead--you need to parse the rest of the statement in order to decide which one it is in the current statement's context.

Generally language designers don't choose to implement a context-sensitive grammar on purpose because they desire some feature that requires it; they strive for a context-free grammar, but end up with a handful of special cases that require context-sensitive parsing because of either convenience or legacy reasons.

Popular programming languages nearly all use custom parsers written in other (or the same) Turing-complete programming languages, so it's not "harder" to parse the context-sensitive rules, it only poses a problem if you want to use a parser generator to implement parsing for a CSG.

This is the reason why ORMs and SQL generators typically quote all column names in statements.

SQL is incredibly verbose compared to dplyr. Modern non-SQL query languages coming out tend to be more more similar to dplyr, based on method chaining or piping data table objects through function calls, like UNIX pipes. It's much more composable than SQL and it just makes intuitive sense as a sequence of data transformation steps in a pipeline.

My favorite feature of dplyr that makes it stand out compared to SQL is that a window function is just a group_by() without a summarize(). There's no separate syntax for "PARTITION BY".

For data analysis, whenever possible, I don't even write SQL anymore, I use the dbplyr package, which is a dplyr-to-SQL compiler.

don't forget the crucial group_by() / summarise()

R's come a long way in the last decade. The tibble and data.table packages both address this issue. data.table (https://github.com/Rdatatable/data.table) is the more strongly-typed of the two, by default it fails loudly when it encounters data that doesn't conform to the column type. It's also quite fast--binds to C code that parallelizes with OpenMP. It has very terse and expressive syntax, I find it so much more intuitive and easy to work with than pandas.

If you're happy with Python, by all means keep using it. I use both languages. Just suggesting that if you gave up on R that long ago, you might be pleasantly surprised by how much better it's gotten since then.

SQL connections can be arbitrarily long-lived. ETL processes and other data-intensive jobs often take multiple hours.

TFA isn't even about SQL the language at all though, it's about the scalability and reliability characteristics of databases, especially in a distributed environment.

Yeah, I need to point at the stove and say "stove off" when I turn it off. A few times I have started driving away from my house and suddenly been hit by a sense of dread that I might have left the stove on after cooking breakfast.

As a data scientist, most of my projects have been individual--I'm generally the only person writing and reading my code. No one tells me which language I have to use. Python and R are the most popular, and I use either one depending on which has better packages for the task at hand. I don't use Julia because I don't see enough of a benefit to switching at this point. But I really don't care, they're just tools, and I will use any language, Julia, Swift, whatever, if I see enough of a benefit to learning it. I would just take a day or two and learn enough of it to write my scripts in it.

So I think that's the good news--because of the more independent nature of the work, you generally can win data scientists over to a new language one at a time, you don't necessarily need to win over an entire organization at once.

Getting a company or a large open-source project to switch from C++ to Swift or Rust or whatever, seems much harder.

RPython is not intended for humans to write programs in, it's for implementing interpreters. If you're after a faster Python, you should use PyPy not RPython.

Numba gives you JIT compilation annotations for parallel vector operations--it's a little bit like OpenMP for Python, in a way.

It's an argument that Python being slow / single-threaded isn't the biggest problem with Python in data engineering. The biggest problem is the need to process data that doesn't fit in RAM on any single machine. So you need on-disk data structures and algorithms that can process them efficiently. If your strategy for data engineering is to load whole CSV files into RAM, replacing Python with a faster language will raise your vertical scaling limit a bit, but beyond a certain scale it won't help anymore and you'll have to switch to a distributed processing model anyway.