kyllo

Love the use of quasiquotation here:

    import Database.PostgreSQL.Simple.SqlQQ (sql)
    ...

    let index :: Handler Markup
        index = do
            tweets <- query_ [sql|
                SELECT "user".name, tweet.contents
                FROM           "user"
                    INNER JOIN user_tweet ON "user".name = user_tweet."user"
                    INNER JOIN tweet      ON user_tweet.tweet = tweet.id
                ORDER BY tweet.time DESC
            |]

This is time partitioning, not indexing, but it's commonly done in data lakes for raw telemetry/log data.

    mydata/
        2019/
            11/
               12/
                   2019111200.csv
                   2019111201.csv

Where each one of those CSV files could be 10-100 GB in size.

Usually you want to process it into a columnar format like Parquet from there, though

Most of the time it's not that data scientists think engineering is "unnecessary," it's that they're expected to deliver analyses and models without the help of engineers, because the engineers are on another team, behind a massive backlog. So they make do with the tools they have.

Optimization often requires knowledge of the behavior at a level of abstraction lower than the one you're currently working at, and exploiting leaky abstractions. Taking away the assumption of two's complement makes code more portable at the cost of making a lot of optimizations impossible. It's a tradeoff, not an absolute rule.

As soon as undefined behavior is guaranteed by a spec, it's no longer undefined, so your statement reduces to "you should never rely on UB." You can say that, but people will still do it because it makes their programs faster.

UB exactly means that it's up to the implementation to decide the behavior because the language spec doesn't. "Implementation-defined" == UB.

> A program with undefined behavior is trivially not compliant with the specification.

Undefined behavior means doing something the compiler spec does not specify, therefore the compiler is free to do anything. There are plenty of programs that exploit the programmer's knowledge of a particular compiler's actual, unspecified behavior as an optimization.

One example is using an int variable after an overflow. Post-overflow, the int's value is undefined behavior, but in practice it is predictable for a given compiler and platform. It is possible to write a program that implements its own program spec correctly, while exploiting out-of-spec behavior of its compiler under the hood.

Much like how it's possible to wrap unsafe Rust code with a safe interface.

Nice. I find it pretty easy to get excited about any prospect of a more efficient and typesafe alternative to R and Numpy.

More likely about trust/security/privacy issues.

I always think this way, but once I make a commitment, now I basically have a deadline, which for me, causes the procrastination impulse to kick in, and it takes effort to fight it.

I think that telling someone I'm going to do something makes me want to do it a little bit less because now I'm doing it for them, not for myself.

Wouldn't that make deletes super expensive, similar to deleting nodes from a doubly-linked list on disk in terms of complexity? How could it delete a node with millions of relationships if it needs to read all those blocks from disk to traverse all the pointers?

And doesn't it cause more cache misses when the on-disk pointers refer to nodes that are spread out across different file blocks?

And why does Neo4j have indexes if it claims to have no need for them? https://neo4j.com/docs/cypher-manual/current/schema/index/

I don't know how Neo4j is implemented, but I'm skeptical that it's purely index-free adjacency, I suspect there is some hybrid data structure backing it.

Learn SQL, yes, and then use your ORM as a handy SQL generator library because life is short and you do not want to write that much SQL.

"the slowness of using b-tree indexes instead of direct pointers to neighbor nodes"

If you don't use a b-tree, you still need some other in-memory data structure like a hash table to hold the pointers. "Direct pointers" don't help you if you have to read a node from disk before you can dereference its pointers.

In my case, I was working with a graph consisting of about 10M nodes and 35M rows of edges (parent-child relationships, where a parent can have many children and a child can have many parents). Many of those 10M parts weren't actively used anymore, so a minority of nodes had a majority of the relationships. I don't have access to the data anymore, but I recall being able to retrieve the transitive closure on a subgraph of ~10k nodes in 5-10 seconds, and the entire graph in about 7-8 minutes. This was on a small, possibly underpowered SQL Server instance. It's possible that Neo4j would do it faster, but I decided it was not worth moving all my data into a different database and introducing that ETL latency, to improve upon performance that was already acceptable for the business need.

I have heard anecdotally that SQL Server's CTEs are more performant than PostgreSQL's, but I don't have evidence of this, and it could be outdated if Postgres has made improvements since I heard that.

It's valid to decide how many buttons the user is going to need, before having any idea what the users might actually want those buttons to do?

There's nothing essentially, inherently slower about recursive CTEs in SQL databases compared to graph database queries, it depends how they're implemented, particularly whether they take advantage of indexing and caching. They're both going to be doing fundamentally the same type of algorithm under the hood.

I've gotten recursive CTEs in SQL Server to perform quite well just by indexing the join key columns.

Anything where you need transitive closure, that is, you need all the nodes that are reachable from a given starting node.

You can do it in SQL dialects that support recursive common table expressions.

In a previous job, I dealt with Bill of Materials data, which basically a parts list for a hardware product. Parts contain other parts, which contain other parts, and so on, recursively.

The data was in SQL Server. I could have moved the data to Neo4j and gotten a much more concise and convenient syntax for traversing the parent-child relationships. But I decided it wasn't worth the trouble of moving data, so I bit the bullet and just wrote the recursive CTE in SQL.

That's the biggest problem with graph databases. If your data's already in a SQL database, it often isn't worth it to _move all your data_ just to get a better query syntax.

I don't think the parent was saying you can use Rust anywhere you can't use C++. I think they were saying, Rust is not only a good C++ alternative, but also a good C alternative.

Yes, exactly. Nail on the head.

When I do have a datatype that needs to be handled differently based on some condition, I tend to use ABC's with only one level of concrete subclasses, just to make that implicit method interface explicit.

> you need a class where natural, convenient, consumer-friendly use of a data structure should leverage features like operators and standard protocols that rely on the existence of particular methods (often special methods.)

I understand and I can't disagree, this just seems much, much narrower in scope than what classes are used for in typical Python code I see every day. Which is perhaps due to influence from other languages like Java/C++ etc.

These are minor annoyances to me. The major annoyance I have with Python is that most Python code makes such heavy use of classes, even when it only needs lists, tuples, and dicts.

Why do I really need a class, when:

- Instance fields can be added and removed at runtime

- Private fields and methods are not actually private ("we're all consenting adults here")

- No pattern matching on types and `if isinstance(foo, Bar):` is an anti-pattern because it's a duck-typed language

- Interfaces are neither available nor necessary due to late binding. Abstract Base Classes are the closest thing to an interface, but they're also unnecessary because again, it's duck-typed.

- Inheritance is widely considered to be a mistake ("Prefer composition over inheritance" etc)

- Classes tend to complicate testing

I can use classes to write custom data structures, I guess, but in a language as high-level as Python, I can just model trees and graphs with dicts. If I need a more performant data structure, I'm not writing it in Python in the first place.

I've seen some very experienced Pythonistas recommend using namedtuple instead of classes for most use cases, and I tend to agree with them. That gets you immutability as well as dot notation for accessing fields.

Clojure programmers seem perfectly happy with just lists and maps, and after experiencing that, classes feel like a bolted-on misfeature in Python.

$170k is top-end for base salary at most tech companies, though--past that point the increases in total comp are all from stocks and bonuses, which are tied to your performance review.

And yeah, a $200K house in Michigan is probably about a $2M house in the Bay Area.

Marxists have a term for what you're describing: "state capitalism."

All beef is grass fed until a short period before slaughter, where American ranchers fatten them up with a diet of corn-based feed. What we call "grass-fed" beef is just "grass-finished" instead of "corn-finished."

This "append-only" style also seems like it could lead to new code being added just to "undo" what the existing code is doing, which is obviously waste.

Reminds me of a process I saw at work years ago. An employee was given an Excel macro to run on a weekly spreadsheet file. The instructions included "Unhide cells D through F". It turned out that the macro had code in it to hide cells D through F, but the employee didn't know how to just remove that code from the macro, so they had to manually undo something the code did, every single time they ran it.

This book is an _epic_ contribution to the field. I've learned a ton from it myself. I had gone through several other tutorials on how to write a parser, but I just wasn't getting how recursive descent works, and yours was the one that finally made me understand it. Thank you so much for writing it!

I like it--I'm a data scientist by day, and I've been following Rust with interest but I haven't found a good use case for it in my job yet. A Rust-based Spark competitor sounds like it could be exactly that excuse to use Rust at work that I've been looking for!

This is really cool! What do you see as ideally the primary API for something like this?

SQL is great for relational algebra expressions to transform tables but its limited support for variables and control flow constructs make it less than ideal for complex, multi-step data analysis scripts. And when it comes to running statistical tests, regressions, training ML models, it's wholly inappropriate.

Rust is a very expressive systems programming language, but it's unclear at this point how good of a fit it can be for data analysis and statistical programming tasks. It doesn't have much in the way of data science libraries yet.

Would you potentially add e.g. a Python interpreter on top of such a framework, or would you focus on building out a more fully-featured Rust API for data analysis and even go so far as to suggest that data scientists start to learn and use Rust? (There is some precedence for this with Scala and Spark)

> Meanwhile, most proper languages have the concept of types that only have two values and can easily use an underlying machine representation for efficiently representing these, without involving the programmer.

Don't other languages still use an entire byte to represent a bool though, since memory access is at the byte level? Having a bool type in the type system is really a language usability concern, I don't think it's at all a performance optimization. And stdbool.h exists now, so that concern has been addressed. When you want a bitmap, you can just use an int of the appropriate length and do bitwise operations on it, instead of wasting space with an array of ints.

You mean if you create the pivot table from data in a sheet? It's not needed for that because you'll run out of memory trying to fit any more than about a couple million rows in a sheet first. Meanwhile you can easily load 10s and possibly 100s of millions of rows into Power Query.

The other thing people don't realize about Excel is that it has a compressed, in-memory, columnstore analytics database built into it. It was called Power Pivot then Power Query, not sure if it changed names again, but it's actually a SQL Server Analysis Services Tabular Model, the same thing that's in Power BI as well. It's going to perform much better than a row-oriented RDBMS at aggregating columns because that's all it was designed to do.

Those python ports of dplyr and ggplot2 are cool, but the problem is that they're abandoned.