4,568 karma · joined September 3, 2010
https://www.chrisaycock.com
The tail ends of a language implementation (parsing and code generation) are a fixed cost; the "middle end" can grow unbounded as more production-quality items are added.
My language: https://www.empirical-soft.com
https://www.hpcwire.com/2022/12/05/new-c-sender-library-enab...
Overlapping communication and computation has been a common technique for decades in high-performance computing to "hide latency", which leads to better scaling. Now standard C++ can be used to express parallel algorithms without tying to a specific scheduler.
There are plenty of other interesting features like auto-FFI, bytecode caching (similar to Python's .pyc files), and "daemonize" mode (similar to mod_perl or FastCGI).
This technique is known as a "tiered JIT". It's how production virtual machines operate for high-level languages like JavaScript.
There can be many tiers, like an interpreter, baseline compiler, optimizing compiler, etc. The runtime switches into the faster tier once it becomes ready.
More info for the interested:
https://numpy.org/doc/2.2/reference/generated/numpy.searchso...
I couldn't figure-out how Arthur's bin matched on symbol though, so I switched to a linear scan on the right table to record the last-seen index for each "by" element. While it worked, my hash table was messy because I relied on Python to handle a whole tuple as a key, which had some issues during initial testing.
The asof join I wrote for Empirical properly categorizes the keys before they are matched. That approach worked far better.
Fun fact: the aj (asof join) function was my inspiration for pandas.merge_asof. I added the extra parameters (direction, tolerance, allow_exact_matches) because of the limitations I kept hitting in kdb.
https://pandas.pydata.org/docs/reference/api/pandas.merge_as...
It's like the quote attributed to Don Knuth: "Beware of bugs in the above code; I have only proved it correct, not tried it."
Basically, use copy-and-patch compilation in a vector language to fuse loops and avoid temporaries. It can be employed for a baseline compiler that will use less memory than an interpreter and will have a much lower startup cost than an optimizing compiler.
There is no magic here. No language can guess the type of anything without seeing what the thing is.
https://www.empirical-soft.com
It can infer the column names and types from a CSV file at compile time.
Here's an example that misspells the "ask" column as if it were plural:
let quotes = load("quotes.csv")
sort quotes by (asks - bid) / bid
Error: symbol asks was not found
Unfortunately, it didn't go anywhere as a project. Turns out that static typing isn't enough of a selling point for people to drop Python. I haven't touched Empirical in four years, but my code and my notes are still publicly available on the website.
I've had a similar thought recently, that there is no demand for rote programmers. No employer is going to hand you a completed spec and tell you to code it up.
Software engineers and data scientists today must be innovative, understand the business they operate in, communicate with users, and work cross functionally. You’ve got to create something original and see it through without having to be told what to do at every step.
But that complaint can be made about any language! "This dynamically typed language won't allow students to understand type safety." "This high-level language won't allow students to learn pointers and systems programming." Etc.
I believe that an intro course should get students coding since the first major hurdle is learning how to construct any kind of program at all. The switch to a more "employable" language isn't going to make education worse.
Society needs to value those attributes if we're going to solve the really hard problems.
I built my own language for time-series analysis because of how much I hated q/kdb+, but Python has been the winner for a bunch of years now.
http://www.rdrop.com/users/paulmck/scalability/paper/whymb.2...
I agree; queries are why table semantics are so useful. Whether you choose an SQL implementation (e.g., DuckDB and SQLite) or a dataframe library (e.g., pandas and polars), expressing searches, joins, and aggregations becomes trivial for tabular data.
In particular, copy-and-patch compilation is still the fastest approach because it uses pre-compiled code, though leaves little room for optimization.
Cranelift uses e-graphs to represent equivalence on the IR. This allows for more optimizations than the copy-and-patch approach.
Of course, the most optimized output is going to come from a more traditional compiler toolchain like LLVM or GCC. But for users who want to get "fast enough" output as quickly as possible, newer compiler techniques provide a promising alternative.
Perhaps there's an approach where we tell an employee to rate everyone, and then use those values only as a self-assessment of the employee.
https://www.getdaft.io/projects/docs/en/latest/faq/dataframe...
Daft is distributed and vectorized with plenty of benefits from lazy evaluation.
I was so burned-out from the experience that I had to stop for my own mental health. I got a normal job three years ago.
Since putting the startup on indefinite hold, I've been able to learn new sports, buy a home, and generally enjoy my life without the constant feeling that I had to be working more and spending less. My life got dramatically better.
So while I would have preferred that the startup succeeded, walking away after four years was far better than continuing down that path. And because the technology I had built was really impressive, it only took a couple months for me to get a job paying more than I ever had before.
- autofree doesn't work
- memory arenas (prealloc) isn't thread safe
- compiler produces large binaries
- coroutines invoke blocking calls
- community behavior is, uh, "unwelcoming"https://johnaugust.com/scriptnotes
It's hosted by John August (Big Fish, Corpse Bride) and Craig Mazin (Chernobyl, The Last of Us).