Another overlooked "performance" feature of terse languages is that of the developer. Vector based, concise languages give you more in one line of code. And I don't find that readability suffers. Yes you need to stare at that one line longer, but there's more going on.
If the article can provoke clear benchmarks against current Dyalog or J then that would be an ideal outcome, even if it proves me wrong. I already see some benchmarks linked here and will try to analyze these.
I thought the same thing but on the other hand, what can you do? If you're not allowed to publish benchmarks you only can rant like this even if it sounds petty.
By getting attention here on HN it helps put pressure on commercial vendors to stop sharing unconfirmable benchmarks otherwise they will lose credibility since they've been called out.
The author does make some good points about the fundamental nature of interpreters too. Although it's hard to confirm there too what these languages do without seeing their source.
I mean that if K or some language is actually interpreting without a virtual machine at some stage, that is indeed going to be slower than any language with code running on a virtual machine.
And also that any code that runs on a virtual machine must be slower than an equivalently intelligent compiled code since it's running on another layer of abstraction that compiled code isn't. ("equivalently intelligent" because yes a virtual machine could be faster than terrible compiled code.)
I think this is not correct. A VM has access to the data that the code is operating on, while an AOT compiler doesn't. It's possible that a VM (eg. JVM) produces faster code than a normal compiler.
Edit: But fair I did say "language with a VM" which would include Java JIT-ed or not.
Edit, edit: Modified the original comment to say "code on a VM" not "language with a VM".
So it's a proprietary lang? What stops benchmarks being run? Just how ridiculous are the licenses? Why has seemingly nobody run them anyway?
https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
Back in the 1980s, there was a paper published[1] that compared a number of database systems, and Oracle felt this paper harmed them. Perhaps if the authors knew Oracle better, they could have made more favourable benchmarks, they would argue, and whilst perhaps DeWitt et al could agree to that, they may point out they had not intended to harm Oracle, and being researchers and not marketers were given no notice or expectation that such a review could harm Oracle.
And so for the avoidance of doubt, Oracle, Microsoft, and many others (probably including KX, although I've never checked) insist in their licensing agreements that they be able to approve benchmarks, ostensibly so that they have an opportunity to correct any misunderstandings the author might have about the software before.
[1]: http://pages.cs.wisc.edu/~dewitt/includes/benchmarking/vldb8...
This is rather an argument why the benchmark code should be made public so that other people can check it and vendors can post modified versions if they feel treated unfairly.
>Back in the 80's (!) computer magazines regularly ran C compiler benchmark results. (At one time I counted 30 C compilers available for the PC.)
>I took a class at Standford on compilers, which included a lot of info on data flow analysis. I decided to implement it. Data flow analysis optimizations basically deleted the benchmark code, because it didn't do anything (lots of dead assignments). This compiler was released as Datalight Optimum C.
>The article writer concluded that the compiler had a giant bug in it, because it ran the benchmarks impossibly fast, because it deleted the dead code. Instead of being lauded as the only C compiler on the PC that did global data flow analysis, it was labeled as a buggy piece of crap.
>By the time we dug ourselves out of that PR disaster, other compilers had implemented it, too.
>Ironically, these days DMD gets repeatedly charged with not doing data flow analysis, and clang is assumed to have invented data flow analysis.
>Just goes to show the power of marketing
tl;dr, while q is orders of magnitude faster that naive python, it is, at best, 5 times faster than optimal python using numpy. Based on those numbers I would be very surprised if an optimal C program wouldn't be significantly faster than both.
I adapted the vectorized Dőtsch solution, which gave the fastest times and is easy to write, to Dyalog APL. Pre-release 18.0 from when I worked there because I can't be bothered to do a real install, but I doubt the arithmetic code's been changed since 17.1. Here are results from a Dyalog session giving times in seconds for 8-byte floats, 4-byte integers, and 1-byte integers. From the article, Q takes 13E¯3 seconds.
)copy dfns cmpx
(sA sB pA pB)←?{1e6⍴ 0}¨⍳4 ⋄ cmpx '(sA×pA≥pB) + sB×pA≤pB'
9.3E¯3
(sA sB pA pB)←?{1e6⍴1e9}¨⍳4 ⋄ cmpx '(sA×pA≥pB) + sB×pA≤pB'
4.6E¯3
(sA sB pA pB)←?{1e6⍴120}¨⍳4 ⋄ cmpx '(sA×pA≥pB) + sB×pA≤pB'
1.3E¯3
Will see if I can get numpy benchmarks running.
[0] https://code.kx.com/q/basics/datatypes/
[1] https://code.kx.com/q4m3/3_Lists/#323-simple-binary-lists
q)`sA`sB`pA`pB set'4 0N#1000000?100
`sA`sB`pA`pB
q)\t:100 (sA*pA>=pB)+sB*pA<=pB
60It appears this benchmark uses vectors of a quarter-million, not a million, elements? My understanding is that 4 0N#a redistributes elements of a into four vectors without repeating them, and this is what ngn/k does. The article says a "sample table of size one million", refers to "one million rows" elsewhere, and later gives timings that scale linearly with the number of columns, so I don't think that's what is meant.
EDIT: Oh, there's generation code near the top. In an image for some reason so the Q version is transcribed below. Definitely a million rows. Seems both Numpy's random.randint and Q default to 8-byte ints? Dyalog would use 1-byte integers if the data fits, as it does in this example.
N: 1000 * 1000
t: ([] pA: N?5; pB: N?5
sA: N?100; sB: N?100) import numpy as np
import time
N = 1000 * 1000
pA = np.random.randint(0, 5, N)
pB = np.random.randint(0, 5, N)
sA = np.random.randint(0, 100, N)
sB = np.random.randint(0, 100, N)
start = time.perf_counter()
runs = 100
for i in range(runs):
bS = sA * (pA >= pB) + sB * (pA <= pB)
end = time.perf_counter()
print((end - start)/runs)You can repeat whole-system benchmarks like STAC[1] or that some bloggers do[2], which specify the hardware and software, inputs and outputs for a problem. You can always do that.
Edit: it's not that these licenses says you cannot ever publish benchmarks it's that they must be approved. So I assume these benchmarks were approved.
Not necessarily. The VM itself will introduce overhead and potentially blow out cache etc.. You can argue, as the article does, that this is a small proportion of execution time if you're executing a lot of loops, but that's inherently workload-dependent. If the language being interpreted is denser than the VM bytecode - something that's believable for a language like K - then interpreting directly makes more efficient use of memory bandwidth than interpreting via bytecode.
Has anyone published a PoC of this (not necessarily using any of these languages) that would give some quantitative credence?
Edit: and nevertheless is quite depressing that as a paying customer you should be afraid
(Edit: if you downvoted this: why?)
I think it may be more convenient to write terse code in BQN over k, because there is greater facility to control function calls in a terse way. Things like under and over are quite common patterns that BQN finds easier (and shorter) than k.
I've not spent a great deal of time yet with BQN, so this could be wrong.
Isn't the burden on proof on the ones making the claim for the speed of their product?
Incredulity should be the default scientific stance here...