Hacker News Reader

> I mean that if K or some language is actually interpreting without a virtual machine at some stage, that is indeed going to be slower than any language with code running on a virtual machine.

Not necessarily. The VM itself will introduce overhead and potentially blow out cache etc.. You can argue, as the article does, that this is a small proportion of execution time if you're executing a lot of loops, but that's inherently workload-dependent. If the language being interpreted is denser than the VM bytecode - something that's believable for a language like K - then interpreting directly makes more efficient use of memory bandwidth than interpreting via bytecode.

That's an interesting idea, I missed that.

Has anyone published a PoC of this (not necessarily using any of these languages) that would give some quantitative credence?

"And also that any language with a virtual machine must be slower than an equivalently intelligent compiled code since it's running on another layer of abstraction that compiled code isn't. "

I think this is not correct. A VM has access to the data that the code is operating on, while an AOT compiler doesn't. It's possible that a VM (eg. JVM) produces faster code than a normal compiler.

I'm talking about where the code is run. If it's run on a VM it's run on a VM. If it's run on the cpu by JIT-ed code it's not running on a VM.

Edit: But fair I did say "language with a VM" which would include Java JIT-ed or not.

Edit, edit: Modified the original comment to say "code on a VM" not "language with a VM".

Sadly it also has to consider the time it takes to compile code like that - we grumble, but not too loudly, at the time LLVM takes to compile things. If the JVM took that same time whilst running live that wouldn't be acceptable.

Surely this depends entirely on the total runtime of the program. A server-style program which will run for a couple of weeks can accommodate quite a bit of JIT compilation overhead.

> If you're not allowed to publish benchmarks you only can rant like this even if it sounds petty.

You can repeat whole-system benchmarks like STAC[1] or that some bloggers do[2], which specify the hardware and software, inputs and outputs for a problem. You can always do that.

[1]: https://stacresearch.com/m3

[2]: https://tech.marksblogg.com/billion-nyc-taxi-kdb.html

Interesting, but I don't understand how this is allowed under the license?

Edit: it's not that these licenses says you cannot ever publish benchmarks it's that they must be approved. So I assume these benchmarks were approved.

I'm not familiar with k, and I'm not understanding this whole "no benchmarks" thing.

So it's a proprietary lang? What stops benchmarks being run? Just how ridiculous are the licenses? Why has seemingly nobody run them anyway?

> I'm not understanding this whole "no benchmarks" thing.

Back in the 1980s, there was a paper published[1] that compared a number of database systems, and Oracle felt this paper harmed them. Perhaps if the authors knew Oracle better, they could have made more favourable benchmarks, they would argue, and whilst perhaps DeWitt et al could agree to that, they may point out they had not intended to harm Oracle, and being researchers and not marketers were given no notice or expectation that such a review could harm Oracle.

And so for the avoidance of doubt, Oracle, Microsoft, and many others (probably including KX, although I've never checked) insist in their licensing agreements that they be able to approve benchmarks, ostensibly so that they have an opportunity to correct any misunderstandings the author might have about the software before.

[1]: http://pages.cs.wisc.edu/~dewitt/includes/benchmarking/vldb8...

These benchmark things reminds me of an anecdote I saw today in the D group:

>Back in the 80's (!) computer magazines regularly ran C compiler benchmark results. (At one time I counted 30 C compilers available for the PC.)

>I took a class at Standford on compilers, which included a lot of info on data flow analysis. I decided to implement it. Data flow analysis optimizations basically deleted the benchmark code, because it didn't do anything (lots of dead assignments). This compiler was released as Datalight Optimum C.

>The article writer concluded that the compiler had a giant bug in it, because it ran the benchmarks impossibly fast, because it deleted the dead code. Instead of being lauded as the only C compiler on the PC that did global data flow analysis, it was labeled as a buggy piece of crap.

>By the time we dug ourselves out of that PR disaster, other compilers had implemented it, too.

>Ironically, these days DMD gets repeatedly charged with not doing data flow analysis, and clang is assumed to have invented data flow analysis.

>Just goes to show the power of marketing

> Back in the 1980s, there was a paper published[1] that compared a number of database systems, and Oracle felt this paper harmed them.

This is rather an argument why the benchmark code should be made public so that other people can check it and vendors can post modified versions if they feel treated unfairly.

Not sure about K, but I know for example that MS didn't allow publishing performance benchmarks of .NET (not sure of this changed with .NET core).

Both Oracle and MS SQL disallow this without approval.

https://stackoverflow.com/a/12116865/1507139

I'm not aware of that ever being true for .net, but it's definitely not true now.

https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

I think the article answers all your questions.

Here are some (micro) benchmarks of q vs python that they did publish: https://kx.com/blog/a-comparison-of-python-and-q-for-data-pr...

tl;dr, while q is orders of magnitude faster that naive python, it is, at best, 5 times faster than optimal python using numpy. Based on those numbers I would be very surprised if an optimal C program wouldn't be significantly faster than both.

Almost! I can't quite replicate these tests because it doesn't give the type (integer or float, and width) of the database columns. There are many possibilities[0] so I tested a few. Fortunately the testing CPU (Intel Xeon E5-2650 v4 @ 2.20 GHz) is comparable to mine (Intel Core i5-6200U CPU @ 2.30GHz). Turbo (one-core speed) is 3.00 GHz versus 2.80 GHz with advantage to the Xeon, but my laptop CPU is a year newer and on a smaller process so could have better IPC. Both have AVX2, and the server CPU has a much better cache, but I don't think Dyalog is bandwidth limited.

I adapted the vectorized Dőtsch solution, which gave the fastest times and is easy to write, to Dyalog APL. Pre-release 18.0 from when I worked there because I can't be bothered to do a real install, but I doubt the arithmetic code's been changed since 17.1. Here are results from a Dyalog session giving times in seconds for 8-byte floats, 4-byte integers, and 1-byte integers. From the article, Q takes 13E¯3 seconds.

          )copy dfns cmpx
          (sA sB pA pB)←?{1e6⍴  0}¨⍳4 ⋄ cmpx '(sA×pA≥pB) + sB×pA≤pB'
    9.3E¯3
          (sA sB pA pB)←?{1e6⍴1e9}¨⍳4 ⋄ cmpx '(sA×pA≥pB) + sB×pA≤pB'
    4.6E¯3
          (sA sB pA pB)←?{1e6⍴120}¨⍳4 ⋄ cmpx '(sA×pA≥pB) + sB×pA≤pB'
    1.3E¯3

If the table consists of floats then Dyalog appears substantially faster, although this could plausibly be due to better IPC on my CPU. It could also be a real increase. Dyalog uses bit booleans for the comparison results which allows it to make smaller reads and writes, and code for packing results to bit booleans and multiplying floats by booleans does have to be written with vector intrinsics ([1] indicates Q doesn't pack booleans). If the benchmark uses 8-byte ints then they should be comparable to floats, and if it's using smaller ints then Dyalog is clearly much better.

Will see if I can get numpy benchmarks running.

[0] https://code.kx.com/q/basics/datatypes/

[1] https://code.kx.com/q4m3/3_Lists/#323-simple-binary-lists

Ran the following numpy program, which resulted in a time of 0.01116s, faster than the article's benchmark but slightly slower than my Dyalog timing of 0.0093s. Assuming relative times are consistent across CPUs, this would put Dyalog on 8-byte floats just ahead of Q on 8-byte ints. Floating-point vector instructions take longer than integer ones; while overflow checking could turn things the other way, K and Q famously don't do it. It's not consistent with a sum of 100 runs as geocar suggests.

    import numpy as np
    import time

    N = 1000 * 1000
    pA = np.random.randint(0, 5, N)
    pB = np.random.randint(0, 5, N)
    sA = np.random.randint(0, 100, N)
    sB = np.random.randint(0, 100, N)

    start = time.perf_counter()
    runs = 100
    for i in range(runs):
        bS = sA * (pA >= pB) + sB * (pA <= pB)
    end = time.perf_counter()
    print((end - start)/runs)

They're 8-byte ints, and the reported times are sum of 100 runs. This is what my 2019 i5 1.6ghz Macbook Air does:

    q)`sA`sB`pA`pB set'4 0N#1000000?100
    `sA`sB`pA`pB
    q)\t:100 (sA*pA>=pB)+sB*pA<=pB
    60

Where do you get this information? Do you have some connection to the article?

It appears this benchmark uses vectors of a quarter-million, not a million, elements? My understanding is that 4 0N#a redistributes elements of a into four vectors without repeating them, and this is what ngn/k does. The article says a "sample table of size one million", refers to "one million rows" elsewhere, and later gives timings that scale linearly with the number of columns, so I don't think that's what is meant.

EDIT: Oh, there's generation code near the top. In an image for some reason so the Q version is transcribed below. Definitely a million rows. Seems both Numpy's random.randint and Q default to 8-byte ints? Dyalog would use 1-byte integers if the data fits, as it does in this example.

    N: 1000 * 1000

    t: ([] pA: N?5; pB: N?5
           sA: N?100; sB: N?100)