Counterpoint: that makes you rewrite it for every architecture and every datatype. A high level language makes it a lot easier to get something more readable, runable everywhere, and datatype generic.
Point being, not only that you can get 2x on top of initial 2x SIMD implementation but usually much more.
Whether or not you see SIMD in the codegen is not a testament of how good the implementation really is.
IPC is the relevant measure here.
I assume you didn't use https://github.com/miguelraz/StagedFilters.jl/blob/master/te... since that benchmark is pretty much flawed for more than a few reasons:
- Operates with the dataset that cannot fit even in the L3 cache on most consumer machines: ~76MB (float64) and ~38MB (float32)
- Intermixes python with the julia code
- Not even a single repetition - no statistical significance
- Does not show what happens with different data distributions, e.g. random, but only uses a monotonically increasing sequence
Also, I don't understand what the issue would be with mixing Python and Julia code in the benchmark script. The Julia side JIT compiles the invocations which we've seen removes pretty much all non-Python overhead and actually makes the resulting SciPy calls faster than doing so from Python itself in many cases, see for example https://github.com/SciML/SciPyDiffEq.jl?tab=readme-ov-file#m... where invocations from Julia very handily outperform SciPy+Numba from the Python REPL. Of course, that is a higher order function so it's benefiting from the Julia JIT in other ways as well, but the point is in previous benchmarks we've seen the overhead floor as so low (~100ns IIRC) that it didn't effect benchmarks negatively for Python and actually improved many Python timings in practice. Though it would be good to show in this case what exactly the difference is in order to isolate any potential issue, I would be surprised if it's more than a 100ns overhead for an invocation like this and with 58ms being the benchmark size, that cost is well below the noise floor).
Though trying different datasets is of course valid. There's no reason to reject a benchmark just because it doesn't fit into L3 cache, there's many use cases for that. But it does not mean that all use cases are likely to see such a result.
Per-se this isn't wrong but you may get the skewed picture of your algorithm efficiency because you're measuring multiple things at once, and perhaps the ones you may not be interested in or even knew that they even exist or are included in the numbers of your experiment, e.g. how good the memory prefetcher understands the access patterns of your experiment.
For the same reason, I think it's better to avoid mixing many different things into the same benchmark run.
I stand corrected wrt statistical significance since I am not familiar with Julia. I missed to understand that part.