Hacker News Reader

Comment by shaklee3 | Hacker News Reader

Has anyone seen similar benchmarks, but with BLIS? OpenBLAS is interesting, but BLIS would possibly perform better.

Based on Zen1 Epyc performance it would seem that BLIS might be significantly more performant even without AMD-specific optimizations (I know AMD has their own fork, but check out the 64 core results for "vanilla" BLIS: https://github.com/flame/blis/blob/master/docs/Performance.m...)

Something is still weird with the gemm benchmarks. That link you sent has BLIS winning all around, but the absolute value of the BLIS is poor. MKL gets around 2TFLOPS on xeons, and this blis benchmark is only hitting around 30-50GFLOPS. Are AMDs hitting over 1TFLOPS anywhere?

To illustrate this, let's assume the GEMM used avx512 perfectly. Roughly that's 2.5e9 (clock) * 16 (ops per cycle) * 2 (FMA per cycle) * 64 (cores). That puts Intel's peak performance over 2TFLOPs, which they're showing. That's about 80x higher than AMD on BLIS...

Everything on that benchmark page is listed as GFLOPS/core so at 30GFLOPS that Epyc 7551 should be pushing around 2 TFLOPS. You can take a look at reproducing them if you want if you have a similar system: https://github.com/flame/blis/blob/master/docs/Performance.m...

I ran into some compile issues, but I care more about numpy so I gave this script a try: https://markus-beuckelmann.de/blog/boosting-numpy-blas.html with some quick/dirty results on my Ryzen 3700X (8C16T) workstation (BLIS doesn't seem to perform very well on this small test). You can see performance basically double on MKL when MKL_DEBUG_CPU_TYPE=5 is used. I'll probably continue to stick w/ OpenBLAS for now:

  ## blis 0.6.0 h516909a_0
  # conda activate numpy-blis
  # conda run python bench.py
  Dotted two 4096x4096 matrices in 2.30 s.
  Dotted two vectors of length 524288 in 0.08 ms.
  SVD of a 2048x1024 matrix in 0.94 s.
  Cholesky decomposition of a 2048x2048 matrix in 0.24 s.
  Eigendecomposition of a 2048x2048 matrix in 6.36 s.

  ## libopenblas 0.3.7 h5ec1e0e_4
  # conda activate numpy-openblas
  # conda run python bench.py
  Dotted two 4096x4096 matrices in 0.41 s.
  Dotted two vectors of length 524288 in 0.02 ms.
  SVD of a 2048x1024 matrix in 0.55 s.
  Cholesky decomposition of a 2048x2048 matrix in 0.14 s.
  Eigendecomposition of a 2048x2048 matrix in 5.53 s.

  ## mkl 2019.4 243
  # conda activate numpy-mkl
  # conda run python bench.py
  Dotted two 4096x4096 matrices in 1.53 s.
  Dotted two vectors of length 524288 in 0.02 ms.
  SVD of a 2048x1024 matrix in 0.51 s.
  Cholesky decomposition of a 2048x2048 matrix in 0.29 s.
  Eigendecomposition of a 2048x2048 matrix in 4.79 s.

  # export MKL_DEBUG_CPU_TYPE=5
  # conda run python bench.py
  Dotted two 4096x4096 matrices in 0.33 s.
  Dotted two vectors of length 524288 in 0.02 ms.
  SVD of a 2048x1024 matrix in 0.29 s.
  Cholesky decomposition of a 2048x2048 matrix in 0.15 s.
  Eigendecomposition of a 2048x2048 matrix in 3.29 s.

Thanks. I completely missed that legend. Does it actually scale linearly by cores? I would think there's a plateau after a certain number of cores.

That benchmark page I referenced shows scaling on a SkylakeX at 1, 26, and 52 cores, on a Haswell at 1, 12, and 24 cores, and on Zen1 at 1, 32, and 64 cores so you can probably map out a ballpark scaling co-efficent from that.

I'm sure there are theoretical and practical limits but it's probably completely dependent on implementation and each system architecture, but that's beyond my pay grade. Maybe start here: https://en.wikipedia.org/wiki/Basic_Linear_Algebra_Subprogra...

If you want to understand implementation and performance, that page won't help you. Read the papers referenced on the BLIS web site, and/or run their benchmark code, which is now referenced somewhere in a README.

Oh, just for fun, I manually replaced the BLIS libs w/ the AMD BLIS 2.0 fork (Ubuntu MT binary on an Arch system: https://developer.amd.com/amd-aocl/blas-library/) and it seemed to work ok. Slightly faster than BLIS 0.6.0 but not actually faster than OpenBLAS on this test...

  ## AMD BLIS 2.0 https://developer.amd.com/amd-aocl/blas-library/
  # conda activate numpy-amdblis
  Dotted two 4096x4096 matrices in 2.33 s.
  Dotted two vectors of length 524288 in 0.08 ms.
  SVD of a 2048x1024 matrix in 0.78 s.
  Cholesky decomposition of a 2048x2048 matrix in 0.23 s.
  Eigendecomposition of a 2048x2048 matrix in 5.84 s.

AMD maintain a fork which is merged back as they make it available, so mainline BLIS typically has that work soon after. That was last done in October, since the current release (0.6.0). The FLAME people encourage you to use the current master branch, but you might bother about correctness, as it was failing BLAS tests recently on POWER and the generic target.

The article says they’d like to include BLIS in the benchmark, but nobody has written a good Numpy Conda package with BLIS hooks yet

The article was specific to AMD's BLIS fork, but just as an FYI, using standard BLIS w/ numpy (which I believe would still outperform OpenBLAS) is pretty straightforward:

  conda create -c conda-forge -n numpy-blis numpy "blas=*=blis"

Are there any reads you can recommend on how to recover from (possibly) failed experiments like these under pip / conda / linux in general? I've been doing R and Linux for two decades, but still am perfectly capable of totally breaking my python/jupyter-workflow by messing up dependencies without knowing how to recover. Would love to learn to remedy that gap and reducing the risk.

When dealing with purely python packages: always always work in python virtual envs. Conduct these kinds of experiments in separate venvs. If worse comes to worse, a reset consists simply in deleting the venv directory and installing from scratch.

If an experiment involves system packages it's a little trickier. You can learn about Docker (which does for the whole system what venv does for python packages), but in most cases you may get away with just taking note of what packages you have installed/uninstalled, so you can revert the process later if needed.

never work as root, just undo all changes you did for it to work

You can actually see in my pasted command that I created a new venv for the blis install. It's as simple as "activate" or "deactivate" to switch around.

That's basically all there is to it, but here are some docs for you to reference from the top page of search:

https://docs.conda.io/projects/conda/en/latest/user-guide/ta...

https://uoa-eresearch.github.io/eresearch-cookbook/recipe/20...

because all of the people "benchmarking" cpus nowadays are just clueless self-advertisers...

Why would you need a Conda package to measure performance or BLIS or another implementation? Assuming an ELF system, either preload an alternative library, or make a trivial shim for libblas.so3, as the Fedora blis package does. See the somewhat old https://loveshack.fedorapeople.org/blas-subversion.html for example.

Reply on news.ycombinator.com