I tend to use either a command-line calculator (to minimise context switching, for simple calculations) or a physical TI calculator for simple matrix calculations, CAS features such as analytical derivatives simple integration, and dimension analysis (which is vital in my corner of Physics and very rarely done properly in software calculators).
It is a bit slow but has decent features, including some physical units support.
I set up Tilda (or another Guake equivalent; I tried a bunch of them and can’t remember on which I settled in the end) to run it automatically when a terminal is opened, when pressing Scroll Lock. It’s quite neat: when I need to do a quick calculation, I just hit Scroll Lock, which spawns a drop down terminal with the focus; then type whatever I want to calculate; then control-D to dismiss the drop down terminal and I am back in whatever I was doing without moving either hand off the keyboard.
I do more or less the same thing on my Mac with Alfred.