But it wasn't what I thought it would be. I guess I expected to be doing more performance oriented work, analyzing numbers and trying to get every last bit of performance out of the cluster. To be honest, they didn't even have any kind of monitoring running. I set some up, and it doesn't really get used. Once in a while we get questions from management about "how busy is the cluster", to justify budgets and that sort of thing.
Most of my 'optimization' work ends up being things like making sure people aren't (usually unknowingly) requesting 384 CPUs when their script only uses 16, testing software to see what # of CPU's it works with before you see a degradation, etc. I've only had the Intel profiler open twice.
And I've found that most of the job is really just helping researchers and such with their work. Typically running either a commercial or open-source program, troubleshooting it, or getting some code written by another team on another cluster and getting it built and running on yours. Slogging through terrible Python code. Trying to get a C++ project built on a more modern cluster in a CentOS 7 environment.
It can be fun in a way. I've worked with different languages over the years so I enjoy trying to get things working, digging through crashes and stack traces. And working with such large machines, your sense of normal gets twisted when you're on a server with 'only' 128GB of RAM or 20TB of disk.
It's a little scary when you know the results of some of this stuff are being used in the real world, and the people running the simulations aren't even doing things right. Incorrect code, mixed up source code, not using the data they thing they are, I once found a huge bug that had existed for 3 years. Doesn't this invalidate all the work you've done on this subject?
The one drawback I find is that a lot of HPC jobs want you do have a masters degree. Even to just run the cluster. Doesn't make sense to me, I'm not writing the software you're running, we aren't running some state of the art, TOP500 cluster. We're just getting a bunch of machines networked together and running some code.