Hacker News Reader

Comment by malux85 | Hacker News Reader

I don’t train LLMs from scratch, but I have:

3x4090s 1xTesla A100

Lots of fine tuning, attention visualisation, evaluation of embeddings and different embedding generation methods, not just LLMs though I use them a lot for deep nets of many kinds

Both for my day job (hedge fund) and my hobby project https://atomictessellator.com

It’s summer here in NZ and I have these in servers mounted in a freestanding server rack beside my desk, and it is very hot in here XD

View on HN

Can you talk about the attention visualization a bit. What’s the benefit. I have been plotting attention masks and even visualizing them over sentences but it obv doesn’t result in a ton of insight due to the sentences changing from words to much more complex vectors that represent those words after being processed through attentions 32x.

I am using attention weighted pooling strategy to generate cross-attentions across N transformer heads, this allows us to visualize which parts of an output were caused by which part of the input -- this is used to debug summarization. Only works for encoder/decoder networks - i.e. ones where you have cross attentions

Very cool website, love the aesthetic! Would love to know how you made the animations and the algos behind them.

Thank you! The animation at the top is a GPU shader of the transition between 1s, 2p and 3d electron wave functions in a hydrogen atom, the animated crystal of sulpher I made using Maya and then animated into/out of wireframe by loading it and spinning it in WebGL, the other ones further down are just screen recordings.

You might like this too - this last weekend I finished writing a topological contour GPU shader for visualizing electron density grids, here is a molecule of buckminsterfullerene - the contour generator is running on the GPU, which is how I can keep the framerate so high.

https://drive.google.com/file/d/1xZkXsWDZtpe0H3BKSkdzvjOA3UE...

WOW! I was undergraduate chemistry (decades ago) and if these visualizations had existed back then... I may have actually done something with my chemistry training! The `SMILES` number text input had me in flashbacks about some of the fledgling research I half-assed waybackwhen.

Thanks for providing such a wonderful resource for future chemists!

Reply on news.ycombinator.com