flutetornado

I was wondering if those ternary bits get expanded into full floats internally in the kernels - you’re probably right about lack of tuned kernels. I’m not sure you could just tune your way out of that easily though. Any suggestions on trying particular solutions?

450 with PTQ_01 and 900 with the other PQ2_0.

GPT Astra did some benchmarking on the DGX Spark. Speed: 34.38 tokens/sec for generation.

Seems like we don't have a drafter model yet so it could not test with speculative decoding on. ngram speculative decoding did not help too much either - not enough accepted tokens.

Smaller size I suppose does not mean better performance in this case - we maybe limited by Spark's low memory bandwidth.

I ended up creating a personal vim plugin for merges one night because of a frustrating merge experience and never being able to remember what is what. It presents just two diff panes at top to reduce the cognitive load and a navigation list in a third split below to switch between diffs or final buffer (local/remote, base/local, base/remote and final). The list has branch names next to local/remote so you always know what is what. And most of the time the local/remote diff is what I am interested in so that’s what it shows first.

My experience with qwen3.5 9b has not been the same. It’s definitely good at agentic responses but it hallucinates a lot. 30%-50% of the content it generated for a research task (local code repo exploration) turned out to be plain wrong to the extent of made up file names and function names. I ran its output through KimiK2 and asked it to verify its output - which found out that much of what it had figured out after agentic exploration was plain wrong. So use smaller models but be very cautious how much you depend on their output.

There are several useful ways of engineering the context used by LLMs for different use cases.

MCP allows anybody to extend their own LLM application's context and capabilities using pre-built *third party* tools.

Agent Skills allows you to let the LLM enrich and narrow down it's own context based on the nature of the task it's doing.

I have been using a home grown version of Agent Skills for months now with Claude in VSCode, using skill files and extra tools in folders for the LLM to use. Once you have enough experience writing code with LLMs, you will realize this is a natural direction to take for engineering the context of LLMs. Very helpful in pruning unnecessary parts from "general instruction files" when working on specific tasks - all orchestrated by the LLM itself. And external tools for specific tasks (such as finding out which cell in a jupyter notebook contains the code that the LLM is trying to edit, for example) make LLMs a lot more accurate and efficient, efficient because they are not burning through precious tokens to do the same and accurate because the tools are not stochastic.

With Claude Skills now I don't need to maintain my home grown contraption. This is a welcome addition!

I prefer it because it forces distillation to core ideas, consumable quickly. Busy people have too little time to read too much verbiage.

Everyone in my entire team - best of engineering as well as every manager left. Underpaying and over subscribing people has become a hallmark over there - it's just a body shop now. Engineers are just numbers on a sheet, to be exploited, chewed and cast aside when they eventually burnout. Upper management has no vision and everyone's constantly firefighting and struggling to catch up with competitors who had long term vision to invest in engineering teams, tooling and infrastructure to scale up the products and people. They want to do in 2 years what took Google and Amazon a couple of decades. Result post-HPE: poor quality, unscalable, cobbled together, barely functional codebase. Before, the startup I worked for had a well balanced rare combination of high performance, modular and well architected codebase. Later the constant push to ship as fast as possible to catch up with competition, completely destroyed the whole thing - teams, codebase and infrastructure. All because they only know how to react and have no idea how to stay ahead of the curve. Buying startups has become their only means of survival as talent stays away from their brand and the only way to justify value to shareholders is to jump from one rock to another, hoping the new one will rocket them away from the black hole they are spiraling into - all they manage to do is stick to the new rock and pull it with them as fast as they were going into the hole they will eventually vaporize in.

I’d do that every time I get a chance! Ex-HPE black label on my resume from a startup I used to work in that they bought. That company is a complete horror show.

I think the performance in terminal apps could come from the fact that terminals can use pre rendered, pre cached in GPU, text glyphs covering a fixed size small grid (of glyph sized blocks instead of pixels.) At least that’s what I have wondered - because the terminal UI experience has been always a lot more slick compared to heavy GUI based programs. Feel free to correct me if someone has done actual performance analysis.

I was able to compile ollama for AMD Radeon 780M GPUs and I use it regularly on my AMD mini-PC which cost me 500$. It does require a bit more work. I get pretty decent performance with LLMs - just making a qualitative statement as I didn't do any formal testing, but I got comparable performance vibes as a NVIDIA 4050 GPU laptop I use as well.

https://github.com/likelovewant/ROCmLibs-for-gfx1103-AMD780M...

My understanding is that top_k and top_p are two different methods of decoding tokens during inference. top_k=30 considers the top 30 tokens when selecting the next token to generate and top_p=0.95 considers the top 95 percentile. You should need to select only one.

https://github.com/ollama/ollama/blob/main/docs/modelfile.md...

Edit: Looks like both work together. "Works together with top-k. A higher value (e.g., 0.95) will lead to more diverse text, while a lower value (e.g., 0.5) will generate more focused and conservative text. (Default: 0.9)"

Not quite sure how this is implemented - maybe one is preferred over the other when there are enough interesting tokens!

GPU workloads are either compute bound (floating point operations) or memory bound (bytes being transferred across memory hierarchy.)

Quantizing in general helps with the memory bottleneck but does not help in reducing computational costs, so it’s not as useful for improving performance of diffusion models, that’s what it’s saying.

Altair is superb. Have used it a lot and it has become my default visualization library. Works in VSCode and Jupyter Lab. The author has a great workshop video on youtube for people interested in altair. I especially like the ability to connect plots with each other so that things such as selecting a range in one plot changes the visualization in the connected plot.

One possible downside is that it embeds the entire chart data as json in the notebook itself, unless you are using server side data tooling, which is possible with additional data servers, although I have not used it, so cannot say how effective it is.

For simple plots its pretty easy to get started and you could do pretty sophisticated inter plot visualizations with it as you get better with it and understand its nuances.

Try neogit plugin for neovim. Its a work in progress.

https://github.com/NeogitOrg/neogit

Yes, on a related note Neovim just got support for WASM plugins and apparently WASM is 100% faster than Lua (neovim's default plugin language runtime) for this use case according to the author. So now plugins in any language that can be compiled to WASM are possible.

Edit: https://github.com/Borwe/wasm_nvim

I found this very helpful when switching to nvim recently. Kudos to the author for having the nvim config on github and making videos explaining how he set it all up:

https://github.com/LunarVim/Neovim-from-scratch

https://www.youtube.com/watch?v=ctH-a-1eUME&list=PLhoH5vyxr6...

JupyterLab

JupyterHub

Jupytext - converts ipynb to py

Nbstripout - strips all output from ipynb

Nbmerge - resolves merge conflicts

Vim-jupytext - vim plugin to auto convert ipynb to py

Papermill - parameterize notebooks

Git

Pandas, Altair - data analysis / Visualization

Phabricator - code reviews of notebooks

Vimdiff + vim-jupytext - diffs in terminal

This solved all my jupyter problems.

I have been using oil infused Ceramic cookware from Calphalon (Classic). The handles are made of stainless steel which stays cool despite all the cooking. Quite convenient. Works very well as a replacement for Teflon based non stick cookware I used to use earlier. It's not perfect but using a bit of butter or oil when cooking gives good results.

More details about various varieties here: https://www.calphalon.com/supportShow?cfid=cookware-use-and-...

This is the best solution I have found so far being a long time vim user. ssh + tmux + vim is all you need. Least amount of potential problems in this setup. vim runs locally on the server where "remote development" is being done, so there is very little friction in terms of getting it to do what you'd like it to do given all the plugins you can add to it for beefing it up.

I have heard stories from coworkers about work being lost when editing is done remotely instead of locally, so never ventured that way.

For a lot of things such as tables, lists, adding links, etc, markdown allows you to do it "inline" while typing, instead of forcing out of band operations.

If space colonization leads to mining resources from asteroids and offloading the environmental costs of such mining from earth or get us building materials strong enough to give us a space elevator or space factories sending their waste into the sun, you wouldn’t be saying this perhaps.

I have always wanted Kindle to support a lot more features than what it does today. Digital books could be a lot more than what they are today. Want to embed a python REPL in a python book for users to test things right in the book, from a common source? Want to embed function graphs in a Math book from a common source? How about a theorem prover right there in the book? How about a Wolfram engine embedded in there to solve arbitrary calculus problems? How about a section in another book - license it from the other publisher and embed it into your book? Embedding a video, a web page, a 3D visualization engine, etc should all be possible in a digital book.

Looks like this maybe able to achieve all of this. Just need to get Amazon Kindle team on board with this.

Not worried about the singularity anymore after listening to the smartest AI’s generated stories.

I like to use the vim plugin DrawIt for ascii diagrams:

https://www.thegeekstuff.com/2009/12/how-to-create-ascii-dra...

https://github.com/vim-scripts/DrawIt

Pretty easy to create comments for code that need a drawing for clear expositions.