Retro_Dev

Indeed - definitely not cost effective to run it on this laptop LOL. It makes me wonder how fast we could run the model if we could fit the weights entirely within CPU cache (assuming a whole ton of CPUs with low latency & high speed IO of course).

I ran it on my laptop, which is a Lenovo Legion 5i (think 32 GB RAM, 4060 w/ 8 GB VRAM, you get the picture). It was a quantized model (otherwise it would not fit on my NVMe 1TB drive) at 4 bits per weight - UD_Q4_K_XL. It ran at about 12 seconds per token (not tokens per second). A fun project, but not worth it. I used 4096 tokens of context cache, and I ran it with llama.cpp - as it supports memory mapping. Because the whole thing could obviously not fit in RAM, I was curious how much it would need to stream from SSD. The answer? For a simple 4 sentence description of who it was, about 1.5 TiB was streamed from disk.

OpenZL is nice, but it's often less useful than you think - it requires that you know the structure of your data, and don't care about inspecting that data outside of your program. I've extracted one too many png files from a word document (by renaming .docx to .zip) to desire OpenZL everywhere... It might be better as a short-term "data in transit" compression than for long term storage.

"open source" means that the code itself (for LLMs - this is training code) is available to the general public. "open weights" means that the weights (trained over time) are available publicly, rather than locked behind a paywalled chat. I do not know of an open source LLM that is not also open weights (unless they never bothered training it). Models like Claude and Gemini are neither open source, nor are they open weights.

Attemping to write my own CDCL SAT solver right now. I've experimented in the past with a DP & DPLL SAT solver. I'm currently somewhat mentally stuck on how to create the derived clause after a conflict, but I'll get there :)

I hope that this brings out a bunch more real study about the qualitative metrics of these models, both to increase the confidence and accessibility of local LLMs, but also to reduce the blind worship that seems to be propagating about their miracle work in all domains.

No questions, but I really enjoyed the article - thank you for sharing. It amazes me how few vacuum tubes these early computers use, compared to the billions and trillions of mosfet transistors used in modern devices.

A gamma wavelength handheld laser would be cool; "and on this petri dish, we see a dot of cells instantaneously develop cancer"

I'm very pleased with the performance of the largest gemma4 model (which I tested through ollama). My singular data point on whether an LLM remembers things well is whether it can translate toki pona to (and from) English. I find it easy to evaluate because I know the language. This local LLM marks the first version that 1) doesn't hallucinate words - at least, for the largest model - and 2) uses common word-phrases that other toki pona speakers use, and most importantly 3) can actually run on my laptop.

There's solid reason for the translation here; the Zig core team is aiming to eliminate duplicated code and C functions, and avoid the need to track libc from multiple sources. In the future, LLMs could serve as part of this, but they are currently quite terrible at Zig (as far as I understand it, it's not a lack of Zig code samples, it's an imbalance of OLD Zig to NEW Zig, as Zig changes quite frequently).

You would need to consider if it is even worth it translating your C code. If the paradigm is identical and the entire purpose would be "haha it is now one language," surely you could just compile and link the C code with libzigc... In my opinion, it's not worth translating code if the benefit of "hey look one language" requires the cost of "let's pray the LLM didn't hallucinate or make a mistake while translating the code."

In the theoretical world where a subset of English could be formalized and proven and compiled, the complexity of the language would reduce my willingness to use it. I find that the draw of AI comes from it's "simplicity," and removing that (in favor of correct programs) would be pointless - because such a compiler would surely take forever to compile "English" code, and would not be too different from current high level languages, imo.

It is my opinion that even if Zig were nothing more than a syntactical tweak of C, it would be preferable over C. C has a lot of legacy cruft that can't go away, and decades of software built with poor practices and habits. The status-quo in Zig is evolving to help mitigate these issues. One obvious example that sets Zig apart from C is error handling built into the language itself.

Totally agree with that - although even right now zig is excellent for gamedev, considering it's performant, uses LLVM (in release modes), can compile REALLY FAST (in debug mode), it has near-seamless C integration, and the language itself is really pleasant to use (my opinion).

As far as I know, Zig has a bunch of things in the works for a better development experience. Almost every day there's something being worked on - like https://github.com/ziglang/zig/pull/24124 just now. I know that Zig had some plans in the past to also work on hot code swapping. At this rate of development, I wouldn't be surprised if hot code swapping was functional within a year on x86_64.

The biggest pain point I personally have with Zig right now is the speed of `comptime` - The compiler has a lot of work to do here, and running a brainF** DSL at compile-time is pretty slow (speaking from experience - it was a really funny experiment). Will we have improvements to this section of the compiler any time soon?

Overall I'm really hyped for these new backends that Zig is introducing. Can't wait to make my own URCL (https://github.com/ModPunchtree/URCL) backend for Zig. ;)

Yep, you are correct! The first example was a bit too simplistic. A better one would be https://github.com/RetroDev256/comptime_suffix_automaton

Do note that your linked godbolt code actually demonstrates one of the two sub-par examples though.

You can change the `target` in those two linked godbolt examples for Rust and Zig to an older CPU. I'm sorry I didn't think about the limitations of the JS target for that example. As for your link, It's a good example of what clang can do for C++ - although I think that the generated assembly may be sub-par, even if you factor in zig compiling for a specific CPU here. I would be very interested to see a C++ port of https://github.com/RetroDev256/comptime_suffix_automaton though. It is a use of comptime that can't be cleanly guessed by a C++ compiler.

Good question! The TL;DR as I understand it is that it won't matter too much. For example, the self-hosted x86_64 backend (which is coincidentally becoming default for debugging on linux right now - https://github.com/ziglang/zig/pull/24072) has full support for most (all?) builtins. I don't think that we need to worry about that.

It's an interesting question about how Zig will handle additional builtins and data representations. The current way I understand it is that there's an additional opt-in translation layer that converts unsupported/complicated IR to IR which the backend can handle. This is referred to as the compiler's "Legalize" stage. It should help to reduce this issue, and perhaps even make backends like https://github.com/xoreaxeaxeax/movfuscator possible :)

Ahh perhaps I need to clarify:

I don't love the noise of Zig, but I love the ability to clearly express my intent and the detail of my code in Zig. As for arithmetic, I agree that it is a bit too verbose at the moment. Hopefully some variant of https://github.com/ziglang/zig/issues/3806 will fix this.

I fully agree with your TL;DR there, but would emphasize that gaining the same optimizations is easier in Zig due to how builtins and unreachable are built into the language, rather than needing gcc and llvm intrinsics like __builtin_unreachable() - https://gcc.gnu.org/onlinedocs/gcc-4.5.0/gcc/Other-Builtins....

It's my dream that LLVM will improve to the point that we don't need further annotation to enable positive optimization transformations. At that point though, is there really a purpose to using a low level language?

To me it didn't really feel like C++; it is much less complicated. Could you explain more what you mean by this?