HNHacker News
TopNewBestAskShowJobs

mmoskal

799 karma · joined May 14, 2015

https://moskal.me
submissionscomments
mmoskal··on Show HN: Prompts as WASM Programs
It's definitely very exciting direction, which we have not explored at all!
mmoskal··on Show HN: Prompts as WASM Programs
Each of these can be implemented on top of AICI. AICI lets you run an arbitrary (Wasm) program to determine which tokens are allowed next - this is strictly more expressible than CFGs.

In addition to constraining output, AICI also lets you fork the generation programmatically as well as dynamically edit the prompt which is important in some scenarios.

The idea is that several LLM inference systems implement AICI as an interface, and Guidance, LMQL, etc implement AICI backend and then everyone wins.

mmoskal··on Show HN: Prompts as WASM Programs
We believe Guidance can run on top of AICI (we're working on efficient Earley parser for that [0], together with local Guidance folks). AICI is generally lower level (though our sample controllers are at similar level to Guidance).

[0] https://github.com/microsoft/aici/blob/main/controllers/aici...

mmoskal··on Show HN: Prompts as WASM Programs
What you describe there is a great example of a custom controller - could be implemented on top of pyctrl, jsctrl, or just natively in Rust.
mmoskal··on Show HN: Prompts as WASM Programs
You generally have to tell the LLM what you want it to generate and then enforce it, i.e, the LLM has to be somewhat aligned with the constraints. Otherwise, for example if you ask for JSON it will keep generating (legal) white-space or when you ask for C code it will say:

  int sureICanHelpYouHereIsAnExampleOfTheCodeYouWereAskingFor;
In this case however, you can just do:

  await aici.gen_text(regex=r"[a-zA-Z\n]+" stop_at = "\n")
Also note that there still a lot of work to figure out how it's easiest for the programmer of an LLM-enabled app to express these things - AICI is meant to make it simple to implement different surface syntaxes.
mmoskal··on Show HN: Prompts as WASM Programs
It's only the controller that runs in Wasm, not the inference.

The pyctrl is a just a sample controller, you can write a controller that takes any kind of grammar (eg., a yacc grammar [0] - the python code in that example is only used for glueing).

Llama.cpp grammars were quite slow in my testing (20ms per token or so, compared to 2ms for the yacc grammar referenced above).

[0] https://github.com/microsoft/aici/blob/main/controllers/pyct...

mmoskal··on Generative AI's environmental costs are soaring – and mostly secret
First hit on google [0] - no idea if it's true but seems plausible... (half a year ago sama said 100m)

[0] https://explodingtopics.com/blog/chatgpt-users

mmoskal··on Fine tune a 70B language model at home
The bitnet paper was showing worse results than fp16 transformer with the same parameter count. The shocking result in the 1.58b paper (same group) is no quality loss compared to fp16.
mmoskal··on Self-Retrieval: Building an information retrieval system with one LLM
This kind of low level strategies can be tried on many models using AICI, including the substring one https://github.com/microsoft/aici/blob/main/controllers/aici...
mmoskal··on Generative AI's environmental costs are soaring – and mostly secret
Training GPT-4 is estimated at 50GWh [0]. Driving a pickup truck consumes about 30MWh per year in gasoline (13500 miles/year, 15 mpg, 33kWh/gallon). So, training GPT-4 is equivalent to driving 1000-2000 trucks for a year, or in other words gasoline consumption of small American town. Doesn't seem very high for a product used by 180m people.

[0] https://www.ri.se/en/news/blog/generative-ai-does-not-run-on...

mmoskal··on Intuitive Machines successfully lands on the Moon
It's 23 rupees to a dollar at PPP. The exchange rate is 82, so a dollar goes 3.5 times further in India not 23 :)
mmoskal··on How much electricity does AI consume?
FWIW a recent The Economist report [0] lists data center usage at 1-2% of global electricity consumption (which is 1/3 of energy IIRC) and AI at 8% of that in 2023 projected to grow to 15-20% by 2028. Given how careful the hyper scalers are about using renewables and offsets (edit: compared to other big energy users that is) I refuse to feel bad about it... [0] https://www.economist.com/technology-quarterly/2024/01/29/da...
mmoskal··on AI Controller Interface (AICI)
I'm a lead dev on this. Welcome comments/questions!
mmoskal··on A 1.55 R⊕ habitable-zone planet hosted by TOI-715
I learned something today :)

Gravity on surface of this planet will be 3(times heavier than earth)/(1.55(radius)^2), so 1.25g, however escape velocity also depends on planet radius so we get sqrt(1.55 * 1.25) = 1.4 earths escape velocity (so 15.7km/s). Assuming Isp specific impulse of engine of around 300s, let's compute m0/mf (initial mass of rocket / payload) = exp(vEsc/(Isp*g0)) (g0=9.8m/s^2 (on both planets)):

For earth, it's 45.

For this planet, it's 208.

So, the rocket (using our current chemical engine technology) needs 4.6 times more fuel than on earth. Much more difficult, but maybe possible...

edit: formatting

mmoskal··on Elevator Expert on How to Move 10k People Up a 118-Floor Skyscraper [video]
Hmm... about 15 seconds per floor (judging from Wikipedia video), times 118 floors = half hour to get to the top! I guess still faster than taking the stairs...
mmoskal··on Stuff we figured out about AI in 2023
All the nvidia/torch stuff is "only" to make it run two orders of magnitude faster or so. For the actual principles behind LLMS, 500 lines seems quite accurate.

How long would it take to explain what an LLMs is doing to someone from 1800s? It is remarkably simple.

mmoskal··on Is Ada safer than Rust?
There is also anyhow::ensure!() (which is more of a requires() but anyhow...)
mmoskal··on GPU Survival Toolkit for the AI age
GPU has 10x the memory bandwidth of the CPU though, which becomes relevant for the LLMs where you essentially have to read the whole memory (if you're batching optimally, that is using all the memory either for weights or for KV cache) to produce one token of output.
mmoskal··on Ask HN: Whats the modern day equivalent of 80s computer for kids to explore?
For the game angle https://arcade.makecode.com may be more of a fit. You can even build a cabinet.

Disclaimer: worked on both.

mmoskal··on Two photographers captured the same millisecond in time (2018)
If the water is moving at 10m/s and you can spot 0.1m difference in the photo then it would need to be within 10ms of each other.
mmoskal··on Guidance: A guidance language for controlling large language models
It updates token logits (probabilities) after every token before sampling. I don't think this is very common yet.
mmoskal··on Mojo is available for local download
I agree (and would add explicit self argument, the fact that all your imports are exported and more) I think the point here is to reuse the Python ecosystem especially in ML.
mmoskal··on Organic Maps
I've been using it for a while and love it but intend to give the open source Organic maps a try. In particular sharing a way point in Mapy.cz seem to require an account with a Czech web portal.

The reason Mapy.cz has all the trails is because they also use open street maps.

mmoskal··on Llama: Add grammar-based sampling
OTOH, AFAIK when running a model locally Guidance does something really similar to what OP is doing.
mmoskal··on Llama: Add grammar-based sampling
There is at least 6 orders of magnitude difference in computation cost of computing a token (pass through a multi-B model) and doing a single step of a program. Even if your validation is really naive, it's hard to beat 6 orders of magnitude.

So no, you're not generating tokens fast enough.

mmoskal··on The Nano ESP32
I bet they have a deal with uBlox so that people prototyping with this board will use uBlox for final mass produced design.

In general you use module instead of chip to save on certification. However for ESP32 the logical choice would be indeed a module from Espressif.

mmoskal··on Arduino Uno R4 WiFi
Well, this M4 is most likely much slower than Rp2040 (2.5x difference in clock speed plus dual core). At the same clock the difference between M0+ and M4 isn't too dramatic (unless you use FPU).
mmoskal··on Arduino Uno R4 WiFi
NRF52 line is Bluetooth LE not WiFi.
mmoskal··on DeviceScript – TypeScript for Tiny IoT Devices
Promised docs here: https://microsoft.github.io/devicescript/language/runtime#wh...
mmoskal··on DeviceScript – TypeScript for Tiny IoT Devices
The consequences of a type unsoundness in TypeScript are typically much more benign (an exception typically) than in C (a buffer overflow, arbitrary code execution, etc.). Also the application logic is more visible in TypeScript just due to it being higher-level - there isn't so much low-level detail (ever done JSON in C?) so less chance for a mistake.
← PreviousPage 5 of 10Next →