om8

Of course it is. GPT-5 is one of the most anticipated things in AI right now. To live up to the hype, it needs to be a reasoning model.

It’s unfortunate that llama.cpp’s code is a mess. It’s impossible to make any meaningful contributions to it.

Sure, but integrated graphics usually lacks vram for LLM inference.

To have a gpu inference, you need a gpu. I have a demo that runs 8B llama on any computer with 4 gigs of ram

https://galqiwi.github.io/aqlm-rs/about.html

I'm currently in Russia trying to get a US visa for my CS PhD. Because I do CS, I got into a thing called administrative processing. For 95% of people, it takes days -- weeks, tops. Because of the colour of my passport, it has already lasted for 3 months. I know people who are waiting for 2 years to pass it.

Why do you think I shouldn't have access to this website in Russia?

I believe he knows

What would you do if some of the deps started to have conflicts in them? Also, what are your plans for migration when you'll need to move from one os version to another?

Implicit solutions like yours have lower cost of entrance, but larger cost of support. uv python scripts just work if you set them up once

I have a demo that runs llama3-{1,3,8}B in browser on cpu. It can be integrated with this thing in the future to be fully local

https://galqiwi.github.io/aqlm-rs

> a local LLM is going to be the way to go

Non-technicals don't know how LLMs work, and, more importantly, don't care about their privacy.

For a technology to be widely used, by definition, you need to make it appealing to the masses, and there is almost zero demand for private LLM right now.

That's why I don't think that local llms will win. There are narrow use cases where regulations can force local llm usage (like for medical stuff), but overall I think that services will win (as they always do)

> 50 tokens is not really very much Yes! And also llama3.1’s tokens are different from Qwen and llama1 tokens. That’s the first model where meta started to use very large vocab_size.

I trust cloudflare more than my ISP, since I live in a place where internet is very state controlled.

Some of the websites just don't open without DoH.

I thing you are just arguing about words, not about meanings. I’d call what you are referring to “secure llm infrastructure ”, not “secure llm”.

But the thing is that we both agree about what’s going on, just with different words

Even human level intelligence (whatever that means) is not enough. Social engineering works fine on our meat brains, it will most probably work on llms for foreseeable non-weird non-2027-takeoff-timeline future.

Based on “bug level of intelligence”, I (perhaps wrongly) infer that you don’t believe in possibility of a takeoff. In case it is even semi-accurate, I think llms can be secure, but, perhaps, humanity will be able to interact with such secure system for not so long time

Here is an example of Rust implementation being faster than C.

https://trifectatech.org/blog/zlib-rs-is-faster-than-c/

Channels are useful when they are really (rarely) needed. IMO Channel API should've been as ugly as reflect API to be considered only in extra cases.

Burden of proof lies on you, since you mentioned corporations first

Not surprising, llama.cpp code is a mess.

It's sad that hacked things that emerge first are way more popular than properly done projects that come later.

You can’t build fault tolerant system without network lag. Your hardware can fail you any second, and every backup will be inconsistent with your local db. To solve it, you need to have a system with proper control plane and consensus algorithm like paxos inside.

SQLite is great for simple projects that can lose data, but, if you need to have no downtime, you need a distributed db.

Sure, what's the issue? Do you think that there is an algorithm that a) does not do this, b) can't be implemented in rust, but can be implemented in C or other language?

Bitwarden is the place where I store stuff safely ><. This update is just awful

Same here. I'm very sad about this 2FA thing. Bitwarden was so easy to use, I could always get an access to my accounts with just my secure master password. Does anybody know good alternative?

Not really a WebNN expert, but looks like it doesn't support CPU inference yet and only works in Chrome. It also lacks support for custom kernels, which we need for running AQLM-quantized models.

When you ask about spec change, do you mean WebNN spec or something else?

This is a demo of what's possible to run on edge devices using SOTA quantization. Other similar projects that try to run 8B models in browser are either using webgpu or 2 bit quantization that breaks the model. I implemented inference of AQLM quantized representation, making model that has 2 bit quantization and does not blow up.

The cool thing about rust is that it forces you to use thread safety primitives. It's called fearless concurrency

Looks like it is a drop in replacement for attention, but models will need to be retrained for this one, yes.

Younger me would've said that WYSIWYG editors were a mistake and that researchers should've used LaTeX.

Now I think these errors are a small price to pay for convenience. One could waste a lifetime fighting small things like this and still lose. It's just the world we live in.

How to you define memorization and reasoning? There is a large grey area in between them. Some say that if you can memorize facts and algorithms and apply them to new data, it is a memorization. Some say that it is reasoning.

More than that -- It's not clear that what humans do is not "just" a memorization. We can always look at human experience mechanisticly and say that we don't think -- we just memorized thinking patterns and apply them when speaking and "thinking"

TLDR: use shitty allocators, win shitty memory leaks

> to minimize the amount of computation

IMO backprop is the most trivial implementation of differentiation in neural networks. Do you know an easier way to compute gradients with larger overhead? If so, please share it.

Single responsibility principle in its finest.