terhechte

Or GitHub. I’m always amused when people don’t want to send fractions of their code to a LLM but happily host it on GitHub. All big llm providers offer no-training-on-your-data business plans.

Just wanted to say that I was only someone who read OsNews daily back somewhere from 2002 to 2007 or 2008. So many interesting things going on.

With the proliferation of web apps and containers, alternative operating systems are actually more feasible today. At the same time, I'm dependent on all the niceties that the macOS/iOS ecosystem offers (the integration, the sync). Something I wanted to look into is just running macOS in a very default way and then using a fast OS (such as Haiku, though their Arm64 support is not very good yet) in a fullscreen VM. With modern Apple Silicon, there's almost no performance penalty.

I don't think the time grows linearly. The more context the slower (at least in my experience because the system has to throttle). I just tried 2k tokens in the same model that I used for the 120k test some weeks ago and processing took 12 sec to first token (qwen 2.5 32b q8).

I just loaded two models of different quants into LM Studio:

qwen 2.5 coder 1.5b @ q4_k_m: 1.21 GB memory

qwen 2.5 coder 1.5b @ q8: 1.83 GB memory

I always assumed this to be the case (also because of the smaller download sizes) but never really thought about it.

To add, they say about the 400B "Maverick" model:

> while achieving comparable results to the new DeepSeek v3 on reasoning and coding

If that's true, it will certainly be interesting for some to load up this model on a private M3 Studio 512GB. Response time will be fast enough for interaction in Roo Code or Cline. Prompt processing is a bit slower but could be manageable depending on how much code context is given to the model.

The upside being that it can be used on codebases without having to share any code with a LLM provider.

Yes, that's what I tried to express. Large prompts will probably be slow. I tried a 120k prompt once and it took 10min to process. But you still get a ton of world knowledge and fast response times, and smaller prompts will process fast.

Sure but the upside of Apple Silicon is that larger memory sizes are comparatively cheap (compared to buying the equivalent amount of 5090 or 4090). Also you can download quantizations.

The (smaller) Scout model is really attractive for Apple Silicon. It is 109B big but split up into 16 experts. This means that the actual processing happens in 17B. Which means responses will be as fast as current 17B models. I just asked a local 7B model (qwen 2.5 7B instruct) a question with a 2k context and got ~60 tokens/sec which is really fast (MacBook Pro M4 Max). So this could hit 30 token/sec. Time to first token (the processing time before it starts responding) will probably still be slow because (I think) all experts have to be used for that.

In addition, the model has a 10M token context window, which is huge. Not sure how well it can keep track of the context at such sizes, but just not being restricted to ~32k is already great, 256k even better.

His purse is empty already, all’s golden words are spent.

– William Shakespeare

How is the moral dilemma of employer-controlled brain implants or eugenics equal to AI? The reason the quote can be applied is because it is a genuinely useful technology to lots of people. That's not the case for eugenics robots.

Did you grow up watching Data (Star Trek), C3PO (Star Wars), KITT (Knight Rider) thinking "Who comes up with these violent sadistic ideas"?

    I've come up with a set of rules that describe our reactions to technologies:
    1. Anything that is in the world when you’re born is normal and ordinary and    is just a natural part of the way the world works.
    2. Anything that's invented between when you’re fifteen and thirty-five is new and exciting and revolutionary and you can probably get a career in it.
    3. Anything invented after you're thirty-five is against the natural order of things.

-- Douglas Adams

Let me guess your age.

A friend of mine build a similar app last year (https://joineat.app) and it didn't go anywhere (even though it is objectively the nicer app). So there's a lot of luck involved here as well (or maybe he was too early).

Came here to say this. I've always been a happy Spotify customer, but this is something I really liked when it was available. Makes me question my subscription a bit.

People end up thinking this, but they're just looking at tech roles. However, the difference becomes clear when you compare other jobs. In tech, the US pays way more than Europe, for other jobs, they don't. Take teachers, for example:

- Median Salary in USA: $~58k [1]

- Median Salary in Berlin: 35k EUR [2]

So while there is a difference, it is by far not as huge a gap as in tech where Europe pays ~60-70k (very and some countries, e.g. Spain or Portugal way less) and in the US 200k-300k are not unusual.

The difference is that US companies understand the huge value creation potential of tech jobs (especially over the past 2 decades) whereas most european companies think value creation comes from having middle aged balding managers.

[1]: https://www.salary.com/research/salary/benchmark/public-scho... [2]: https://worldsalaries.com/average-teacher-salary-in-berlin/g...

Check out Floneum, it's basically ComfyUI for LLM's, extendable via plugins

https://floneum.com/

Scroll down a bit on the website to see a screenshot.

Man, I feel you. I did the same, the issue is that I had to regularly update my wrapper types as new exceptions occurred

There was a discussion about this on the Rust Reddit. Can't find the link right now as I'm on a very slow internet connection.

Not everybody has this luxury. I once worked on a project where the BE dev had a fundamental issue understanding types (or his PHP code did, who knows), such that, for example, a json array would change its type when empty, e.g.

  // Non-Empty
  {"objects": ["a", "b"]}

  // Empty
  {"objects": {}}

and many other such issues.

Here's a 2022 video that shows it in action

https://www.reddit.com/r/Games/comments/v42611/dario_on_twit...

Which LLM did you use with Goose? That really affects the quality of the outcome

You can, with ish (https://ish.app). It is a bit slow though and doesn't support the newest releases. (Well and by default it runs Alpine instead of Debian)

I have a set of tests that I can run against different models implemented in different languages (e.g. the same tests in Rust, Ts, Python, Swift), and out of these languages, all models have by far the most difficulty with Rust. The scores are notably higher for the same tests in other languages. I'm currently preparing the whole thing for release to share, but its not ready yet because some urgent work-work came up.

That's cool, than you, but does MLX support the Qwen 1M context yet?

I used to work together with Mert at the same company but choose to leave on my own 2 years prior to his layoffs because of similar issues as the one pointed out in this post. Most notably, the lack of vision and the excel table. The company had been dragged down by unbelievably incompetent leadership for some time. You could tell by the composition of teams that employees were not considered based on their qualifications but based on a moving-resources-around-in-an-excel-table strategy. There were also no attempts to gain knowledge from engineers or consider their feedback on decisions. This isn't ultimately something every company needs (bottom up is a double edged sword) but mixed with incompetent management it is a fastlane into chaos.

I gave it a 446433 token input, then it calculated for ~4 hours, and gave me a reasonable response. The content was a Rust / Typescript codebase where Typescript is the frontend and Rust is the backend. I asked it which backend apis are currently not used by the frontend. I haven't checked yet, but the answer looked correct.

Running this on a M4 max

I bought an M4 Max with 128g of ram just for these use cases. Currently downloading the 7b

Yes. It requires a lot of ram, and even on a M4 with a lot of ram, if you give it 1mio tokens the prompt processing alone (that is, before you get the first response token) will probably take ~30min or more. However I'm looking forward to check if indeed I can give it a whole codebase and ask questions about it.

Its also the name of the AI in Terminator Zero https://villains.fandom.com/wiki/Kokoro

I'm not sure if that is related here.

Long context windows are a problem. I gave Qwen 2.5 70b a ~115k context and it took ~20min for the answer to finish. The upside of MoE models vs 70b+ models is that they have much more world knowledge.

In addition to the ones listed by others, WizardLM2 8x22b (was never officially released by Microsoft but is available).