mgreg

I'm interested in their Planner -> Implementer -> Reviewer -> Verifier process they used for this transition to Rust. I've see similar but curious how they actually implemented this.

Curious how this could be applied to greenfield coding rather than just making a copy in a new language or performance optimizing.

Link to repo for those interested: https://github.com/ZhangYiqun018/AvengersPro

There's also Rancher Desktop (https://rancherdesktop.io/). Supports moby and containerd; also optionally runs kubernetes.

This is quite the loss for OpenAI.

And no one seems to have heard what Ilya Sutskever's status is at OpenAI as well.

Unsurprising but disappointing none-the-less. Let’s just try to learn from it.

It’s popular in the AI space to claim altruism and openness; OpenAI, Anthropic and xAI (the new Musk one) all have a funky governance structure because they want to be a public good. The challenge is once any of these (or others) start to gain enough traction that they are seen as having a good chance at reaping billions in profits things change.

And it’s not just AI companies and this isn’t new. This is art of human nature and will always be.

We should be putting more emphasis and attention on truly open AI models (open training data, training source code & hyperparameters, model source code, weights) so the benefits of AI accrue to the public and not just a few companies.

[edit - eliminated specific company mentions]

I think this is built into Excel already.

In the Excel menu bar: Data -> Data from Picture

It's a great observation. People simply want their free stuff.

The potential challenge arises in the future. Today's models will probably look weak compared to models we'll have in 1, 3 or 10 years which means that today's models will likely be irrelevant in years hence. Every competitive "open" model today is tied closely to a controlling organization weather it's Meta, Mistral.AI, TII, 01.AI, etc.

If they simply choose not to publish the next iteration of their model and follow OpenAI's path that's the end of the line.

A truly open model could have some life beyond that of its original developer/organization. Of course it would still take great talent, updated datasets, and serious access to compute to keep a model moving forward and developing but if this is done in the "open" community then we'd have some guarantee for the future.

Imagine if Linux was actually owned by a for-profit corporation and they could simply choose not to release a future version AND it was not possible for another organization to fork and carry on "open" Linux?

Applying the term "open source" to AI models is a bit more nuanced than to software. Many consider reproducibility the bar to get over to earn the label "open source."

For an AI model that means the model itself, the dataset, and the training recipe (e.g. process, hyperparameters) often also released as source code. With that (and a lot of compute) you can train the model to get the weights.

This does look like a truly open model with all the components needed to replicate under Apache 2. This seems to be a fine-tuned version of their CrystalCoder model.

Kudos for releasing a fully open model that will (hopefully) foster collaboration in the community. Looks like they are also planning to release a 65B model (see Diamond model: https://www.llm360.ai/).

CrystalCoder Dataset (including prep): https://github.com/LLM360/crystalcoder-data-prep CrystalCoder Training code: https://github.com/LLM360/crystalcoder-train CrystalChat Model & Weights: https://huggingface.co/LLM360/CrystalChat

I understand where you're coming from but what they provided DOES make their work reproducible. You can use the data, source code, and recipe to train the model and get the weights.

It would be nice if they provided the weights so it could be USABLE without the effort or knowledge required.

We (I think) would all like to see more _truly_ open models (not just the source code) that enable collaboration in the community.

I very much appreciate that the authors not only published their code (https://github.com/llm-random/llm-random) but included the dataset they used (available on Huggingface - https://huggingface.co/datasets/c4) as well as the training process and hyperparameters they used so others can replicate and build on their work. The only thing really missing is the weights which would be nice to have on huggingface as well.

There are now several open source models that are fine tuned for function calling including:

* Functionary [https://github.com/MeetKai/functionary]

* NexusRaven [https://github.com/nexusflowai/NexusRaven-V2]

* Gorilla [https://github.com/ShishirPatil/gorilla]

Could be interesting to try some of these exercises with these models.

The online models to a decent job of proving up-to-date info. Simple inputs like "who won the football game last night" provided the correct score and a little detail on the NFL's Monday Night game. Did well with some other queries that require current info.

Their blog [1] states they use their own index: "In-house search technology: our in-house search, indexing, and crawling infrastructure allows us to augment LLMs with the most relevant, up to date, and valuable information. Our search index is large, updated on a regular cadence, and uses sophisticated ranking algorithms to ensure high quality, non-SEOed sites are prioritized. Website excerpts, which we call “snippets”, are provided to our pplx-online models to enable responses with the most up-to-date information."

Anyone know what their bot name is or any insight into their indexing? Impressive that they are not relying on Bing/Google/Brave/?.

1. https://blog.perplexity.ai/blog/introducing-pplx-online-llms

Some details that might interest you from SemiAnalysis [1] just published yesterday. There's quite a bit that goes into optimizing inference with lots of dials to turn. One thing that does seem to have a large impact is batch size which is a benefit of scale.

1. https://www.semianalysis.com/p/inference-race-to-the-bottom-...

Agreed on 1; people in the court case are saying Google is better.

Given that, why wouldn't Google negotiate a lower rev share?

Consider this from Apple's Eddie Cue: "Cue implied that Bing’s technology was inferior to Google’s, saying that he doesn’t 'know what we would have done” if negotiations with Google ever fell apart.' " [1]

If I were sitting across from Cue at the negotiating table and knew he felt that way I certainly would be looking at a lower rev share and saving some billions of profit for Google...

[1] https://www.bloomberg.com/news/articles/2023-09-28/microsoft...

It seems unclear if Microsoft was willing to pay more. From Bloomberg's coverage: "Microsoft business development executive Jon Tinter said that his company weighed making a multibillion-dollar investment in its relationship with Apple in 2016, an effort to outspend Google and make Bing the default option on Apple devices." They also offered to sell part of Bing to Apple it seems.

As to Samsung why else would they consider switching from Google to Bing? According to the WSJ [2] they changed their mind "given concerns over how the switch could be perceived by the market as well as the impact on its wide-ranging business relations with Google." This suggests Microsoft was willing to pay more.

[1] https://www.bloomberg.com/news/articles/2023-09-28/microsoft... [2] https://www.wsj.com/articles/google-is-spared-a-search-engin...

There’s been some good points raised by both sides in the case but there’s still some logic that doesn’t flow.

* Google pays a rev share to Apple for being _default_ search provided and default was key: “no default, no deal”. Apple turned down more money from Microsoft including offers to buy all/part of Bing. Samsung also dallied with Microsoft for more money but also turned it down.

* This suggests that there’s some other reason aside from money that Apple is going with Google. After all, as Google points out, if competition is just a click away why wouldn’t Apple click away to another partner? This could be:

Google is just better. If Apple switched their users would notice and suffer/leave Apple. There’s additional leverage that Google has that it is leveraging e.g. search ranking penalty? Access to other Google services (especially for Samsung), Agree not to compete in other areas (like their previous no-poach agreement they had).

So a couple of key questions:

1. Is Google really that much better to the _average_ internet user (note that HN crowd is not average in this regard)? Is there much risk to Apple to take the money from Bing and make them default?

2. Somewhat dependent on 1, but ff Google knew/know Apple or others are unlikely to switch because Google is that much better than why not exercise their market power and lower the rev share % they pay to others?

Feels strange for both things to be true - that Google is that much better AND that they don’t negotiate down their rev share.

Very cool and thanks for sharing.

To me a killer feature would be easily running different models simultaneously such as one for embeddings and another for completion (e.g. Chat). This likely can be done already by specifying the model parameter in Ollama (and others) but I've not explored it much yet.

I'm actually using Ollama for it's Rest API endpoint. Llama.cpp does now have it's server implementation. Unfortunately they do have different endpoints and behave a little differently.

* https://github.com/jmorganca/ollama/blob/main/docs/api.md

* https://github.com/ggerganov/llama.cpp/blob/master/examples/...

Thanks to the folks at MLCommons we have some benchmarks and data to evaluate and track inference performance published today. Includes results from GPUs, TPUs, and CPUs as well as some power measurements across several ML use cases including LLMs.

"This benchmark suite measures how fast systems can process inputs and produce results using a trained model. Below is a short summary of the current benchmarks and metrics. Please see the MLPerf Inference benchmark paper for a detailed description of the motivation and guiding principles behind the benchmark suite."

https://mlcommons.org/en/inference-datacenter-31/

For example the latest TPU (v5) from Google scores 7.13 queries per second with an LLM. Looking at GCP that server runs $1.2 / hour on demand.

On Azure an H100 scores 84.22 queries per second with an LLM. Couldn't find the price for that but an A100 costs $27.197 per hour so no doubt the H100 will be more expensive than that.

7.13 / $1.2 = 5.94 queries/second/$ 84.22 / $27.197 (A100 Pricing) = 3.09 queries/second/$

[edited to include GCP TPU v5 and Nvidia H100 relative performance info for LLM Inference]

That is a very good idea. Any thoughts as to why this hasn't happened already?

This is a very good point. There are many other ways to use OpenAI GPT including Bing Chat, Perplexity, Phind, You.com, to name just a few. Some of these services focus on the coding use case as well and may be more fit for purpose for that use case.

That's a good point. A parallel can be drawn with AWS / Amazon perhaps. After all FB uses a lot of AI in their products already so you could argue they probably know how to run them in production at scale. Following the Amazon model why couldn't they offer AI (generative LLMs in this case) to third parties?

"Meta released its own language model, known as LLaMA, to researchers and academics earlier this year, but the new version will be more widely available and customisable by companies, three people familiar with the plans said. The release is expected imminently, one of the people said."

Will be curious how they've improved on LLaMA (which is only about 4 months old) for their next release and what they've gained from the open source adoption of the model.

From the blog:

"Our paper on I-JEPA (https://arxiv.org/abs/2301.08243) will be presented at CVPR 2023 next week, and we’re also open-sourcing the training code and model checkpoints today."

A few but a search would no doubt turn up more:

Arc https://thebrowser.company/ - Perhaps not open source?

Orion https://browser.kagi.com/ - Mac only (built on webkit)

Vivaldi https://vivaldi.com/ - Chromium based

Opera https://www.opera.com/ - Chromium based

Ungoogled Chromium https://github.com/ungoogled-software/ungoogled-chromium

And this is a critical point. Not only has Microsoft raised the pricing for their Bing Search API massively (a cost to Neeva and others) there are rumors that they were starting to enforce some terms of service that prevents the use of the API for many things including training LLMs.

Note - they took down their page on the pricing update that was here: https://www.microsoft.com/en-us/bing/apis/pricing-update; article about it at https://www.ghacks.net/2023/02/17/microsoft-increases-bing-s... and the pricing is still published at https://www.microsoft.com/en-us/bing/apis/pricing.

"Microsoft threatens to restrict data from rival AI search tools" https://money.yahoo.com/microsoft-threatens-restrict-data-ri...

Bing Search API ToS: https://www.microsoft.com/en-us/bing/apis/legal

edit to add links

Looks like Bing chat is using GPT-4 already:

"Good news, we've increased our turn limits to 15/150. Also confirming that the next-gen model Bing uses in Prometheus is indeed OpenAI's GPT-4 which they just announced today." - Jordi Ribas, Corporate VP @ Bing/Microsoft

https://twitter.com/JordiRib1/status/1635694953463705600

It's called "predatory pricing".

https://www.ftc.gov/advice-guidance/competition-guidance/gui...

Craigslist and Stackoverflow come to mind.