huac

I sense that most folks will end up using this via a Python wrapper, if at all, much like how HF has Transformers written in Rust but with a Python wrapper.

This library is targeting production, a much smaller subset of engineers. See the 'why candle' section: https://github.com/huggingface/candle#why-candle. So you could use PyTorch to build models and Candle to serve them.

I'm writing a blog post with some more reasoning but my view is that it can be useful for certain simpler tasks (eg unstructured -> structured, basic summarization) and not more complex things (eg generation).

The tricky thing is that finetuning makes a big difference, and while it should be possible to hotswap LoRA adapters (at some cost to performance), I haven't figured that out yet.

you need an inference server. I am doing ~400 tokens/sec on 7B with a 4090 with multiple concurrent (streaming!) requests.

it's reasonably straightforward for me to host this and serve public requests, but would likely just be a base model -- not sure if hosting (eg) 13B chat can serve peoples' use cases

agree! and notably - there's a difference between pre- and post-filtering -- left as an exercise to the reader :)

dev experience is very very important, but I think it's so subjective -- if you are really pushing the boundaries, maybe you want a super powerful platform which can use all the bells and whistles, but if you want to hack a weekend project together, maybe you just want some API calls.

I would suggest that anyone trying a real comparison of vector DB's consider the following

    - necessary functions / use cases (eg prefiltering, dense search)
    - embeddings version management
    - anticipated embedding size (the article only considers glove-100 on ANN-benchmarks, which is quite different from openai-ada-002 1536 - both in terms of their output distribution and the vector size)
    - required precision / recall
    - required ingestion speed 
    - required ingestion throughput / time to ingest periodic updates
    - required query speed (percentiles, not average!)
    - required query throughput
    - required RBAC, data privacy, active-active, etc

...and so much more. ANN-benchmarks is a good start for thinking about this but remember that actual throughput is quite different from whatever you see in the algorithms benchmarking!

You should test yourself on your own use case (eg vector dimension, prefiltering, throughput, target latency). In my testing, using identical HNSW configs between OS and a purpose built vector DB, I saw 10x+ better performance with the vector DB, despite much smaller CPU usage for vector DB and even including internet latency for the vector DB (but not OS).

This may not matter if you are not doing high throughput / have tight latency requirements, but in my case, it did. Of course you should weigh that versus the convenience of preexisting ES/OS clusters and so on. You can also use ES/OS together with a separate vector DB. (these tradeoffs are, of course, what make a static benchmarking post like this one so hard to think about).

did they announce that anywhere? it does appear like progress has slowed down quite a lot.

I've written a version of this in Golang (tied to OpenAI API, mostly): https://github.com/stillmatic/gollum/blob/main/dispatch.go

Define a struct and tag it with golang's json comments. Then, give it a prompt and ...

    type dinnerParty struct {
        Topic       string   `json:"topic" jsonschema:"required" jsonschema_description:"The topic of the conversation"`
        RandomWords []string `json:"random_words" jsonschema:"required" jsonschema_description:"Random words to prime the conversation"`
    }
    completer := openai.NewClient(os.Getenv("OPENAI_API_KEY"))
    d := gollum.NewOpenAIDispatcher[dinnerParty]("dinner_party", "Given a topic, return random words", completer, nil)
    output, _ := d.Prompt(context.Background(), "Talk to me about dinosaurs")

and you should get a response like

    expected := dinnerParty{
        Topic:       "dinosaurs",
        RandomWords: []string{"dinosaur", "fossil", "extinct"},
    }

no, because the compute intensity scales with the number of classes which you wish to classify to. if you have n classes, you need to do n gzip compressions at inference time. in the embedding world, you only call the embedding model once on insert, and only need to dot product at inference time.

the same logic extends to using a self-hosted embedding model, which tend to be as good as Ada on most benchmarks, and yes, can be finetuned over your private data.

how would you build an index over the gzip encoded data? seems quite different from building indices over vector embeddings.

The gzip approach is much slower at inference time because you need to compute the gzip representation of the concatenated strings (query + target). Intuitively, this should be significantly more than a dot product of two embedding vectors.

it's marketing :)

from somebody who bought 2500 of them: https://news.ycombinator.com/item?id=36313960

It would be really cool to support more vendors (eg Lambda Labs, Runpod, etc) - getting GPU availability feels like 'any port in a storm' sometimes :)

It is, in fact, a try-catch. https://github.com/BerriAI/reliableGPT/blob/main/reliablegpt...

I think it's inevitable that the LLM providers will build out more and more of the dev tools around their stack, such as this Azure service. The model providers can offer significantly better products via their direct access to the model weights, and retrieval augmented generation (what's going on here) is one good example. Think of them as AWS but with the world's best EC2 instances. How many of these MLOps / infra companies will be able to outsell OpenAI?

https://hua.substack.com/p/openai-is-a-hyperscaler

does that scale to high-dimensions? IIUC the photon mapping procedure typically uses a kd-tree to find closer/farther candidates, and if you consider something like ada embeddings (1536 dim), that would be very computationally intense. photon mapping feels like a 3D problem

I have gotten 10x speedups with SIMD on modern hardware - goes from fast to blazing. https://github.com/stillmatic/gollum/blob/main/vectorstore.g...

FWIW - Weaviate's pure Go implementation of cosine similarity is probably as fast as you can possibly get in pure go, without SIMD -- https://github.com/weaviate/weaviate/blob/2f44dbee52196656bd...

nice, if I'm reading this right, you do random projection as a first stage filter, then exact search over the filtered subset? that's clever, did you test precision/recall at all?

I'd be curious how this stacks up on ANN-benchmarks (https://ann-benchmarks.com/).

FWIW exhaustive search is still probably good enough for most use cases. IMO the exhaustive search should use a heap https://github.com/fennel-ai/fann/blob/main/src/main.rs#L12 as you're only looking for top-k, it reduces time and memory complexity greatly. On a relatively unoptimized Golang implementation (though much beefier hardware) I get ~100ms to process 1M vectors of 300 dim. Still quite a bit slower than approximate, of course, but in absolute terms probably good enough for most use cases, especially because many use cases don't have 1M vectors.

Per card? DGX is 8 H100, so ~$300k per node?

Arguably better async primitives too. Python's main advantage at this point feels like a stronger ecosystem of packages calling into native code (eg numpy et al mostly call optimized C routines). If Typescript adds more of those libraries (like this one), then I could see it gaining a lot of adoption.

I don't quite understand the convenience, R2 and the other services are already quite easy to use. I wrote a simple wrapper that exposes the same API: https://github.com/stillmatic/storage-wrapper/blob/main/stor...

The wrapper is pretty straightforward, I can see how it's a teensy bit simpler to have Vercel manage things for you, but it's not really much easier...

The model isn't particularly clever and isn't generating something useful here. Better model will for sure get better results - for now this is running free and without signup (and is quite simple). You can easily imagine that a paid tier would solve them though!

GPT-4 gives a better response fwiw:

``` graph TB Start(Start) --> A[Insert one end of the shoelace into the bottom left eyelet] A --> B[Make sure the lace is even on both sides] B --> C[Cross right end over the left end] C --> D[Insert the right end into the second eyelet on the left side] D --> E[Pull laces tight] E --> F[Cross left end over the right end] F --> G[Insert the left end into the second eyelet on the right side] G --> H[Pull laces tight again] H --> I[Continue lacing the shoe in a crisscross pattern] I --> J[Reach the top of the shoe] J --> K[Tie laces in a bow or knot] K --> End(End) ```

Great to hear it was helpful! What did you try to generate?

Agree the formatting is not great. The tool is not intended to be 'perfect', with respect to formatting or to content, so if I continue working on this, I think the most useful step would be to add "export to Figma" -- and let you tweak there. I think this is most useful as a very fast prototyping tool.

I found myself context switching back and forth between my terminal and ChatGPT, so I wrote a very simple CLI to talk to the ChatGPT model directly from your terminal. It's not free (uses your API key) but the costs are basically de minimis.

There are quite a few other similar CLI's, but this is the only one that I'm aware of which implements streaming completions and has zero external dependencies (self-contained build, OS-agnostic).

Shameless plug - made a very similar app (no sign-in): https://autodiagram.vercel.app/

I wrote this piece about the different subscription models a few years back and think that it held up very well: https://hua.substack.com/p/subscriptions-as-price-discrimina...

In particular re Medium:

> On the backend, the way [Medium's] partner program works is that users’ membership fees are allocated proportionally to the writer based on how many articles the user read. Fine. Also there is something about clapping. I guess if I really like a writer I could make sure to read a lot of their articles so they can get some of that membership fee. But, as a consumer, I: a. can’t subscribe directly to the writer, and b. can’t signal that I value their content at more than $5 per month. No amount of claps can make up for that!

Writers and readers both don't get the right amount of value from their framework.

Did you have prior Go experience to the rewrite? My experiences line up with this quite closely, similar tasks are much better suited to Go, concurrency management is night and day vs Python. But Go can seem scary to folks without previous experience and that turns lots of folks off - even though the language and syntax don't feel _that_ different and there is a similar batteries included approach, though Go's batteries are generally better.