huac

I would, at the very least, wrap the errors being returned inside the process function https://github.com/neurocult/agency/blob/14b14e50a7570189388...

Or, I suppose the user must handle exception behavior in their custom `OperationHandler`

yeah, I just think it's a question of picking the right set of things to abstract away (and we're all figuring out what set that should be). I chose to not add much abstraction over the go-openai library to support maximum customizability, and better support as openai adds functionality. I think the custom abstractions (eg Message, Operation, etc) are helpful iff you add additional non-openai providers.

fwiw I really like agency's interceptor funcs, good place to add tracing, metrics, logging, etc.

definitely serve different purposes, I basically think abstracting away the flows (agency/langchain) is the wrong layer of abstraction, and that it's more useful to abstract away certain problems to solve (eg vector retrieval) or useful patterns (eg function dispatch). another comment mentioned that langchain had hardcoded prompts, which make it difficult to experiment and iterate (easier today, perhaps, when I first used it, it was certainly a mess). where is the complexity hidden?

imo using someone else's abstractions for your entire workflow is a nice way of making a hackathon project, but a terrible way to build a full product.

I like Go a lot for working with OpenAI etc, it's 'just' API calls and Go is great at that. I've opensourced some bits here: https://github.com/stillmatic/gollum -- in particular, function dispatch (given a prompt, return an arbitrary Go struct) is really nice, as is a very fast in-memory KNN index.

I concur that, for the use cases that OpenAI's voices cover, it is significantly better than Eleven.

It is theoretically possible to train a model that, given some speech, attempts to continue the speech, e.g. Spectron: https://michelleramanovich.github.io/spectron/spectron/. Similarly, it is possible to train a model to edit the content, a la Voicebox: https://voicebox.metademolab.com/edit.html.

to zoom out a bit, this is known as the 'cocktail party problem' and is very much considered unsolved (particularly when there is an unknown number of speakers or overlapping speech).

nice work! generating good example data is the most important part of finetuning.

imo summarization is also a fairly simple task -- I wouldn't be surprised if a fine-tuned open source model (eg llama 13 / mistral 7) would get to similar performance.

"it depends" https://kipp.ly/transformer-inference-arithmetic/

10k H100's cost around $400M at sticker price. Sometimes I wonder if a massive setup like this is less expensive (volume discount) or more.

the nice thing too is that because you are freezing almost all the parameters, and generally in lower precision (eg QLoRA loads the full model in 4-bit), it's super low gpu memory usage. a free Colab will suffice for finetuning a 7b definitely, renting a 3090 is less than 50 cents an hour, pretty low barrier to entry to try something!

I don't have any access to their financials, so this is speculative, but while they do 'give away' GPT-3.5-turbo in the free ChatGPT, the rest of the business is likely extremely profitable. If you want to estimate cost of serving those free requests, consider how much it costs to do that via API. A 10 message conversation, where ChatGPT outputs 200 tokens each time, is $0.002, or two-tenths of a cent. I believe their API usage to still be positive margin for them. (Of course, now consider how much markup there is in ChatGPT pro!!)

There is a difference between pricing aggressively and pricing at a loss. Their pricing for gpt-3.5-turbo now matches leading public providers for Llama-70B ($1/million tokens). Rumors are that 3.5-turbo is actually a 20B model, but even let's assume that it is larger than 70B: OpenAI can still price more aggressively than Llama-70B providers because they have better throughput and utilization of the same hardware.

I think this is one of the most important possible works for open source LLM's, really glad y'all pushed this forward!

That's not hyperbole. Why is OpenAI able to charge so little for their API's? I have heard rival mega LLM company CEO's complain that OpenAI's prices would be a loss for their rivals. But I think it's still positive margin, and that they can charge low prices for API because they've invested more into managing the infra, sure, but most importantly because they have the best utilization of their existing hardware.

If it costs everyone $X/gpu/hr to serve models, the company that has the most throughput wins on price. In a world without finetunes, the most capable model, the one that can zero- or few-shot the most tasks will have the most usage. Finetuned open models can reach parity with GPT on narrow tasks, but until now, having public providers serve the models was expensive. Your private finetune is only going to be queried by you, not everyone, so it's super expensive to serve on a per token level. With hot swappable LoRA adapters, that calculus changes, and the cost per token can go way down. Super, super exciting!

you're thinking of 1455 market st. this is 1455 third st. (the coincidence amused me)

the 'low resource languages jailbreak gpt-4' paper (https://arxiv.org/abs/2310.02446v1) shows that the models, when prompted in a low resource language, return highly coherent and sophisticated responses. it's possible that RLHF does not do a great job here but the base models are clearly cross-applying intelligence from high resource languages to low resource languages.

15 million GPU hours is interesting, llama 2 had about 1.7 million hours for the 70B (3.3M for all). Their GPU hours are on LUMI which is AMD MI250 GPU, which Mosaic reports is about 75% of the A100 (https://www.mosaicml.com/blog/amd-mi250) but AFAIK hasn't been tested at this kind of scale. so, validating if the MI250 in a cluster can effectively train a large scale LLM will be useful. not that it really matters to any other group, you won't get these in the cloud anytime soon.

either way - say that they have about 10M A100 hours. if they use half their budget for the biggest model as FB did with Llama (rest on testing / smaller models as they scale), then they'll budget 5M A100 hours for the biggest model, or 3x llama-2's 1.7M A100 hours (2k A100 for 21 days).

However, why wouldn't Llama-3 also be 3x bigger than Llama-2? FB's H100's are coming online; they apparently have 4k H100 in a cluster (https://www.nextplatform.com/2023/09/26/meta-platforms-is-de...), and each H100 is about 3x the speed of an A100 (https://www.mosaicml.com/blog/coreweave-nvidia-h100-part-1). So if they provided the same 'calendar time' of 21 days over a single cluster of 4096 H100, that would be about 6.2M A100 hours for a 70B model.

there is an entirely separate question of, LLM's are functions of compute _and_ data, will they also find enough high quality _and_ kosher tokens in their low resource languages?

rebuttal: compute optimality matters https://arxiv.org/pdf/2203.15556.pdf

We used Triton Inference Server (with a Golang sidecar to translate requests) for model serving and a separate Go app that handled receiving the request, fetching features, sending to Triton, doing other stuff with the response, serving. This scaled to 100k QPS with pretty good performance but does require some hops.

In general writing pure Go inference libraries sucks. Not easy to do array/vector manipulation, not easy to do SIMD/CUDA acceleration, cgo is not go, etc. I wrote a fast XGBoost library at least (https://github.com/stillmatic/arboreal) - it's on par with C implementations, but doing anything more complex is going to be tricky.

I would love some examples of how to do non-trivial data interop between Rust and Python. My experience is that PyO3/Maturin is excellent when converting between simple datatypes but conversions get difficult when there are non-standard types, e.g. Python Numpy arrays or Rust ndarrays or whatever other custom thing.

Polars seems to have a good model where it uses the Arrow in memory format, which has implementations in Python and Rust, and makes a lot of the ndarray stuff easier. However, if the Rust libraries are not written with Arrow first, they become quite hard to work with. For example, there are many libraries written with https://github.com/rust-ndarray/ndarray, which is challenging to interop with Numpy.

(I am not an expert at all, please correct me if my characterizations are wrong!)

yeah, I hear you that the decoder-only infilling approach is 'weird' -- I just don't know if I agree that it's manifestly worse at language understanding / performance than the BERT appraoch

where is that reported? in Table 14 I see PSM performing much better than SPM. I also see a note about the SPM performance which attributes the degradation to the tokenizer edge cases

> As an example, our model would complete the string 'enu' with 'emrate' instead of 'merate' which shows awareness of the logical situation of the code but incomplete understanding of how tokens map to character-level spelling.

that doesn't really feel like a failure of language modeling to me

this comment is downvoted but it is correct. while fill-in-the-middle is less obvious in the decoder-only paradigm, it is still possible. one example is code-llama https://ai.meta.com/blog/code-llama-large-language-model-cod... it is a variant of llama 2 (GPT-style decoder-only) but it supports infilling

> [W]e split training documents at the character level into a prefix, a middle part[,] and a suffix with the splitting locations sampled independently from a uniform distribution over the document length. We apply this transformation with a probability of 0.9 and to documents that are not cut across multiple model contexts only. We randomly format half of the splits in the prefix-suffix-middle (PSM) format and the other half in the compatible suffix-prefix-middle (SPM) format described in Bavarian et al. (2022, App. D). We extend Llama 2’s tokenizer with four special tokens that mark the beginning of the prefix, the middle part or the suffix, and the end of the infilling span

Tokens are super powerful :)

I am, with many optimizations eg dynamic batching, tensorrt, but most important is batch size > 1 -- cheating a little bit in the comparison ;).

batch size 1 is more like 200 tok/s, still a very large improvement. the reason why this is still an interesting comparison is that it's comparing consumer hardware vs scaled datacenter setups. datacenter setups are much more efficient in large part because they are designed to process many inputs at once, fully utilizing the hardware. there are advantages to scale in inference, for the gpu-rich.

The OP notes that they can infer a 7B model at 2s/tok using their CPU setup. Equivalent inference on a 4090 ($1600) is up to 400 tok/s, or 0.0025 s/tok. This is an 800x difference.

(to be fair, you can do better than the OP's performance, but we _are_ talking several magnitudes of performance, not to mention energy efficiency, which matters at scale)

because on a per-inference level, it's _still_ cheaper to use GPU than it is to use CPU, even if you ignore latency and only consider offline batch applications.

I wanted a very simple feature flagging solution, so I wrote my own with Google Sheets as the backend: https://github.com/stillmatic/flagsheet

It's slightly more complex than a code push but still very easy to use.

Finetuning teaches skills. RAG provides 'facts.'

Net effect: fewer clicks to source websites, fewer clicks on ads on said websites.

I feel like this is a linear optimization problem (maximize benchmark score for games and/or for work) constrained on price / part compatibility, not something which you can just feed into ChatGPT. Unfortunately, the compatibility constraints are tricky, as well as understanding what the benchmark score would be with a nonlinear set of items.

What I would do is classify each item to the best item year (similar to FB's best device year), find benchmark scores for representative sets of item-years, and use that as a 'oracle' to guide the solver.

Is it possible to have a 'knob' here? some topics might need low latency even if most don't. My sense, reading this, is that while most topics / use cases will be fine on Warpstream, that some will not be.