GPT-4 API General Availability
openai.com
openai.com
If you use GPT, you're giving OpenAI money to lobby the government so they'll have no competitors, ultimately screwing yourself, your wallet, and the rest of us too.
OpenAI has no moat, unless you give them money to write legislation.
I can currently run some scary smart and fast LLMs on a 5 year old laptop with no GPU. The future is, at least, interesting.
I have an old laptop with 16GB RAM and no GPU. Can I run these models?
https://huggingface.co/TheBloke
There's a LocalLLaMA subreddit, irc channels, and a whole big community around the web working on it on GitHub nd elsewhere.
edit: I forgot to directly answer you: yes you can run these models. 16GB of plenty. Different quantizations give you different amounts of smarts and speed. There are tables that tell you how much RAM is needed per which quantization you choose, as well as how fast it can produce results (ms per token). e.g. https://github.com/ggerganov/llama.cpp#quantization where RAM required a little more than the file size, but there are tables that list it explicitly which I don't have immediately at hand.
It might have been a contract violation for the guy who redistributed it, but I'm not a party to that contract.
I'm calling it now. Based on what I see today. Europe will position itself as a leader in AI legislation, and its economy will give way to the nations that want to enter the race and grab a chunk of the new economy.
It's a Catch 22. You either gimp your own technological progress, or start a war with a nation that does not. Pretty sure Russia and China don't really care about the ethics behind it. There are plenty of nations capable enough in the same boat.
Now what? OK, so in some hypothetical future China has an uncensored model with free reign over the internet. The US and Europe has banned this. What's stopping anyone from running the Chinese model? There isn't enough money in the world to enforce software laws.
How long have they tried to take down The Pirate Bay? Pretty much every permutation of every software that's ever been banned can be found and ran with impunity if you have the technical knowledge to do so. No law exists that can prevent that.
If it did, OpenAI wouldn't exist.
Forms of this argument get tossed out a lot. Laws don’t prevent, they hopefully limit. Murder has been illegal for a long time, it still happens.
Wouldn’t that violate the Nemo dat quod non habet legal principle and so you cannot hide behind the claim that you weren’t party to the contact?
The contracts in these cases are somewhat similar to an NDA, without the secrecy aspect. Restricted disclosure of public information. You can agree to such a contract if you want to, and a court might even enforce it, but it doesn’t affect anybody else’s rights to distribute that information.
Contracts are not statutes, they only bind the people directly involved. To restrict the actions of random strangers, you need to get elected.
It is the same as any confidential data. Logs, readings from sensors, etc etc. If it's confidential and given to a 3rd party through a contract that doesn't mean that it's suddenly not confidential data for the rest of the world, even if the 3rd party leaks it.
And if you really have a lawyer trying to tell you that some, at best, extreme grey area, is fine to build a business on, I think you should find a new lawyer.
All of these are surmountable problems.
We can beat OpenAI.
We can drain their moat.
I've got an AI powered sump pump if you need it.
Update: That is only true for the legal system I am currently residing in. No idea about e.g. the US.
MPT, OpenLLaMA, and Falcon are probably the most generally useful.
For code, Replit Code (specifically replit-code-instruct-glaive) and StarCoder (WizardCoder-15B) are the current top open models and both can be used commercially.
While you can run all kinds of GPTs locally, GPT-4 still smokes everything right now – and even it is not actually good enough to not be a lynchpin for a lot of cases yet.
I would argue that it creates better images too.
Not if you want it to write adult (graphically pornographic or violent) content.
Here's a handy model comparison chart (this is a coding benchmark, so coding-only models tend to rank higher): https://i.imgur.com/AqSjjj2.jpeg
It beats Claude and Bard.
You could probably get a 4bit 15B model going in 16GB of RAM and be approaching GPT4 in capability.
...on an old laptop, lol
Let's eat OpenAI's lunch! They deserve it for trying to steal this tech by "privatizing" a charity, hiding scientific data that was supposed to be shared with us by said charity whose purpose was to help us all, and dishonestly trying to persuade the government not to let us compete with them.
I'd post a more well rounded benchmark but the problem is that all non-coding benchmarks are currently more or less complete garbage, especially the Vicuna benchmark that rates everything as 99.7% GPT 3.5 lol.
The situation for the little guy is wildly better than most people imagine.
There are exceptions of course, as there are a few models (e.g. Vicuna, Baize) that don't do well at coding at all but otherwise perform well for chat, and the coding models I mentioned that game the benchmark by sacrificing performance in all other areas.
If you exclude those, it's very a accurate overall reasoning level comparison, at least it fits most to what I've seen their performance was for various tasks when testing out individual models. The only other valid benchmark that isn't coding are the SAT and LSAT tests that OpenAI runs on all of their models, but afaik there isn't an open version that would be widely used.
You can rent time on a hosted GPU, sharing a hosted model with others.
It used to be this took a few people to come up with writing actual responses to forum posts all day, or marketing operations plans, or pro- or anti-thing propaganda plans.
But now, you could astroturf a movement with a GPU, a ChatGPT clone, some bots and vpns hosted from a single computer, a cron job, and one human running it.
If you thought disinformation was bad 2 years ago, get ready for fully automated disinformation that can be targeted down to an online community or specific user in an online community...
No, he says it because its true and concerning.
However, just because AGI has a good chance of making humanity extinct does not mean we're anywhere close to making AIs that capable. LLMs seem like a dead end.
How? I mean surely it will lead humanity down some chaotic path, but I would fear climate catastrophe much much more than anything AI-related.
That's one of the better outcomes.
A worse outcome is that an unrestricted AI helps walk a depressed and misanthropic teenager through the process of engineering airborne super-AIDS.
Or that someone suffering from a schizophrenic break reads "I Have No Mouth And I Must Scream" and tasks an unrestricted AI to make it real.
Or we have a bug we don't spot and the AI does any of those spontaneously; it's not like bugs are a mysterious thing which only exists in Hollywood plots.
So what we have ongoing for half a century?
I honestly don’t see what changes here — super-human intelligence has limited benefits as it scales. Would you suddenly have more power in life, were you twice as smart? If so, we would have math professors as world leaders.
Life can’t be “won” by intelligence, that is only one factor, luck being a very significant other one. Also, if we want to predict the future with AIs we probably shouldn’t be looking at “one-on-one” interactions, as there is not much difference there compared to the status quo — a smart person with whatever motivation could easily do any of your mentioned scenarios. Hell, you couldn’t even tell the difference in theory if it happens through a text-only interface.
Also, it is naive to assume that many scientific breakthroughs are “blocked” by raw intelligence. Especially biology is massively data-limited, which won’t be any more available to an AI than to the researchers at hand, let alone that teenager.
The new dimension such a construct could open up is the complete loss of trust on the internet (which is again pretty close to where we stand today), which can have very profound effects indeed I’m not trying to diminish. But these sci-fi outcomes are just.. naive. It will be more of a newfound chaos with countless intelligent agents taking over the internet with different agendas - but their cumulative impact might very well move us back to closed forums/to the physical world. Which will definitely turn certain long-standing companies on its head. We will see, as this is basically already happening, we don’t need human-level intelligence, GPT’s output is more than enough.
Humans being the dominant life form on Earth may suggest otherwise.
> I honestly don’t see what changes here — super-human intelligence has limited benefits as it scales. Would you suddenly have more power in life, were you twice as smart? If so, we would have math professors as world leaders.
Intelligent humans by definition do not have super human intelligence.
Also, a big tongue in cheek but someone like John von Neumann definitely had superhuman intelligence.
Smart people get what they want more often than less smart people. This can include positions of power, but not always — leadership decisions come with the cost of being responsible for things going wrong, so people who have a sense of responsibility (or empathy for those who suffer from their inevitable mistakes) can feel it's not for them.
This is despite the fact that successful power-seeking enables one to get more stuff done. (My impression of Musk is he's one who seeks arbitrary large power to get as much as possible done; I'm very confused about if he feels empathy towards those under him or not, as I see a very different personality between everything Twitter and everything SpaceX).
And even really dumb leaders (of today, not inbred monarchies) are generally above average intelligence.
A smart AI that also doesn't want power is, if I understand his fears right, something Yudkowsky would be 80% fine with; power-seeking is one of the reasons to expect a sufficiently smart AI that's been given a badly phrased goal to take over.
I don't think anyone has yet got a way to even score AI on power-seeking, let alone measure them, let alone engineer it, but hopefully something like that will come out of the super-alignment research position OpenAI also just announced.
I would be surprised if the average IQ of major leaders is less than 120, and anything over 130 is in the "we didn't get a big enough sample side to validate the test" region. I'm somewhere in the latter region, and power over others doesn't motivate me at all, if anything it seems like manipulation and that repulses me.
I didn't think of this previously, but I should've also mentioned there are biological fitness constraints that stop our heads getting bigger even if the IQ itself would be otherwise helpful, and our brains are unusually high power draws… but that's by biological standards, it's only 20 watts, which even personal computers can easily surpass.
They also tend to have some level of autonomy in not following the orders of idiots and psychopaths.
Except fully automated, cheaper, and with the capacity to fluently respond to each and every person who cares about the topic.
At GPT-4 prices, a billion words is only about 79800 USD.
> Life can’t be “won” by intelligence, that is only one factor, luck being a very significant other one.
It doesn't need to be the only factor, it just needs to be a factor. Luck in particular is the least helpful counterpoint, as it's not like only one person uses AI at any given moment.
> Especially biology is massively data-limited, which won’t be any more available to an AI than to the researchers at hand, let alone that teenager.
Indeed; I certainly hope this isn't as easy as copy-pasting bits of one of the many common cold virus strains with HIV.
But homebrew synbio and DNA alteration is already a thing.
We should absolutely care about nature and our impact on it but climate alarmism is not a way to go.
To oversimplify to the point of wrongness: Essentially how humans dominated our world, by being smarter.
> LLMs seem like a dead end
These would seem contradictory. If you really think that both are true and Altman knows it, then you're saying he's a hype man lying for regulatory capture. And to some extent he definitely is overblowing the danger for his own gain.
I really doubt they are a dead end though, we've barely started to explore what they can do. There's a lot more that can be extracted from existing datasets, multimodality, gains in GPU power to wait for, fine tunes for use cases that don't even have datasets yet, etc. Just the absolute mountain of things we've learned since LLama came out are enough to warrant base model retrains.
Only if you believe that LLM is a synonym for AI, which OpenAI doesn't.
The things Altman have said seem entirely compatible with "the danger to humanity is ahead of us, not here and now", although in part that's because of the effort put into making GPT-4 refuse to write propaganda for Al Quaida, as per the red team safety report they published at the same time as releasing the model.
Other people are very concerned with here-and-now harms from AI, but that's stuff like "AI perpetuates existing stereotypes" and "when the AI reaches a bad decision, who do you turn to to get it overturned?" and "can we, like, not put autonomous tasers onto the Boston Dynamics Spot dogs we're using as cheap police substitutes?"
Both can be true. It is extremely convenient to someone who already has an asset if the nature of that asset means they can make a convincing argument that they should be granted a monopoly.
> LLMs seem like a dead end.
In support of your argument, bear in mind that he's making his argument with knowledge of what un-nerfed LLMs at GPT-4 level are capable of.
While this is absolutely true, it's extremely unlikely that a de jure monopoly would end up at OpenAI's feet rather than any of the FAANGs'. Even in just the USA, and the rest of the world has very different attitudes to risks, freedoms, and data processing.
Not that this proves the opposite — there's enough recent examples of smart people doing dumb things, and even without that the possibility of money can inspire foolishness in most of us.
Possibly. The Microsoft tie-up complicates things a bit from that point of view. It wouldn't shock me if we were all using Azure GPT-5 in a few years' time.
It's been a month or two since I've tried but the results were depressingly slow and useless for more or less every task I tried.
Every time a model is claimed to be "90% of GPT-3" I get excited and every time it's very disappointing.
(On that note, after using GPT-4, GPT-3 now seems disappointing almost every time I interact with it.)
Text generation style instead of chat style is another avenue that makes the feedback time not so annoying for a developer.
at 100ms/token, it's faster than most people type, I think. That's what you might get on an old laptop with a 7B model.
There's a useful leaderboard here to help you pick a model: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb...
It really depends on your task, lots and lots of natural language type tasks give great results, the models seem to have extensive knowledge of many fields. So for some kinds of Q&A bot (technical or not), for copy blurbs, for fiction, game NPCs, etc, the models (especially 13B and up) can be breathtaking, even moreso considering they run on bottom-dollar consumer hardware (I paid $250 for the laptop I'm developing on).
There are of course some things that neither the local LLMs nor GPT4 can do, like create useful OpenSCAD models :)
Things keep getting better, newer quantization methods give you more smarts in the same amount of RAM at basically the same speed -- the models are getting better, there are more permissively licensed ones now.
Like, not vaguely hand wavey stuff, specifically, what model and what inference code?
I get nothing like that performance for the 7B models, forget the larger models, using llama.cpp on a pc without an nvidia GPU.
But for LLM I have no where idea where to start quickly. Finding a model on a leaderboard, download and setup then customising it and benchmarking is way too much time for me, I'll just pay for GPT4 if ever need to instead of chasing and troubleshooting to get some magical result. It'll be easier in the future I'm sure when an open model merges as the SD1.5 of LLM
I get around 5 tokens a second using the webui that comes with oogabooga using default settings. If I understand correctly, this does not get me 8k context length yet, because oogabooga doesn't have NTK-aware scaled RoPE implemented yet.
Using the same model with the newest kobold.cpp release should provide 8k context, but runs significantly slower.
Note that this model is great at creative writing, and sounding smart when talking about tech stuff, but it sucks horribly at stuff like logic puzzles or (re-)producing factually correct in-depth answers about any topic I'm an expert in. Still at least an order of magnitude below GPT4.
The model is also uncensored, which is amusing after using GPT4. It will happily elaborate on how to mix explosives and it has a dirty mouth.
Interestingly, the model speaks at least half a dozen languages much better than I do, and is proficient at translating between them (far worse than deepL, of course). Which is mindblowing for a 8GByte binary. It's actual black magic.
Could you elaorate on what you mean by that, like, are you telling it to write you a short story and it does a good job? My experiments with using these models for creative writing have not been particularly inspiring.
I tested it to create NPCs for fantasy role playing games. I think its the primary reason cobold.cpp exists (hence the name).
You give it a (ideally long, detailed) prompt describing the character traits of the NPCs you want, and maybe even add back and forth dialogue with other characters to the prompt.
And then you just talk to those characters in the scene you set.
There's also "story mode", where you and the model take turns writing a complete story, not only dialogue. So both of you can also provide exposition and events, and the model usually only creates ~10 sentences at a time.
There's communities online providing extremely complex starting prompts and objectives (escape prison, assassin someone at a party and get away, ect.) for the player, and for me, the antagonistic ones (the models has control over NPCs that don't like you) are surprisingly fun.
Note that one of the main drivers of having uncensored open source LLMs is people wanting to role-play erotica with the model. That's why the model that first had scaled RoPE for 8k context length is called "superhot" - and the reason it has 8K context is that people wanted to roleplay longer scenes.
I don't know anything so I left default settings.
I get about 450ms/t with airoboros-7b and 350ms/t with orca-mini-3b.
edit: with oobabooga webui
On that laptop CPU from 2017, using a copy of llama.cpp I compiled 2 days ago (just "make", no special options, no BLAS, etc):
./main -m models/WizardLM-7B-uncensored.ggmlv3.q4_0.bin -n 128 -s 99 -p "A short test for Hacker News:"
llama_print_timings: sample time = 19.12 ms / 36 runs ( 0.53 ms per token, 1882.65 tokens per second)
llama_print_timings: prompt eval time = 886.82 ms / 9 tokens ( 98.54 ms per token, 10.15 tokens per second)
llama_print_timings: eval time = 5507.31 ms / 35 runs ( 157.35 ms per token, 6.36 tokens per second)
./main -m models/WizardLM-7B-uncensored.ggmlv3.q4_0.bin -n 128 -s 99 -p "Sherlock Holmes favorite dinner was "
llama_print_timings: sample time = 54.37 ms / 102 runs ( 0.53 ms per token, 1875.93 tokens per second)
llama_print_timings: prompt eval time = 876.94 ms / 9 tokens ( 97.44 ms per token, 10.26 tokens per second)
llama_print_timings: eval time = 16057.95 ms / 101 runs ( 158.99 ms per token, 6.29 tokens per second)
the code I was running: https://github.com/ggerganov/llama.cpp
and the model: https://huggingface.co/TheBloke/WizardLM-7B-uncensored-GGML
There are other models that may perform better, I'm going to be doing a lot of screwing around with OpenLLaMA this weekend.
There's definitely some prompt magic openai does behind the scenes that helps beat the raw style local llms usually go for. With proper prompting you can get chatgpt like answers.
You can take a look at a list of evals here: https://llm-tracker.info/books/evals/page/list-of-evals - for general usage, I think home-rolled evals like llm-jeopardy [2] and local-llm-comparison [3] by hobbyists are more useful than most of the benchmark rankings.
That being said, personally I mostly use GPT-4 for code assistance to that's what I'm most interested in, and the latest code assistants are scoring quite well: https://github.com/abacaj/code-eval - a recent replit-3b fine tune the human-eval results for open models (as a point of reference, GPT-3.5 gets 60.4 on pass@1 and 68.9 on pass@10 [4]) - I've only just started playing around with it since replit model tooling is not as good as llamas (doc here: https://llm-tracker.info/books/howto-guides/page/replit-mode...).
I'm interested in potentially applying reflexion or some of the other techniques that have been tried to even further increase coding abilities. (InterCode in particular has caught my eye https://intercode-benchmark.github.io/)
[1] https://github.com/turboderp/exllama#results-so-far
[2] https://github.com/aigoopy/llm-jeopardy
[3] https://github.com/Troyanovsky/Local-LLM-comparison/tree/mai...
[4] https://github.com/nlpxucan/WizardLM/tree/main/WizardCoder
If people are mostly using their LLMs for specific tasks, then using cloud providers (Vast.ai and Runpod were cheapest last time I checked) can be cheaper than dedicated hardware, especially if your power costs are high. If you're needs are minimal, Google Colab offers a free tier with a GPU w/ 11GB of VRAM, so you can run 3B/7B quantized models easily.
There are reasons of course irrespective of cost to run your own model (offline access, fine-tuning/running task specific models, large context/other capabilities OpenAI doesn't provide (eg, you can run multi-modal open models now), privacy/PII, BCP/not being dependent on a single vendor, some commercial or other non-ToS allowed tasks, etc).
[1] https://gptforwork.com/tools/openai-chatgpt-api-pricing-calc...
Is exllama an alternative to llama.cpp?
As far as I can see llama.cpp with CUDA is still a bit slower than ExLLaMA but I never had the chance to do the comparison by myself, and maybe it will change soon as these projects are evolving very quickly. Also I am not exactly sure whether the quality of the output is the same with these 2 implementations.
There are a couple big difference as I see it. llama.cpp uses `ggml` encoding for their models. There were a few weeks where they kept making breaking revisions which was annoying, but it seems to have stabilized and now also supports more flexible quantization w/ k-quants. exllamma was built for 4-bit GPTQ quants (compatible w/ GPTQ-for-LLaMA, AutoGPTQ) exclusively. exllama still had an advantage w/ the best multi-GPU scaling out there, but as you say, the projects are evolving quickly, so it's hard to say. It has a smaller focus/community than llama.cpp, which also has its pros and cons.
It's good to have multiple viable options though, especially if you're trying to find something that works best w/ your environment/hardware and I'd recommend anyone to HEAD checkouts a try for both and see which one works best for them.
oobabooga recently did a direct perplexity comparison against various engines/quants: https://oobabooga.github.io/blog/posts/perplexities/
On wikitext, for llama-13b, the perplexity of a q4_K_M GGML on llama.cpp was within 0.3% of the perplexity of a 4-bit 128g desc_act GPTQ on ExLlama, so basically interchangeable.
There are some new quantization formats being proposed like AWQ, SpQR, SqueezeLLM that perform slightly better, but none have been implemented in any real systems yet (the paper for SqueezeLLM is the latest, and has comparison vs AWQ and SpQR if you want to read about it: https://arxiv.org/pdf/2306.07629.pdf)
Depending on your pre-prompt, your fine-tune (i.e. which model you downloaded), and your specific task, the results can be startlingly good, it's crazy that you can do this on a $250 laptop. I stay up nights working on it lately, it's so interesting.
More importantly, things change by the day. New models, new methods, new software, new interfaces... the possibilities are endless... unless we let OpenAI corrupt our government(s).
[1] GPT4 is 8 x 220B params = 1.7T params: https://news.ycombinator.com/item?id=36413296
I would guesstimate that the great majority of prompts are trash. People playing with a toy and amusing themselves. The platform sends those to the trash models.
For the other tiny percentage that produces a prompt the size of a paragraph, using the techniques published by OpenAI themselves, they likely get the higher tier models. This is also why I believe many are recently complaining about the quality of the outputs. When your chat history is filled with "have waifu pretend to be my girlfriend" then whatever memory the model is maintaining will be poisoned by the quality of your past prompts.
Garbage in, garbage out. I am certain that the #1 priority for OpenAI/Microsoft is lowering the cost of each prompt while satisfying the majority.
The majority is not in HN.
That's certainly true, but it's hard to deny the quality of gpt 4. If the issue is the training data, let's just use their training data, it's not like they had to close up shop because of using restricted data.
I think the issue is more on the financial side, it must have been extremely expensive to train gpt 4. Open source models don't have that kind of money right now.
I'll finance open source models once they are actually good, or show realistic promises of reaching that level of quality on consumer hardware. Until then, open source will open source.
I've never bought any kind of subscription or paid api costs to openai, but if gpt 4 finally reached the point where I feel like it's a lot better than just good enough, I'll happily pay for it (while still being on the lookout for open source models that fit my hardware).
a 1billion parameter model beats 175billion parameter GPT3.5
OpenAI wants us all to drink the kool-aid.
Most of peoples' problem is watching the AI type, it's not instant, but then not all (or even most) applications need to be instant. You can also avoid that by having it return everything at once instead of streaming style.
Local absolutely can scale. All kinds of fun things can be done on a machine with 16GB of RAM, or 8GB if you work harder.
Funny, for me it is the complete opposite. I created an interface in Matrix that does just that: return everything at once. But the lag annoys me more than the slow typing in the regular chat interface. The slow typing helps me keep me focused on the conversation. Without it, my mind starts wandering while it waits.
But to address both: is it very relevant what LLM you use right now? Local or hosted, openAI or other?
It seems like the interface has converged around chat-based prompts.
New ideas for tuning or improving the efficiency of foundational models are published almost every week.
If one wants to build a product on top of of generative AI, why not simply start with what’s free or works with one’s dev environment?
Presumably, the interaction with or API to text-based gen AI will be very similar no matter what engine is best for your use case at any given time.
This would imply these backends will be swappable, the way web services are that copy AWS S3 APIs.
So, to return to the point, can’t people just build their product with openAI or other and plan to move away based on the cost and fit for their circumstances?
Couldn’t someone say prototype the entire product on some lower-quality LLM and occasionally pass requests to GPT4 to validate behavior?
It seems far-fetched to believe this tech can be constrained by legislation.
OpenAI can lobby all they want, it won’t necessarily buy them anything. Look what happened with FTX.
Since LLMs can be run locally and the engines be black boxes to the user, how could a legislative act really prevent them from being everywhere—-especially given the public utility.
This, infact, might be a better way to do inference anyway: https://twitter.com/Francis_YAO_/status/1675967988925710338
> So, to return to the point, can’t people just build their product with openAI or other and plan to move away based on the cost and fit for their circumstances?
Depends. There are signs that folks are buying into GPT-specific APIs (like function calls) which may not be as easy to migrate away from.
It can be done -- it is the basis for assisted generation and related work. It does require full access to the model, to be time and money-efficient. See https://huggingface.co/blog/assisted-generation
Disclaimer: I'm the author of the blog post linked above.
> I can currently run some scary smart and fast LLMs on a 5 year old laptop with no GPU.
And, something useful or just playing? I played with local models, and will keep playing, training, experimenting. It's interesting, but not a solution, not yet.
Their moat is that they had access to data sources which have since been clamped down on, eg reddit and twitter apis.
https://github.com/imartinez/privateGPT is great if you want do it with code.
I sympathize with the idea of wanting to run a local LLM, but IMO, this would require building a desktop with a GPU and plenty of horsepower + silent cooling and put it somewhere in a closet in my apartment. Running LLMs on my laptop is (to me) clearly a waste of my time and its battery/cooling.
Is there a local solution that is at least as intelligent as GPT 3.5 in that regard that I can run in a container?
This will end my usage of openai for most things. I doubt my $5-$10 API payments per month will matter. This just lights more of a fire under me to get the 65B llama models working locally.
*"They have just locked down access to a model which they basically realized was way more valuable than even they thought - and they are in the process of locking in all controls around exploiting the model for great justice?"*
What are funs things we can with it until it sunsets on January 4, 2024?
When using davinci the onus is on you to construct prompts (memories) which is fun and powerful.
====
97% of API usage might be because of ChatGPT's general appeal to the world. But I think they will be losing a part of the hacker/builder ethos if they drop things like davinci-003, which might suck for them in the long run. Consumers over developers.
I used the "Complete" UI (from the Playground) for a bit before the "Chat" interface was available; I don't really think there's anything you couldn't do in the "Complete" UI that you couldn't also do in the "Chat" UI.
One supposes openai has a 6 month notice period vs a 12 month period for azure. This might generally effect one’s appetite in choosing which endpoint to use for any model.
I guess they'll give you early access to it.
But with davinci at the same price point as GPT-4 I'm hoping the latter is enough of a step up in its variety of vocabulary and nudgeable sophistication of language to be a drop in replacement.
Though in general I think there's an under appreciation for just how much is being lost in the trend towards instruct models, and hope there will be smart actors in the market who use a pre-optimization step for instruct prompts that formats it for untuned models. I'd imagine that parameter size to parameter size that approach will look much more advanced to end users just by not lobotomizing the underlying model.
Trying the same prompts that gave nerfed "I am just an AI I can't speculate about the future" bs on completion API gave somewhat better results, but most of the time they were flagged as breaking the guidelines which is a TOS breach if done enough times.
This can be solved other than open models. The same thing happened with stable diffusion. Good thing it's open so you can still use the pre-nerfed 1.6 models.
I know it might be edgy or unpopular but I don't think one entity should decide how we can use this powerful tool. No matter its implications and consequences.
FOS for the win.
Lots of my favorite and frankly the best litterature in the world have elements that are obscene, grotesque, bizarre, gritty, erotic, frightening, alternative, provocative - but that's just too much for chat-gpt, instead it has this - in my eyes - way more horrifying smiling-borg-like nature with only two allowed emotions: "ultra happiness", and "ultra obedience to the status quo".
But mostly it has to do with the fact that LLM do what they've seen. And if they've been fine-tuned to not respond to some classes of things they'll misapply that to lots of other things. That's why most people go for the "uncensored" fine tuning datasets for the llamas even for completely sfw use cases.
More info:
https://platform.openai.com/docs/model-index-for-researchers
Wait, they're not letting you use your own fine-tuned models anymore? So anybody who paid for a fine-tuned model is just forced to repay the training tokens to fine-tune on top of the new censored models? Maybe I'm misunderstanding it.
It's not cost-effective, but it may be part of a valid business plan.
It's somewhat rare to have a task and good enough dataset that you can finetune something else to be close enough in quality to GPT-4 for your task.
That said I think efficiencies will dramatically improve over the next few years and over investing now probably captures very little value beyond building internal competency - which doesn’t grow with anything but time and practice. The longer you depend on OAI, the longer you will depend on OAI past your point of profound regret.
Absolutely. A large consortium of companies could each contribute 0.2-2% of the total cost and fund something much larger than OpenAI.
Just sell access at a higher price than you get it
Either directly, on on average based on your user stories
Do you have any recommendations for good open models that businesses could use today?
From what I've seen in the space, I suspect businesses are building fine tuned models against closed models because those are the only viable models to build a business model on top of. The quality of open models isn't competitive.
On fine-tuning:
> We will be providing support to users who previously fine-tuned models to make this transition as smooth as possible.
On embeddings:
> We will cover the financial cost of users re-embedding content with these new models.
Based on the language it sounds like they'll do the same when that launches.
I assume if you reach out they throw some credits at you
The extra trust gained by seeing another customer treated that way easily pays for a few credits for a small number of users.
For example, AWS was one of the first cloud providers. Now there are alternatives, but I still pick AWS because I trust them not to break my dependencies way more than, say, Google
I have much better performance by "prompt tuning" - when question arises, I search 30 similiar examples in training set, and send it to non-tuned GPT and ask the question and get much better performance than fine-tuned older models.
It's a deal-breaker for many.
gpt-3.5-turbo is the model behind ChatGPT. It's chat-fine-tuned which makes it very hard to use for use-cases where you really just want it to obey/complete without any "chatty" verbiage.
The "davinci-003" model was the last instruction tuned model, but is 10x more expensive than gpt-3.5-turbo, so it makes economical sense to hack gpt-3.5-turbo to your use case even if it is hugely wasteful from a tokens point of view.
The difference between them is that the chat models are much more... chatty - they're trained to act like they're in a conversation with you. The chat models generally say things "Sure, I can do that for you!", and "No problem! Here is". The conversation style is generally more inconsistent in it's style. It can be difficult to make it only return the result you want, and occasionally it'll keep talking anyway. It'll also talk in first person more, and a few things like that.
So if you're using it as an API for things like summarization, extracting the subject of a sentence, code editing, etc, then the chat model can be super annoying to work with.
Have you tried getting your formatted JSON out via the new Functions API? I does cure a lot of the deficiencies in 3.5-turbo.
Think of the properties you want in the JSON object, then send those to ChatGPT as required parameters for a function (even if that function doesn't exist).
# Definition of our local function(s).
# This is effectively telling ChatGPT what we're going to use its JSON output for.
# Send this alongside the "model" and "messages" properties in the API request.
functions = [
{
"name": "write_post",
"description": "Shows the title and summary of some text.",
"parameters": {
"type": "object",
"properties": {
"title": {
"type": "string",
"description": "Title of the text output."
},
"summary": {
"type": "string",
"description": "Summary of the text output."
}
}
}
}
]
If you're interested, I wrote a blog post with more detail: https://puppycoding.com/2023/07/07/json-object-from-chatgpt-...
OK maybe I'm stupid but I am a paying OpenAI API customer and I don't have it yet. I see:
gpt-3.5-turbo-16k
gpt-3.5-turbo
gpt-3.5-turbo-16k-0613
gpt-3.5-turbo-0613
gpt-3.5-turbo-0301
Edit: Probably my problem is that I upgraded to paid API account within the last month, so I'm not technically a "paying API customer" yet according to the accounting definitions.
Same for me. I signed up only a few days ago and was excited to switch to "gpt-4" but I haven't paid the first bill (save the $5 capture) so I probably have to continue to wait for this.
I made a very simple command-line tool that calls the API. You run something like:
> ask "What's the opposite of false?"
https://www.pastery.net/ccvjrh/
It also does streaming, so it live-prints the response as it comes.
Let us know what you find.
It's around 1k lines of python, audit the code if you care and rotate your keys.
Or don't use it.
OK sorry I didn't know them.
I mean I usually use software that came with my computer or ones that I apt-install from the official ubuntu distribution. I know it's not perfect security but at least it's more than a hacker news link to a github pip. If I had to use other ones then it's usually from people I know.
1. chat subscription only
2. i have paid for api calls but don't have a subscription
and only #2 currently has gpt4 available in the playground
The mass adoption of the ChatGPT APIs compared to the old Completion APIs proves my initial blog post on the ChatGPT API correct: developers will immediately switch for a massive price reduction if quality is the same (or better!): https://news.ycombinator.com/item?id=35110998
However if you try the gpt-4 API, it's possible it will be much better.
So still waiting to be on the same 32 pages...
Medical writing is the opposite, with unstated premises, semi-random associations, and rarely a meaningful sentence.
That makes sense. The labor impact research suggests that law will be a domain hit almost as hard as education by language models. Almost nothing happens in court that hasn't occured hundreds of thousands of times before. A model with GPT-4 power specifically trained for legal matters and fine tuned by jurisdiction could replace everyone in a courtroom. Well there's still the bailiff, I think that's about 18 months behind.
I built notionsmith.ai on 3.5: for some time I experimented with GPT 4 but the result was significantly worse to use because of how slow it became, going from ~15 seconds per generated output to a minute plus.
And you could work around that with things like streaming output for some use cases, but that doesn't work for chain of thought. GPT 4 can do some tasks without chain of thought that 3.5 required it for, but there are still many times where it improves the result from 4 dramatically.
For example, I leverage chain of thought in replies to the user when they're in a chat and that results in a much better user experience: It's very difficult to run into the default 'As a large language model' disclaimer regardless of how deeply you probe a generated experience when using it. GPT 4 requires the same chain of thought process to avoid that, but ends up needing several seconds per response, as opposed to 3.5 which is near-instant.
-
I suspect a lot of people are building things on 4 but would get better quality of output if they used more aspects of chain of thought and either settled for a slower output or moved to 3.5 (or a mix of 3.5 and 4)
For instance, with MATH dataset, my own n=500 evaluation showed no difference between GPT-3.5 (w/ and w/o CoT) and GPT-4. I was pretty surprised by that.
My project takes transcripts from YouTube, which don't have punctuation, splits them up into chunks, and passes each chunk to GPT-4 telling it to add punctuation with paragraphs. Part of the instructions includes telling the model that, if the final sentence of the chunk appears incomplete, to just try to complete it. Anyway, GPT-3.5-turbo works okay for several chunks but almost invariably hits a case where it either writes a bunch of nonsense or spells out the individual letters of words. I'm sure that there's a programmatic way I can work around this issue, but GPT-4 performs the same job flawlessly.
{
"isIncomplete": [true if the chunk seems incomplete]
"completion": [the additional text to add to the end, or undefined otherwise]
"finalOutputWithCompletion": [punctuated text with completion if isIncomplete==true]
}
And what open LLM you used it with / how successful you've found it?
poe law
I mostly use it for generating tests, making documentation, refactoring, code snippets, etc. I use it daily for work along with copilot/x.
In my experience GPT3.5turbo is... rather dumb in comparison. It makes a comment explaining what a method is going to do and what arguments it will have - then misses arguments altogether. It feels like it has poor memory (and we're talking relatively short code snippets, nothing remotely near it's context length).
And I don't mean small mistakes - I mean it will say it will do something with several steps, then just miss entire steps.
GPT3.5turbo is reliably unreliable for me, requiring large changes and constant "rerolls".
GPT3.5turbo also has difficulty following the "style/template" from both the prompt and it's own response. It'll be consistent then just - change. An example being how it uses bullet points in documentation.
Codex is generally better - but noticeably worse then GPT4 - it's decent as a "smart autocomplete" though. Not crazy useful for documentation.
Meanwhile GPT4 generally nails the results, occasionally needing a few tweaks, generally only with long/complex code/prompts.
tl;dr - In my experience for code GPT3.5turbo isn't even worth the time it takes to get a good result/fix the result. Codex can do some decent things. I just use GPT4 for anything more then autocomplete - it's so much more consistent.
Where 3.5 excels is with programmatic access. You can ask it for 2x as much text between setup so the end result is well formed and still get a reply that's cheaper and faster than 4 (for example, ask 3.5 for a response, then ask it to format that response)
From using voice in the ChatGPT iOS app, I surmise that Whisper is very good at working out what you've actually said.
But it's really annoying to have to say my whole bit before getting any feedback about what it's gonna think I said. Even if it's getting it right at an impressive rate.
Given this is how OpenAI themselves use it (say your whole thing before getting feedback), I don't know that the API is set up to be able to mitigate that at all, but it would be really nice to have something closer to the responsiveness of on-device dictation with the quality of Whisper.
Using built-in text input showed quite good results since ChatGPT is still understanding the ask quite well
Pretty crazy stuff — perfectly understandable translations.
Superficially, I think this will work very well, but slightly worse than whisper (with the advantage ofc being that its better at real-time transcription).
[0]https://machinelearning.apple.com/research/attention-free-tr...
Wait for what!? Christmas? When we can open our presents and have a GPT 4 inside?
It's like they took a leaf from Google's "how to guarantee the failure of a new product" marketing. That is: restrict access, ensuring that to word-of-mouth marketing can't possibly work because none of your friends are allowed to try the product.
The announcement here is "general availability" of the GPT-4 model...
...but not the 32K context model. Not the multi-modal version with image input. No fine-tuning. Only one model (chat).
As of today, I can only access GPT 3.5 via Azure Open AI service and the Open AI API account that I have.
What's the point of all these arbitrary restrictions on who can access what model!?
I can use GPT 4 via Chat, but not an API. I can use an enhanced version of Dall-E via Bing Image Creator, but not the OpenAI API. Some vendors that have been blessed by the Great and Benevolent Sam Altman have access to GPT-4 32K, the rest of us don't.
Sell the product, not the access to it.
Don't be like the Soviet Union, where you had to "know someone" to get access.
That makes no sense.
What makes much more sense -- especially if you listen to his interviews -- is that Sam Altman doesn't think you can be trusted with the power of GPT-4 via an API unless it has first been aligned to death.
And having such a big anchor tenant, its reasonable that you would prioritize them if GPUs are in short supply.
Yeah, but Microsoft already gets 75% of the profits OpenAI makes, it's not the same price for them as the rest of us.
It is SO amazing that we have such a driving force (LLMs/consumer-AI) for this (instead of stupid cryptocurrencies mining or high-performance gaming). This should drive innovation pretty strongly and I am sure the next "leap" in this regard (processing hardware) will put technology in a completely different level.
So they open things carefully, pull back when necessary like when they limited use of the public GPT-4 version of ChatGPT. That doesn’t seem too unreasonable. And yes sure, some amount of it might be attempts to manufacture scarcity to increase the hype. It’s an old tactic and hardly comparable to Soviet Russia.
If there's not enough GPUs at a certain price point, raise prices. Then lower prices later when GPUs become available.
They did it with GPT 3.5, so why not GPT 4?
I don’t see why such a thing should be embarrassing. Or, at least no more so than being acute or obtuse. Just as long as nothing is askew.
"Lick my boots, in person, and you can be one of the privileged few" is very much the behaviour of a Communist dictatorship, not a capitalist corporation.
I can spin up an Azure VM right now in almost any country I choose... except China. That's the only one where I have to beg the government for permission.
I know exactly how this works.
When someone has power, they will use it. In small, petty ways, or big “do me favours for access” ways.
It's both, and more besides. Veblen goods are absolutely a thing in capitalism.
Not that "giving people access to GPT 4 APIs because they attended a conference" should be controversial enough to even get worked up about, let alone to compare to a dictatorship, but Google did much the same at developer conferences: I/O 2012 swag list included a Galaxy Nexus, a Nexus 7, a Nexus Q, and a Chromebox.
If we guesstimate that every 100 customers needs 1 NVIDIA GPU (completely random guess), then that means OpenAI needs to buy more GPUs for every 100 new customers using GPT-4. The problem is there's a GPU shortage so it's hard to add more GPUs by just throwing money at the problem.
https://www.fierceelectronics.com/electronics/ask-nvidia-ceo...
Infrastructure.
> It's like they took a leaf from Google's "how to guarantee the failure of a new product" marketing.
Yeah, an infamous guaranteed failure: GPT-4. (canned laughter)
I've had completions with it that had character and creativity that I have not been able to recreate with anything else.
Brilliant and hilarious things that are a permanent part of my family's cherished canon.
https://raw.githubusercontent.com/thomasdavis/omega/master/s...
Had it hooked up to speech so you could just talk at it and it would talk back at you.
Gave incredible answers that ChatGPT just doesn't do at all.
I've been working on LLMs for creative tasks and believe a mix of chain of thought and injecting stochasticity (like instructing the LLM to use certain random letters pulled from an RNG in a certain way at certain points) can go a long way in terms of getting closer to human-like creativity
It relies on WebSockets, Js, and a reasonably stable connection to run since it's built on Blazor
I am especially interested in gpt-3.5-turbo-instruct, as I think that the hype surrounding ChatGPT and "conversational LLMs" has sucked a lot of air out of what is possible with general instruct models. Being able to fine tune it will be phenomenal as well.
I do not really understand the efforts that went on behind the scenes to train GPT models on factual data. Did humans have to hand approve/decline responses to increase its score?
"America is 49 states" - decline
"America is 50 states" - approve
Is this how it worked at a simple overview? Do we know if they are working on adding the rest of 2021, then 2022, and eventually 2023? I know it can crawl the web with the Bing addon but, it's not the same.
I asked it about Maya Kowalski the other day. Sure it can condense a blog post or two, but it's not the same as having the intricacies as if it actually was trained/knew about the topic.
OpenAI will never fulfill the entire market, and their moat is in danger with every other company that has LLM cash flow.
They want to become the AWS of AI, but it's becoming clear they'll lose generative multimedia. They may see the LLM space become a race to the bottom as well.
There's still no news about the multi-modal gpt-4. I guess the image input is just too expensive to run or it's actually not as great as they hyped it.
https://help.openai.com/en/articles/7102672-how-can-i-access...
The decision of burying these extra information in a support article, not cool!
We already know they have a SOTA model that can turn images into latent space vectors without being some insane resource hog - in fact, they give it away to competitors like Stability. [0]
My guess is a limited set of people are using the GPT-4 with CLIP hybrid, but those use-cases are mostly trying to decipher pictures of text (which it would be very bad at), so they're working on that (or other use-case problems).
There was no question something happened in January with ChatGPT, weirdly would refuse to answer questions that were harmless but difficult(Give me a daily schedule of a stoic hedonist)
Every once in a while, I see redditors complain of it being nerfed.
Sometimes I go back to gpt3.5 and am mind boggled how much worse it is.
Makes me wonder if they keep increasing the version number while dumbing down the previous model.
With an API, being unreliable would be a deal-breaker. Looking forward to people fine-tuning LLMs with GPT4 API. I'd love it for medical purposes, I'm so worried of a future where the US medical cartels ban ChatGPT for medical purposes. At least with local models, we don't have to worry about regression.
The current versions is close to the original 3.5 version, while 3.5 has become horribly bad, such a scam to not disclose what's going on, especially for a paid service.
The fact that rumors about GPT-5 were quickly suppressed and the models were dumbed down even more cannot be entirely explained by excessive demand. I think it's more likely that GPT-3.5 and GPT-4 demonstrated unexpected capabilities in the hands of the public leading to a pull back. Moreover, Sam Altman's behaviors changed dramatically between the initial release and a few weeks afterward -- the extreme optimism of a CEO followed by a more subdued, even cowed, demeanor despite strong enthusiasm from end-users.
OpenAI cannot do anything without Microsoft's data center resources, and Microsoft is a critical defense contractor.
Anyway, personally, I'm with the crowd that thinks we're about to see a Cambrian explosion of domain-specific expert AIs. I suspect that OpenAI/Microsoft/Gov is still trying to figure out how much to nerf the capability of GPT-3.5 to tutor smaller models (see "Textbooks are all you need") and that's why the API is trash.
the one from a MS researchers that has been recommended to all of us probably. Good video btw.
My guess is that they began to restrict ChatGPT because they can't sell that. They probably want to sell you CodeGPT or other products in the future so why would they give that away for free? ChatGPT is just a teaser.
This is my experience also. I have not formally benchmarked the different releases, but specifically for Python coding ChatGPT 4 got considerably worse with the latest updates.
Is there any actual evidences other than some user subjective experiences?
One would pay because what they get out of chatGPT provides value, of course. Keep in mind that the users of these 2 products can be (and in fact are) different — chatGPT is a lot friendlier (from a UX perspective) than using the API playground (or using the API itself).
They are killing text-davinci-003 btw.
Noobie follow up question: Should we put any trust into “Sparks of intelligence” I thought it was regarded as a Microsoft marketing piece, not a serious paper.
Also notable that the team behind that paper wasn't involved in designing/building the model, but they did get access to prerelease versions.
This is the double edge sword of being so ridiculously closed.
You can apparently have it be nice or smart, but not both.
Even on this forum I've seen AI enthusiasts claiming AI will be the best psychologist, best school teacher, etc.
https://news.ycombinator.com/item?id=35402777
https://www.businessinsider.com/widow-accuses-ai-chatbot-rea...
buffing is the opposite, when an item gets better.
?
https://chat.openai.com/share/04c1dbc0-4890-447f-b5a5-7b1bc5...
But unfortunately, you have to read a ways into the doc and understand a lot of details about the benchmark. Here's a direct link and excerpt of the relevant portion:
https://aider.chat/docs/benchmarks.html#the-0613-models-seem...
The benchmark results have me fairly convinced that the new gpt-3.5-turbo-0613 and gpt-3.5-16k-0613 models are a bit worse at code editing than the older gpt-3.5-turbo-0301 model.
This is visible in the “first attempt” portion of each result, before GPT gets a second chance to edit the code. Look at the horizontal white line in the middle of the first three blue bars. Performance with the whole edit format was 46% for the February model and only 39% for the June models.
But also note how much the solid green diff bars degrade between the February and June GPT-3.5 models. They drop from 30% down to about 19%.
I saw other signs of this degraded performance in earlier versions of the benchmark as well.
Has anyone fared better? I might be doing something wrong but I can't see what that could possibly be.
We use it to generate automatic insights from survey data at a weekly cadence for Zigpoll (https://www.zigpoll.com). This makes getting an instant response unnecessary but still provides a lot of value to our customers.
> We recognize this is a significant change for developers using those older models. Winding down these models is not a decision we are making lightly. We will cover the financial cost of users re-embedding content with these new models. We will be in touch with impacted users over the coming days.
You don't need very large models to generate usable embeddings.
The API will make you wait up to 10 minutes, and then time out. What's worse, it will time out between their edge servers (cloudflare) and their internal servers, and the way OpenAI implemented their billing you will get a 4xx/5xx response code, but you will still get billed for the request and whatever the servers generated and you didn't get. That's borderline fraudulent.
Meanwhile, their status page will happily show all green, so don't believe that. It seems to be manually updated and does not reflect the truth.
Could it be that it works better in another region? Could it be just my region that is affected? Perhaps — but I won't know, because support is non-existent and hidden behind a moat. You need to jump through hoops and talk to bots, and then you eventually get a bot reply. That you can't respond to.
My support requests about being charged for data I didn't have a chance to get have been unanswered for more than 5 weeks now.
There is no way to contact OpenAI, no way to report problems, the API sometimes kind-of works, but mostly doesn't, and if you comment in the developer forums, you'll mostly get replies from apologists that explain that OpenAI is "growing quickly". I'd say you either provide a production paid API or you don't. At the moment, this looks very much like amateur hour, and charging for requests that were never fulfilled seems like a fraud to me.
So, consider carefully whether you want to build against all that.
https://learn.microsoft.com/en-us/azure/cognitive-services/o...
Borderline!? They're regularly charging customers for products they know weren't delivered. That sounds like straight-up fraud to me, no borderline about it.
I meant it's not normal to start selling a feature in 2016 and delivering it in beta seven years later.
If you want better latency and sane billing you need to go through Azure OpenAI Services.
OpenAI also offers decreased latency under the Enterprise Agreement.
However, I'm virtually certain somethings wrong on your end, I've never seen a wait even close to that unless it was completely down. Also the thing about "small prompts"...it sounds to me like you're overflowing context, they're returning an error, and somethings retrying.
Very sorry to hear about these issues, particularly the timeouts. Latency is top of mind for us and something we are continuing to push on. Does streaming work for your use case?
https://github.com/openai/openai-cookbook/blob/main/examples...
We definitely want to investigate these and the billing issues further. Would you consider emailing me your org ID and any request IDs (if you have them) at atty@openai.com?
Thank you for using the API, and really appreciate the honest feedback.
OpenAI has an amazing core product, but in the span of six months:
* Went from an amazing and inspiring open company that even put "Open" in their name to a fully locked up commercial beast.
* Non-existent customers support and all kinds of borderline illegal billing practice. You guys are definitely aware that when there's a network error on the API or ChatGPT, the user still gets charged. And there's a lot of these errors. I get roughly one per hour or two.
* Frustratingly loose interpretation of EU data protection rules. For example, the setting to say "don't use my personal chat data" is connected to the setting to save conversations. So you can't disable it without losing all your chat history.
* Clearly nerfing the ChatGPT v4 products, at least according to hundreds or even thousands of commenters here and on reddit, while denying to have made any changes.
* Use of cheap human labor in developing countries through shady anonymous companies (look up the company Sama who pay Kenyan workers about $1.5 an hour).
* Not to mention the huge questions around the secret training dataset and whether large portions of it consist of illegally obtained private data (see the recent class court case in California)
What is wrong about injecting millions into developing nations?
The rest I agree with, although I don't think it was ever really 'open' so its not getting shitty, it always was. Thankfully, "there is no moat" and other LLMs will be open, just a few months behind OpenAI
Please don't try to reframe this to make exploitation a positive thing. See my other comment here.
Is the newer 64B models up there? 1 year, 2 years? Can't wait until i get back the crazy quality of the orig model.
We need something open source fast. Thanks open-ai for giving us a glimpse of the crazy possibilities, too crazy for the public i guess.
If you pay a developing country developed country wages what you'll get is 1. inflation and 2. the government mad at you because all their essential workers/doctors/government officials are quitting to work for you.
Note that I feel I have quite deep understanding of this issue, and feel strongly about it, because I live and work in a developing country and I see this happening a lot. Westerners come over here and treat local workers like shit, pay them peanuts for 80 hour weeks while making loads of money themselves and then justify it because "it's the local norm". It's sickening, frankly. We westerners doing business in developing countries are in a position of privilege and should be leading by example, not jumping on the first excuse to dump a hundred years worth of the fight for workers rights.
Just curious.
If you had a morally consistent way to live that life, you'd have my respect. But no, you had to deflect the topic to a phrase I wrote and make presumptions about what I really meant.
FYI, I'm morally at ease with myself, I don't need to justify anything to anyone.
"Complaining on HN will get you access. You have know people or "complain in the right forums."
THERE SHOULD BE NO LOGIC.
No qualifying rules. No access checks. No gates. No hoops.
Sam Altman has gone on a worldwide interview tour claiming he wants to "democratise access to AI", meanwhile OpenAI is the least open company I have ever dealt with, or even heard of.
Oracle is more "democratic" and open, for crying out loud.
Thanks for offering help, I will contact you directly.
I tried again just now and I got "Oops! We ran into an issue while authenticating you." but it works on chromium.
I’m pretty sure they tuned the Cloudflare WAF rules on GPT 3 and forgot to increase the request size limits when they added the bigger models with longer contest windows.
as far as I know OpenAI only has one region, that is out in Texas.
even more hilariously, as far as I can tell, Azure OpenAI -also- only has one region.. cant imagine why
https://learn.microsoft.com/en-us/azure/cognitive-services/o...
It's definitely limited, but there's currently more than one region available.
(I happen to be working at the moment on a location-related fix to our most popular Azure OpenAI sample, https://github.com/Azure-Samples/azure-search-openai-demo )
I too had an issue and put in a request. Took about 2.5 months to get a response, so 5 weeks you are almost half way there.
https://www.reddit.com/r/ChatGPT/comments/14ruui2/comment/jq...
I've heard LLMs described as "setting money on fire" from people that work in the actually-running-these-things-in-prod industry. Ballpark numbers of $10-20/query in hardware costs. Right now Microsoft (through its OpenAI investment) and Google are subsidizing these costs, and I've heard it's costing Microsoft literally billions a year. But both companies are clearly betting on hardware or software breakthroughs to bring the cost down. If it doesn't come down there's a good chance that it'll remain more economical to pay someone in the Philippines or India to write all the stuff you would have ChatGPT write.
If you assume an A100 lives at the cutting edge for 2 years, that's about a million minutes, or $0.01 per minute of amortized HW cost.
In the crazy scenarios, I've heard 10 A100s per query, so assuming that takes a minute, maybe $0.1 per query.
Add an order of magnitude on top of that for labor/networking/CPU/memory/power/utilization/general datacenter stuff, you get to maybe $1/query.
So probably not $10, but maybe if you amortize training, low to mid single digits dollars per query?
It's very possible the web UI is using a nerfed version of the model evident by its different versioning, but not the API which has more distinct versioning.
It's fraudulent, full stop. Maybe they're able to weasel out of it with credit card companies because you're buying "credits."
I suspect it was done this way out of pure incompetence; the OpenAI team handling the customer-facing infrastructure have a pretty poor history. Far as I know you still can't do something simple like change your email address.
In our own tracking, the P99 isn't exactly great, but this is groundbreaking tech we're dealing with here, and our dissatisfaction with the high end of latency is well worth the value we get in our product: https://twitter.com/_cartermp/status/1674092825053655040/
For example exponential back off is the first step, then adding retrying on timeouts (we use streaming and if there are 30 seconds in between getting data back we retry the whole request - rare but happens), then fixing anything else that pops up
It is possible to have a stable production app on top of it
Just fix your code and stop expecting OpenAI to hold your hand
:-)
My code does retry and the entire application is written to detect and work around breakage. But eventually I do need to get enough content from OpenAI API to be able to make progress, and I am not.
At the moment, for example, all requests just time out after 12 minutes (on my side). No amount of "fixing my code" will help, and I don't want OpenAI to hold my hand, I just want it to a) return some data at least sometimes, b) not charge me for data not delivered.
Let's look at my billing page: over the last hour it shows 8 requests. A total of 52584 tokens. Not a single response made it back to me.
We spend over $20k/mo with them and don't have any issues like this.
12 minute timeouts make no sense because first of all why are you even waiting 12 minutes?
Get off HN and go fix your shit
I'm not as excited about this as I am about most new tech. I'm sure there will be cool uses eventually but right now it seems like the primary use is to cheat at exams and write bad articles, and to ask the kind of questions Google could have answered 5 years ago.
I do think it's cool that it can debug and review code though.
Recalibrate where you think AI provides "real" value
Unfortunately it's still too expensive and the completion speed is not as high as GPT-3.5 but I hope both problems will improve over time.
I had the silly idea of trying to change the signin method of my account. Which isn't possible. So I figured to just delete the account and create a new one with the correct signin method.
Turns out they don't delete anything. Both the email address and phone number are held hostage. As you try to create a new account, it will point out that those are in use. I can easily change my email address but not my phone number, I only have one.
I've contacted support 4 times, but it's just bot replies. There's entire Reddit threads full of us perma-banned potential customers. Money in hand, but permanently locked out.
What a ridiculous company.
But yeah whining on HN would be more productive
Things that GPT-4 would easily, and correctly, reason through in April/May it just doesn't do any longer.
I checked many of the links you posted and, although I am a fluid programmer in several languages, I lack the specific python background that many of these links seem to state as a requirement.
Would any kind soul put me in the direction of an easy solution to run LLMs, potentially in AWS, that answers questions about your own docs? (I use Confluence but I can happily export pages.)
Thanks a lot in advance!!!
All of them are open source minus the gpt part so you can get a feel for how it works.
It's also a shame that the API is so cut down, and removes all the good options that text completions had.
I only hope their competition are better.
nooooo they are deprecating the remnants of the base models
The chat models constantly argue with you on certain tasks and are highly opinionated. A completion API was a lot more flexible and "vanilla" about a wide variety of tasks, you could start a thought, or a task, and truly have it complete it.
The chat API doesn't complete, it responds (I mean of course internally it completes, but completes a response, rather than a continuation).
I find this a big step back, I hope the competition steps in to fill the gaps OpenAI keeps opening.
Comparable, perhaps.
OpenAI even has a whole repository specifically for this - GPT-eval. No one uses it.
I'm not saying the theories are wrong. Maybe there is something behind the hunches that so many people seem to have about degradation. But there isn't _any_ proof. None. Whatsoever. And people are taking _internet comments_ as that proof instead? I mean, sure, it's easy to be cynical about companies in this day and age; which is why I would ultimately believe someone if they provided actual evidence. But, again - not a single ounce of proof has been provided in any one of these threads.
Furthermore, the lack of rigor being applied even with the various anecdotes is appalling.
Which version are you talking about? GPT-4 or GPT-3? Are you using the API or the web interface? Are you aware that output is non-deterministic? Are you aware that your own psychological biases will skew your opinions on the matter? One or more of these questions tend to go unanswered.
Just please, show me some robust proof. If you can't because you didn't think to; you _surely_ must realize that many people are building entire businesses on top of this tech and at least _one_ of them is running these types of evaluations. Furthermore, the model is state-of-the-art for research now as well and if you can _prove_ that there is degradation in the model that they are lying about (in a research paper), you will get citations. And yet, there is nothing. Zilch. Nada.
For anyone who wants to quickly try this out in VSCode for your custom prompts - https://marketplace.visualstudio.com/items?itemName=ppipada....
So need to pay to fine tune again?
I made up an example here to illustrate, but it's just very annoying because sometimes it puts at the beginning, slowing down my interaction, and it now refuses to obey my prompts to leave caveats out.
https://chat.openai.com/share/1f39af02-331d-4901-970f-2f4b0e...
looking at your sample and using character count as a rough proxy for tokens, (465/(1581-465))*100 means they added ~42% token count cost to your response explicitly adding caveats which you dont want. fun!
500 {'error': {'message': 'Request failed due to server shutdown', 'type': 'server_error', 'param': None, 'code': None}} {'Date': 'Thu, 06 Jul 2023 20:48:07 GMT', 'Content-Type': 'application/json', 'Content-Length': '141', 'Connection': 'keep-alive', 'access-control-allow-origin': '*', 'openai-model': 'gpt-4-0613', 'openai-organization'Sadly, that niche is still better served by openAI.
So I could build long prompt scripts and run them against the api?
Now, I am not superstitious, but...
GPT-4 is on a completely different level of consistency and actually listening to your system prompt than chagpt-3.5. It trails off much more rarely.
If only it wasn't so slow/expensive... (it really starts to hurt with large token counts).
What is the difference? Replacing evil with another evil.
This is just behemoths exchanging hands.