Hacker News Reader

Comment by airgapstopgap | Hacker News Reader

Not really. They have a way of squaring this circle, by changing their inference code. Speculative sampling [1] would still make their first claim a lie – sure, there'd still be the original GPT-4 model, plus a smaller draft worker. But early exit decoding [2] allows you to get almost as good results for much cheaper from exactly the same checkpoint. We know that this line of research for large-scale inference is going strong [3] so it stands to reason that OpenAI, with their wealth of talent focused on GPT-4 throughput&inference [4], large contexts and aggressive pricing policy, would also develop something like that. And of course it's "smarter" that way – in a very deceptive sense of the word.

1. https://arxiv.org/abs/2302.01318

2. https://arxiv.org/abs/2207.07061

3. https://arxiv.org/abs/2307.02628

4. https://openai.com/contributions/gpt-4

View on HN

I don't get why you're jumping to cloak and daggers style operations: OpenAI would not kneecap their commercial offering by randomly changing how it works.

At the end of the day 99% of the confusion comes from people using the web interface, which undoubtedly does change much more often than the API versions they share.

The web app they host isn't a simple API wrapper, it does summarization, has some sort of system prompt, and calls the moderation API. That's undoubtedly being updated all the time.

> OpenAI would not kneecap their commercial offering by randomly changing how it works.

> As of July 3, 2023, we’ve disabled the Browse with Bing beta feature out of an abundance of caution while we fix this in order to do right by content owners. We are working to bring the beta back as quickly as possible, and appreciate your understanding!

https://help.openai.com/en/articles/8077698-how-do-i-use-cha...

Thank you for confirming my point?

> At the end of the day 99% of the confusion comes from people using the web interface, which undoubtedly does change much more often than the API versions they share.

The API does not offer any browsing features, that's the web app.

No, it makes sense to secure engagement with the most expensive implementation and then cut costs, this kind of stuff is pervasive in the industry. Besides, we have Brockman on record saying that they do "a lot of quantization"[1][2] so it's not paranoia to suspect other optimization schemes when there's a clear performance drop, which they have also denied a few times.

1. https://chat.openai.com/share/44a0c5b6-c629-470a-992f-8cdbbe...

2. https://www.youtube.com/watch?v=_hpuPi7YZX8

Paranoia would be charitable: it's FUD.

If you intentionally smear the line between their web app which is chock full of optimizations to even let it function as it does (the web app's max conversation length exceeds the context window) and the API which is versioned and iterated on in the open... it's either a lack of understanding or FUD.

> OpenAI would not kneecap their commercial offering by randomly changing how it works.

Have you seen the 25 messages/3 hours limitation for GPT-4? Why do you think they did that? Of course they would make more money scaling up the volume, but how to do that when compute is so limited? Of course, by using some kind of approximation - quantised model or speculative sampling come to mind. It's hard to pinpoint model regressions, but scaling up volume is great, one more incentive to do it.

You realize that's a limitation in the web application right?

The web app is a consumer app (B2C) the api is commercial (B2B). They tinker with the B2C app because it's already a lossy approximation of using the model between the summarization and system prompt.

They cannot mess with the commercial offering willy-nilly: People are building businesses predicated on it behaving a certain way. That's why there are dated version that you can pin to with the API. The web app changes whenever they feel like it.

You keep repeating that. You don't even know if the people commenting to you use the API or the "web app". I use the API and I noticed the same stuff others have.

> Have you seen the 25 messages/3 hours limitation for GPT-4?

If can't tell if that's about the API or the web app, I don't think you're familiar enough with the subject to speak on it.

I don't need to be familiar with it to see that you're not being genuine in your interpretation of peoples' comments and in the way you're responding to people in this thread. Case in point: your reply to me.

What was I supposed to reply to your factless baseless anecdote with?

Millions of dollars in spend from products predicated on the API not randomly changing?

Arguably, both ChatGPT and API are consumer apps. That includes researchers. Pay as you go, no strings attached, "oh yeah no no, we're not changing anything, follow our CEO on Twitter if you want to know more". That kind of stuff.

The actual B2B offering is handled by Microsoft, via Azure OpenAI. Same models, but deployed on Azure - meaning they come with SLA and all the right protocol and compliance stuff, so that your people can negotiate with their people - and if you're willing to spend enough, you'll get the models for yourself. Not the weights, of course: just no training on your inputs, not even retaining inputs for 30 days "because ${legal reasons}" - instead, you can pick and chose, fine-tune and deploy OpenAI models on your own tenant, and basically manage everything except the weights themselves.

Maybe arguable if you don't know what consumer apps are? Also sounds like you haven't actually used Azure OpenAI:

- It has the same 30 day retention for legal reasons unless you manually request (just like OpenAI)

- You can't fine tune any models that you can't fine tune on OpenAI, and in fact default access is a subset of what OpenAI offers.

- "and if you're willing to spend enough, you'll get the models for yourself" is a bit of nonsense, Azure OpenAI forces everyone to make a "tenant", that's just for VPC stuff to work. Outside of that it's bog standard fine tuning and at most "on your data" which is a wrapper for chunking + vector embeddings

- Azure OpenAI has a narrower built in filter that you can't modify without again, a separate request.

Azure OpenAI overall is mostly for companies that need to signal to other companies that they're using Azure: it's no more commercial than the OpenAI offering.

> Also sounds like you haven't actually used Azure OpenAI

On the contrary, I am using Azure OpenAI daily at work, and I'm explicitly not allowed to use "regular" OpenAI offerings.

> It has the same 30 day retention for legal reasons unless you manually request (just like OpenAI)

It doesn't, at least not for us.

> Azure OpenAI has a narrower built in filter that you can't modify without again, a separate request.

I'm not sure if it's narrower, but it is there and I have a strong suspicion that MS is just trying to extract additional rent from companies that really want to turn the filter off.

> Azure OpenAI overall is mostly for companies that need to signal to other companies that they're using Azure: it's no more commercial than the OpenAI offering.

No. Azure OpenAI is for companies that don't play fast and loose with data - their own data, and their customer data. Of course, most companies don't give a damn, but for big enough companies, or those operating in certain industries, there are actual, severe legal consequences for mishandling the data, and such companies don't have the option to just not give a fuck and dance with OpenAI - they need to sign an actual contract with a serious entity that understands regulatory compliance, and how corporations tick. Microsoft is such entity. OpenAI isn't.

> It doesn't, at least not for us.

So then you filled out the request because the default is exactly the same as OpenAI: retained unless you manually apply for an exception.

https://customervoice.microsoft.com/Pages/ResponsePage.aspx?...

> I'm not sure if it's narrower, but it is there and I have a strong suspicion that MS is just trying to extract additional rent from companies that really want to turn the filter off.

You don't need to question if it's narrower, OpenAI used to surface it as an API separate from the moderation API and it's much stricter by design.

> No. Azure OpenAI is for companies that don't play fast and loose with data - their own data, and their customer data...

I don't know if you actually believe this or you're just not aware, but the companies that don't play fast and loose aren't using OpenAI period: Azure flavored or otherwise.

OpenAI has SOC2, GDPR and CCPA compliance. They comply with HIPPA and offer BAs. They sign DPAs on a case-by-case basis same as Azure.

You're pretty much proving the value of Azure in your comment: it's a veneer of familiarity that coaxes people who are convinced the new kid on the block must be untrustworthy.

If OpenAI can't promise something Azure can't either: They're entirely dependent on OpenAI for this. Every idiosyncrasy behind Azure OpenAI maps back 1:1 to OpenAI.

> I don't know if you actually believe this or you're just not aware, but the companies that don't play fast and loose aren't using OpenAI period: Azure flavored or otherwise.

They do, and that's the biggest value proposition of Azure OpenAI right now: strong contractual guarantees, from a reputable partner (that's easy to hit with lawsuits should they go rogue :)).

The current situation is that it's pretty unwise for any company to ignore GPT models. OpenAI itself is a wildcard, but getting the same from Microsoft isn't "playing fast and loose with data" any more than using Windows and Office 365 across the organization is. Most large corporations and governments have been building their office work and communication around those tools for decades now, so - questions of antitrust aside - all the kinks have been worked out. I don't think you appreciate how big a difference this makes.

I mean, it's either that or all the company communication I got on this was bullshit.

> OpenAI has SOC2, GDPR and CCPA compliance. They comply with HIPPA and offer BAs.

That's the first I hear of it, but since I never dealt with OpenAI itself on that level, I accept this was my ignorance speaking; thanks for clarifying.

> You're pretty much proving the value of Azure in your comment: it's a veneer of familiarity that coaxes people who are convinced the new kid on the block must be untrustworthy.

I think you're underestimating the importance of this. What you call "veneer of familiarity" translates to billions of dollars of differences in terms of security risk.

As mentioned before, MS has been in this space for a while, and has decades of trust and experience built with governments and corporations and other big organizations. Microsoft is a known, trusted quantity. That alone is worth a lot.

But then, there are also technical aspects too - like how deploying to a tenant on Azure integrates properly with all the other services you use to run half the company. In practical terms, this means all use is monitored and auditable by in-house teams, and all the in-house policies are being enforced. OpenAI can't begin to offer this level of integration - they have neither technical nor legal resources for that.

> If OpenAI can't promise something Azure can't either: They're entirely dependent on OpenAI for this. Every idiosyncrasy behind Azure OpenAI maps back 1:1 to OpenAI.

None of that matters here. The models are what they are - peculiar large matrix multiplication as a service. By themselves, they're pretty much pure functions. The part that matters is operations - both technical and legal aspects - and this is where Microsoft and OpenAI are independent and have different offerings.

Also, looking at the way money flows, I think it's OpenAI that's dependent on Microsoft right now, not the other way around. They kinda pretend to be just friends with benefits, but it's obvious who the dependent party is.

I'm advising on calls with firms that have existed since the 1800s: their clients don't even want LLMs involved in output, regardless of who's hosting what.

Companies that don't play fast and loose are not using LLMs yet. They use "old school" ML at most with much narrower scope because at this point it's simply less of a liability.

You seem to think I'm underestimating what Azure's name adds to OpenAI: I fully understand how bureaucratic organizations work off vibes under the guise of name recognition and my point is I simply have no respect for it.

If you genuinely care about customer data, then the value of being able to sue MS instead of OpenAI is moot. You also probably aren't going to use a service that shouts from the roof tops about not using your data then quietly keeps it for 30 days unless you manually opt-out. You probably don't use some model with unsolved copyright/PII questions. And a million other unknowns

> The part that matters is operations - both technical and legal aspects - and this is where Microsoft and OpenAI are independent and have different offerings.

You might want to check OpenAI's subprocessor list if you think that they're not the same technically...

https://platform.openai.com/subprocessors/openai-subprocesso...

And Azure's subprocessor list is a superset of that list, not a subset.

Reply on news.ycombinator.com