What about "illegal" things? You could live in a country that punishes gay people and want to ask questions about being gay. Maybe you live under a tyrannical government and want to ask questions about the government or politics without raising any red flags on your name or IP. Maybe you overstayed your visa in the US and are worried about ICE putting out honeypot sites to find your location when you are trying to research options on how to get a work visa or how to escape the US without getting arrested at the border and sent to prison.

Is it being overly paranoid? Maybe in most cases yes. But we know the capability and desire exists for certain governments and agencies to locate and punish people and we know it does happen, even if it isn't super common.

"requires anonymity" is already invalid assumption / invalid question territory.

You do not have to justify, and no one has the right to even ask for a justification, let alone judge the answer, of why ones thoughts should be allowed to be private by default.

Businesses submitting data that may be of interest to their competitors is one concern. Assuming that a business trusts how OpenAI handles the data, more parties will have access to it. That increases the probability of it being leaked. (The trust doesn't have to be intrinsic. There will be contracts in place, so those businesses can be held accountable via the courts.) Something that they should not be doing, in my opinion, but probably are: submitting confidential data of other parties. That adds another layer that may face legal repercussions if data is leaked.

On the level of individuals, "embarassment" may be more than emotional discomfort. It may lead to discrimination. Given some of the things happening in the US at present, it may lead to prosecution (or worse). It is possible that none of the repercussions will be legal, but they will happen nevertheless.

That’s the wrong way to think about it. Everything should be anonymous and private by default against up to nation state actors, so that dissidents and journalists can hide easily among the masses without giving up their anonymity

Why don't you want me to install a camera in your house watching you sleep and get dressed? Just because it would be embarrassing to be watched all the time?

Like, theres so many good reasons to want privacy by default it isnt funny.

But just consider that the dumbest country on the planet, elected its greatest moron ever devised, and he is already screening people crossing the border based on social media history. Its not a terribly far leap from screening social media history, to screening literally any other data the government can get its hands on.

There are a whole host of things people could (and probably are) using LLMs for. Think about what the tool fundamentally is. An enabler. A helper. A teaching assistant. A more ergonomic search tool.

Now think about all of the dumb, violent, ideological, illegal things that folks do.