XMPP, a comeback story | Hacker News Reader

XMPP, a comeback story

takebackourtech.org

380 pointsupofadown302 comments

We tried our hardest back in the early 2000's to make this stick. I cofounded a startup building servers, clients, and SDK's. Maybe the world is finally ready for it now. The issue we ran into was customers/consumers just didn't care about federated systems. And various implementations had all sorts of quirks making interoperability between domains and clients difficult, at best. There are just SO MANY THINGS you have to do to make good messenger software that lining up all the pieces with open standards proved impossible in the ~8 years I was working on it. Again, maybe it's better now, but I'm skeptical, probably jaded. :)

It's not about the protocol. Most people don't care about XMPP, RSS, or any of those things.

They have things they want done, like chatting with a friend. They want a solution that delights them in doing that. What delights people has generally changed (like more people care about others not listening in these days).

Whose the user and what's their experience? This is where XMPP always failed me. The clients had a poor experience.

If someone wants that to change now... focus on a great experience for the people you want to be users.

Yes, this is generally the issue with open networks. Very, very hard to get complex and disparate client/servers to play nicely with each other. Perhaps with the capital that is being put behind web3 someone will figure this out.

Yeah but then we gotta pay for em!

I’m expecting “web3” to go the other way: all of those speculators are looking to get their money back at a healthy return and that tends to favor building up barriers to switching. I’d expect claims that being open makes it too hard to prevent spam or slows innovation by adding features which, ooops, they totally slack on making interoperable.

How can it go the other way, when it is specifically defined otherwise?

> Web3 revolves around the idea of a decentralized Internet. Proponents often contrast this to Web 2.0, where large amounts of the web's data and content are centralized in a fairly small group of companies (often referred to as Big Tech). [0]

[0] https://en.m.wikipedia.org/wiki/Web3

Yes, I've seen the marketing materials, too. The thing to think about is what you're actually using — the very next paragraph from your quote would be a good place to start:

> Specific visions for Web3 differ, but all are heavily based in blockchain technologies, such as various cryptocurrencies and non-fungible tokens (NFTs). Some visions are based around the concepts of decentralized autonomous organizations (DAOs), which seek to enable many people to have equal ownership and governance in an organization. Decentralized finance (DeFi) is another key concept, in which users exchange currency without bank or government involvement. Self-sovereign identity allows users to identify themselves without relying on a centralized authentication system like OAuth.

Note how the common thread is that these are all working around the scalability and reliability issues inherent to the popular blockchains. It's perhaps survivable for an identity system to be that slow when making changes if you don't do that very often, although it's definitely a risk for the overall network, but there's no way you're building an XMPP competitor on top of it even before you think about handling audio/video content, and privacy would of course preclude that as well.

That means that you're really building a system which looks a lot like the competitors except using something like a wallet ID to login. Beyond being rather less disruptive than the Ethereum salespeople like to tell you, that's going to look a lot like XMPP — a traditional networked service running a protocol of some level of standardization, only with a different login option. There are some natural trends which are hard to avoid: users will gravitate to the more reliable service offered by the larger companies who can invest more time in optimization, various parties will add extensions which either aren't well documented for competitors or favor their infrastructure choices, and abuse will be a problem which leads to some combination of arbitrary moats being created or users leaving to services where they don't get spam.

That last point is key: imagine if your web3 messaging startup starts to see enough usage that spammers descend en masse? Your real users will leave unless you do something about it. You can add entry fees, which will tank your user growth (which is probably already dangerously limited if you require a blockchain ID), or restrict federation to peers whom you trust to control abuse by their users.

And through all of this, big tech will fight you tooth and nail with the billions we gave them the last decades. And when web3 has burned down to a pile of rubble, they'll welcome the users back to be locked in their warm and safe walled gardens. (Terms and conditions and no-privacy-policy apply)

I don't really see this taking off and succeeding in the long run sadly. Don't get me wrong, I'm rooting for it. But I'm pretty sure the game is already rigged.

I don't want to sound like an apologist for the large tech companies — they certainly can afford to pay for their own — but I think it's a mistake just to view this as a rigged game.

Decentralization sounds like a great idea but I've been watching that generally fail since the 90s and I think there are some challenges beyond aggressive acts by big players. Part of this comes down to how the terms are used to cover different levels: most of the advantages of decentralization at the infrastructure level are less pronounced in an era where a single entity can easily run code around the world and there are some drawbacks related to things like caching or security. “Distributed, run by one party” is probably going to outperform “Distributed, run by many parties” for most users and the things which are going to make them switch to something else would be high-level things like decisions about the product direction.

The higher-level organization trade-offs are less clear cut, I think. There's an advantage in having more than one party making decisions but it also harms your overall reaction time somewhat substantially. Other than the web and email there aren't all of that many success stories and I wouldn't feel too secure about email long-term. That suggests to me that we might be prone to give the drawbacks insufficient weight when we're thinking about decentralization, especially when it comes to the challenges of controlling abuse. I ran my own Jabber servers for a while and it worked well at work (centralized) but basically all of my personal messages came either from people on Google Talk or spammers. It wasn't just that Google put more money into running servers or building apps but that there wasn't a great path forward for a lot of features which federated services offered.

Crypto's target audience is cyberpunks, how do you expect those people to accept a walled garden solution? And spammers are already on the big tech platforms and nobody leaves those.

A tiny fraction of users of a system with minimal real-world impact do not have much leverage. As we’ve seen, most crypto users are comfortable using exchanges and there’s no reason to think that will change. The majority of users are speculators who are thinking about returns and don’t seem bothered by control accumulating in a few companies or mining pools.

Re: spam, think a bit longer about that — how much spam do people get in email versus Facebook or Twitter? Those platforms have a lot of users who value their control of user signups, monitoring, and ability to arbitrarily remove spam messages after delivery.

Email is one of the most distributed systems in the world but a rather large fraction of users self-centralized on Gmail. That wasn’t just spam control but I think it’d be really useful to think about why some new system wouldn’t follow the same arc: if your argument is “cyberpunks wouldn’t accept it”, remember that we had that ethos arguably even stronger back then and it didn’t have the predicted effect.

Before exchanges can do something shady, crypto must become really ubiquitous. I think analogy for exchanges is banks. Some people use big banks even though those provide poor service, in my experience switching banks has zero friction, but I evade biggest banks.

On Android I use Conversations[1] (from F-Droid), on Linux I use gajim[2]. For me it works, for others, I have no clue. Gajim is available on macOS and Windows, too. I dunno if by default it has the OMEMO[3], and PGP plugin though. I think they are a must-have. Make sure that "XEP-0384: OMEMO Encryption" is supported by the server you pick.

[1] https://f-droid.org/en/packages/eu.siacs.conversations/

[2] https://gajim.org/download/

[3] OMEMO is an XMPP Extension Protocol (XEP) for secure multi-client end-to-end encryption based on Axolotl and PEP. Might want to go through https://dev.gajim.org/gajim/gajim-plugins/-/wikis/OmemoGajim....

The sheer amount of permissions required for Conversations is mine boggling. What's Bluetooth and WiFi permissions for?

You could check out the source code[1].

Check out these files:

- src/main/java/eu/siacs/conversations/ui/RtpSessionActivity.java

- src/main/java/eu/siacs/conversations/services/AppRTCAudioManager.java

- src/main/java/eu/siacs/conversations/services/AppRTCBluetoothManager.java

For the most (or all?) part it seems like it is related to Bluetooth audio. There are stuff about establishing connection to headsets and whatnot.

[1] https://f-droid.org/repo/eu.siacs.conversations_42023_src.ta...

I miss the "golden age" of chat where I could talk to anyone (google/fb/msn messenger/aol?) with my own jabber server. Pidgin was a great client at the time.

I've been looking for a modern self-hosted chat solution. I want it to be XMPP so badly (running XMPP with something like prosody is so convenient). The clients are really letting me down though. Desktop clients are old-fashioned and awful (I could get behind that, but I don't think my kids could). The mobile situation is even worse.

The new-fangled chat solutions (Matrix, Zulip, etc.) are interesting, but the operational requirements are insane (in the case of Matrix, I'm also very nervous about reliability).

I can't say I'm super hopeful, but I'm rooting for you, XMPP.

Too bad AIM went away. AOL should've open sourced it. In the early 2000's, several companies I worked for used AIM like we use Slack today.

When I did tech support in the early 2000's, we used the corporate version of ICQ for internal communications. It actually served quite well.

This golden aged failed when it missed the boat to go mobile. Ridiculous to think that none of the messengers from this time could envision to sync with the mobile phone's address book. I don't think we would have WhatsApp or Signal if any of the desktop chats had allowed that.

The Palm Pre actually had a really nice mobile messenger. It synced with Address Book, supported SMS, Jabber, Facebook Messenger (back when it was jabber), aim, and others. It would seamlessly switch protocols based on how the contact messaged you. Like most things with the Pre, it was way ahead of its time.

Arguably, they were killed by the mobile platform-owners.

Sending push notifications, which is broadly necessary for achieving both performance and battery efficiency, has been strictly centralized and locked down (understandably, I'd hate to try to block abuse if I were running it). Federated systems with multiple apps were pushed out because there was no sane way to achieve that with the push architecture, and that's still the case. You see the same thing with nearly any third-party app, they're nearly always partly-crippled experiences because the first-party will not send push notifications to anything but their own app.

You absolutely need push on iOS and modern Android. S60 and Android before doze didn't because you could just have a connection in the background; afaik, BlackBerry didn't restrict background data either. But it's not like there was a lot of good mobile XMPP clients that died out when push became mandatory.

If there was, it wouldn't take much to make an XMPP extension or three for the client to provide a push url to the server. Client devs would (probably) need to standup a push proxy, so xmpp server pushes to client server pushes to client, because I don't think many platforms give out push urls that can be securely passed on to 3rd parties. There's some expense there, but not a whole lot, IMHO.

XEP-0357: Push Notifications <https://xmpp.org/extensions/xep-0357.html> has been a thing for while. A lot of servers support it too: <https://compliance.conversations.im/test/xep0357/>.

While a lamentable situation, this has been addressed to a considerable degree and it's not all that bad. For example Matrix has its Push Gateway API (https://matrix.org/docs/spec/push_gateway/r0.1.1#id4) which allows any client to register any kind of push service. This means that any given iOS app, for example, can maintain its own backend for feeding events into APNs like Apple requires. I'm not as familiar with it but I believe XEP-0357 does something similar.

Nah, they were killed by XMPP's braindead protocol design that made it useless for multiple clients. Apparently the idea that people might want to read messages on both their phone and computer, or on two computers, was completely unimaginable for XMPP, despite every other messaging system doing the sensible thing. Eventually there was an XEP ("Message Carbons") to give the basic sensible functionality, and eventually eventually most servers more or less supported it and it more or less worked most of the time, but that killed XMPP on mobile long before push notification platforms were even a thing.

That indeed was a problem that hurt it badly, but XMPP is also old. If I remember correctly, supporting multiple simultaneous clients at all was more than other contemporary IM systems did.

From my memory at the time AIM and Skype would both Just Work. I think maybe MSN would forcibly log you out if you logged in on a second computer? (Still better than continuing to route messages to your other computer, which is what XMPP does by default).

Exactly. XMPP's a little younger than AOL, but the idea of having more than one device connected to the Internet and using them simultaneously, was sheer opulence at one point. Combined with XMPP being an open protocol and difficulty incrementing protocol versions, lead to the shortcomings described.

This is certainly not a problem with XMPP now. I run my own XMPP server and use XMPP to share things between my different devices. I have multiple accounts logged in from multiple machines all day every day.

What I really like about XMPP is that I can do precisely this. Also, setting up and running prosody is dirt simple if you have any UNIX experience. IMO it's even simpler than running your own web server.

I feel like I must be misunderstanding what you're saying here, at least in the last sentence. I think of "first party" as "platform owner" and "third party" as, uh, not the platform owner, but third-party real-time messaging apps obviously can send push notifications on iOS and Android.

While it's been an awfully long time now, I seem to remember that services like AIM didn't successfully move to mobile because they required stateful connections similar to telnet/SSH, and that was something that just wasn't (and largely still isn't) possible under iOS.

> services like AIM didn't successfully move to mobile

AIM is a lot like AIM, and it worked fine on BlackBerries.

It's quite possible the Blackberry AIM client supported stateful connections. IIRC, it worked well on the Danger Hiptop / T-Mobile Sidekick, too.

Blackberry had their own software (BES/BIS) running as an intermediary between email and their devices. Were they using the same method for chat apps too?

Since we're talking about a federated system (xmpp as a whole), I mean "first party" as in "an xmpp ecosystem runner" like google used to be. And "third party" as "any other implementation".

Google had an xmpp system (google talk) for which they have a first-party app which they send push notifications to. No other xmpp app can take advantage of that for google accounts - they all need to run, essentially, an xmpp-bouncer + their own push notification infrastructure and client implementation.

In ye olden days, a google-free version for google-free xmpp accounts could not be built in many cases, as the mobile device's APIs wouldn't let you hold open connections in the background like that - that was unique to the platform-owner (Android / iOS). Now you basically can, but each implementation increases battery use, so there's a fair bit of pressure to use the built-in one instead... which you can't do, Google won't send your app any push notifications that are intended for their apps.

---

The same thing applies to email apps, for example. IDLE just plain doesn't work well enough on phones, so email apps build their own email backends so they can use google's push notification infrastructure to push notifications for google-owned email account events. Google is the first party here, owning the email account and infrastructure, but not sending notifications to other email applications.

And also for basically any third-party app for basically any service. E.g. Talon (a third party twitter app) has to resort to polling Twitter's API to do anything. But those at least aren't federated systems, so though I'm still a bit sad about that, it's much more understandable that it happens.

> battery efficiency

yep, that's the downside of xmpp, isn't it? How could that be mitigated?

It's mitigated in several ways. But first of all consider that XMPP is hardly inefficient by today's standards, when mobile devices can stream video continuously for hours on a single charge.

Still, battery is important to consider, so what have we done in XMPP to improve energy efficiency in the past 15 years?

One of the first things we did was define a way for clients to reconnect a session that suffered a broken connection - the classic "I just went through a tunnel" scenario. With this optimization we gained two things: a way to quickly reconnect without resynchronizing everything, and a guarantee that no traffic was lost in the process due to the network interruption.

Secondly, we've implemented traffic optimization in servers. Clients will tell the server when they are idling in the background, and the server will stop sending any traffic that isn't time-sensitive. As soon as the user focuses the app, it tells the server and the connection switches back to "real-time" delivery. This behaviour helps keep network activity low when the client is in the background, and prevents continuously waking up the device's radio (such wake-ups can be quite power hungry over the course of a day).

Finally, modern mobile XMPP clients support push notifications, which is the Apple/Google sanctioned way of solving this problem. It means that you can safely close the XMPP client entirely and still get notified when you receive messages.

There's probably more besides these things that I've forgotten but I think these changes are the highlights!

thank you, that's quite something.

> ...has been strictly centralized and locked down (understandably, I'd hate to try to block abuse if I were running it).

Not understandably at all, the reason for the lockdown is not for your benefit even if it is portrayed as such. Push notification can be run just like SMTP and is subject to the same type of abuse with the big difference that it actually easier to keep the spammers out since you know up front which senders are allowed to send notifications. It would be fully possible to self-host a push notification daemon which connects to those services which are meant to send you notifications, to which your devices connect for notifications. If you don't want to self-host it would be possible to choose a notification service just like you can choose an email provider. Things could be just as hands-off as they are now without being forced to open an Apple/Google/Microsoft account and hand over your communications to them.

Don't be fooled by these 'for your protection' lockdowns, they are for the benefit of the one doing the lockdown, not for your own.

Those kinds of systems tend to have been created and popularized before any single large actor appeared and started using it. Push notifications were single-actor from the very beginning. Decentralized setups in that situation are much more expensive to build and run - with the exception of "from the goodness of their hearts" projects, a business has little reason to make the sole implementation vastly more complex and open to abuse.

And tbh I'm not sure SMTP serves as all that positive of an example. It works, sure, but the reputation system and extreme complexity that has sprung up around it makes it difficult for newcomers to join - I can't just host one from my house, most email hosts will reject my emails. It works in spite of all this because it has too many large actors already using it for any new ones to stand any chance of rejecting (or improving) it.

It also failed because of this

> Signup for an XMPP service anonymously

Unmoderated networks quickly degenerate spam pits from Hell. It worked in 90s when everyone was still polite in Internet.

People weren't polite on the internet in the 90s! There was plenty of trolls even back then. Plenty of spam too.

I'm ashamed to admit this but I even wrote software what ran on Windows 95 and would spam chat rooms with thousands of phrases generated on the fly from a dictionary. The intention was it would run over a RAT (also of my creation) and get other people booted off the chat room. ie the spam would be posted by other people in chat rooms against their will and channel admins would then kick them. I wrote this because at college we "only" had dual ISDN lines and kids in chat rooms would slow down my warez downloads.

I know, I know, the justification is worse than the tool itself. But we were young, I've grown up since and these days the lessons I've learned from my college exploits are put to good use :)

> This golden aged failed when it missed the boat to go mobile.

There were plenty of mobile applications that supported multiple chat protocols. There might even still be. The issue is proprietary protocols would often change breaking support -- I suspect intentionally breaking support for 3rd party client.

Linux / Unix nerds running FOSS would be tolerant of that but I suspect your average smart phone user who might have spent 99c on the client would be less understanding.

dino.im is the new up and coming desktop client. I have a few friends that are using Gajim on Windows.

kaidan.im is also a new KDE client.

dino.im is very promising, It's a little buggy right now and missing features, but I love the approach.

The clients are really letting me down though

Yeah! I ran my own XMPP server on my home network and used it to accept status messages from other boxes on my network. Worked really well. The clients were generally horrible though, especially the IOS ones. I've now switched to MQTT.

1000% agree, so much so that I started such a project and wrote a blog post about this very problem[1] and launched what has become a full-time project to solve it (Snikket).

These days I believe very much in identifying and serving a specific set of users. XMPP is just a protocol, it has lots of potential applications, but it's nothing unless you channel that into a tangible usable product for real people.

This epiphany came to me after more than 10 years of working on XMPP as a server developer and spending time promoting open communication protocols to people. It was soul-crushing that at the end of a day I would still see my own family communicating with each other via WhatsApp.

Six months after the initial Snikket prototypes, and after a little user testing, I migrated my family over to XMPP by sending a simple invitation link (the start of the Snikket onboarding journey). They're now using it daily and totally happy with it. I'm beginning to hear similar stories from others who have repeated this with their own families/groups and their own Snikket setups.

All it took was to look at the issue through the eyes of "normal people" for a moment, and it changed my approach (and success rate) significantly.

[1]: https://snikket.org/blog/products-vs-protocols/

This is very cool. I moved our family to signal, but this looks better. I'll be trying it out for sure.

This is great. I think it would benefit from having a large catch-all public server ehich people could use if they don't want to set up their own, shile retaining the choice to do so. That would improve the UX substantially, I think, although the cost may be too high.

Indeed, https://quicksy.im/ follows this approach, it's as trivial as it can be.

My focus with Snikket is on making it as easy as possible for people to run their own server (hosted or self-hosted), and to avoid a repeat of the Google situation where 99% of the network was controlled by a single (volatile when it comes to messaging apps) entity.

Snikket may not be for everyone, but that's kind of the point of XMPP. Snikket users can text and call Quicksy users with E2EE, and vice-versa, despite using two different systems.

I badly want to move off of discord, but I need something that follows the "voice channels" concept from discord (which comes from teamspeak/ventrilo/mumble).

What do you think the best path is to that feature? Element seemed to have audio calls already, but not voice channels.

Also basic features like push to talk and noise thresholds are key.

Who's going to proxy that voice? That's not a free service. Discord burns investor's money, but most services can't afford that.

Afaik Matrix uses whatever homeserver you are using to proxy the voice. Video calls as well

The fun part about WhatsApp is that it uses XMPP internally. It's just not federated, and probably modified to the point that it's not compatible with anything else. But I do remember seeing JIDs in logs while reverse engineering it several years ago.

Thank you for making Snikket! I run my own XMPP server for friends and family and train people to use Adium. I just downloaded Snikket and it looks so much nicer.

looks great, does it support OTR?

It doesn't, OTR was deprecated by OMEMO a while back.

Pidgin doesn't have a non-"highly experimental" OMEMO plugin yet and Gajim has major issues e.g. changes to privacy preferences silently failing. A lot of people I know are still on OTR.

Pidgin is quite simply a terrible XMPP client with none of the features of modern XMPP. I haven't had any issues with OMEMO with my contacts on Gajim.

Well I can believe that Gajim+OMEMO works, but I didn't trust it to setup an account after noticing the undesirable defaults in preferences->advanced->privacy then realising that unticking them had no effect (they're reticked when I open the preferences again).

Do you know any alternative Linux clients?

I use dino.im and kaidan.im also exists (though they implemented a newer version of the spec which isn't backwards compatible: https://invent.kde.org/network/kaidan/-/issues/254 ).

Thanks, I did have a look at dino-im recently and was surprised it has no options at all - just a 'set up account' wizard that doesn't offer proxy settings before making connections. It wouldn't let me add a fake@127.0.0.1 account because it couldn't connect...

I'll have a dig in the conf files, perhaps I can configure an account manually.

Yes, someone should fix those iOS clients... I have Siskin, Monal and ChatSecure all running in parallel on my iPhone and none works as it is supposed to do.

Siskin seems to be the best of these three, but is neither as reliable nor as feature-complete as Conversations on Android or Gajim on the Desktop.

I think Android is the only platform that currently has a good user experience with Conversations or Quicksy for the less technical people.

https://snikket.org/blog/snikket-ios-public-release/ - needs a prosody module for reliability though.

I realise it’s superficial but I took one look at the screenshots of the Snikket iOS client and thought “the person who built this has no concept of spacing, padding or text sizes”. It’s really difficult to want to use a client that looks ugly and this is where so many iOS XMPP clients seem to fall down.

Yeah, it seems like a small thing, but this immediately put me off the first time I tried out Snikket. All the spacing and sizing just feels janky and out-of-place compared to other iOS apps.

If aesthetics are the main issue, then that's validation that we've come a long way :)

The iOS app hasn't yet been through a thorough visual review, though it is planned in the coming months (a number of fixes are already in beta builds though).

Thanks for the feedback!

Plenty of people willing and able to build that experience, but not for free.

If we got a nickel from everyone that complained about XMPP client fragmentation, we would have funding to pay enough man-hours to get high-quality clients for every major platform and even the minor ones.

I think the writing was on the wall for XMPP once people started chatting on Facebook and Facebook got past their brief XMPP compatible phase. Then Google also deprecated it.

It's a nice protocol, but if you're missing all the conversations it doesn't help. When it was popular, it was the system that let you connect Yahoo to Google to MSN to Facebook to Zephyr to... it's network effects, people have to be on it for anyone to want to use it.

Google was federating from the start. Then Facebook started winning customers over, and they weren't. So Google also closed. When that happened, the writing was on the wall for XMPP at Google. If you are not federating, there's no reason to follow the spec.

It's a variation of the prisoner's dilemma. If one party shares and the other doesn't, the market becomes lopsided. It's why every big permissively licensed non-copyleft product so far hasn't managed to build an ecosystem outside of a few select distributors.

With protocol specifications the problem is much harder. Market dominance can be maintained by slight incompatibilities.

Back in the day I love the trillio or something like that client that connected a bunch of different networks together in a single app for the very reason you talk about. I wanted to chat with my friends, and others but some where on AOL, some on MSN, some on gchat, etc etc

So I could connect to all the networks from one app

Today the would connect Facebook, Teams, Slack, Whatsapp, signal, etc etc...

That would be the dream....

There is a lot of parallels with Matrix here. I'm sad to say that the Element client just isn't polished enough to win me over Discord.

Most people, including myself, don't really care about the proprietary nature of Discord. I can fairly trivially move off Discord if it fails me so I don't really see the point in working harder to use Matrix.

Give Cinny[1] a go. It's a new client built to be familiar to Discord users. I've switched to it as a daily driver, even tho it doesn't implement 100% the protocol. It has all the main features in a nice UI which is more than enough for me to daily drive it.

[1]: https://cinny.in

No voice channels :(

Cinny is pretty, but with no link previews, no video embeds, no custom emoticons, and very limited user options, you'd be quite hard pressed to get people to leave their current Matrix clients for it, let alone Discord.

What are you missing from Discord?

This is why I think the Matrix approach is superior. Matrix seems to prioritize features first and then makes changes to the spec based on the features to support. There's a minority that likes to criticize Matrix for this approach claiming it makes it hard for third-party implementers, but the point is that only people that write software would think about a spec first and the usecase second. No non-technical user cares.

The thing I struggled with was how the XMPP chat based app I was trying to use was very much tied to the domain. If you weren't the domain admin you were SOL trying to set it up. Setting up two on the same domain--we were attempting to make it robust against disruption, think multiple servers on the same network where the interconnects may go down unexpectedly, but all local users need to remain connected and messages need to be relayed once the connections are re-established. They had previously used IRC which worked ok, but was "insecure" and had a mandate to switch to XMPP for security. Also had to be a COTS solution.

In the end I left the project before they got it working. I have no idea if it ever got deployed.

XMMP was and is very commonly used in finance for various trading systems. Pretty much any exchange or a large trading firm with a white label product has a chat function in the desktop client and they all almost always used XMMP because amongst other things you needed federation.

It’s not falling out of fashion a bit with many of the trading clients moving to web based interfaces but it have a sense it will still stick around for a long time.

Any names you can name using federated xmpp? In my little corner of the trading world everybody seems to use either ICE Chat and/or Bloomberg. My guess would be ICE Chat is xmpp under the hood, but I don't think it's federated.

CME’s chat client for CME direct uses XMPP they also support connection to other XMPP servers it’s usually mostly server to server or inter domain federation tho (can’t login with just any set of credentials but you can talk to other firms), CME is or at least was interconnected with other financial messenger services like Eikon/Reuters Messenger which is also based on XMPP.

And there are federated networks like the one from Markit

https://www.businesswire.com/news/home/20131007005587/en/Glo...

I don’t think there is a single major financial messenger today that isn’t based on XMPP.

> The issue we ran into was customers/consumers just didn't care about federated systems.

Quite the opposite: users actively do not want federated systems, as federated systems allow anyone on the internet to spam them. This is why Google Talk stopped supporting federation: 99.9%+ of users never used it to talk outside of Google instances, and 99.9%+ of the inbound messages from the outside world were spam.

The spam problem was exaggerated and used as an excuse to avoid FTC scrutiny. Even if 99.9% of federated inbound messages were spam, that doesn't mean it all reached the client. Most e-mail is spam but very little makes it to the user. (AFAIU, the primary motivations were 1) lack of MSN reciprocity, and 2) Google's endless dithering and floundering attempts to capture users the same way Facebook, WhatsApp, etc had and/or were going to.)

Large walled garden IM platforms have the spam problem potential. It's solved by using whitelists (via invites) instead of blacklists. The same mechanism is trivial to accomplish using XMPP, and I'm sure there are already XEPs for that. (Multiple, even!) You can end up with invite spam, but that's exactly the problem you see on Facebook, WhatsApp, etc.

As for me, I ran my own XMPP server for years along side an SMTP server and never received XMPP spam. Both my XMPP and e-mail addresses were the same, and my e-mail address had been in use since circa 2000 and exists on countless spam lists. I doubt the spam problem could have been very significant. I never used a GMail or GTalk address, but of the people I know who did, some of whom I chatted with over federation, I never once heard of GTalk spam.

Anybody who cares about privacy should care about federation and should advocate for federation. Being cynical about the potential is counterproductive. If you think federation is impractical because centralization is required to prevent abuse, then you must admit that privacy beyond the current status quo is also impractical; just give up. If you don't want to give up on privacy, don't give up on federation. The technical problems to federation will never go away, but the will to solve them is entirely voluntary. (Though, if we legally mandate federation then that's at least a substantial, long-term commitment.)

I have used XMPP since forever, and actively used it for GTalk federation. I don’t think I remember any spam messages, but there might have been one or two.

Facebook/Messenger? Not a week goes by without some sexy lady wanting to have intercourse with me.

This does not seem like it’s a federation issue.

Two things spring to mind reading this.

First, I can't remember who said it but there's a quote that goes something like (paraphrased) "open standards are for losers" (side note: if anyone has the original quote, please let me know). It's a pithy way of pointing out that the winners in any market don't care. Others want open standards to level the playing field.

Second, and I can't stress this enough, literally nobody wants federated systems apart from a handful techno-enthusiasts. It doesn't solve any problem users care about. In fact, it tends to create problems users do care about.

Look no further than the POTS system. Part of the reason you have spam and robocalls is one provider is pretty much free to send or forward traffic to anyone and to essentially spoof the sender such that things like caller ID do very little. Plus there's all the routing issues. Users tend to want to keep their numbers so you can't use a phone number as an identifier anymore. You have to use something lower level (think of this as the difference between a domain name and an IP address) and then deal with all the porting issues and the security issues that creates (eg SIM jacking).

Yes -- if you want a good open data system, you're in a bad place.

As long as people continue to use the Internet for their own benefit, that's fine. The internet that will be the ultimate "open standard" for the masses of people.

That's because with proprietary standards users are losers, and open standards try to fix that.

I tried to run an open source community on XMPP at around the same time and found it equally crap.

The XML schema felt bloated (baring in mind a lot of people were still on dial up, and later the data caps of early feature/smart phones), chat client support was rubbish, linking to other servers was non-trivial (they'd often be running extensions you weren't. In fact the whole extension system was broken). And IIRC the leading XMPP server was written in Haskell and the only way to configure it was to edit the source. It was just a horrible experience all round.

I had Skype, MSN, AOL IM, IRC, and another (possibly IRQ?) all linked together -- most of which are proprietary -- under one server and the only protocol I couldn't get working properly was XMPP. Which really says a lot about the state of XMPP at that time.

Maybe things have improved. But now that Google Talk, Facebook Messenger and others don't use XMPP any more, there is even less reason to bother trying.

The only other reference I can find to a Haskell XMPP server is your other comment at https://news.ycombinator.com/item?id=27297137

Suffice to say; go and download a modern client like conversations.im and install ejabberd.im on a machine somewhere (or just use a throwaway account on a public server). The ecosystem has significantly improved since you last tried it... 15-20 years ago?

I'd already answered why I haven't

> Maybe things have improved. But now that Google Talk, Facebook Messenger and others don't use XMPP any more, there is even less reason to bother trying.

There's no shortage of chat protocols out there. As far as I'm concerned XMPP had it's chance and failed to deliver. While I'm not a fan of NIH (not invented here) syndrome, I'm even less of a fan of polishing a turd, because that's what leads us to monstrosities like STARTTLS, FTPES and so on and so forth.

I saw that, and it's simply a ridiculous and narrow minded viewpoint (even more so giving you're judging on purely historical data).

I myself have rediscovered XMPP in the past two years (whilst using other tools like Matrix for other roles). The experience has revolutionized the way I view chats apps and even I host my own server on my own bare metal.

The thing quite simply flies and is rock solid. I've also seen multihour WhatsApp and Signal outages go by whereas my friends and I have kept chatting on my reliable server. Never had an issue with STARTTLS.

XMPP has silently evolved with the times and is far better than ever and should simply be the go to if you want to have control over your own comms.

> I saw that, and it's simply a ridiculous and narrow minded viewpoint

No need to be abusive. And lets not forget I'm basing my views on technical arguments, which is the opposite of being narrow minded. If you want to convince me that XMPP is worth trying then you need to provide me with technical reasons why I should.

> I myself have rediscovered XMPP in the past two years (whilst using other tools like Matrix for other roles). The experience has revolutionized the way I view chats apps and even I host my own server on my own bare metal.

Good for you. I wasn't suggesting nobody should use it though. I was only answering why I wouldn't be investing any more of my own finite amount of free time on it. That doesn't make me narrow minded. That just makes me busy enough that I need to make trade offs with my time.

> The thing quite simply flies and is rock solid. I've also seen multihour WhatsApp and Signal outages go by whereas my friends and I have kept chatting on my reliable server.

You can't compare a personal chat server for friends with a service with multi-national infrastructure used by millions. Problems grow exponentially with scale.

> Never had an issue with STARTTLS.

Then you've not spent enough time working with SMTP. :) But my point wasn't that STARTTLS sucks (though it does) but rather that STARTTLS should have been a warning sign that we need to abandon SMTP for something better.

> XMPP has silently evolved with the times and is far better than ever and should simply be the go to if you want to have control over your own comms.

Maybe but I haven't yet heard what any of these improvements are from you though. You've talked about how XMPP makes you feel, you've talked about how my previous post made you feel, but you've not actually talked about XMPP itself. For a technical conversation your reply lacked a lot of detail about the tech and a lot of abuse towards the person you're trying to persuade.

> And lets not forget I'm basing my views on technical arguments

You're not, you just simply have a strong opinion about something you haven't even tried (in the last decade) and thus basically know nothing about it. It's a view based on ignorance.

You're just throwing insults around again and still haven't posted a single technical argument. You haven't even posted a rebuttal to my own technical points aside saying "it's better and you're a dick for not trying it" -- to paraphrase. And frankly it is my personal time we are talking about so it is none of your business whether I do decide to try it again or not (for reference, I'm busy working on open source projects to benefit people like yourself while also holding down a full time job and raising two young kids. So I make trade offs of what interests me and what doesn't or might cost my productivity. That is why I don't have time for all the fanciful projects that random people on HN claim I should be doing. Not that that is any of your business).

If you want a grown up conversation then I'm happy to engage. But don't waste my time with petty insults and vague claims of greatness when you're unwilling or unable to add any technical detail to back up your bluntness.

> still haven't posted a single technical argument

Have you even tried reading the original article that this thread is literally linked to at the very top?

Yes. It's an interesting article but doesn't specifically address the points I've been making and nor does it explain your point.

- They talk about evaluating extensions but extensions was one of the things I raised as being a turn off.

- They talk about clients, but there's no shortage of other chat solutions out there with decent clients these days. A list of platforms and links to clients doesn't tell me whether XMPP clients are buggy or well polished. Let alone why I should use XMPP over another solution with a well polished client.

- They talk about privacy but even on XMPP that's not a given and there are other solutions out there if you're interested in privacy.

- They talk about voice and video calls, and that at least is new from when I last tried. So that's one thing in favour of your point. However I already have a half a dozen platforms that do that -- platforms that already have my friends and family on it. Why should I set up another? And moreover how am I going to convince my friends and family to install this other chat client if you can't even convince me, a fellow technology geek, to try it?

This is why I talk about XMPP missing the boat. People gave XMPP a try and moved on. And for all my asking, neither the article nor yourself can explain why XMPP is a cut above the rest. Worse yet, all the talk in the article of "evaluating extensions" and "reviewing privacy" doesn't convince me that it's any less annoying to set up either.

> - They talk about evaluating extensions but extensions was one of the things I raised as being a turn off.

They can be daunting at the start (as it is with learning any new technology ecosystem), but once configured; it's not something one thinks about.

> - They talk about clients, but there's no shortage of other chat solutions out there with decent clients these days. A list of platforms and links to clients doesn't tell me whether XMPP clients are buggy or well polished. Let alone why I should use XMPP over another solution with a well polished client.

My personal favourite is conversations.im, but these days, with the notable exception from Pidgin, they all tend to interoperate fine. iOS clients still has issues with push notifications (thanks to the OS level) that are being fixed: https://snikket.org/blog/snikket-ios-public-release/

> - They talk about privacy but even on XMPP that's not a given and there are other solutions out there if you're interested in privacy.

Disagree on this point:

- Other solutions generally require a number or email address. On XMPP, you can simply create an account by pointing a client at a server and asking for an account (you can even create a completely random pseudonym).

- Other solutions do not generally allow you to self-host (Slack, WhatsApp, Signal, etc), so they're not private even to begin with.

- Other solutions don't generally have the flexibility of federation to let you talk to other servers (such as Mattermost).

- XMPP clients also have optional support for Tor, and servers can be hosted on hidden services and can talk to servers over Tor too - which is much better than Signal and others should you desire an elevated level of privacy (I do not bother with this point, but if I met someone who did - good to have as an option)

- Things like Briar that do P2P do exist, but their clients are lacking in other areas such as voice calls, multi-platform support, etc.

> - They talk about voice and video calls, and that at least is new from when I last tried. So that's one thing in favour of your point. However I already have a half a dozen platforms that do that -- platforms that already have my friends and family on it. Why should I set up another? And moreover how am I going to convince my friends and family to install this other chat client if you can't even convince me, a fellow technology geek, to try it?

It's just chat at the end of the day, and as I said - I did not want to rely on a third-party for something as crucial as my personal, private comms.

Another thing that I enjoy about XMPP is that the clients support multiple accounts as a first-class feature (something you cannot do on Signal/WhatsApp). Theoretically; from the same app; I could have accounts for personal, work, a neighborhood chat, and an account over at sfconservancy[1], whilst being able to enable/disable as and when I want to.

This last point is more pertinent as people become fatigued and bored of having to have multiple apps for different contacts across multiple platforms (a pain point made worse as new chat apps come into life every other year and contacts decide to move over). In my case, I know that there will be open-source XMPP clients that I can use to log into my server and still have the same contact list.

Tools like https://quicksy.im/ also exist to make it easier for normal people to adjust to an XMPP client before moving them over to non-number based accounts.

[1]: https://sfconservancy.org/blog/2021/jun/21/chat-options/

I really appreciate you taking the time to post that. It was a really interesting read and filled in a lot of blanks I had as to why people are still pushing XMPP. Thank you :)

It's a little odd. Facebook and Google talk both allowed use of standard xmpp clients when they were using xmpp underneath - so one could get away with a single app to talk to people over xmpp, Facebook chat and gtalk. And eg OTR worked fine.

But both Facebook and Google refused federation, and then phased out xmpp.

I assume all the big players are equally eager to kill email, but haven't quite figured out how yet.

I reverse-engineered the comms for my cheap Ecovacs robot vacuum and was surprised to discover that, like some angsty teen, it spent all day hanging out in an XMPP chatroom waiting for somebody to talk to it: https://github.com/wpietri/sucks/blob/master/developing.md

XMPP is a part of TR069 as some sort of rendevzous point for device and controller:

https://www.qacafe.com/resources/2015-03-26-using-xmpp-for-t...

It's little known that XMPP is a big hit with military and police organizations. (e.g. the Baltimore police department has XMPP-based chat and feeds the messages into Lotus Notes)

Vendors in that market heavily use extension mechanisms. For instance commanders of infantry units might fill out forms every day about what kind of contacts they had with the enemy and get them sent to a central location.

> Vendors in that market heavily use extension mechanisms. For instance commanders of infantry units might fill out forms every day about what kind of contacts they had with the enemy and get them sent to a central location.

How does XMPP help with this over a HTTP form?

HTTP would be fine as well, you could GET an XML with the form to fill and POST the XML with form data.

But: - HTTP forms might be less flexible when it is about structured data. - HTML needs a complex renderer software

Remember that this stuff might need to work on low resource devices and over low bandwidth connections.

Sounds similar to WinLink forms on ham radio, used to make sending structured data easier: https://www.winlink.org/WinlinkExpressForms

I happen to have been an avid user of the Lynx web browser. A long time ago yes but I still remember it.

HTML forms are pretty flexible and could be designed in a low CPU environment.

On the battlefield you don't have good internet so the cycle of GET the HTML form and POST the fields is not reliable.

XMPP Forms are deeply asynchronous. I'd assume that the "blank form" already exists on the soldier's laptop and the message gets stored-and-forwarded until it gets to the destination.

Isn't this just an issue of saving the POST until you successfully deliver it?

HTML is an async protocol. We pretend it is sync through a mountain of cookies. But innately, HTTP and HTML are all async. Without PHPSESSID, there really isn't an official way to correlate GETs with POSTs.

XMPP is synchronous. It's a chat interface. I'd assume it's be harder to do async comms with it. IIRC, it's a stream of XML that coincides with the servers state.

heh, back in my day that role was filled by IRC over a low bandwidth SIPRNet link - with each room being a different command or area of operation. You can see evidence of this from Manning's tactical data dump.

> meaning people from different servers can talk to each other while no central authority can have influence on another server unlike Matrix

Elaborate please. Matrix doesn't have a central authority server which can influence other servers.

Also, the bridge system in matrix is solid. Most people do not want to ban all other messengers, but they want all their communication in one place. Matrix-based messengers like Element or Bleeper seem like the way to go. And if people then start using matrix based chatrooms, all the better.

But I doubt you can reach critical mass with a system that is not open to other chat systems, even if those are closed.

XMPP has bridges.

Yes, but it does not sell them well. Look at Beeper (beeper.com) or Element One (element.io), they actively market the fact, that you can use whatsapp or signal or telegram with them. Using matrix is nearly an afterthought, it's more like using pidgin or trillian back in the day. And I think it's clever because it lets me switch away from whatsapp etc, but I don't have to bring all my friends over at once.

And, once I am on board, I can start to look into the features provided by matrix. People don't switch to a protocol, I think, they want to leave a service or provider. And yes, you are still part of whatever walled garden you choose to use, but whatsapp can run on an old phone and never touch my new phone, never getting the data on that device.

Beeper seemed cool but trying to find out more it just put me on a waitlist. Element One is extremely limited in bridges. Element Home seems better in that regard but still doesn't have a major one for me Google Chat/Hangouts. Neither seems clear on if I can use my personal domain while they host and manage the homeserver+bridges. Element One mentions subdomains and beeper mentions self hosting but neither seem to comment on personal domains which seem a natural fit to use for a federated service.

I've been so excited for Beeper since the beginning of the year but just like you, I'm on the waitlist and have been for almost 10 months now.

I'm dying to try it, and if it functions how they've described it only makes me want it more. A few weeks back I got an itch and started watching any videos online and came across a few from the founder and their main developer (the one developing all the matrix bridges that I believe even Element One is using? maybe?).

For someone in my situation, if it works its priceless and if they wanted 3x or 4x the monthly fee I'd still be more than okay with it. Hell, I'd pay extra just for early access. I've always struggled to keep up with my communication and a big one for me is iMessage. But aside from that everything else I use for family (WhatsApp,Telegram,FBMessenger), work (Skype,Slack), and friends (iMessage,Discord) is stressful for me everyday switching between all the apps when looking for something, remembering who I need to respond to, switching to phone (I don't use Mac, so iMessage only when I'm actually on my phone which is not often).

XMPP is decentralized in practice (as opposed to Matrix, which is decentralized in theory, with everyone using the same client on the same server using the sole available server implementation). There are hundreds of public servers and dozens of clients. Nobody runs marketing campaigns.

Judging by the state of most transports, actual long-term interest in that feature is limited. I don’t know if Matrix does something in a sufficiently different way for bridges (apart from ICQ) to be actively used.

Commercial services, by necessity, try to differentiate themselves. That prevents full interoperability.

XMPP was developed as an open protocol to bridge to proprietary communication platforms, just as Matrix is today.

There are still such bridges (more commonly called "gateways" or "transports" in XMPP), such as for WhatsApp, Signal, and others. However my perception is that over the long period of time XMPP has been around, focus shifted to the "native" XMPP-XMPP use-case, because it's easier to build a better experience that way.

My personal prediction is that Matrix will certainly go the same way - using a focus on bridging to bootstrap the network in the initial years, and then gradually shifting to focus on "native Matrix" for most use-cases after reaching a certain critical mass.

> the bridge system in matrix is solid

Solid? They crash or malfunction all the time.

Sorry, not a native speaker. Yes, they are a bit wonky, with solid, I meant well established and (comparatively) easy to set up.

Can't confirm, using xmpp and Telegram bridge here

Isn't the bridging software pretty much the same?

Last time I check both used Spectrum, at least for evertything libpurple related. So at least in theory they should be equally stable, unless it is just a question of code maturity which should improve over time.

Yup, Matrix as a network has no central authority at all, although it's a common untruth which gets passed around, rather depressingly.

Things which could be at the root of this misunderstanding include:

* The fact that we operate a *strictly optional* logically centralised directory service to look up matrix IDs by email address or phone number. I really wish we'd never added this to Matrix; it causes way more grief than it solves - we'd have been better off waiting off for someone to build a decentralised keybase service and use that instead.

* Element runs a centralised integration manager (aka app store) to make it easier to add bots/bridges/widgets to rooms. Again, this is completely optional - although back in 2019 we had a stupid bug in Element Web which meant it checked the integration manager even if you weren't using it. This was fixed in June 2019 (https://github.com/vector-im/element-web/issues/6802).

* Element has to query your server to find out what authentication mechanisms are available before you log in. Therefore if its config points to the default matrix.org homeserver, you end up pinging that every time you log in, even if your homeserver is elsewhere. The solution for now is to change your default homeserver in the config to avoid this.

* Servers need to be able to validate events from servers even if those servers are offline. To do this you need to grab the public key for the offline server from somewhere; by default Synapse uses matrix.org as a fallback, but loudly nags the admin to pick a better default when you install the server.

None of these actually allow a "central authority to have influence on another server" though; I've just listed the only places I can think of where a typical self-hosted Matrix implementation might interact with centralised services run by Matrix.org or Element.

Unfortunately I think the real root of the misinformation is that it's just a bit too tempting and easy to throw around FUD. Some day we may get to the point where alternative FOSS projects will realise that the real enemy are centralised proprietary user-exploiting services rather than one another...

Matrix is a lot better than every proprietary service, but it's poor XMPP compliance currently makes it impossible to have essential features like encrypted communications or adding Matrix users to multi-user chats.

Unfortunately XMPP<->Matrix end-to-end encryption is a contradiction in terms. You have to speak the same protocol from end-to-end, otherwise you'd have to break the end-to-end encryption to translate from one protocol to the other. The closest you could get would be to run the Matrix<->XMPP bridge clientside alongside your Matrix client... but at that point you might almost as well be running a multihead messenger that speaks both Matrix & XMPP.

In terms of adding Matrix to MUCs; i believe we're working on it in Bifrost, which is in active albeit slightly sporadic dev: https://github.com/matrix-org/matrix-bifrost.

> Unfortunately XMPP<->Matrix end-to-end encryption is a contradiction in terms. You have to speak the same protocol from end-to-end, otherwise you'd have to break the end-to-end encryption to translate from one protocol to the other.

You'd only have to speak the same encryption protocol. There already exists standards like the Signal protocol (on which OMEMO builds) or OpenPGP (which many XMPP clients support besides OMEMO). If messages can cross the border, so can public keys and encrypted messages.

Unfortunately this isn't true. Olm (Matrix's encryption protocol) is the same as OMEMO, which in turn is the same as libaxolotl. In fact, the OMEMO XEP even recommended Olm as an implementation for OMEMO at one point due to licensing concerns with libaxolotl/libsignalprotocol!

But the fact that both protocols use the same E2EE encryption protocol doesn't help you with interoperability - given that you have to break that E2EE in order to convert between Matrix & XMPP; either on the server (ugh, defeats the whole point of E2EE) or on the client (ugh; you might as well just have a multiheaded messenger).

I'm not sure I understand. If in fact the same underlying encryption protocol is used, why can I not retrieve the public key of a Matrix user, encrypt the message and send it? Nothing here requires the server to break E2EE.

Because the message format is different. In XMPP, the message is an XML document according to the message schema. In Matrix, it's JSON.

You can exchange the private keys, but using what format? Once that's done, you can encrypt XML on one side, but Matrix has to convert it into JSON on the other (in its own, separate format with its own fields). This can be done on the server side after decryption (no more E2EE), or on the client side (after decryption). In the latter case, you're not running a bridge anymore, you're just running an ad-hoc client for XMPP inside your matrix client.

In XMPP, encrypted messages can either contain encrypted XML

> <message type="chat"><body>Hi there!</body></message>

or raw text

> Hi there!

In Matrix, it's always encrypted JSON

> {"type": "m.room.message", "content": {"msgtype": "m.text", "body": "Hi there!"}}

It would be possible for Matrix to indicate that a room is a "raw room" and that clients should send and expect raw messages in that room - losing a few features on the go. Such raw room could be bridged over to XMPP fully end-to-end-encrypted (with small changes on either side to adjust for the differences between OMEMO and OLM).

But I doubt anyone at Element will invest time into doing it and for the XMPP side it's much more work to do it "on their own" (by implementing parsing support for the encrypted JSON of Matrix as shown above).

Having raw text in the encrypted content is by the way still the normal way how things work in XMPP world and is supported by every client doing E2EE - because it is basically free to implement (barely any additional code needed).

Even if you could make Olm/OMEMO interoperate otherwise, the Signal protocol supporting client requires access to a server to pick up pre-keys. So you need to have access to the pre-key server of your correspondent or your correspondent needs access to the pre-key server you are on. The Signal Protocol is not generally suitable for this sort of thing mostly due to its complexity.

If you did something stateless like OpenPGP then you could do E2EE over bridges with no problem. The cost would be that each and every message would be complete in itself. So sending a single emoji would take several hundred bytes. That might be less important these days...

> but at that point you might almost as well be running a multihead messenger that speaks both Matrix & XMPP.

Yes please! There's sooo much work people put in the clients, and there are multitude of them for each network. I understand they are different and a specialized client may be better suited for unique protocol features, but I also think there should be more general clients with protocol plugins, as it was with Pidgin, Miranda and such.

> You have to speak the same protocol from end-to-end, otherwise you'd have to break the end-to-end encryption to translate from one protocol to the other.

Why is this any different from WhatsApp, which has a Matrix bridge? Or does that break encryption too?

I have no idea how the Whatsapp bridge works but it cant be end to end encrypted.

Matrix to whatsapp bridge can be encrypted, whatsapp bridge to whatsapp can be encrypted, whatsapp to whatsapp is encrypted. But you have to break the chain at some point between matrix and whatsapp.

Encrypted != End to End Encrypted.

Seems to be my understanding as well.

> The fact that we operate a strictly optional logically centralised directory service to look up matrix IDs by email address or phone number. I really wish we'd never added this to Matrix; it causes way more grief than it solves

(Not a Matrix user so maybe help me understand here.) I think the reason why I have about 30 contacts (instead of like 3) on Signal is due to phone number discovery. It's not decentralized, which is of course subject to criticism, but exchanging & verifying public keys between O(N^2) people hasn't "taken off" in the X decades since it was invented.

> we'd have been better off waiting off for someone to build a decentralised keybase service and use that instead.

Do you have an example of any mainstream product that has ever achieved that, without either being partly centralized? Decentralized identity is really, really, really hard, and mostly the layer-8 implementation (discovery, name-squatting, impersonation, abuse, spam, key-management and exchange, etc).

And it is also the reason why I'd never considered signal for anything.

because it makes it easy for you to find contacts?

I do not want people to know I use X when they have me as a contact.

I also do not want my account to be tied to my phone number at all, because my phone number is tied to a contract.

well, keybase got quite far - it could have been decentralised but they chose to centralise it for commercial reasons. but yes, decentralised identity is hard, hence us cutting that corner (and then getting abuse about being too centralised…)

Amazing work. Bravo. I'm very much hoping to see P2P Matrix arise. That is no small task, I realize (and I'm grateful to those who have contributed to this). What are you thoughts on that aspect of the ecosystem?

P2P Matrix is a moonshot, but it's coming together really well. The only bits left at this point are:

* Making Pinecone resilient to attacks. It's working well for clear weather scenarios, but we need to properly try to break it and get it audited. Need to test it at scale too.

* Improving Dendrite so that it's more stable and efficient clientside

* Metadata-protecting store-and-forward, so that clients can pass messages to each other without needing to be online at the same time. You can cheat and run a serverside P2P node to relay, but then it stacks up metadata - whereas half the point of P2P is reduce Matrix's metadata footprint.

* Account portability. Currently each node is a separate account; need to support the same logical account being replicated over multiple nodes.

* Better than full-mesh routing. This should be pretty easy as we can just follow Pinecone's topology to route traffic between nodes rather than having each node try to simultaneously post to every other node in a given room.

You can get testflight & APK builds from #p2p:matrix.org if you want to experiment.

Arathorn, thanks for responding to comments here. Public discourse comes in all flavors and I appreciate you taking time to respond to the queries and comments. :)

AFAIK, there are 6 server implementations of Matrix protocol: Synapse (Python), Dendrite (Go), Conduit (Rust), Construct/Charybdis (C++), Plasma (Elixir), Jeon (Java). Are there any more that I have missed? Which ones other than Synapse are most mature? An unbiased comparison post (that remains updated) would be really useful.

A second question is: are the APIs between a homeserver and the frontend defined and standardized? I've noticed this in the ActivityPub ecosystem where it is usually very easy to use one frontend with another backend. Are matrix servers also written in the same way?

Not OP,I'm just a happy matrix user.

Yes the matrix spec is all standardized. You can use any client with any homeserver.

There are a few other server implementations (polyjuice in Elixir https://gitlab.com/polyjuice/polyjuice_server; https://github.com/clecat/ocaml-matrix in OCaml; the abandoned mxhsd in Java) - but the most stable ones alternatives to Synapse right now are Dendrite (Go) and Conduit (Rust).

Matrix's API surface is big, and servers deliberately handle all the heavy lifting in order to make clients trivial to write, so writing servers is not trivial. But Dendrite & Conduit are both very usable and making good progress right now.

And yes, the APIs are all standardised (while also constantly evolving) as per https://spec.matrix.org :)

Thank you! Those are extremely important points, and I'm glad to hear it. Rambling from a madman in the desert, so please feel free to ignore me or tell me buzz off (and, if so, it was an honor speaking with you!):

* This might be a worthy tool for simulating aspects of scale: http://imunes.net/

* I assume power usage is a non-trivial concern (looks like yall have been worried about this too). I've tried my fairshare of P2P apps on mobile, and this can be a dealbreaker for many folks. I assume this will be something that has to be dialed in on mobile clients (and probably provides options to the user). I'm sure there are many trade-offs to consider, and I'm hoping logical accounts will provide significant advantages here. Are you feeling confident this will succeed?

* The Store-and-Forward problem looks like a really big deal. Defaulting to friends (for those folks who are lucky to have them) or a chosen subset to proxy seems reasonable. Having the clear ability to pick out who we trust here (including volunteer networks) seems solid (I'm a huge fan of Retroshare's F2F concept). I take it that, ideally, with logical accounts and excellent NAT traversal (also a sore spot usually), one's own cloud of machines will obviate some of the desire to run anything in a datacenter (and even perhaps make friends a fallback). I'm excited to see Dendrite and Conduit grow!

* As part of that portability, at least to my ideal, I'd be able to join my network from scratch using a password hashing algorithm for the private key, as otherwise I may be reliant upon secondary networks to move keys. This is often one of the first applications I want to install on a system to bootstrap myself in.

* Pinecone's Topology looks like a good idea (Yggdrasil is dope too). That metadata problem may still raise its ugly head here. Do you predict the use of onion routing, at least with Tor and i2p, will be relatively easy to integrate in addition? Do you foresee Matrix enabling its own onion routing some day?

* If you had to take a wild guess at the first stable moon landing, when do you think that might occur? Do you think 5-years may be acceptable?

Lots of interesting stuff there - thanks :) We're using https://github.com/mwarning/meshnet-lab rather than imunes.net for network simulation currently, but will take a look.

Power usage is looking pretty positive so far; as long as we route the Matrix traffic over the routing topology rather than going full-mesh it should minimise radio usage (the main battery suck, other than screen).

For store-and-forward, honestly using P2P Nodes as intermediaries is an okay approach other than exposing metadata to them. Our plan in the longer term is to switch to loopix-style mixnets to obfuscate the store and forwarding, a la nym.

In terms of joining the network by deriving a private key from a passphrase... yup, that could be cute, although slightly terrifying in terms of the risk of weak passphrases :)

We're hoping to get the P2P network stable in the coming year (although we were also aiming for this year originally :P)

You are exceptionally polite speaking with lunatics like me (I've seen that multiple times from you). I'm [[grateful]] for your candid responses. In case it may be useful, I will continue thinking about it here: https://philosopher.life/#2021.11.17%20-%20HN%20Log%3A%20Ara.... You already know, but it is my duty to acknowledge that you work on behalf of Humanity and quite a bit may ride on your success. You have a difficult ship to steer without enough resources, and I wish you had military-level funding for what is ideally a military-grade project. If you ever directly need something from me, please let me know. Thank you again for your time and willingness to listen to me, sir.

Every Matrix protocol client I've ever tried, including yours, seems to connect to your "strictly optional" centralized directory service. I've never once wanted to use it, and, despite your claims of it being optional, 100% of the times I have logged in to a Matrix instance, via the web or via an app, my computer has talked to your computer.

Something's broken here.

See https://news.ycombinator.com/item?id=29263797.

I just tried logging in to Element.io with an account on my own homeserver and I saw several requests to vector.im before the privacy policy popup, including what seems to be some kind of token exchange.

I don't think the intent is malicious, but the system is definitely leaky. My homeserver config doesn't mention vector.im (perhaps it's a hidden default somewhere?), I checked.

From a protocol standpoint, no identity server is necessary. From a practical endpoint, all feature-rich clients (the ones I know support e2ee) seem to send requests to vector.im. Matrix is decentralized, but a lot of data still seems to get leaked to a centralized place by accident from my short experiments. It's not a great scenario, even if I'm fine with a few requests to the 3PID server every now and then.

Client issues are ecosystem issues if all usable clients suffer from them.

> I don't think the intent is malicious, but the system is definitely leaky. My homeserver config doesn't mention vector.im (perhaps it's a hidden default somewhere?), I checked.

IIRC, identity server is something primarily set on the client (element.io here) rather than the homeserver.

It really needn’t and shouldn’t ping an identity server as part of the sign-in process (esp before/without privacy policy agreement)

FWIW it seems the issue is that there’s a fallback process for identity server; if neither your account nor your homeserver has one set, it defaults to the client default. if you host your own Element-web instance you can change this.

https://github.com/vector-im/element-web/blob/develop/docs/c...

Or check out ma1sd. You don’t need to enable all the bells and whistles if you don’t want to.

So I've just double-checked this by opening https://app.element.io in an incognito tab and looking at the network inspector. What I see is what I was referring to in the grandparent as:

The same applies to the identity lookup server as well as the homeserver: the app checks the one in your config to ensure that it exists before the user tries to log in on it. It does this with a single request (a GET to https://vector.im/_matrix/identity/api/v1). There is no token exchange there.

The workaround is to configure the app with the correct default homeserver & identity server (if any) for your deployment. The underlying bug (i.e. don't validate the config until the user tries to log in) is https://github.com/vector-im/element-web/issues/11655, and it simply hasn't got to the top of the list yet; we've been focusing on narrowing the UX gap between Element & Discord/Slack/Teams rather than reimplementing the login flows. As you can see on the bug, it's been picked up off the backlog 9 days ago and is currently being worked on.

Separately, I'm seeing a related bug that Element Web's guest mode is incorrectly kicking in on the login page. Guest mode exists so you can deeplink straight to something like https://app.element.io/#/room/#matrix-dev:matrix.org and read scrollback without needing to register an account - but it looks like we're spinning up guests on the default homeserver needlessly on the login page. This is https://github.com/vector-im/element-web/issues/11533, and we've put it on the radar too.

TL;DR: this is slightly leaky, but is it really any different to Chrome's default config loading up Google as a default homepage? Or Firefox loading up Mozilla.org? Or Thunderbird loading up Thunderbird.org? Eitherway, we'll fix the behaviour, but PLEASE understand that this is a million miles away from the bogus "central authority can have influence on another server" smear from the original post.

> TL;DR: this is slightly leaky, but is it really any different to Chrome's default config loading up Google as a default homepage? Or Firefox loading up Mozilla.org? Or Thunderbird loading up Thunderbird.org? Eitherway, we'll fix the behaviour, but PLEASE understand that this is a million miles away from the bogus "central authority can have influence on another server" smear from the original post.

I understand the whole "central server" stuff is bogus.

However, I consider the forced homepage opens every browser seems to do today an invasion of my privacy, especially with the way these pages often includes trackers. Mozilla is especially bad at this; I've had to disable checkboxes on all my devices to make sure I don't get their stupid VPN ad, which is tracked with Google analytics and which addons can't properly touch because certain Mozilla domains are exempt from addons (rightfully so!).

Google knows when I open Thunderbird. Google knows when I update Firefox. Google knows when I do anything in Chrome. Microsoft tracks even more than Facebook when I accidentally open Edge because I was foolish enough to click on a help icon in Windows 11. Element knows when I log in to Matrix. All of this data collection is completely unnecessary, and while this comes from a sick hunger for data in Mozilla's/Google's/Microsoft's cases and a configuration quirk in Element's case, it's still a worrying development in my opinion.

I know you folks are working hard to fix things like these and I originally wanted to call balogne on the claims several people made in these threads, but I couldn't because their observations about the identity server were right (though their conclusions were strongly overstated).

I'm glad that this is on your radar and I'll happily keep using Matrix while this is being fixed in the background, so don't get me wrong here. I love Matrix and the possibilities it represents, and I hope the clients and features will continue maturing so that I can recommend my friends and family to get it.

The identity server is probably what is being referenced here.

The identity server can be and is self hosted by a lot of people.

True, but because of the architecture it is not very useful self hosted unfortunately.

I worked on a large XMPP system for a while and realized it's actually not a very good protocol, for one reason: presence. "Presence" is the feature that shows a little indicator next to each one of your contacts indicating whether they're online or away.

The problem with presence is that it generates a LOT of traffic. It's N^2 with the average number of contacts each user has - every time a user changes presence, you have to send a message to every one of their contacts to update them about the presence. This happens whether they're paying attention to your client or not. So on a large system, the big majority of the traffic is just presence messages flying back and forth saying who is at their keyboard and who isn't. This is really wasteful in general. Worse, it's not a super important feature for a lot of people, so it's definitely not worth the effort. And ever since mobile computing became ubiquitous, it's not even clear what this is supposed to represent.

So, I say let XMPP die. It was a fine idea for its time. But the protocol has no place in the modern world.

That's afaik a problem with pretty much any system that supports presence, even though it can be somewhat optimized potentially (where I don't know how many options exist in XMPP, but I don't think would be fundamentally impossible to add)?

It seems like it would be much more efficient for the presence to be stored somewhere where clients can fetch it if needed.

if needed=each time presence change.

So you d have an enormous graph with one node in the center refreshing presence for everyone. And end up worse.

Fetch has the downside that with the usual idea of presence you want to know immediately if presence changes, so you'd be polling constantly. What could make some sense is caching server-side and allowing the subscription to be quickly enabled/disabled, so that a client could save traffic while it's not on screen etc. When it returns, it asks for the last message for each contact to be delivered from cache and the subscription to be resumed (in exchange for having to wait for the data to come in, and until then not showing the correct information).

Yeah, any kind of aggregation would help. But XMPP dictates that you "MUST" broadcast presence messages to all subscribers every time it changes. (Many open standards are very forceful in their language about such things.)

Most chat systems have done away with it. Slack has it, but that's about it. For anything you use from your phone, what does it even mean?

oh, it's a core thing and not open for extensions changing it or being disabled? That's indeed unfortunate.

Core indeed. The first P in XMPP standards for Presence.

> Most chat systems have done away with it. Slack has it, but that's about it.

Slack, Discord, Matrix, FB Messenger, Instagram all have it to name a few.

The only ones that don't have it are SMS replacement apps (Whatsapp, Signal) and IRC.

I for one greatly rely on presence. It very often informs if I should or should not send a message to someone.

IRC has away-status at least.

> I for one greatly rely on presence. It very often informs if I should or should not send a message to someone.

Do you have an example? I certainly don’t operate this way. I just fire off a message. The status only informs my expectation of when a reply might arrive.

In the absence of a status I have no expectation of a reply. With mobile devices I still get immediate replies frequently because of notifications. So I don’t rely on status.

In my experience away status is uninformative. It's often inaccurate - transitions too early, you can also send the message in the gap between afk and status transition. On the other hand when I'm at the keyboard, it doesn't mean I'm free for chat and can easily reply 10 minutes after I receive the message.

+ Teams, Jabber, Sametime, Hipchat, FB workplace

i.e. basically every corporate chat system

Huh? 100% of instant messaging apps I use daily have presence. Telegram has it, VKontakte also has it, Discord has it. Not sure about Matrix, but I open it maybe once a month.

Matrix has it, but it's e.g. disabled on the matrix.org instance because of performance issues.

Ah ok. I'm indeed on matrix.org, I did check if there's presence before posting that comment, and didn't see any.

I do keep wondering though what's the deal with the performance of the Matrix protocol. From reading the overview in the spec, it doesn't feel like there's anything that could have a significant performance cost. It's essentially a standardized but extensible REST API that passes JSON objects over HTTP. Is it that the existing server implementations aren't optimized very well?

It used to be that Synapse (the server) hadn’t been optimised much, but we have been going through doing so. Presence just hasn’t come to the top of the list yet as we think there are more important features to prioritise.

The reason there is stuff to optimise is that Matrix passes JSON around over HTTP in a byzantine fault tolerant manner: every time you send an event you have to sign it and justify why you’re allowed to send it, and every time you receive one you need to execute that proof effectively and do a merge resolution algorithm to authenticate the event and store it in the right context - very similar to how Git works, but with events rather than commits. A naive implementation of this can be very heavy. We’ve improved it by 3-4x over the last year or so, and there are other big algorithmic improvements in general on the horizon too.

My client has status settings that allows to disable automatic away status, so online-offline is the only status change unless you manually set away status.

Actually, it is an important feature for a lot of people, with virtually every popular messenger implementing it. Telegram and Whatsapp both go one step further and not only show when a user is at his keyboard, but also if he's currently typing. In the grand scheme of things, a few strings being routed regarding presence information don't pose a big problem for server providers (I know two of those and they didn't complain nor think of it as an issue whatsoever).

You shouldn't impose your use case to "a lot of people", and more importantly also shouldn't conclude that it's "definitely not worth it" from that. The industry disagrees with you, and so does my mom.

When somebody is typing or not does not get broadcast to every contact. It’s only sent to the individual who is about to receive the typed message. O(1) communication vs O(N). That is a different feature which is not XMPP presence and I agree way more relevant in the modern world.

Even if the server load from presence is acceptable (we clearly disagree here) it’s not okay to bombard somebody’s mobile device with these messages all the time. If your phone’s CPU and radio need to wake from sleep every time any one of your contacts picks up or puts down their phone, your battery life is ruined. You may think these messages important, but they are not urgent.

Honestly, I can't disagree with you on technical terms, because what you're saying makes sense and on other topics this discussion would be over.

But in this case, I have Conversations on my phone and can say from experience that it didn't degrade my battery life whatsoever. I know that android restricts this behaviour heavily by default and for good reason, because lots of apps will drain the battery in a matter of hours. No idea how the conversations devs do it, there must be a quirk to it I don't know which makes modem usage acceptable. I'm not a zealot in this case and I accept the flaws, but I can't leave you thinking mobile xmpp hurts the battery in a relavant way because that's (in my experience) not true.

I'm not deep into xmpp xep standards, but I'd imagine theres a way to tell the server "I don't care about event type x please don't send those until I ask for them again"

What do you suggest replacing it with?

I wonder why people jump fast to conclusion that to "let XMPP die", when the protocol can also be iteratively improved. Presence is not required in XMPP, its an optional feature. Everything you said has been considered in newer XMPP extension protocols (like XEP-0396: MIX).

Why improve when you can start over? Modern tools for pub sub and websockets are so much better. We don’t need to use XML any more. (For the love of god please let XML die.) The old code and implementations are more of a liability than an asset, especially if you want to take out major functionality. Unless interop with AOL instant messenger’s install base is a key feature or something. (Don’t know if AIM uses XMPP but the timing is about right and representative of what you’d get.)

Why start over when you can improve? You'll most likely end up with the second system effect, probably slowly rediscover all the valid reasons for the things you disliked, probably re-make all the mistakes.

I remember using XMPP for a site I built that connected tenants and landlords, a reverse MLS of sorts. Back then, I choose XMPP because I had heard this is what Facebook was using at the time for their new messenger and it provided services like online status, typing status, and live messaging out-of-the-box. I don't recall it being terribly difficult to implement and it was awesome to see it working for the first time!

This was almost 10-years ago now, it's incredible to see this technology still moving forward.

Interesting.

I had the impression the general opinion about XMPP is that it's resented as the SOAP of messaging.

This was my experience. It did end up doing what we needed and being able to use off the shelf libraries/servers at the time was convenient, but holy heck is the protocol chatty and annoying to debug at the trace level.

The primary similarity between the two is the use of XML. Maybe the general opinion is that XML sucks?

It's just not a great protocol.

Many years back I was working on a desktop client for XMPP.

The first problem I encountered was that XMPP does not actually send individual XML documents each time it sends a message. Instead it starts with a tag that is not closed until you actually turn off your client and close the connection. So you couldn't use a simple XML document parser for those relatively simple messages as it would read the start tag and never finish, but had to do some kind of streaming-based thing.

Instead of all the web3 nonsense, which still requires implementations working with interoperable protocols like this, that energy should be put into advocacy for standard protocols, or building apps that use them.

Yup, I second this. The whole web3 idea is completely pointless if the ecosystem is designed to be heavily centralized. Everyone is and will be fighting for a pie in the “platform” market. We need decentralization in the application side first, and that is only possible with standard protocols.

What an earnest and well-meaning BS post.... I mean seriously, XMPP is fundamentally flawed, and it's not coming back. What's more, we have matrix which is the protocol re-written from the ground up with the knowledge of XMPP and better design.

Matrix IS XMPP 2.0.

> XMPP is fundamentally flawed

how is it fundamentally flawed, can you elaborate?

I really wanted to like XMPP back in the days, but honestly I always ended up feeling that the protocol is just bad. This idea of opening an XML document at the beginning of the stream, then only allowing a subset of XML and all the mess with the xmlns, all that bringing really nothing to the table except complexity.

I think ideally in a good protocol, the server should not have to parse the content for the messages that are not targetted to itself (only the metadata useful for routing). the XML mess makes it impossible to do that since you have to validate the full document.

At the time I think this page was a good summary of the issues https://about.psyc.eu/XMPP No idea if this is still relevant though.

I've built an XMPP client for an internal application. My impression was the protocol was overly complex, starting with the lower level problems you describe ("streaming XML.")

It's been years since I've looked at. Maybe things are better now.

Well, I guess it kinda makes sense? Beats having to reinvent a tokenization format from scratch?

Of course, smaller messages (à la Matrix) probbably make more sense.

Ultimately the issue is that XML is a document markup language, not a purpose built streaming protocol. You can kind of force it into that role, but the parser ends up being overly complex and issue prone. It's like basing a chat app around creating a MS Word document of every message and sending it across the network.

There were several members of the core team, pre-XMPP effort in IETF, who wanted to change it to have framing. If not a length-prefixed model, a nil-separated one.

There were ideas to separate out the addressing/routing from the actual messaging, so that servers did not need to process XML and so that messages did not necessarily need to be XML.

There were even some very preliminary ideas on using the servers to potentially negotiate peer connections for arbitrary traffic.

Back in the early 2000s there were a _lot_ of self-hosted Jabber servers though, and there was push-back from early commercial interests on any protocol-breaking changes resetting adoption to zero. This resulted in Jabber 1.0 pretty much becoming the basis of the XMPP RFC, with XMPP adding new authentication techniques and internationalized JIDs.

Later on, there were efforts to establish alternative transports to accomplish some of these alternative transports and forms - HTTP endpoints to poll for messages, JSON mappings of the core messages, etc.

I would argue against PSYC's claims (or would have, back in the day) that the usage of XML in XMPP is not proper, however. Its proper, it just wasn't the best idea.

I'm not going claim that it's fundamentally flawed, but here's an anecdote. Many years ago, when XML was having its day in the sun, long before it was sidelined by the simplicity of REST and JSON, I was at a Java One convention listening to a speaker present on some new XML parsing API. After the talk, I approached the presenter for some post-talk Q&A to ask how one might use the API to parse the Jabber protocol, which may or may not be relevant to what XMPP is today (I haven't been keeping up.)

The presenter was unfamiliar with the protocol, so I had to describe how the xml document was opened when you establish a connection, and how elements keep getting appended to it, and how the "xml document" isn't really completed until you're all done and the connection is terminated.

They looked at me like I had two heads.

To them, XML didn't make any sense at all unless you have the entire document available all at once. After all, how on earth could one ever apply an XSLT transform to it, right!?

Good times.

That's what Jabber is? Streaming XML? Whoa boy...

> After all, how on earth could one ever apply an XSLT transform to it, right!?

There is streaming APIs for XML. Just as XSLT 3.0 can do streaming. Saxon has implemented it, for example[1]. I am aware, that you are talking about the past, but also the XML world moves forward, albeit slowly, since the community has gotten much smaller.

[1]: https://www.saxonica.com/html/documentation10/sourcedocs/str...

It was common to have to parse below XML (angle bracket counting) to convert each stanza to an XML element, then parse those as separate XML documents.

You'd also have to explicitly turn XML namespace support _off_, since so many systems didn't actually support them. XML defines well-formedness and namespace-well-formedness as two different things, and you didn't want to completely drop communication because the other side was sending a message that didn't meet the more stringent requirement.

Some implementations would figure out ways to incrementally disrupt and extract elements from the DOM - but this would sometimes cause resource leaks due to the design of the W3C DOM itself.

The expat had explicit support for parsing Jabber/XMPP messages very early, and was by far the most often XML component used for making libraries.

I still find it amusing that XMPP by design violates XML spec, thus requiring its own custom XML parser.

It uses a subset of the XML spec, it does not need a custom parser, any existing parser will do.

It does not violate the spec, but it violates the expectations of a lot of XML tooling for sure.

Matrix is not even a protocol. It is by far inferior to xmpp, especially regarding extendability, and the only flaw in XMPP is a rather lethargic council of elders that are not really interested in its development. This council is too be ignored, the base of the protocol is healthy.

That is interesting. Do you mean that the xmpp.org foundation is not a good custodian of the specs?

I think they are in some kind of lethargy, waking up each year and updating the compliance suite xeps.

TBH, (federated) chat/messaging is a solved problem since 1988 (IRC) or 1999 (XMPP), so I don't know what you expect

- group chats are shit

- message formatting is a mess

- device management is absent

- iOS apps can't really work with stock xeps

And the list is long, these are just the top problems. Luckily, this will likely change soon.

>Luckily, this will likely change soon.

Please elaborate.

Group chats, nicely working and with fallback for legacy clients. They are usable (without all features, of course) in Gajim/etc without doing anything:

Web client: https://xmpp.redsolution.com/upload/4bddf4f264f5c6577f16551f...

iOS client: https://xmpp.redsolution.com/upload/4bddf4f264f5c6577f16551f...

Message formatting (from web client, but is supported by all, actually): https://xmpp.redsolution.com/upload/4bddf4f264f5c6577f16551f...

Device management. Newly connected device is given a token, with a mechanism to prevent token duplication, and tokens can be revoked, resulting in device disconnection:

https://xmpp.redsolution.com/upload/4bddf4f264f5c6577f16551f...

This all is a work in progress, and we're aiming for an iOS version release in december or early next year.

On the Matrix side of this: rather than spending time responding in detail I'll link to last week's iteration: https://news.ycombinator.com/item?id=29152551.

The TL;DR seems to be the concern that Matrix was created by an existing team of folks who worked together (we built VoIP stacks for telcos), and because 50% of the people in the Spec Core Team who define Matrix's direction are from that original team, they have a disproportionate influence over how the protocol develops.

The reality, as far as I can tell, is that we are incorporating the best proposals from across the whole community, wherever they come from, and the protocol is progressing steadily under this open governance model, and folks are constantly experimenting with new features under prefixes, which they propose MSCs (Matrix Spec Changes) for, and then get incorporated into the official spec once they're approved by the Spec Core Team (which requires 75% consensus).

We spent ages setting up https://matrix.org/foundation and trying to enshrine a fair and neutral governance process for the spec, and as far as I can tell it's overall working. The thread linked above gives examples of ways in which the process is working as intended.

Translation from prspeak: one closely integrated group has outsized influence on specification, and development is dominated by over commercial entity.

This is not how healthy federated networks work.

It's how they all start. "Rough consensus and running code."

The question is where to take it from that.

You say Matrix is not a protocol, and then you go on and compare it to another protocol. By your own logic, you are comparing apples and oranges here. Make up your mind, either it is a protocol or it isn't.

Oh, and Matrix is plenty extensible, for example via custom event types: https://spec.matrix.org/v1.1/client-server-api/#types-of-roo...

Matrix is a product developed by one entity that can be installed on premise and has an api. Different instances can communicate with each other. Mattermost also has it but we don't call it a protocol.

This is just not true. There are four main Matrix server implementations today: Synapse, Dendrite, Conduit and Construct. Conduit & Construct are written by entirely independent different parties; all four share zero code; and yet they all can communicate together (modulo beta-ness in some places).

(And aside from that, Mattermost doesn't have federation; different instances can not communicate with each other. Although we hope they'll implement Matrix eventually :)

Both third party implementations have beta status and all entirely independent implementations will be screwed and will stop working each time you decide to significantly change the spec in your monolithic protocil.

I know, you consider xep structure of XMPP to be a problem, 'because it is hard to orient in it', but in fact it is what makes XMPP so durable and capable to work in a real federated environment.

Matrix is working in a real federated environment between hundreds of servers

... between server instances provided by a single dominant vendor, whose idea of federation is for everyone to update to the latest version of the spec or fall behind [1].

I can see zero problems with this decisive approach. /s

[1]: https://news.ycombinator.com/item?id=19421978

> between server instances provided by a single dominant vendor

No, any server instances provided by any vendor

> by a single dominant vendor, whose idea of federation is for everyone to update to the latest version of the spec or fall behind [1].

Server instance provider is not the same as software provider. Of course, you fall behind if you don't update. Time progresses, so does software.

> Of course, you fall behind if you don't update

In proper federated networks (email, xmpp), you don't.

Define proper :D

What I value with Matrix is progress

What you perceive as progress is just a sign of immaturity of an ecosystem. One player will ALWAYS move faster because he doesn't have to check with other participants, single-handedly self-defining specs and implementing them in own software. But reliance on this model will never allow another participants to enter, and it will always be a one-vendor vehicle.

Proper federated networks are ones that are already past this initial phase of fast progress and where participants have learned to work with each other.

Having progress (and thus the possibility of falling behind) doesn't say anything about the speed of progress.

> and it will always be a one-vendor vehicle.

You are clearly not speaking of Matrix: e.g. Element, Beeper, Dataport

How no progress ends up, can be observed with email and xmpp

> Proper federated networks are ones that are already past this initial phase of fast progress

That's made up by you. I claim proper federated networks are ones where you can fall behind

How is Matrix not a protocol? Because it has not been externally standardized?

I use both XMPP and Matrix regularly, and run servers for both. I wouldn't say Matrix is better than XMPP across the board, but that they have different trade-offs.

The main architectural difference between them is that Matrix provides "eventually-consistent cryptographically secure synchronisation of room state across a global open network of federated servers and services", whereas XMPP just passes messages from point to point (where one of the endpoints may be a multiuser room).

The main practical benefit of Matrix is that multi-user rooms don't have a "home" server; they're distributed across the homeservers of every member of the room. This means that though everyone may have signed up for a room identified as #shitposting:matrix.org, if the matrix.org homeserver goes down, the chat still works. People can't join it until a new name is published for it, but it works for everyone already in it. In XMPP, chats are hosted on a particular server, and if the server a chat is on goes down, the chat goes down.

The main practical benefit of XMPP is that it is much more lightweight than Matrix. Being based on synchronization of room state means that Matrix stores a lot of data, generally all messages and attachments back to when the room was created (or possibly the first time someone on your homeserver joined the room). Apparently it's possible to prune room history, but it's not done by default, and as far as I know, it's not officially supported. It's much easier to control how much data your XMPP server stores.

XMPP is also a simpler protocol, which means there is a wider variety of clients; while there are several vaguely viable Matrix clients, if you're not using Element, you are much more likely to have problems, especially with encrypted rooms. Of course, the flip side of this is that a lot of XMPP clients don't support all the extensions which make modern XMPP useful, either.

Both of them have trouble with multi-device E2EE key management, though Matrix has the edge. But again, if you're not using Element, you are likely to have problems.

> whereas XMPP just passes messages from point to point

So does Matrix. All the other qualities are build on top of that. XMPP also enables e2ee, message synchronisation and an "global open network of federated servers and services".

I'll say that we did have efforts early (pre 1.0 Jabber) to do leaderless communication and build rooms on top of it.

Back around 2000 getting information on building eventually consistent systems was much harder than it is today, and leaderless versions of that doubly so. Those who worked on it felt like they were inventing half the techniques themselves, had difficulty recruiting (teaching) new people, and eventually burnt out.

If those involved had gotten it to work, it would have just been core infrastructure for eventually consistent ordered events, with UX concepts like "rooms" built on top as common business logic. Or in other words, distributed apps.

It'd take the flawed XMPP protocol any day over the ad-hoc implementation of Matrix, which hardly even deserves to be called a protocol. Matrix feels like something made by making things up as the wrote the code, there is no coherency at all.

> ad-hoc implementation of Matrix, which hardly even deserves to be called a protocol.

Please explain why you believe this.

This is the spec: https://spec.matrix.org/latest/

These are the spec change proposals: https://spec.matrix.org/unstable/proposals/

Jabber was written by making things up as we wrote the code, and then pitching a prototype to a core group for revisions.

XMPP didn't really pitch any client or server protocol breaking changes.

Since the messages themselves were just addressed XML elements, we were able to do revisions over time, such as the first group chat being replaced by MUC or adding in the ability for XHTML-formatted rich text.

What would truely make XMPP stick and be a killer app, would be if enterprise interoperability could be shown between apps like Microsoft Teams, Slack, and Discord. The amount of friction that enterprise consumers like myself have to deal with because none of these things interoperate is immense. Making them interoperate at a basic level via XMPP does not seem like it would be overly challenging.

the technical challenge, or lack thereof, is not the reason why it's not done I suppose.

It has been done. Chat apps once did work together and you had multi client apps that would connect to everything. Then they slowly died off and became discontinued because people don't actually care that much about having multiple chat apps. Thanks to modern push notification services, you can receive messages from all IM services without having to have those apps actually running.

You're looking for Matrix. XMPP wants nothing to do with 'enterprise' apps that rely on potentially backdoored encryption standards. The people who care enough to enable OMEMO on their chats aren't going to do so with the intention of undermining their communique for a slightly better level of compatibility.

but what if they don't want to interoperate with each other

Idea: market interoperability as a metaverse thing.

Once upon a time, google chat (neé gchat) _was_ that killer app. For a brief time there was glorious federated communication between several IM vendors, self hosted open source servers, and gchat.

Then google killed it all by dropping xmpp/jabber support in gchat, and others soon followed.

I don't think there ever was federation support? Or maybe you were an earlier adopter than me, and talk of a time before they killed federation?

At one point with Apple’s Continuity I had Gchat, Facebook messenger and SMS on a single window in my MacBook.

Yes, but with only a single xmpp account, that federated access to gchat, fb?

Yes, at one point I ran my own prosody jabber/xmpp server and regularly chatted through it with friends who were gchat users. Even had presence (idle , busy, etc) support! It worked amazingly.

First they killed federation, then they killed xmpp support entirely.

Teams, Slack and Discord don't want to talk to each other. They want their own silo with their own users. XMPP, Matrix, or anything, for that matter, can't fix that.

Have you looked at Mio? https://youtu.be/0mAXRUgJtgg Basically does what you're talking about

Reminds me how Slack killed off their IRC bridge...

The whole issue with XMPP is that yes, in theory you could do e2ee accross multiple servers and devices but only if all the servers support the right extensions and the clients support them properly.

This is not a problem with XMPP, but any open ecosystem. There's no way to force third-party developers to implement stuff, especially when they are open-source volunteers working in their free time.

XMPP does have this feature parity issue, though there is a good selection of modern active XMPP clients across platforms with important features like end-to-end encryption and calls. But you're right - there's no way to stop people trying to use clients like Pidgin, which have been essentially frozen in time for a decade.

Matrix is newer, so has less diversity, and lots of resources to put into the Element clients. However there certainly is exactly the same problem growing in the Matrix ecosystem too - there are many features supported by Element that are not (yet?) implemented in popular alternative clients such as FluffyChat.

The best you can do is ensure that when two clients communicate without the same set of features, that you degrade gracefully and securely (e.g. the worst case I can imagine would be E2EE that silently becomes unencrypted if not supported by your contact - thankfully that's not how it's done) to the best common feature set between the two.

I think a major problem is there isn't (last I looked) a clear set of feature tiers or a collection of XEPs that are given names and tests.

Instead of "oh MS teams supports up to xmpp2017" it's just a crapshoot.

Here's what you're looking for: https://xmpp.org/about/compliance-suites/

Compliance suites are reviewed, updated and published annually with the recommended set of features across a range of different categories.

Well there go. I guess now that I think of it, last I looked was MUC support in 2012 or so.

Looks like it's on the list but as "* Support can be enabled via an external component or an internal server module/plugin."

So...a crap shoot, haha.

That just means that some servers may have it built in (all of the maintained ones that I'm aware of do), but that you can still claim compliance if you support a general plugin system and delegate your multi-user chat implementation to an external plugin (because of the way it's designed it doesn't necessarily have to live on the same server or in process with the XMPP server). That seems like a fine implementation detail to mention and not a problem…

You can say that..but it was a problem. It was not trivial, despite what the spec says or what is claimed now. If it was, more people would have used it.

I don't know what you mean "is a problem"; what's a problem, setting up groupchat? Everything uses multi-user chat and it's built in to all the servers I know of (but can generally be run as a standalone server if you prefer, which is what this means)? You can just enable it in your config and it just works on at least Ejabberd and Prosody, probably others.