Slack is down
status.slack.com
status.slack.com
Here I sit, with my colleagues, connected to the same network, with the same application open, but we cannot write to each other, because some backend server in the US have a problem.
How in the world could this not have been fixed yet? Our internet backbone addiction is seriously crippling us when things go sour. Not thinking specifically about Slack, but other more mission-critical tools that just... stops working because they cannot ask the time from their backend server.
We seriously need P2P systems (without blockchains please) that can be used by normal applications to solve issues like this. Offline and local network communication should be obligatory today.
Using a service like slack has downsides, as you are seeing now, but running your own service or using P2P has other downsides that we have deemed are even worse.
Maintenance, lack of client support, network access requirements, lack of features, etc.
Obviously right now the downsides of a central service is front and center, but we can't ignore the downsides of the alternatives.
IMHO a huge problem with the Internet since the late 90s is that business models have replaced protocol RFCs.
As my full-time work is working on P2P systems, I would love to hear more about what you think gets even worse with P2P systems compare to hosted services.
Lack of client support, features and so on is more a sign of lack of adoption, rather than something fundamentally worse with P2P than hosted services.
Maintaining history is also tricky in a P2P system. Can I search and view all chat history from when I am not there? A hybrid system could, but then you would still be limited by the host being online and accessible.
Also just the amount of moving parts. With P2P, you can get strange issues where some people can talk to some people and not others. It can be hard to track down where those connectivity issues are.
My company moved from self-hosted IRC to slack a few years ago. No one had time to maintain the system (backups, updates, troubleshooting, etc), and it was cheaper and easier to pay slack than to hire someone to maintain an IRC server, or to pull someone from their actual job to maintain it.
The STUN protocol solve most of this issue[2], webRTC is working fine in the majority of enterprise network.
> Maintaining history is also tricky in a P2P system.
It's perfectly possible to build a distributed database, actually that's exactly what blockchains are (OK, I know , we said no blockchain but I merely use it as an example ;)).
> My company moved from self-hosted IRC a few years ago. No one had time to maintain the system (backups, updates, troubleshooting, etc), and it was cheaper and easier to pay slack than to hire someone to maintain an IRC server, or to pull someone from their actual job to maintain it.
This isn't p2p, p2p shouldn't be confused with self hosted. Nobody has time to set up and maintain a server of any kind, but everybody was seeding torrents in the late 2000 and nobody complained about it being too much work.
[2] while the TURN protocol gives you the “hybrid system” for free (development-wise, you still need a hosting plan in that case, but it can be mutualized between all your applicative needs).
- Security in a P2P system is harder to get right. The system is distributed and each member has to trust more parties. This significantly increases the attack surface compared with a client that just needs to trust a single server.
- Not everyone is connected all the time, so network partitions happen constantly. It's hard to maintain a consistent state such as a chat log, not to mention anything more real-timey like the state of some real-time game.
- STUN and other NAT traversal techniques don't always work. NAT is horrible, creating connectivity between NATed clients is always a dice roll.
- Bandwidth - A dedicated server can have a lot of upstream bandwidth (to the clients). In homes, it typical to find (in some countries at least) connections that are heavily biased towards the downstream. So a 100Mbps connection (downstream) might have a 2Mbps connection upstream. Now say you want to have an 8 people video conference call, and all of your users have a 2Mbit uplink and need to send 7 copies of their video stream... What if it's a 20 person video call? What if half of them are on mobile?
- Bootstrapping - every P2P system still needs to have dedicated servers for bootstrapping. There are no pure P2P systems AFAIK.
- Client versioning and patching - a lot harder in a P2P system. With a central server - you just update the server and everything works. With a network of clients, things are trickier and you run the risk of partitioning the P2P network between different client versions.
- Monetization in P2P is hard(er).
- the bandwidth problem is real, but there is no more technological limitation[3], it's merely a commercial problem: client have no use for huge uplinks, then providers don't sell them. It's a bit of a chicken & egg problem.
- the monetization problem is the root of all sins in that case …
[1] on the client versioning & patching side, you underestimate the complexity of distributing a web client, with CDN, browser caches and clients who never reload their page, you can also end up with a lot of client lagging behind in updates.
[2] and centralized system also have their own issues (scaling anyone ?)
[3] ADSL was asymmetric because the asymmetry widened the reception bandwidth, which would otherwise be too low. With optical fibers, you don't have such limitations.
We used to run Microsoft Communicator (OCS) and availability of that service was far worse than what we've seen with Slack.
NAT's are the real pain for this sorta stuff and why people use stuff like skype, slack, discord, etc. When the majority of folks have IPv6 connectivity a lot of things should become much easier (at least for home users and people not behind a frustrating firewall they don't control)
I'm surprised maintaining a small IRC server was that much of a time sink. User problems is understandably another story.
Anyway, this is at least the 7th or 8th time Slack has been totally or partially down during working hours.
This is very, very far from "rare outages".
We can have the best of both worlds.
I just don't see it being worth the extra effort of self-hosting a redundant copy of the system, and then keeping that system up to date with the latest software for security patches / features / bugfixes. Especially when the cloud version has n x 9's of uptime.
>If we fall short of our 99.99% uptime guarantee, we’ll refund customers on the Plus plan 100 times the amount your workspace paid during the period Slack was down.
Source: https://get.slack.help/hc/en-us/articles/204113126-Plus-plan...
Maybe an emergency mode in the Slack application. When users cannot reach the Slack servers (when they're down or bad network connectivity) the application exposes a P2P connection and looks for other P2P instances.
It would still need a centralized tracker (like a torrent tracker) to help P2P connections find each other through the internet.
All this is easier said, then done!
Why have a mothership at all? Why couldn't software clients simply talk to each other? We already have hash-based addressing systems that can find known peers anywhere in the world (e.g. Tor).
I'm also pretty sure all of us wrote this in CS 101. I sure did.
We manage over 900 virtual systems on many, many hosts and file systems and have next to no downtime because we work really hard to make that the case.
SIP for instance is perfectly happy to do ip<->ip calls (well ignoring NAT issues - one day we will all have ipv6 addresses....)
This really is a solved problem, people for various reasons have decided to rely on external services they have no control over. I don't think we need to throw yet another protocol at the problem, instead we should focus on making what we have better and easier to setup and use.
But, to be brutally honest, I doubt the backbone of our internet will change at all until we have a serious, blatant cyber war wherein the vulnerable aspects of our HTTP-based internet is taken advantage of.
Problem still stands, when your self-hosted instance/software goes down, I still can't communicate with people sitting literally meters away from me, connected to the same WiFi. It's incredibly stupid.
That doesn't seem like a problem that software can solve.
P2P networks can also work even if you're not connected to the same network. Point being that it's a direct connection to the person that you're communicating with, with no indirection.
* At one point we had to re-login to the client, because our ssl cert expired (let's encrypt), I think.
These issues might be because we ran the server from a home internet connection, but I run a lot of things from home without issue.
It doesn't use a global blockchain, but there is a more fractal internal append-only log structure that logs data from peers you've connected with.
Now if we can combine these apps with routerless ad hoc or mesh networks, that would be a very good start towards an unstoppable communication platform.
* https://www.freebsd.org/cgi/man.cgi?query=talk
* http://netbsd.gw.com/cgi-bin/man-cgi?talkd
(-:
We really need more systems that can do P2P well (centralized is just a special case of P2P - you need a few robust, replicating nodes with high connectivity). The routing, replication and persistence of messages is completely orthogonal to the physical nodes and TCP channels. Unfortunately systems are not usually designed this way.
The problem has been solved to a large degree by some systems in the past - Croquet comes to mind, though there are probably others. IPFS is getting there is a roundabout way. Another interesting protocol/system scuttlebot.io.
However, others in my team are very dependent on the part of Slack that is less like XMPP or IRC: the built-in persistence that lets them scroll back or search for stuff that happened while they were away. This is clearly important to some people, who also bodged together persistence for XMPP or IRC using various bots or artfully deployed console clients in screen/tmux sessions on remote servers. I always found it strange, since I expect chat to be ephemeral.
For asynchronous communication, I wish all these folks had learned how to use email or usenet properly, since I think threaded discussion on mailing lists or newsgroups was far superior to what we have now in any of these popular replacements.
I still don't quite understand whether there is a majority opinion that "rooms", "channels", or serialized message boards are somehow a better UX for the average mind, or if it's really just the distortion of rent-seeking businesses always building this form that is easier to control and monetize for some reason.
Have you had issues with your client taking a long time to negotiate a connection over Slack's XMPP gateway? I had that issue with Thunderbird's chat client and ended up having to switch over to the IRC gateway instead.
I'm not saying that an IRC experience comparable to Slack can't be achieved. I'm saying that it is so much work to do so that, for many organizations, just paying Slack a few bucks a user makes a lot more sense than sinking scarce and vastly more expensive engineer time into reinventing that particular wheel.
You mean like communicating with people?
>"relatively rare expertise"
Do you realize that IRC has been around for 30 years and that a lot of open source projects have their own IRC channel?
Plenty of people were able to use IRC in the past without IT help or training. They just installed a client like mIRC and figured it out from there.
"And yet so many people seem to prefer IRC! Why is that, do you think?"
You do realize that IRC is practically synonymous with Hacking/UNIX/FOSS culture no?
Since you don't seem to be big on providing any actual data for your claim, here are some substantial projects in the FOSS community that use IRC:
Debian, Docker, Prometheus, Django, Angular, ReactJS, FreeBSD Node.js, Mozilla, Haskell, Golang, Openstack, Erlang, Ubuntu.
Why do you think that is?
My last comment was pointing out the further absurd and unsubstantiated claim you made that "so many people seem to prefer Slack!" I think it's a pretty good illustration of "so many people" who don't prefer Slack.
Probably, but the OPs case was that they were trying to get work done.
> doing so requires so much relatively rare expertise [..] support investment [..] not cost-effective
"apt-get install ircd"?
That's Debian for, "I'm getting too old for this shit."
If you are sitting in the same room with your colleagues, what do you do when there's a power outage? Sometimes there really isn't anything to do but wait. If those kinds of outages were less rare, you'd need to figure something out to smooth productivity. Whether that is buying a generator, or finding things to do regularly that don't require electricity depends on your business.
We have distributed systems that are used by normal applications to solve issues like this. When we need to, we put an ActiveDirectory replica in every satellite office. Frequently organizations forget why they spent the money on so much redundancy in the first place, re-centralize stuff, and it might be years before they experience a big enough outage that reminds them why they were "wasting" those dollars on redundant systems.
As you say, there are plenty of systems that could be made less fragile than they are. Slack is specifically interesting because they made a number of "more central" design decisions, even though chat as a protocol is easier to make "more distributed". What did they get out of this tradeoff? A ton of well packaged features and integrations with other systems.
Could the same thing be recreated in a more decentralized fashion? It is theoretically possible, but it could be years before we see a product/system anywhere near as functional. I would love to see an on-prem Slack product. Atlassian offers the closest suite I've seen that fits this bill. It is not easy to operate these systems at scale.
There are a lot of tools that should work better offline. But a remote collaboration tool is not a good example of that. I've done work on secure private P2P messaging systems, and when you want that feature set, you really want that feature set.
But when I use Slack, it's because I don't want to have to be in the same building as all the people I'm working with. There is no way for this to gracefully degrade with a network partition. My argument is that central coordination and audit (with Slack) is not an edge case, it's the primary case.
With a hosted service you don't get that level of control, your software updates when the provider wants it to, not when you want.
Unfortunately we use Skype for business. Uptime is great but other features (like sending a god damn message) are frequently broken.
The mental acrobatics to justify not changing the status are often astounding: "One customer can load a partial page from Timbuktu, so we're not down."
Compare to Amazon... an hour, two?
But it's Smmooooootttthhh Sailing :sunglasses:
I think these kind of pages are usually updated manually by incident response teams.
* https://news.ycombinator.com/item?id=16109335
* https://news.ycombinator.com/item?id=16109132
* https://news.ycombinator.com/item?id=16109140
Granted - I've never built or run any system on such a scale as Slack - but I think that I would have either 1. learned from past experience 2. built some anti-fragility into the thing...