Twitter and the Internet War
vanityfair.com
vanityfair.com
This is probably a new low in journalism.
> If Twitter’s co-founders had known what it would become, a third former executive told me, “you never would have built it on a Fisher-Price infrastructure.” Instead of building a product that could scale alongside the platform, former employees say, Twitter papered over its problems by hiring more moderators. “Because this is just an ass-backward tech company, let’s throw non-scalable, low-tech solutions on top of this low-tech, non-scalable problem.”
It sounds like the original people who spoke to the journalist left the impression that there were deficiencies in the base technology which may have led to the current circumstances.
A good example is the way the 'quote tweet' feature is often used to start a pile-on. It's not a bad feature in itself, but there is a significant portion of usage that lends itself to starting food fights. Could things be improved by, for example, a setting that limits quote-tweets to people who follow you? There's a lot of dynamics like that which can be explored.
However I think twitter used to have a lot of weird secret features like how DM's could accidentally become public if you formatted them wrong, and another secret rule to twitter could be a bad idea.
> Twitter was designed, from Day 1, to enable any random person to send messages directly to any public figure. In other words, from Day 1, it was designed to be an abuse and harassment engine. It's not a bug. It's a feature. All that abuse and controversy is how it gets clicks and money.
http://jeff-vogel.blogspot.com/2016/04/how-i-deal-with-haras...
Twitter's feature set is perfectly optimized for harassment and abuse. Changing it to prevent harassment and abuse would kill the product.
Which topics engage people's identity depends on the people, not the topic. For example, a discussion about a battle that included citizens of one or more of the countries involved would probably degenerate into a political argument. But a discussion today about a battle that took place in the Bronze Age probably wouldn't. No one would know what side to be on. So it's not politics that's the source of the trouble, but identity. When people say a discussion has degenerated into a religious war, what they really mean is that it has started to be driven mostly by people's identities.
...
More generally, you can have a fruitful discussion about a topic only if it doesn't engage the identities of any of the participants. What makes politics and religion such minefields is that they engage so many people's identities. But you could in principle have a useful conversation about them with some people. And there are other topics that might seem harmless, like the relative merits of Ford and Chevy pickup trucks, that you couldn't safely talk about with others.“
At the same time, her defenders say, Harvey has been forced to clean up a mess that Twitter should have fixed years ago. Twitter’s backend was initially built on Ruby on Rails, a rudimentary web-application framework that made it nearly impossible to find a technical solution to the harassment problem. If Twitter’s co-founders had known what it would become, a third former executive told me, “you never would have built it on a Fisher-Price infrastructure.” Instead of building a product that could scale alongside the platform, former employees say, Twitter papered over its problems by hiring more moderators. “Because this is just an ass-backward tech company, let’s throw non-scalable, low-tech solutions on top of this low-tech, non-scalable problem.”
Setting aside the whole "Ruby/Rails is slow" discussion, I would've loved to see more elaboration by these sources about how a web framework makes moderation and policing "nearly impossible". Compared to what? The PHP scripts that became Facebook?
My understanding is that Twitter moved away from Ruby/Rails back in 2010-2012 anyway, making the comparison more confusing.
https://www.google.com/amp/s/techcrunch.com/2008/04/23/amate...
Of course, it is a stretch to suggest that in ten years they couldn’t reinvent the architecture.
It took them many years to get beyond the “fail whale”. And it is surprising who they thought of as experts at scaling:
“And this week they hired two scaling experts – John Kalucki and Steve Jenson (“known for his work scaling Blogger and Blogspot”).”
No offense to those guys, but Blogger and Blogspot were also notorious failures at scaling. Bloggers problems were not fixed till a year or two after they were bought by Google.
Steve and John were both good developers that wrote some seriously critical components of the system and were a big part of getting twitter more stable. But reading Techcrunch to figure out what was happening inside of Twitter is asking for serious misinformation. At that time it was very common for people with no idea what they were talking about to write articles about what was going on inside of Twitter, yet do so with authority that they didn't really have. By the time a team of reliability focused people started to get hired Twitter was finally getting smart about not letting employees/outsides control the messaging sent to Techcrunch and the like.
Twitter has completely reinvented the architecture since I left so that is not impossible by any means.
Lets be clear about the rate that Twitter was growing though. In any 9 month window while I worked there they never failed to grow an order of magnitude in request volume. When I started we would have spikes of a few thousand requests per second. We had a major site outage at 451 tweets per second and we simply couldn't keep up. 9 months later we were spiking 50k requests per second, and we never dropped below 451 tps (barring outage). By the time I left both of those numbers simply couldn't be seen in the graphs anymore.
Every time we got technology in place to deal with the scaling issues we were already behind the curve because of how fast things were growing. It felt like being in a car that has amazing 0-60 time, but it just kept accelerating well beyond 60mph.
There are financial systems that handle millions of transactions per second because their designers sat down and designed systems that could scale by magnitudes.
Ultimately it wasn't "build a db that handles x writes" though. That is actually a fairly known and solvable problem. Our issue was that we were migrating from a design model that encouraged rabid development by using existing patterns to a model that would scale up much better but required a lot more planning and design, all while dealing with management that expected the feature development cycle to not slow down at all during or after the transition.
https://job-openings.monster.com/Full-Stack-Senior-Software-...
https://job-openings.monster.com/Senior-Web-Engineer-BLAW-Ne...
Bloomberg is fairly well known for having top notch technical capabilities (in their general space). What they build tends to work, work well, and have high reliability.
Whoever was quoted has precisely Zero current understanding of moderation and the challenges it faces.
And worse - they haven't reflected on what "technical acumen" resulting on anti-harassment tools means.
Effective anti harassment tools can suss out intent, and by nature are also censorship tools.
Effective tools which can identify intent are so far past invasion of privacy, that they may as well be categorized as assaults on specific ideas.
If twitter does come up with this magical tool, then it's a sure bet that other nation states will also have this magic algorithm.
This rule applies to all lesser tools for "anti-harassment".
The solution sadly IS hiring more moderators, and journalists vetting.
Which I suspect is impossible at the scale and message type twitter operates on.
But what if following had only been bi-directional for example? What if the platform never displayed retweet or star count? What if muting and blocking had been built in from the start? What if you could only @mention people who follow you?
It’s very possible that twitter would never have become Twitter if these things had been different, but that’s also my point. So while it may not be a technical root cause in the classic coding sense, I can buy the argument that many of twitter’s early technical and design decisions (as well as their early platform policies and who the first adopters were) led to their present difficulties
The problem wasn't ruby. The problem was the way that Twitter used Ruby. We had one big mono repo with every single function and every form of business logic baked into a single place. That logic relied on monkey patching and all sorts of crazy horrible glue to keep it working together. Every time we had to scale up we would glue infra in place to keep things working while we came up with a real solution (which never really materialized).
In my time there we had memcache instances which held timelines. Populating them took hours/days and while they were unpopulated the site was offline. Rebooting/restarting the caches was simply not an option. We had a data sharding strategy that was temporal. We would spin up a new database cluster every few weeks to handle all of the incoming tweets and failing to spin up a new cluster in time meant we would have a global site outage. Don't even get me started on the "load bearing mac mini".
In reality the only problem rails really contributed on its own was that it could only process a single request per process at a time. Each machine would spin up 16 or 32 processes to handle requests in parallel but each process needed its own connection to the database, to memcache, etc. At one point we had something like 100k processes all trying to talk to a single mysql master. Much of this could have been mitigated by better design of course, but rails encourages models that don't scale up to crazy dimensions.
In reality moderation was virtually impossible because we were in a 24/7 fight with ourselves about how to keep the system alive for the next couple of days. Constant infighting, managerial changes (I had 9 different managers in 3 years), focus changes (we didn't finish the last major site redesign before starting the next one) and a general unwillingness to pause features long enough to stabilize the system meant we were always on the losing end of a infra battle.
Every so often a person would have a brilliant idea on how to solve our scaling issues. They would then disappear into a corner to invent yet-another-bird-themed-datastore. After a few weeks/months they would appear with a magical new thing that would fix all our problems and would make everybody happy. Every single time it would fail.
Having a team that is not the main team design something means that they likely didn't understand the state of the thing that they were replacing. The thing they were replacing was a bucket of edge cases non of which they knew about. The scale never looked like what they expected because in the meantime the load had changed. This was compounded by the constant desire to hire somebody external that could solve the problem for us. They would come in with ego and a feeling that they had a mandate to replace it all. Eventually they would learn just how fragile and complicated the system was, only to then be considered old guard enough to be replaced by the next wave of experts. =/
But the number one killer was that every single thing was baked into the mono repo so it wasn't like they could have just easily shimmed in something to replace the old thing. All the while that they are building in a change to the data store another dev has added 15 new features that they now have to port over. In the time it took to port those over another 20 had been added.. etc.
Just getting the okay to pause feature development was like pulling teeth and it only bought you a few weeks at best.
Can I get this on a T-shirt?
We had an API service, a web interface, the legacy web interface that was still used for select devices because the new UI didn't quite work right on them, the even older legacy interface that was necessary because a bunch of badly behaved early day clients still relied on the functionality and they were popular enough that turning them off would cause outrage, the "zero" interface used in countries with low bandwidth capabilities, the mobile interface.
Each interface had to implement all the different variations on functionality. Timelines with inline tweet rendering (automatic expansion of images, etc), list (alternate view time lines), the whole following graph (duplicated for lists as well), verified users and all the infra around that, search, public/private designations, direct messages, notifications via email, text message, and mobile app, favorites, retweets, replies, plus a slew of statistics and information tracking data integrated directly into the site.. Thats only the user visible stuff. There are a TON of experiments and projects that run behind that interface in a way the user will never completely see.
We heard over and over that twitter was so simple that it could run on a laptop and every time it reminded me just how clueless most developers are when it comes to seeing the body of work needed to make something like twitter work, even more so at the scale we are talking about.
I'm not sure if Twitter can ever put pandora back in the box, but at the very least, requiring a mobile number is a start. It's hard(-ish) to generate thousands of bots if you have to have a unique phone number with multifactor for each signup.
Yes, the platform is a failure in so many ways - failure to protect the identity and safety of the people on it. (Though as someone who has served a subpoena to Twitter, let me tell you something - the legal team isn't exactly handing over data easily. It was a huge pain in the ass with tons of individual privacy concerns the whole way.)
And the platform is failing revenue-wise, yes.
But if it's about the Internet War, so to speak? Twitter is at the top of it all. No one has to like the externalities - I sure don't - but their influence is undeniable.
What value does Twitter provide to ordinary people? I have friends who are celebrities in their industries or trying to market themselves-- but other than that it seems like the most hostile place imaginable. Why would I want to participate in that?
Exactly what this "stuff" is comes down largely to your own ability to curate a decent list of providers of "stuff". This doesn't necessarily differentiate it from any other large provider of "stuff" hosting enormous groups of humans messaging each other and broadcasting "stuff", e.g. reddit, but it is still host to certain groups of humans for which Twitter is their platform of choice to air this "stuff" out.
Product announcements, celebrities, certain subsets of certain industries (e.g. I used to follow a curated collection of Mac indie developers back when I was 15 and wanted to be one), activists, customer support reps, politicians, whatever, if you can think of it, you can probably find it has a place somewhere on Twitter and that is what makes Twitter valuable to you. It might be entertainment value, knowledge, news, lifehacks or whatever, and maybe for you personally you have much better sources for all of that, but Twitter just happens to be the place others get some or all of this from.
Then there are the people that treat it like a diary, or journal, or a place to vent. For them, Twitter isn't for following a curated collection of people, for them Twitter is just a place to talk. You know, a messaging platform.
Full disclosure: haven't personally used Twitter in years other than as a file transfer host from a particular video game console lacking decent file transfer options. It is valuable to other people, I just prefer my "stuff" in a different format than what Twitter provides.
I don’t tweet. I do read newspapers. Twitter appears to have made journalists quicker at spotting trends. (By making it easier to find quotes, it also makes them more productive.)
And lots of people tweet when they're pissed about things, so Twitter search is the first place I check to see if a major service is down, and why.
A lot of information security folks tweet as they working or to release information. Heck, just following the Google Zero folks on Twitter had given me early hints of major security issues.
There are some hilarious folks on Twitter, many who are just random folks. Especially if you like absurdist humor, or dumb jokes (or both), it's amazing.
There are people of deep expertise posting and conversing on their subjects. Early on in the DNC hack saga, @pwnallthethings described in great detail the public evidence that it was Russia.
Finally there is local stuff like, will any of my favorite food trucks will be near my office today? What are the custom flavors at the gelatto shoppe today? When exactly is likely to start raining in my area?
There's a lot more. But to get a lot of value out of Twitter, each person needs to be following the accounts that are best for their interests, or know how to use the search well, or both. Either way, there is a learning curve.
The fast way up that learning curve is to experiment--be quick to follow folks who look interesting, but also be quick to unfollow the ones you don't like.
Their business is in the best condition it has ever been in. They're finally profitable. Their business appears to have reached full self-sustainability. Dorsey has been driving it toward that since he returned. And not just at a low level, it looks like they can punch upwards of $450m-$500m in net income on an annualized basis in the near future. They also have $3.8 billion in cash.
Can they drive much revenue growth from here, without user growth? That's going to be very tough. The profitability gives them a lot of breathing room to work from.
Using Facebook, I have always felt that the website tried to "force" me into where it thought I belonged in the social graph. Twitter has a much lower barrier to entry. I prefer the lurker-first philosophy. Facebook tries way too hard to engineer interactions (so does Twitter, but I think it's not as bad).
If they really wanted to solve the problem all it takes it consuming their own firehose API and writing back to delete flagged content.
You really don't need to insert the moderation into the "backend", you just have to want to solve the problem, instead of accepting the problem because it fuels your growth.
The odds are against Harvey being the first person in human history to solve this problem without false positives and negatives.
I know where I stand on this one: be biased against the mean-spirited.
To be fair to Twitter they are consistent here. What looks like false results to an outsider is clearly policy.