Collaborative filtering doesn't work for us | Hacker News Reader

Collaborative filtering doesn't work for us

about.chatroulette.com

157 pointsandreyk104 comments

This doesn't even come close to the power of (not even particularly) modern collaborative filtering algorithms.

CF doesn't need to assume that A talks to B well and B talks to C well implies A talks to C well- rather, it typically tries to learn an embedding for A, B, C that predicts their interactions effectively.

In the simplest form, this could be applying SVD or nonnegative matrix factorization to the user interaction matrix, truncating to get a low-rank symmetric approximation. More advanced forms would try to predict the interaction as some function of the two user embeddings, possibly even a neural network.

This only shows that a particular feature based on 2-chains of interactions isn't useful, whereas the standard approaches I described can try to infer information across any number of connections, and don't necessarily assume that 2-chains as described are indicative of a particular result.

Skip all that and cut to "a neural network." Presto.

I wish I understood this simplicity when I first got into ML. I was always annoyed with claims like "a NN will solve your problems." Oh really?

Yeah. Really.

You can learn the theory later. Sure, applying SVD or nonnegative matrix factorization to the user interaction matrix, truncating to get a low-rank symmetric approximation makes sense now. (I have to pause and think about it for a little bit, but I do indeed understand each part.)

The cool thing was, I didn't need to. The theory helps inform my future ML training runs. But I was able to get most of the magic without knowing a damn thing.

Literally just throw Pytorch at the problem and screw around until it starts giving results. You'll be amazed and shocked and delighted how far that gets you. (And all the professional mathematicians and statisticians will be disgusted, but who cares?)

Actually its one of those problems where it is worth not skipping. I have seen people applying neural networks to transform embeddings before doing a final dot-product where it could be proven (and indeed was confirmed in practice) that all of that was equivalent to a direct embedding.

Furthermore, SVD type of algorithm are orders of magnitude faster than NN to train (which is why they become used to train word embeddings, see Glove).

If a NN is equivalent to $sophisticated_algorithm, that's a positive, not a negative. It means you can blindly let the NN do your thinking for you, and come out ahead.

You're wrong about training times. You can get great results from far less than 117M params. My tiniest music model (similar to https://soundcloud.com/theshawwn/sets/ai-generated-videogame... but not the same one) was around 2M params. Trained it a couple days on my laptop's old CPU. Worked fine.

I think people love to exaggerate their own importance, and in the importance of theory. There's a certain myth of The Smart Hero, where intelligence alone is the decisive factor that saves a project.

Nope. It's determination. Bet on determination every time. And determined people can get very, very far with NNs.

This is totally counter to my experience. NNs are a real pain involving architectural searches, hyperparameter searches, etc. They take forever to train. Sometimes it's even difficult to get them to converge. The easy magical things are GBTs. If you have a classification problem that isn't super non-linear, try a GBT first. There's hardly any tweaking and the training is fast.

I'm surprised our experiences differ so greatly. It's true that you have to be particularly determined to get results, but it's not true that they take forever to train. At the height of my productivity, I was aiming for at least three completed experiments per day, and always made sure to have at least one going while I was sleeping.

It's a bit like saying it takes forever to grow tomato plants. Well, yes. Yes it does. Welcome to gardening.

ML is problem-space gardening. Yep, it's hyperparameter searches. Yep, it's architecture searches. No, it doesn't take nearly as long as you're implying, because you can quickly narrow the problem space in log(N) steps. (It would be foolish to do a hyperparam search in linear increments.)

Resources have also never been more plentiful. TRC literally gives you 100 TPUs when you sign up. https://jaxtputest.vercel.app/ That means you can (and I did) run 100 training sessions simultaneously. https://www.docdroid.net/faDq8Bu/swarm-training-v01a.pdf

Tbh it sounds like you got lucky on your first outing with ML and your first NN model worked well enough to deliver a success. Congratulations on achieving this rare milestone.

While I’ve seen many data scientists fall into the trap of spending 1 week per iteration even when it’s clear the change produces negligible result. If you stick with the field long enough you’re likely to hit a dog of a problem that should have worked, but ultimately did not.

The areas where ML is advancing fastest also suffer from elusive solutions. Language models and image recognition are only recently tractable problems - the folks trying to advance these fields do not experience magic NN improvements.

Oof. I completely agree that one week per iteration isn’t good. Thanks for pointing this out — I need to write up something about how to be effective.

1 week is fine for a production training run. But that’s after you’ve locked in your hyperparams on small prototypes. Small prototypes should take no more than two days to give results. If your prototype absolutely must be a large model, you should have many (perhaps even a dozen) attempts training simultaneously.

It’s hard to imagine someone spending an entire week waiting to see whether their latest tweak worked, then tweaking again and waiting another week…

I had models like that, but they were exploratory side projects, not my primary research focus. It’s stuff you try for fun, not for work. That way a week is “whatever” if it doesn’t work out.

But when you’re hunting for a result, no way. Prototype results are measured in “N completed runs per day”. Which means you should have multiple runs going at all times.

It gets tricky to manage, but… I don’t know. Either you’re right and I got incredibly lucky, or determined programmers can do the same thing I did. I’m not that smart.

> one week per iteration isn’t good.

Try training Imagenet. For some reason this is called a "medium" sized data set.

I do want to point something out though that's a big problem with a lot of your comments. We're seeing the results of what happens when you "just implement a DL pipeline." Lots of people are using these powerful tools without understanding. It's like handing a monkey a machine gun. We're seeing that models don't generalize like people think they do. We're seeing people use models pre-trained on ImageNet or CelebA and think that there is no bias in them. When people are saying you need to learn theory, this is really what they are concerned with.

If you can train an NN in a day, then you can probably train SVD in minutes/hours- so why not start with it? Otherwise you might not even recognize the symptoms of bad hyperparameters fo r the NN resulting in poor performance.

This is spot on with my own observations, especially as we get into modelling more 'abstract' ideas.

As more NN methods become viable, some more savvy data scientists complain to me "this NN is just approximating SVD/PCA/POD/etc!" Wonderful, that's explicitly the point! The network we're applying to this problem compares/combines multiple approaches to dimension reduction. The network created a latent space that makes way more semantic sense than just PCA or SVD for this problem (No Free Lunch). It still takes effort and understanding, but the value I've personally gotten over just applying PCA for my problem-sets has been incredible. In fact I'm certain it has made my career. Turns out diagonalizing covariance matrices aren't the only dimension reduction game in town!

It seems so strange to me as a software/data engineer to read things like this from a fairly adjacent space and understand none of the words.

I've been doing it for two years and am barely past the "understand none of the words" phase.

It helps to think of each term as an interesting puzzle. For example, SVD. It's fascinating if you dig into it. Most people don't want to, because it feels like work. For me, it's neat understanding ... whatever it is, ha.

I think it's finding the basis eigenvectors in a higher dimensional space, which basically just means that e.g. the eigenvectors of a cube are the X, Y, Z axes you're used to. If you skew it along the X axis, the Y axis bends a bit, along with the cube.

The eigenvectors form a shape that, when you find the volume of it, is the area of the resulting form. So the determinant of a cube's SVD is the volume of the cube.

In higher dimensional spaces, it's the same thing, except it's called "eigenvectors" (named after Sir Eigen of Eigenmadethisup) because mathematicians have reasons for using complicated language, some of which is valid. But as you see from me muddling through this, the underlying concepts are all small simple pieces that fit together.

Or I was nowhere close to the explanation of SVD. But it was close to something interesting, since it leads to the question of "What's the SVD of a sphere? How about a point cloud?" It was easy to figure out for a cube. Not so easy when it's an arbitrary shape. "And why is it useful?" Because it gives a lot of hints about what that object is. In the optimal case, in StyleGAN for example, the SVD can even be the basis vectors like "smile", "age", and so on. (You know in Faceapp how you can drag the "Age" slider and make yourself look older or younger? That's a basis vector in higher dimensional space. It's orthogonal -- more or less -- to "smile", because if you drag the "smile" basis vector around, it doesn't cause you to age older or younger. Except it's not quite orthogonal, because it's a higher dimensional weird-ass shape and therefore can't be orthogonal, so sometimes when you make someone older their hair turns grey even though "blonde hair" is orthogonal to "age" in theory.)

Yada, yada. Rinse and repeat and dive in for a couple years. You'll find it's fun once you jump in.

P.S. All the people reading this that feel offended like "No, you really must start with theory; you can't possibly learn anything if you don't know what you're doing," you better read this: http://thecodist.com/article/the_programming_steamroller_wai...

That steamroller is coming for you. Once the legions of javascript programmers realize that hey, I can do DL just like an ML researcher, you're gonna be doomed. Because a 17yo JS programmer has roughly 10x as much determination as even I can muster these days, let alone someone who clings to the idea that theory is the only path forward.

As someone who's spent 20 years tuning my own genetic algorithms, being swamped by newbs who spout fancy language and don't even want to know how to write the code themselves just feels like what it is - a new generation of recent business grads who swapped "blockchain" for "ML". Soon to be separated into "founders" and real estate agents, while the rest of us toil in the vineyard. So goes it.

Neural networks let anyone bullshit a good game until things get tricky. Back in the day frontpage was going to kill web development because anyone could make a website. Now we can slap newer tech we don’t understand together and profit will ensue.

It’ll democratise but it’s not there yet.

> savvy data scientists complain to me "this NN is just approximating SVD/PCA ..."

It wouldn't be "approximating" anything. An optimal one layer linear NN "autoencoder" is PCA. There are other learning algorithms for PCA than gradient descent, but the infrastructure for learning NN:s with big data sets makes it painless.

As soon as you add activations and layers, you're improving on SVD/PCA. For dimensionality reduction, it means the "manifold" is more complicated than just a linear projection.

> As soon as you add activations and layers, you're improving on SVD/PCA

You're expanding the space of realizable functions, which is an improvement in a specific sense, but not in all senses! The SVD, since it is better understood theorist theoretically, is a more straightforward problem to solve robustly. There are fewer hyperparameters (like learning rate) to choose, and you aren't left wondering whether your solution is at a bad local minimum.

I think it's wrong to think that it's an obvious improvement.

SVD is not a "sophisticated algorithm". It's about the simplest machine learning tool there is, and certainly simpler than a neural network.

> It means you can blindly let the NN do your thinking for you, and come out ahead.

I'm assuming by "ahead" you mean score the best on some arbitrary metric?

Doing this approach you will not come out "ahead" when your model runs into problems in production and you need to diagnosis and get it performing as soon as possible.

You also won't come out "ahead" in 5 years when a magical matrix of weights is the core of your predictive modeling efforts and no one on the team really understands exactly what was being done but the output of that model is crucial to another team... except it doesn't seem to be working.

You also won't come out "ahead" when an event like the pandemic strikes, and the assumed distribution of all of your data no longer holds but you still have to keep your teams performance up for the month.

I've been in the top 100 globally on kaggle so I understand pushing a metric higher, I've also worked for 10+ years doing DS work where a minuscule bump in a metric is not worth all the associated cost of a complex model. There have been at least three times where I have walked into a company that was running some mess of a neural network and replaced it with a one or two parameter model (that took seconds not days to train) and gotten practically identical performance to a NN that was causing massive headaches to maintain, train and debug.

One of the most important principles of engineering that seems frighteningly lost on many data scientists I've met is that simplicity is the goal, and we only take on complexity when absolutely necessary.

There are certainly cases were NN are the best path and you should let a million parameters do the thinking for you, but there cases are relatively few and getting solid performance here requires serious experience and skill.

How does using these embeddings (of sequences of interactional partners) compare with using embeddings of words/tokens?

They should be quite similar. In the end you coax your embedding space to amount to some consistent measurement of what causes samples to diverge from one-another.

You can do similarity search and all the sorts of things you do for word embeddings on embeddings generated for other scenarios.

P-hacker news?

You’re standing on the shoulders of a lot of giants, telling them you’re coming for them. Where do you thing the leverage is here? Those guys are driving cars and writing JavaScript automatically. Tuning NN knobs can also be done automatically. It often is.

We could all do with a little more humility here. We’re surrounded by insane skills who write the simple knobs we get to turn.

I'm a big NN proponent as well, and I generally agree, but as so many things can go wrong training I prefer to have a baseline (such as SVD in this case) first to make sure the NN is actually doing what you want.

Now, you actually do have to understand how to apply the NN correctly for this problem- each user has a embedding, which is a trainable parameter, and you train it on pairs that have interacted. I don't think your average PyTorch MNIST tutorial will teach this.

If you use a standard distance metric for your embedding space, won't the triangle inequality still imply a slightly weaker version of these chains? I.e. if A-B and B-C are both smaller than epsilon, then A-C is smaller than 2*epsilon?

Yes

The embedding needs to embed a social graph with similarity weights and predict ties. Of course, such partly cyclical graphs can not be guaranteed to embed in a low dimensional (aka far away from N) Euclidian or even Hyperbolic space.

Which means, among other things, that throwing any old Pytorch NN at the problem will not work, similar to how you can’t just Word2Vec to GPT performance since the latter employs many parallel Euclidian embeddings (or, in other words, any selection or aggregation of layer embedding is contextual).

Not that you can’t do it on Graphs, it’s just that understanding the issues is very helpful.

For example, understanding that Euclidian embeddings imply a triangle inequality while social graphs of similarity do not, is a crucial thing to know before throwing stuff at pytorch.

I like the analogy to multi-head attention. Is there work on something like it for collaborative filtering embeddings, eg splitting up the vector into multiple subvectors and calculating distances separately?

Not sure about CF but in terms of representation learning there are, of course, transformer approaches to embedding graphs ( the article essentially talks about the predictive power of closed triads in a graph)

So, a good starting point might be SDNE type graph embeddings with a transformer architecture, since these are already encoder/decoder setups.

There’s a ton of ways to get node similarity in a graph, but it turns out that all of them have a substantive theory basis (walks versus neighborhoods, higher order structure vs proximity etc) and none of them are really just “NN, embedding and done“

Perhaps also because it is really simple to see ex ante that a flat Euclidian embedding can not work.

That's right, but we don't need to use the Euclidean distance directly, and in fact most often we don't. For example, we usually use dot product, and predict success between two users as sigmoid(A.B), then we can have A.B > 0 and B.C > 0 but A.C < 0, so there's no problem. We can get the same with Euclidean metric too, by (soft) thresholding- anyway we have to map from our distance to a prediction of the interaction somehow.

None of this solves the basic problem of what features in the world you want to put into the model.

Heights of A, B, C? Age? Gender? Educational history? Reading or watching preferences? Handedness? Favorite color? You have absolute no idea which, if any, of the possible characteristics you could use to describe A, B or C, and so you only choices are to be vaguely selective and hope your guess is right, or throw everything you can think at the model and hope that there's still a clear signal. You may still get it wrong.

These extra features could help it narrow things down more quickly, but CF doesn't need any of this to get good quality- what I described needs _no_ features, only the observation of interactions between users. It tries to find a pattern to explain the successful interactions, by generating a set of features for each user and optimizing it to fit the data.

This is Chatroulette. Burn the matchmaking algorithms and keep it true random?

For example, if every dating site in existence did this the world would be a better place. Instead, we typically see layers of opaque algorithmic shit governing human interaction. The platform's interests are then in effect prioritized, not the user's.

Did anyone ever stop and think: "Is aggressively matching what we believe to be compatible or like-minded people actually a good thing for society? Is it possible that's actually a pretty terrible thing? Do we even know?"

There's a certain beauty to true randomness in a matchmaking context. It's a fair playing field for everyone. You start using highly imperfect and frankly arbitrary algorithms to govern who matches with who, and suddenly certain people aren't going to meet certain people for reasons they're probably not even aware of.

"Hey, this person could have been your significant other or best friend, but you never met them because some engineers at some tech company decided to get fancy. Fuck you." Or, in so many dating app cases: fancy and greedy.

Every time these algorithms are employed, there's always people that fall into the cracks; the dead zones. There they languish. Perhaps months or years later the algorithm is adjusted and affords them a reprieve. Perhaps never.

---

This isn't meant to be an indictment of Chatroulette. On the contrary, good on them for reporting negative results. Obviously some filtering must be done for a myriad of reasons, such as curbing abuse. Their stated mission is even to match people "At random." They've thought about this problem a lot more than I have.

That said, I thoroughly believe the road to hell is paved by matching users based on metrics such as projected compatibility or similar interests.

Just to answer that point: people (academically) look at whether it is a good idea to match similar people, what such a social structure means and how preferences for homophily, imposed or intrinsic, may affect the social structure.

Do social media websites do think about it? I am sure some of then have. But then, their goals typically differ vastly from what you would consider good for society.

Thanks for this. Admittedly my tone was somewhat hyperbolistic, but you bring up an excellent point that I'd like to build off:

It pains me to think there's large bodies of deep academic research in the social sciences that could guide certain tech platforms in a much more responsible direction. Instead, the drive for market domination and profit ensures that a mangled society is left in their wake (ironically, economics is also a social science).

The hubris certain tech leaders have in making the decisions they do astounds me. Virtually every major tech platform has the money to have a veritable army of social scientists pondering the ethics of anything that company does with its platform. Certainly some of these companies do employ ethicists, though I can't help but feel more often than not it's a dog and pony show.

Reality seems to be that very few pairs of people have mutual interest in each other. Matching is just increasing the probability, but it's still random.

I find that it's also very possible to make concerted effort and end up fine in a relationship with relatively anyone: I've been with a student who spent her life in the same region as me, studied in the same class at uni and had exactly the same age for 6 years, a Pakistani 4 years younger, practicing muslim who was born and lived in Hong Kong, for 2 years, and ended up marrying a Chinese citizen 2 years older than me, and it's been 5 years and a child now...

So I think none of the match making matters as much as the capacity for patience in relationship as well as rational understanding you build a partnership not a lifelong burning passion :D

>For example, if every dating site in existence did this the world would be a better place. Instead, we typically see layers of opaque algorithmic shit governing human interaction. The platform's interests are then in effect prioritized, not the user's.

Doesn't that go against the less random nature of match-making in real life? People have preferences and biases, and these influence or even dictate the people they meet. They tend to frequent people in certain circles, and these circles have an imbedded bias (proximity, socio-economic, cultural, scene, etc).

>"Hey, this person could have been your significant other or best friend, but you never met them because some engineers at some tech company decided to get fancy. Fuck you." Or, in so many dating app cases: fancy and greedy.

You don't need engineers to do that. Match-making in real life is not that random. You hang around places your "could have been your significant other" does not go for any number of reasons. Even in medium sized cities (say 8 million people), the graph is pretty interesting.

Unless your real argument was that Chatroulette's raison d'être was to "break out" of the non-random nature of match-making in real life by introducing the randomness of a roulette, and that introducing match-making algorithms makes it more similar to real-life which defeats the purpose of Chatroulette, which I believe makes more sense, and thus, I believe is your argument.

>Doesn’t that go against the less random nature of matchmaking in real life?

No. Holy cow. Those are my biases. I can choose to act on them or not. I get pigeonholed after a few weeks on every dating platform, it’s brutal and very depressing. In the wild I can choose to go to a different grocery store, go to a different park, hang out in a different cafe, go play mini golf… whatever. I am very open minded, I go out of my way to talk to a lot of different people from every walk of life, despite having a healthy group of loving friends and not being particularly extroverted, but I can only talk to those who are willing. I can try to talk to those who catch my fancy, and believe me I do, but our “stranger danger”, dopamine addled, earbuds-in, narcissistic and anxiety ridden culture makes this a fairly poor strategy. Perhaps in another life this would have been how you meet people, but generally speaking as a male in the modern heterosexual dating culture of millennials and gen z, you aren’t going to have much luck meeting people “in the wild” because talking to people unprovoked —-if you even can, everyone is plugged in all the time— is either creepy, weird or annoying- or at least that’s been my experience living throughout Northern California and the Bay Area. And covid has made this phenomenon significantly worse. And I know what this sounds like, so I’ll just come out and say it: I’m conventionally good looking, tall, fit, well dressed and a good conversationalist, not a creep. In settings where it’s appropriate, like, say, a music festival or a party, I’m good at meeting people, so I really don’t think it’s me. But on apps it’s just the same people. With apps there’s basically two options, I give a lot of people a chance and my MMR tanks and I only see a certain kind of person, or I’m selective for people I think would be a good fit and I see a different higher MMR group, which eventually devolves into the first group as it seems MMR decays over time.

I’m currently single again and almost 30 and I recognize a lot of the women I get in my feeds from dating at other points in my life. I use many apps, have no filters and cast my net quite wide. If I didn’t understand that this is essentially to be expected from social media for profit, I’d be befuddled and confused, but it’s really just upsetting. Romantic love is a huge piece of the human experience being gate kept by mega corps.

OkCupid getting bought by Match Group was horrible; now it’s search functionality is gutted and it’s got the same matchmaking bullshit all the others do, and matches me with the same groups of people and inactive accounts just like the rest.

Anyway, what you wrote triggered me, and I felt quite compelled to scream into the void about it. Modern dating apps feel like I have overly aggressive parents trying to put me in an arranged marriage. Its downright suffocating and makes this whole messed-up-planet, extremely socially divided, and ethically bankrupt life feel that much more dystopian.

Rant over.

>No. Holy cow. Those are my biases. I can choose to act on them or not.

I know that you want to believe this. I know that I want to believe this. I'm not sure it's a good idea.

"OkCupid getting bought by Match Group was horrible"

OkCupid was seriously excellent at its job, and a unique jewel in the landscape. There was nothing like it before, and as far as I know there has been nothing like it since its destruction under match.com.

I think what OkCupid did was quite profound, even if they stumbled across the social mechanisms by accident at first.

It's ability to find uncannily good matches for friendship and conversation unexpectedly via free-form attitude common ground discovery, driven by people who joined curious about romance or sex, touches on something so much more important than mere romance and dating.

It was like an electronic assist for what people instinctively do in the offline world in a healthy community. Partnering instincts driving the interest, but emergent behaviour if you were up for it was friendship networks with light flirting and sometimes decent conversation. Not at all like the gate-keeping, manipulation and unhappy stress seen on other sites.

With people so affected by and dependent on social sites these days, and the apparent ever-increasing polarisation, we might need to reinvent aspects of OkCupid again.

We must've been on a different website. Outside of the forums (which were mostly full of trolls), I don't see how you could've been making friends on that website in the way you convey. I certainly wasn't and neither were any of the people I knew who used it either. Hell - even getting a date on it was quite difficult (I never got one but I know my female counterparts were getting 50+ messages/day and never complained for a lack of suitors). The stats they released backed that up too. After all it wasn't just an effortless swipe to see if you were physically compatible with one another before moving forward... You had to craft a message and pray whatever attempt you used worked. (And then do that a lot)

I never saw friendly messages going between same-sex people or anything of that sort.

Could have been different experiences, or could have been different timeframes. It has changed a lot. Last time I looked a few years ago, it was very offputting and not at all interesting. But I gather they do keep changing it, so who knows what it's like now.

In my experience, which was pre-2007, OkCupid is the only social media site where I've ended up meeting people and where I first encounted people who would become new friends. It was uncanny how it also ranked me very highly with people I already knew though.

I had many chats on there with both males and females. It's true that most didn't reply, and I did receive some unsavoury approaches, but some were interesting conversations, and as I said, I met a few people IRL. Nothing sexual though.

By comparison, Facebook is supposed to be social media, but I don't thikn I've ever struck up a decent conversation or friendship on Facebook with someone I didn't already know from elsewhere.

I've been thinking about this a bit recently, after watching one too many Dhar Mann videos about not judging a book by its cover of all things.

I think filtering out options through heuristics that are sometimes wrong but on average skew towards desired outcomes makes a lot of sense when you have more opportunities than you could ever follow up on.

Going back to your example of online dating, if attractive women had a conversation with every man that messaged them, let alone a date, they'd be spending 100+ hours a week talking to men on the off chance that someone surprises them.

Given that, it makes perfect sense from their point of view to completely ignore a guy for seemingly insignificant/unfair reasons like him wearing a Star Wars shirt in his DP or an algorithm believing there's a 24.8% chance that he's a creep.

She can only follow up on so many leads - might as well spend her time on the promising ones.

Heuristics like this make a lot of assumptions that are heteronormative. Tinder for example took years to finally come around, and in the interim if you were trans, bisexual, polyamorous, or otherwise different, you were basically garbage as far as their algorithm was concerned.

That said, I think it makes sense to just re-frame your argument as examining the merits of platforms catering to desirable users. I've seen arguments attempting to justify why this is necessary, most having to do with keeping the platform alive, else the platform experiences a flight of said desirable users and everybody loses. I don't buy it.

The real reason is monetization. If a platform takes steps to ensure only the most desirable users match with the most desirable users by making it a zero-sum competition, then it's essentially preying on users' psychology such that they keep dumping money into the platform in hopes of obtaining what's seemingly out of their reach. It's a really profitable albeit sick business model.

Tinder in particular is notorious for this. New accounts get good results until they don't, but suddenly do again once money is supplied. However, the amount of money users need to surrender to sustain those results keeps going up over time, and they'll fuck with your account visibility in a completely opaque manner in order to keep that money coming.

I haven't used online dating in almost a decade (OKCupid circa 2012), so aren't familiar with their predatory practices, but I don't think you need an algorithm or even any form of gatekeeper at all to lead to a situation where undesirable people have almost no opportunities and desirable people have more opportunities than they know what to do with.

This has been the case for basically all of human history (if not longer) across a broad spectrum of different domains.

If the platforms are deliberately fostering this dynamic or exploiting it for profit at the expense of their users' mental health, that's of course unethical. I don't think it's a new thing that they've created, though.

>If the platforms are deliberately fostering this dynamic or exploiting it for profit at the expense of their users' mental health, that's of course unethical.

They are. Some platforms do this, most notoriously Tinder. Did their engineers sit there and say "Aha, let's be evil!"? No, of course not. Things like this happen via a combination of apathy, and focusing primarily on business metrics. It's a black box with every aspect built for monetization. Whether its ill effects satisfy the definition of deliberate is semantics at that point.

The tragic part, is these dating apps have become so widely popular that they've nearly monopolized the dating scene. Most non-technical users don't care about the nuance of this stuff, so that's how they get away with it.

The crux is this: these systems have created conditions where desirable people fall into the cracks for reasons that are purely algorithmic, and there's nothing those people can do about it.

If you don’t narrow it down at all you have a dating site which is showing me a list that’s 50% men, when I’m only interested in women. By similar logic, it makes a lot of sense to further filter down to the types of people I’m likely to be interested in.

I'm not against users having the ability to explicitly filter, I should have made that more clear in my original comment.

The problems arise when this option isn't present because a matchmaking algorithm that makes many incorrect assumptions takes its place, with no ability to opt out.

The problem with dating apps is that everyone hates a bad match. It's awkward, may feel like a waste of time, and generates a lot of rejection. People hate feeling rejected.

While a few apps have tried to do blind matching to get people in person faster, the fact they don't stick around implies that they weren't very successful.

There is some basic level of matchmaking thats required to get any relationship off the ground and for many people older than their early 20s this tends to involve some element of shared goals and shared lifestyle.

>There is some basic level of matchmaking thats required to get any relationship off the ground and for many people older than their early 20s this tends to involve some element of shared goals and shared lifestyle.

I think the key here is that the users should be able to decide what that is, and be given the tools to explicitly seek it out for themselves if they so desire.

Black box algorithms that apply to everyone are a far cry from that. Although I never had the privilege of using it, that's probably why a lot of people speak so highly of the old OkCupid.

> -roulette. Burn the matchmaking algorithms...

Heh. True.

The road less traveled was foraging.

Prior to relevant search and recommenders, information science people (eg librarians) facilitated user directed discovery.

Instead, most effort has been put preventing user agency. We know what you want. We'll tell you. Leave the details to us; no need to worry yourself about our algorithms.

IMHO, web directories (eg curlie, neé DMOZ) are maybe the closest digital analog of foraging. goodreads.com was pretty good. I'm struggling to think of other examples. Maybe BBSs like craigslist and metafilter.

> I thoroughly believe the road to hell is paved by matching users based on metrics such as projected compatibility or similar interests.

I have an anecdotal data point against this: my ex found her current husband on OKCupid, based on their question system.

Their questions ask what YOUR answer is, as well as what you want your MATCH to answer. Additionally, there is some leeway (you can choose multiple acceptable answers) and different importance levels for questions.

They have been married for 4 years, and I believe online dating can work.

OkCupid was good at that. It was unique though.

The GP liked OkCupid (until it was bought by match.com and ruined), so I think the GP is referring to the kind of corporate-monetising and "it's out of your hands" matching done everywhere else as the road to hell, with OkCupid's method being significantly different.

I believe the key to OkCupid wasn't just that it asked what you want your match to answer. I believe the main key was that OkCupid did not choose all the questions; the questions themselves were quite free-form, provided by users, and ranked by a statistical method to sift the most helpful questions. So they reflected real-life interesting factors that really affect people's compatibility, instead of what dating site designers think you should be asked.

And there were simply a lot of questions with which to build a rich picture without a prescribed schema. It was fun and thought-provoking to answer them because the system generally seemed to promote good questions. You felt like answering them revealed something useful that you wouldn't have thought of asking about. It promoted curiosity, and it seemed worth it to find out what non-obvious matching outcomes would look like.

Essentially, OkCupid's matching questions ended up being refeshingly honest and effective, it uncovered interesting characteristics that didn't seem obvious until they did, it had an air of curiosity rather than commercialisation, and it gave people a chance to find each other who would never had met under the schemas used by other sites.

And because you could see how well you matched with people and on which general axes ("selfishness", "inquisitivness", "religiousness" etc), and some questions, you were free to think about and reconsider your answers. They also gave you talking points.

For example, an emergent line of questions was about polyamory. Most sites don't ask, with the result that some kinds of poly people never find compatible people on those sites. Or, "are you bi" may be asked, but OkCupid's emergent questions would end up asking what kind of bi, the kind who must be with different people, or the kind who are gender-indifferent (and only want one person)? Those are distinct compatibility groups. Maybe you yourself only want one partner, but don't mind your partner snuggling/kissing/everything other people. Or you do mind, a lot. Or, are you asexual, or mostly so, but still want romance in your life? What kind of romance anyway, what does that mean for you in practice? These sorts of questions make a big difference to some people.

Then there were useful non-sexuality things such (making these up to give a flavor) as how often do you like to go on holidays, when was your last camping trip, what is your idea of a good Sunday morning, how tidy do you like your home to be, would you support a partner who wanted to go to college if you had to pay, have you lent money to a friend then been happy to write it off, which boardgames are best, should cheese on toast include onions, what's a good bedtime, do you support intervention in Afghanistan, would you bend the knee, spliffs, brownies or no drugs, how many children would you prefer, what about adoption, is black the only proper color for clothes, vi or emacs, rust love it or hate it, etc. So much that other sites miss.

>The GP liked OkCupid (until it was bought by match.com and ruined), ..."

I ain't your grandparent, nor did I say anything about OKC. :)

You're probably referring to this: https://news.ycombinator.com/item?id=28256462

>... so I think the GP is referring to the kind of corporate-monetising and "it's out of your hands" matching done everywhere else as the road to hell, with OkCupid's method being significantly different.

Nonetheless, I'd agree with that.

>Most sites don't ask, with the result that some kinds of poly people never find compatible people on those sites.

Speaking from experience, this is a special kind of hell I'd wish on no one.

Swipe-based apps rank your desirability by how many people swipe left/right on you, and bin accordingly. Polyamory obviously isn't as popular as monogamy, so you get a lot of left swipes regardless of whether you're generally regarded as both kind and beautiful/adorable by people in real life. You could have looks approaching model quality with a hardcore commitment to physical fitness—the charm and personality to match—and none of that would matter because as far as the algorithms are concerned, you're not attractive.

So what happens is despite putting all of the effort in and then some, you rarely ever find other attractive poly people, and instead you're binned in with low-effort monogamous people that aren't even poly, and might even be homophobic to boot. Your dating experience is then beset by a mountain of self-esteem issues that shouldn't even exist in the first place, and you feel extremely alone and apart from society.

Why does this happen? Because the algorithm supposedly knows better than the user, and thus the users aren't supplied with the tools to match themselves in situations where edge cases exist.

This exact same scenario plays out not just with polyamory, but with queer orientations too. Bonus points if you're both. It's swipe purgatory, or at least it used to be (these days it isn't quite as bad).

> I ain't your grandparent, nor did I say anything about OKC. :)

> You're probably referring to this: https://news.ycombinator.com/item?id=28256462

My apologies, you're right.

I might have misunderstand the blog post but it seems to me that everything that was tried used a "2-chain" which works on the assumption that the same type of people enjoy talking together.

From my personal experience although this can often be true it can also often be completely wrong. Concretely the friends I have are not a copy of myself and I am not necessarily friends with those that are very similar to me.

Another way to put it is to put people's personality as a binary, they're either of type I or type II.

In the example given, person A and C both had a non-trivial conversation with person B. So person B is type I and person A and C type II. In this case type II can be interpreted as "has non-trivial conversations with type I". There is no reason for person A and C which are both type II to have a non-trivial conversation.

Specifically, type I and II are probably male and female. Isn't it immediately obvious that model doesn't make sense if people are talking to the opposite sex? No, Bob and Carlie don't want to have a "meaningful chat" because they both chatted with Alice.

s/Carlie/Charlie/, I assume?

(Carlie, where I grew up, is a woman's name so this threw me for a bit).

Yes, Charlie, sorry.

With this in mind, the conclusion of the blog post is hilarious.

Quotes from the blog post:

> [The] rough idea is that if, say, Alice spoke to Bob for a long time and then Alice also spoke to Carol for a long time, then Bob and Carol are more likely than not to speak for a long time too.

> [...]

> The data have clearly shown us that, despite proving a successful technique for many other products across a range of contexts, collaborative filtering simply wasn’t a match for Chatroulette.

Basically:

1) make an arbitrary assumption about your data,

2) apply a widely-used recommendation algorithm,

3) notice that the results are really bad,

4) reject the algorithm... rather than the assumption.

Admittedly, I don't know whether the algorithm requires this assumption, or whether a slightly different one could be used. For instance, if Alice spoke to Bob and Charlie, then Bob and Charlie don't want to talk together. Or, if Alice spoke to Bob and Charlie, and Carol spoke to Bob, then Carol should be recommended to Charlie, etc.

I came here to say this very thing. This was the first thing that popped into my mind when I started reading. I hope the people responsible understand the motivations behind their customers using their product. It's a very superficial, and obvious observation to make that a large number of people use this product for romantic interactions.

You are so right.

The last person I want to chat with is someone like me.

(I do think Chatroulette should kindly ask men to refrain from the dick picks though. Then again, it seems like their users only go their to expose themselfs? I remember when it first started, and it was a great idea. Then the penis appeared.)

>I remember when it first started, and it was a great idea. Then the penis appeared.

Thus said EVERY WOMAN who has been married

I know a few women that are into penis.

Comments like this are frowned upon I guess but I can't not bring up a great quote from 30 Rock's Jack Donaghy:

"Wigco.com? Sheinhardt Wig's user-generated content site. All that anybody ever posted on it were penises."

Wouldn’t that imply that there wouldn’t be groups of conversational friends larger than 2 people? Or are you suggesting that many people might be fine with conversations with people of either type?

The basic assumption is that if M1 talks with F1 for a long time, and M2 talks with F1 for a long time then M1 and M2 will talk for a long time.

Assuming it's based on hetero dating type situations, you can see why this wouldn't work. But there could be many other factors in play also. For example, if the M's above were talkers and the F's above were listeners, of any gender, you can see that two talkers or two listeners would not have a long conversation. But a talker and a listener is a good pair.

Instead they should look at this. If M1 talks to F1 and F2 for a long time, and M2 talks to F1 for a long time, then will M2 talk with F2 for a long time?

I think this would have a better chance of working.

I must have missed the mentions of gender in the actual paper. No wait, I didn't. There is no mention of gender in the actual paper. It has nothing to do with dating ... the paper is about chatroulette.

I think GP's point is that chatroulette's article is incomplete because it doesn't analyze how gender impacts chat duration.

Like, the first question that came to me reading this was "Is there a pattern of men systematically skipping other men until they're connected with a woman?". It's a little weird the article doesn't ever address that kind of question.

I think it would be more accurate if they said that the types of collaborative filtering they tried didn't work. It seems like they only explored quite a small part of the solution space.

nice to see some data but a little more qualitative analysis seems called for. like, i don't know what the current userbase of chatroulette is like, but from the reputation 10 years ago i would guess that a significant proportion of users are looking for someone they find attractive. if this is the case, and given that people are on-average heterosexual, then it would not be at all surprising that a 2-chain-based system would perform much worse than random. again, i'm not sure that this theory is well-founded, but it would be nice to see some demographically-informed discussion.

also, saying that 2-chains didn't work doesn't seem to justify the title. collaborative filtering is broader than a single technique.

That's very insightful yet kind of obvious. It would be interesting to see if 3-chain data fixed this, where a central meaningful conversation BC and meaningful conversations AB and CD tended to imply anything about AD.

>Unfortunately, the results showed a true-negative rate of 78%. i.e. most of the time the model failed to predict when a meaningful conversation was about to occur.

>...

>The model is significantly worse than a coin-flip. Obviously, this is not good; and given that the majority of conversations on the site are trivial

1. This isn't correct use of the term "true negative". A true negative is when your model predicts a negative, and the label is actually negative. "the model failed to predict when a meaningful conversation was about to occur" would correspond to a false negative (the model predicts negative, the the label is actually positive)

2. "The model is significantly worse than a coin-flip" - Talking about coin flips hardly makes sense if the majority of conversations on the site are trivial.

A better way to think about this question would be from a Bayesian perspective. Define a random variable L for linkage. L=True if there exists a user B such users A and C both had non-trivial conversations with B. L=False otherwise. Then define a random variable T for whether the conversation is trivial or not. Then you can compute P(T=False|L=True) and P(T=False|L=False) from your data.

It could be that what this article is trying to say is that P(T=False|L=False) > P(T=False|L=True), i.e. the presence of a linkage decreases the probability of a nontrivial conversation.

This could occur if, for example, most users of Chatroulette are using it as a dating site, and since heterosexual people outnumber gay people, most nontrivial conversations occur between men and women. So if there's a linkage between A and C, that predicts that A and C are of the same gender, and they won't be interested in talking to each other.

Anyway, I didn't read any further than that, but I would suggest just ranking users based on their propensity for nontrivial conversations and connecting users with other users who like to have nontrivial conversations (or perhaps even better, compute the mean/median conversation length for each user to get a sense of their preferred conversation length, then connect users to other users who have a similar preferred conversation length).

I suspect there's room to use machine learning to improve Chatroulette a lot, even to the point where it could be the next Facebook. People already pay thousands of dollars to attend Harvard etc. for the networking. You just have to use machine learning tools correctly and find a way to capture some of that value.

I’m shocked chatroulette even still exists. It makes me wonder what their dedicated user base is like. My gut says it would be either some of the coolest and most interesting people on the internet or some of the creepiest and most shocking trolls.

Release the data and let the community figure this out!

This particular analysis is rather limpid and, ummm, misses a key feature of the most common chatroulette use case.

If I recall a few years ago Chatroulette really needed a "Hotdog/Not-Hotdog" classifier. I hope they solved that tiny problem using big data.

As hilariously as the comment is worded - it's honestly the truth and the fact that this blog post is focused on "creating conversations longer than 30 seconds" and not on reducing conversations quicker than 5 seconds (because that's obviously a dong I'm looking at) feels to me like they've really missed the mark.

As uh, depressing as this may sound to the AI that needs to learn it, a hotdog focused neural network (let's just assume we'd call it "DongMind" because obviously we would) would probably greatly increase Chatroulette's value proposition.

I heard this a long time ago but if I recall correctly, the Diablo 2 devs had a way to find cheaters and used match making to put them in games with one another. If a similar strategy was applied here...

Jinwang would make more sense (pun on the name of the character who makes the hotdog/not hotdog classifier in Silicon Valley which the gp is referencing)

Why not combine the "hotdog" filter with the collaborative filtering to make much better predictions about mutual engagement by specific users?

can you elaborate?

Some people want to see nudity. Some people don't. Could use presence of nudity and preference for nudity in the model.

Great to see them reporting a negative result. We don't get enough of that in our industry.

I could repost what I recent said about onlyfriends here. I think for many people, the whole point of Chatroulette is "show me yours I'll show you mine". That's why the gender thing that others have commented on is not surprising (if Anna chatted with Bob, and Bob with Charlene, it does not follow that Anna wants to chat with Charlene - because people are looking to chat with the opposite sex.) If they want to pivot and sanitize it, they'll loose their core audience.

Maybe it would be worthwhile to include two different populations in the model... One that looks for "meaningful conversations" and one that looks for nudity...

This is really interesting but I'm highly skeptical. I'm not sure we're given enough information to really get at why they're getting the results they are.

They report that if A-C have a non-trivial conversation and B-C have a non trivial conversation, then A-B have a "true-negative rate" of 78% of having a trivial conversation, by which I interpret a 78% over the base line of having A-B have a trivial conversation. The fact that it's so heavily biased against having a non-trivial conversation seems like it should be a huge red flag.

How are they conducting the study? Are they informing participants A and B that they're being matched because they've had a non-trivial interaction with C? Is it random?

If they are being informed and it turns out that there is a gender component to it, then that might be a big piece of the puzzle in that jealousy and competition start to come into play. Also, if gender is a big factor, then that might inform a "3-chain" collaborative filtering method in that if M0-F0 have a non-trivial interaction, M0-F1 have a non trivial interaction, M1-F0 have a non trivial interaction, then maybe M1-F1 have a higher chance of having a non trivial interaction.

I think psychologists call these "confounding factors"? They get a massively strong signal of something but then disregard and and say "doesn't work for us". Something's going on, it's be nice to dig deeper into what it is instead of dismissing it offhand.

Maybe by and large guys like to talk to girls, but guys don’t like to talk to guys and girls don’t like to talk to girls. This would make a 2 chain model look bad I think. Maybe try a longer chain? i.e. “Bob liked Alice, and Charlie liked both Alice and Diane, so maybe Bob will like Diane”

(Ik this is heteronormative — sorry)

Hi, this is CR founder.

Pleasantly surprised to see HN take interest in this.

I hope you will not mind if I use this opportunity to say that I am looking for people who can help me on this and other interesting problems.

at@chatroulette.com

I think what doesn't work is their assumption that amount of time talking means it was a "good match." It's quite possible that it took some time for someone to figure out the other person was awful, or even that the entire conversation was negative, such as an argument. The algorithm doesn't know that because it is purely based on talk time, rather than taking into account such things as if the users gave each other good ratings.

Wouldn't that trend towards optimal over time? Perhaps with a decay or bias for recency.

Valuable connection = what you put time into, most latest

The hypothesis they chose is not the most obvious one to attack, and it ignores the most obvious application of collaborative filtering.

Their hypothesis is that if A likes B, and A likes C then C will like B.

They should instead attack this hypothesis (much more common application of collaborative filtering, think predicting what book you might buy on Amazon):

A likes B, A likes C, D likes B -> D has higher probability of liking C as well

But to them, A liking B and B liking A are one and the same, since all they are measuring is talk time.

Regardless, I think it would do much better if it took into account ratings. Even better if they allow you to rate them on several things and then give an overall rating.

I personally hate it when things try to learn from my behavior rather than my explicit declarations. For instance, if I click on clickbait (hey, I'm human) it shouldn't assume I actually like that thing or want to see more of it.

Would be cool if they published some data so that others could give it a try. I bet someone out there can build a model that is better than a coin flip.

> the rough idea is that if, say, Alice spoke to Bob for a long time and then Alice also spoke to Carol for a long time, then Bob and Carol are more likely than not to speak for a long time too.

This is not how collaborative filtration works. If Alice independently spoke to Bob and Carol for a long time, and Dwayne spoke to Bob for a long time, then chances are Dwayne may speak for a long time to Carol.

Why just 2-chain?

If you wanted to predict matches and lengthy coversation between random tinder users (if there was no rigid separation for genders) you'd fail in similar fashion.

Try 3-chain.

Has someone made Chatroulette with only audio yet? And no typing obviously. Seems it would solve the problem.

Everything old is new again. Phone chatrooms were like this in the 80s.

We had Clubhouse in the 80s? Are you sure?

In the 1880s? They were called "party lines."

https://en.wikipedia.org/wiki/Party_line_(telephony)

Extremely good HN comment :-D

Funny and my history knowledge got a serious boost!

Yes, but it was called "club house". It had audio, video and haptics.

I thought chatroullete was the premier platform in the 'exposing yourself to strangers' industry? Before you needed to take the city bus to experience this level of public exposure but now you can do so from the comfort of your own home.

Is anyone else bothered by that font? The paragraphs feel really stretched out horizontally, and the headings seem one step removed from Comic Sans.

TIL Chatroulette is still a thing

They could've saved a lot of time if they just read the Terence Tao blog (I'm being a bit cheeky here since it's a rather counterintuitive insight and doesn't necessarily hold for all cases).

"If X correlates with Y and Y correlates with Z, then this does not imply X correlates with Z."

https://terrytao.wordpress.com/2014/06/05/when-is-correlatio...

Reply on news.ycombinator.com