Comment by jillesvangurp | Hacker News Reader

Comment by jillesvangurp | Hacker News Reader

Tagging and categorizing are two subtly different things to do. Having dealt with a lot of real world data, all I can say is that getting your hands on consistently tagged or categorized data is hard and gets harder the more data sources you have.

A real world example of how tagging can be both super useful and get out of hand is open street maps. The only meta data allowed in there are tags. The OSM community depends on people using tags correctly. But of course the tagging is incomplete, inconsistent, and subject to regional and data source specific variations. Which makes interpreting the tags a bit of a dark art.

But it still adds up to a very complete and rapidly evolving world map. So, there's that.

I've had the pleasure of seeing the documentation for Navteq's internal meta datas schema that they used for their maps while I still worked for Nokia Maps, which at the time owned Navteq. These days the whole thing is known as Here Maps. This was a PDF of around 4K pages. Thousands of attributes. Lots of weird little details related to traffic lights, subway entrances and exits, and other features you have on maps. This stuff gets complicated quickly.

Two very different approaches to the same problem. I think I like the OSM way a bit better. Neither is easy.

My job at the time was trying to align some of that data with data we got from external data sources like TripAdvisor, Qype, HRS, and a few others. Way too much time got invested in dumbing down categories, mapping one to the other, and trying to make sense of stuff. We had lots of issues with duplicate POIs because of all sorts of subtle differences in how different data sources were annotated with categories, tags, etc. Some of the data just isn't that good, complete, or consistent and you have to deal with that.

My thinking is that categorization is a dead horse. Here is a problem: take a car and a truck. Now slowly, bit by bit, morph the car into the truck. At what point is the car a truck? Or is it ever a truck? I, personally, personally cannot see this as a problem that can be solved. Maybe you are better.

My conclusion is that composition is a better method. Has a. And composition by capability is even better. Not great, but better. I was part of several efforts by standards groups to come up with SNMP standards for storage devices. What is a disk? And it has a controller right. Wait. Now the controller is in the disk. Wait now there are RAID systems. Wait...

Trying to build a system of composable tags for functionality seems like it might have a chance. Can it store something? Can it retrieve it. Does it have a retrieval time? Etc.

Another annoying problem is the categories for Craiglist. Where do cooking pots and pans go. For sale. A locality. Household items. Really? Nothing under cooking or even kitchen? I'm not saying this is wrong, I'm just thinking about how do I go about finding stuff other than just "search title".

> I, personally, personally cannot see this as a problem that can be solved. Maybe you are better.

Both library science and cognitive linguistics have solutions for/discuss this problem.

I would say what they call 'composition by capability' is a theory of categorization, in fact similar to Wittgenstein's family resemblance theory.

That said, it's not obvious that the underlying data model needs or should map to a theoretical model for how human cognition works. Maybe it should and it's worth considering the theoretical landscape before setting out, but there are other features that come into play.

I know very little about library science, but I assume they take a more practical (for this task) approach to this that's worth looking into.

You should definitely look into it, especially if you like pedantic arguments.

This runs the very real risk of thinking you can overload all "tags" onto something that you want.

That is, embrace the restrictions that come from a taxonomy. It is a bit of a lie, but a lot of it is defined on contrasts and these contrasts help build things.

What distinguishes a car from a truck? One is more faceted to holding people than to holding cargo. But if you don't know either a truck or a car, you are unlikely to know from just that description. Indeed, you could wind up with a cargo van. Which is different from a passenger van, in much the same way.

To that end, we present the taxonomy as a heirarchy of descriptive properties. When, in many cases, it is a heirarchy of representative samples. In programming, we distinguish between class based and prototype based object oriented representation. In reality, most taxonomies are presenting class style based relationships, but using prototype based representation from a population.

Not disagreeing but unfortunately a lot of data sources have different opinions on this. And sometimes categorizations are actually useful indicators to tell what is what.

Composable and namespaced tags is exactly what we do in my company. It gives us a lot of flexibility and it mirrors what OSM does. You can do a lot of duck typing against such a system and it will work as good as your data is without breaking completely when you get bad data. You just loose some features.

> At what point is the car a truck?

when you can use it as a truck (ie. when it can do what the car isn't adequate for).

Discrete categorization of continuous phenomena is always going to be inherently arbitrary. It's important to keep in mind what your goals are. This sort of categorization shouldn't be done with the goal of finding some sort of fundamental platonic truth; that's not going to work. Rather, discrete categorization is performed simply because humans find these categories useful when communicating.

The fuzzy line between truck and car isn't a problem when you approach categorization with this mindset. If you see a bank robber fleeing in an El Camino, you can tell the cops they fled in a "truckish car" or a "carish truck"; you don't have to neatly categorize an El Camino as one or the other to get your point across, but the arbitrary categories still help you communicate the idea.

If you're tagging pictures and come across an El Camino, and you don't have a "coupe utility vehicle" tag, you can simply tag the El Camino as both a truck and a car.

Reply on news.ycombinator.com