Hacker News Reader

Comment by okr | Hacker News Reader

I can think of many reasons, why a limit is somehow hardcoded/assumed throughout the entire system including the history and probably a lot of optimizations around it. You can not think of any? In databases you also have limits, like character limits, like 16k for a varchar or something. Or memcaches with bucket sizes and so on. It is not just changing a number, i can imagine.

View on HN

(ex-twitter engineer) I left before this project got started, and do not have any insider info on how they did it. Given what I know about the number of places tweet length assumptions were built into, it must have been a large, cross-team effort. It likely required thoughtful problem-solving.

Take, for example, search. An early iteration of Twitter search relied on this limit to pack term positions into 8 bits (Source: https://www.umiacs.umd.edu/~jimmylin/publications/Busch_etal...). 280 > 256, so if this was still around, the whole approach had to be rebuilt, and the indexes recreated. That's ... non-trivial. And that's just one subsystem.

Wow, sounds like Y2K.

At the volume Twitter works at, you take the constraints you can find.

Even more of a reason not to make this large, system breaking change.

And i can imagine, when they just doubled the amount of possible characters, they just correlate these tweet buckets now. Like, maybe each tweet already had space for attributes/opcodes, so one can express things like "deleted". Maybe there is another attribute now: "has successors". And in the API one just return many tweets as one. In that way, one does not have to rewrite history. But who knows!

There are databases with arbitrary limits on strings? That strikes me as really poor design.

You must be really new to databases.

They aren’t arbitrary.

Almost every SQL database has arbitrary limits on text. Personally I find it ludicrously anachronistic.

I mean, there’s good reasons to limit the size of fields in database tuples - relational databases are made for storing small to medium amounts of normalized data, you can abuse them (and I have, storing a couple million 60-200K PDF’s in a Postgres database at work) but you can shoot yourself in the foot by creating tons of 60% full pages and the stupid amount of extra IOPS those wide columns create.

Still, not allowing unrestricted text and binary columns is still silly, so I agree for the most part - but you still need to know what you’re getting yourself into when you start making wide tables.

Why you are being down voted: As far as I know in most databases TEXT and BLOB types have no "arbitrary" length parameters but are stored at the end of the database file making access slower especially on spinning rust, VARCHAR(x) can only be x characters long but is stored inline with the table so it's much faster. If you know the length of a string won't exceed a relatively small number of characters VARCHAR is the right tool for the job.

Fun fact: Postgres is one of the few databases to do this right. "varchar" is just an alias for the "text" data type, which has a limit of 2GB (last I checked). A small amount of text is stored inside the row, and the overflow is compressed and stored in a series of linked child rows. The system is called TOAST, and quite fast. So the underlying storage is completely abstracted, and setting a limit doesn't impact storage; it only enforces a limit on what input is allowed. MySQL's text type, on the other hand, is like a binary blob and has (last I checked) different semantics than the varchar type.

Not string lengths, but definitely other things. For example, Oracle is notorious for only allowing 1000 columns per table:

https://asktom.oracle.com/pls/asktom/f?p=100:11:0::::P11_QUE...

They bumped it up to 1000 from 255 a few years ago. It was a major engineering effort to support that. It's unlikely that they will be willing to do it again.

Reply on news.ycombinator.com