markpapadakis

Please see earlier comment. Specifically re: sys calls and concurrency you may want to start from this issue https://github.com/phaistos-networks/TANK/issues/14

It is a service, and a library(C++). You can always check the implementation, but obviously that's the wrong answer to that question :)

The README states that replication is not implemented yet. I encourage you to also check this issue https://github.com/phaistos-networks/TANK/issues/14 for some details on the I/O semantics, and the Wiki for answers to other questions: https://github.com/phaistos-networks/TANK/wiki

I am the core developer. It's not abandoned at all. There are updates that haven't been pushed upstream, but no new features. Was going to support replication via an external consensus service (etc, consul, etc) -- but looks like implementing Raft directly into Tank for interfacing with other cluster nodes is a better idea, all things considering (no external deps., simplicity).

The reason this hasn't happened yet is that, other than lack of free time to pursue it, we(work) haven't really needed that feature yet. We run a few instances and they are very idle and we can also mirror to other nodes (via tank-cli).

John Carmack's said "Focus is a matter of deciding what not to do". I think that quote describes a fairly efficient heuristic for keeping your mind focused.

We build our own. All events are published on Tank( https://github.com/phaistos-networks/TANK ) , and we have a bunch of consumers that consume from various Tank topics. They process the data and either publish to other Tank topics(to be consumed by other services), or they update state on various services.

- For data exploration, we use memSQL. We keep the last day’s worth of data there(we DELETE rows to keep the memory footprint down), and because most of the time it’s about understanding something that has happened recently, it’s almost always sufficient. Each row contains the event’s representation as JSON, and we also have a few more columns for faster lookup. memSQL’s JSON support is great(we used mySQL for that but it was too slow), so we can take advantage of joins, aggregations, windowing etc.

- For data visualisation, we use ELK (but it’s pretty slow), a tool our ops folks built (“otinanai”: https://github.com/phaistos-networks/otinanai) and we have a few smaller systems that generate graphs and reports.

- For alerts and tickets, our ops folk built another tool that monitors all those events, filters them and executes domain-specific logic that deals with outliers, notifications routing, and more.

This solves most of our needs, but we plan to improve this setup further, by monitoring even more resources and introducing more tools(Tank consumers) to get more out of our data.

Sometimes things just don’t work out. Twitter certainly can’t be blamed for trying all kinds of different ideas, either by putting their own resources to work on them, or just acquiring other companies (Vine, Periscope, and many more - see list here: https://en.wikipedia.org/wiki/List_of_mergers_and_acquisitio... ).

They have a product that is very valuable to many people, but those people are not enough, and the things Twitter tried just didn’t improve the situation. They could expand and try to compete with Facebook, but that’s not going to work and will alienate their current core users, they could reconsider the 140characters limit (ditto), they could just stick to offering the same service in perpetuity with minor changes here and there, and hope for the best, and that’s what they are doing. They need to expand their users base and by extension increase their profits (more users, more ads served, and there is this whole network effect to consider).

The fact is, potential buyers are aware of that implicit users-base wall Twitter has hit a long time ago, and are probably not interested that much in contributing their own ideas in order to succeed where Twitter’s many leaders in the past failed to do. They should know that significant changes to the product will morph Twitter into something else, it will no longer be Twitter, and like I suggested earlier, Twitter doesn’t stand a real chance as a competitor to Facebook and other emerging/established networks.

I love Twitter, I use it every day and it provides me with a lot of value. I suppose it’d be best for me and others like me if they stay the course, so that we can all use it like we do today, and hope one of their future ideas will work out, all the while they are losing market share, and then eventually, inevitably, agree to a sale. The alternative would be someone else buys it now and Twitter will soon become something, and that will undo the company.

I imagine Yahoo! Mail engineers being royally pissed about this. Well, I suppose that includes all Yahoo! folks who are still putting real effort into improving Y!'s services. Every odd day something surfaces about Y!'s execs questionable practices and decisions, every even day problems, leaks, bad press. Moral must have hit rock bottom.

Maybe the Yahoo! Board should have surveyed the startups scene, looking for founders who bootstrapped successfully and proven their worth, and recruit the best they could get. I am not very familiar with management of people and aspects of running a business, but I believe there is a lot more to it than being a smart person with computers.

Maybe TANK ( https://github.com/phaistos-networks/TANK ) would have been a good alternative on there. No features parity with Kafka, but setting it up is a matter of running one binary and creating a few topics, and it is faster than Kafka for produce/consume operations. (disclosure: I am involved in its development).

In Greece, elections are always held on Sundays.

No provision for submitting votes by any other means other than physically going to election posts(in practice, the government uses schools), entering a tiny makeshift voting booth (which in theory protects you from prying eyes and gives you privacy, but it really doesn't), selecting one paper sheet among the pack you received before entering the booth (one sheer for each political party), placing it in an envelope, sealing it, exiting the booth and dropping it into a box. It doesn't get more analogue than that. Also, there are penalties for those who do not vote - in theory you will be prosecuted, in practice, AFAIK, it rarely happens, but either way, it adds to the ridiculousness of the whole thing (of course the parties want you to vote, and they conspired to force you to vote by making it illegal not to.. This is Greece).

I recommend Robin Hood open addressing scheme. It's particularly trivial to implement, and because it relies on linear probing, so it's memory-cache friendly.

See more here: http://www.sebastiansylvan.com/post/robin-hood-hashing-shoul...

Jeff Preshing has written many interesting blog posts about the subject. For example, see his blog post on Leapfrog Probing( http://preshing.com/20160314/leapfrog-probing/ )

It was called Trinity -- it was initially developed for Pathfinder.gr, and soon thereafter was the search provider for in.gr, and was also accessible at trinity.gr for some time.

In the past we built and operated Greece’s largest search engine(Trinity), and we would crawl/refresh all Greek pages fairly regularly.

If memory serves, the frequency was computed for clusters of pages from the same site, and it depended on how often they were updated(news sites front-pages were in practice different in successive updates, whereas e.g users homepage were not, they rarely were updated), and how resilient the sites were to aggressive indexing (if they ‘d fail or timeout, or it ‘d take longer than expected to download the page contents than what we expected based on site-wide aggregated metrics, we ‘d adjust the frequency, etc).

The crawlers were all draining multiple queues, whereas URLs from the same site would always end up on the same queue(via consistent hashing, based on the hostname’s hash), so a single crawler process was responsible for throttling requests and respecting robots.txt rules for any single site, without need for cross-crawler state synchronisation.

In practice this worked quite well. Also, this was before Google and its PageRank and social networks (we ‘d probably have also considered pages popularity based on PageRank like metrics and social ‘signals’ in the frequency computation, among other variables).

I used Instapaper for a long time before I switched to a Pocket.

There were issues with their parser they never fixed, has been consistently slow, articles would often be lost even though they were "saved" according to the apps that used the Instapaper API(or its browser extensions), it would take forever to download content over 3G ( maybe the client was downloading it? ). It just never got better for me.

They did introduce a killer feature(narration) and I switched to Instapaper from Pocket again for that feature alone (2 hours or commute time). Still annoying to use and as soon as Pocket implemented the same feature, I jumped ship again.

Pocket seems to be fast(at downloading content, and the iOS app is very responsive), their parser is improving and it's now by far my most used app on my phone.

I hope they get the resources they need to really improve their service though. They have lots of happy users and that must mean they are still doing something right.

I live in Crete, Greece. Almost all families here own olive trees groves and produce oil - the island, and I guess this is true for most regions of Greece, is full of olive trees. We consume a lot of oil - we use oil in everything, and it’s delicious - I personally have a hard time eating anything that doesn’t have olive oil in it.

I ‘ve tasted olive oil when I had the chance to travel to other countries, except it barely resembled the olive oil I know. It tasted bad, and it didn’t even look like the real thing. This could have only been fake or otherwise diluted olive oil. I doubt any kind of olive tree produces that kind of oil anyway.

By the way, I noticed from my time in some European cities, that olive oil is sold for a very high price. I understand that imported goods are expensive, but the prices I saw at super markets were just ridiculous ( and it’s possible even that was for ‘fake’ olive oil).

You may also find this interesting: https://github.com/phaistos-networks/TANK/issues/14

This is a message broker. You can publish messages to it, and it will retain them for as long as you want(it supports retention and compaction semantics), and you can consume the messages, starting from any message(messages are identified by sequence numbers). This is similar to Apache Kafka and - to a lesser extent - Azure Events Hub )

You are right, I should have elaborated on those claims, and I plan to add a relevant Wiki page to address this shortcoming. However, you can use the tank-cli bm command to benchmark client to broker to client (end to end) latency.

We get sub-ms (a few hundred microseconds) latency here, and even on a very busy Tank broker (it takes effort to get there), I don't foresee situations where you can get over 1ms latency for any operation.

For throughput, Tank relies on sendfile() and readahead() for very high throughput - and thanks to the very tight packing of the messages, you will likely saturate your disk/network easily.

This may be because I have always admired John Carmack, and because I love video games - although I am drawn to all kind of different book genres, but "Masters of Doom" had a profound effect on me.

I read it every few months, and I have gifted it to family and friends, and most of them loved it as well.

It’s not the writing, nor the story per se that stand out, it’s not just about how it beautiful highlights and highs and the lows of the Johns’s symbiotic relationship and their accomplishments, it’s not even about how their skills, strengths and weaknesses play into their success and failures (which I am sure is typical of most co-founders stories).

It’s about empowering the reader to believe that everything’s possible, and how smart, hard-working people can build technologies that affect the lives of many.

This book works wonders for when I am going through burn-outs, or I am not motivated enough to pursue a problem or a project. When I am done reading the book, I am excited and eager to get back into the game. I can’t recommend it enough.

Excellent. I was genuinely interested to learn about their product.

On perhaps unrelated note, I think it'd be important for NetSuite to make sure their site's up the day Oracle buys them out, otherwise it 'd make both them and Oracle look bad. ( http://www.netsuite.com/portal/home.shtml, click on Free product tour - Oops )

You may want to read Google's recently released SRE book, watch the many conferences(monitorama, @scale, and many more) videos on YouTube and/or hunt down slides, check various engineering blogs (e.g Twitter posts about observability, Uber posts about their Atlas outliers detection system etc) and talk to people who are already dealing with those kind of problems.

That said, I think what's more important is to log everything ( use Kafka ), even if you don't know in advance how to make use of them. You may need them later. And you need to break this down into 3 different problems. Data exploration, data visualization, and alerts. And you probably shouldn't or won't be able to find one product that will help you address all those needs - or maybe will be too expensive. You should instead use multiple and all should consume from your events feed/stream. If you want to do it yourself, and you don't have the hardware resources to handle lots of data points, then use two differ sets of services. One that will retain say upto 24 hours worth of data so that you will get predictable, consistently fast query, processing and visualization time and another that may hold data for a longer period. Chances are you will want to understand issues and that happened recently so optimize for that. Good luck :)

This ML thread's only real argument is that some OS/Kernels may not support pread/pwrite. The readahead argument makes little to no sense IMO. Unless there are too many uses of random access IO in the codebase, they should use pread and friends if available there. Especially considering most people run it on Linux not some exotic OS nowadays.

Great write-up. A few observations:

1. The encoding and translation schemes of Postgres and mySQL/InnoDB are well described in the blog post, and I would also agree that InnoDB’s design is, all things considered, better for all the reasons outlined in the post.

2. I don’t understand why anyone still uses lseek() followed by read()/write() and not pread()/pwrite() syscalls. It’s trivial to replace the pair of calls with one. Aerospike is another datastore that resorts to pairs of seek/red-write instead of pread/pwrite calls.

3. Process/connection model makes no real sense nowadays - although to be fair, there is, today, practically almost no difference in terms of footprint between OS threads and OS processes (other than memory and FDs sharing semantics, they are practically the same). It’s still more appropriate to use threads (although I ‘d argue maintaining a pool of threads for processing requests and one/few threads for multiplexing network I/O is the better choice).

4. ALTER TABLE is obviously a pain point with mySQL, although I am not really sure many users with large datasets care; they probably figured out long ago it’s going to be an issue and they designed and expanded accordingly. It’s also a relatively rare operation. That said, other than using mySQL (or any other RDBMS) to build the data plane for an elaborate, distributed KV store, one should consider Salesforce’s approach too. Their tables have some 50 or so columns, and the column names are generic (e.g column_0, column_1, … ). They have a registry where they assign column indices (e.g column_0) to a specific high-level entity type (e.g customer title, or price), and whenever they need to query, they just translate from the high level entity to the actual column names and it works. They also, IIRC, use other tables to index those columns (e.g such an index table can have just 3 columns, table id, column index, value) and they consult that index when needed (FriendFeed did something similar).

5. Cassandra should have no problem supporting the operations and semantics of Shemaless ass described in their blog posts. However, given they already operate it in production, they probably considered it and decided against it.

I 'd like to believe all those unfortunate calls on their part, the huge benefits of switching to x86-64 and standard h/w components(for the most part) for PS4/XB1 developers(and their makers), and their predicament they are in now, have informed their guided their decisions this time around.

I understand it is about tradeoffs, and that the benefits and limitations of every language(in terms of syntax and semantics, and its associated runtime/VM -- if any) definitely define to a large extent the final product's characteristics.

I am only saying that Mongo is not unreliable because C++ was selected; it's unreliable because of architectural choices and lack of care which may or may not have been intentional.

The language, especially the languages you enumerated, rarely have anything to do with the actual final product's qualities (including it performance, reliability and ease of use). Mongo is unreliable because too much effort was spent making it easy to install and use it immediately (it works!) and little to no care apparently went into anything else. Nothing to do with the choice of C++ for Mongo, or the choice of Erlang for Couch.

Orleans is just fantastic.

343 Industries famously used Orleans for Halo 4 backend services infrastructure. Caitie McCaffrey's presentation can be found here: https://www.infoq.com/presentations/halo-4-orleans

You may be interested in Obit ( https://github.com/orbit/orbit ). It is inspired and based on Orleans, except it's written in Java. Similar concepts and semantics. ( a BioWare project )

I ‘ve come to realise that no matter what your data engine choice is(Storm, Spark, Flink, DataFlow, Ruby or Bash scripts, whatever) it is extremely beneficial to persist incoming data first to a distributed log.

Even if all you want to do is accept events and immediately persist them on some data store (Cassandra, mySQL, text files, etc), you re far better of first publishing them on a distributed log, and then consuming from it and persist on, say, a mySQL or Cassandra cluster.

You decouple the data flow from processing - and this means you don’t need to necessarily directly attach your firehose to the processing systems. You can just accept them as they come and deal with them later, if ever - in fact many distinct processing engines can each, asynchronously and independently, access and scan those previously collected event streams, and they can do so at whatever pace makes sense(i.e depending on how fast they can process messages/events).

You are probably familiar with the the log abstraction anyway. See https://engineering.linkedin.com/distributed-systems/log-wha... and http://www.confluent.io/blog/stream-data-platform-1/

Using Kafka as a core infrastructure technology, and for persisting any incoming and generated messages/events to it should be the default strategy. Also, it’s extremely unlikely you will hit service capacity limits, because publishing to Kafka, or a Kafka like service; publishing is mostly about appending data to a file and consuming is mostly about streaming (sequential scan) from files — all very fast and low overhead operations.

Shameless plug: if you want a Kafka like service, with, currently, fewer features but with better performance and far less requirements and a standalone operation mode(no need for ZooKepeper), you may want to check https://github.com/phaistos-networks/TANK

Anyway, I can’t recommend Kafka and investing on logs enough. Also, Kafka Streams is very elegant; likely based on Google’s DataFlow design and semantics. If you are using Kafka or plan to use it, you may want to evaluate it and adopt it over other heavier footprint and more complicated systems (e.g Storm or Spark).

The TimerWheel abstraction, implemented for example: - https://github.com/facebook/folly/blob/master/folly/io/async... (Facebook/Folly) - https://github.com/real-logic/Agrona/blob/886d7744fa15fc3de1... (Real-Logic/Argona) and in turn based on the Linux Kernel's "Timer Wheel" data structure (see here: https://lwn.net/Articles/646950/) are particularly elegant algorithms; great performance characteristics, except that you really can't get in O(1) the time when the next timer is going to fire, in order to implement a tickless scheduler, etc.

In that article, a new timer wheel draft is described - that solves most of the shortcomings of the original TW design.

I suppose we all have stories to tell and advices/tips to share; I too have gone through intense burnout, and initially it was just too hard to deal with.

Eventually, it turned out what really worked for me (it also worked with some friends who tried this) is to just do something else for a change.

Work on small, contained projects - maybe on unrelated problem domains, and consider changing your habits and do more things for fun, but whatever it is you do, don't try to 'force' yourself to do it. Working out at the gym work wonders. Getting some good night's sleep also helps.

I would also snap out of it after a few days. This never failed me, but I realize that it may not work for everyone.