HNHacker News
TopNewBestAskShowJobs

NyxWulf

933 karma · joined November 7, 2008

submissionscomments
NyxWulf··on Log Everything as JSON. Make Your Life Easier.
I've seen several articles like this, and there are a number of things to consider.

Logging to ascii means that the standard unix tools work out of the box with your log files. Especially if you use something like a tab delimiter, then you typically don't need to specify the delimiter.

As an upside you are aren't storing the column definition in every single line, which if you are doing large volume traffic definitely matters. For instance we store gigabytes of log files per hour, grossing up that space by a significant margin impacts storage, transit and process times during write (marshallers and custom log formatting). Writes are the hardest to scale, so if I'm going to add scale or extra parsing time, I'd rather handle that in Hadoop where I can throw massive parallel resources at it.

Next you can achieve much of the advantages of json or protocol buffers by having a defined format and a structured release process before someone can change the format. Add fields to the end and don't remove defunct fields. This is the same process you have to use with protocol buffers or conceptually with JSON to have it work.

Overall there are advantages to these other formats, but the articles like this that I've seen gloss over the havoc this creates with a standard linux tool chain. You can process a LOT of data with simple tools like Gawk and bash pipelines. It turns out you can even scale those same processes all the way up to Hadoop streaming.

NyxWulf··on Fast ID Generation
Can you elaborate on a few things? You say that UUID version 1 was designed for an outdated purpose. What do you mean by that? UUID V1 does sort by time, so I'm not sure where the outdated purpose is relevant to that consideration.

You say you cannot allow the number of bits a UUID spans. Is that a hard constraint, or is that an intuitive desire? in my experience limiting the number of bits is usually to fit on a single machine or in limited memory. However designing a distributed id system is all about scale and minimizing complexity from my point of view.

When you start to scale out to dozens of machines, the weight of these complex methods really starts to become evident.

I resisted UUID's for a long time, but I finally gave in, and you know what? It's one of the best decisions I've ever made. The design simplicity with v1UUID's is amazing. With the simple_uuid ruby gem I can generate 40kish UUID's per machine on a modest ec2 instance. Using NTP our clock drift is in the 10s of microseconds range IIRC. Using Cassandra and Hadoop we don't have to worry as much about fitting into a single machine. The overall improvements have been staggering though. The simplest and most important benefit is none of the services have to read or request information before generating an ID. Now it's generate an id, send the event on it's way, next. I'll leave it as an exercise for the interested reader to map out the synchronization process for generating unique id's without locking across multiple threads and multiple machines.

tl;dr Just use version 1 UUID's and NTP.

NyxWulf··on 500px Terms of Service
I've been through the process of litigating a contract in court. Not fun, but definitely was an exquisite education in the school of hard knocks.

Lawyers throw all kinds of language into a contract, that doesn't mean the intent was there. Vague, unfair, or overly broad language that was not clearly understood by both parties tends to get reduced, removed, or modified in accordance with what both parties reasonably understood to be the intent. So if they tried to get overly sneaky and do something where the basically version says X, but the full clause says we do X,Y,Z, then the courts may very well rule the implied intent was X but Y and Z were not implied and agreed to.

If they understand what they are doing, and plan to strictly abide by the basically section, this is a fantastic idea. If they at some point decide they want to get sneaky and put something in the detailed version that is not covered in the "basically" section, they may well find themselves in a world of trouble if it comes to court.

My experience was a company trying to sue me, despite them breaching the contract. The judge took a dim view of the crazy clauses in there and was pretty sharp with them. I expect the same thing would happen here. A normal user would probably abide by the "basically" section and the courts would probably take that interpretation. Of course YMMV.

NyxWulf··on 500px Terms of Service
I'm not a lawyer, but I have been through the process of litigating a contract in court.

The law does cover this. Contracts are about a meeting of the minds. Do both parties understand and agree to the terms. Things like email, other written communication about the contract or terms are all reviewed at trial. It's amazing how much of the "sneaky" verbage just get thrown out or ignored as unreasonable or not understood.

NyxWulf··on Why NoSQL Equals NoSecurity
Initially it was not encrypted and was stored in plain text in the database. Then we started encrypting sensitive information at rest, both the database files and the backups. We also encrypted certain sensitive information with a secondary encryption. The next massive undertaking was to find all of the processes that use the data, how they use it - both internally and externally, and then ensure the data was protected throughout the entire process. The company was still working through that process when I left. They had identified many places that were problematic and were actively working on fixing them. The team I managed was the payment processing and cash management team. There were a number of legacy systems written in access, fox pro, DTS, Excel even. Many of those were the most painful, and frankly the most problematic.

Many people were put in a situation where the requirements for getting something implemented through the IT teams would take months, yet they were still held accountable for getting things done, so they would build an Excel process to get the job done. They would build some fantastic process, and get promoted, then that Excel system would get dumped on either someone in IT or someone replacing them.

I'm getting off track, but looking at it superficially the data was encrypted. If, however, you watch the data through it's entire value stream, there are many places it is vulnerable. Given the state of affairs in most companies and how few of them think in terms of managing value streams, I expect my observations are fairly widely applicable.

NyxWulf··on Why NoSQL Equals NoSecurity
We use the authentication in our apps. It's simple, and as Zephyr suggested I wouldn't recommend using that as your only security - it's not our only security, just part of the overall approach.
NyxWulf··on Why NoSQL Equals NoSecurity
I'll play. I worked at a financial institution for several years. On the one hand you'd think we were secure because we used a relational database, but we also had a heavily segmented network with all of the databases in their own VLAN's with highly restrictive permissions. On top of that each application was locked down to exactly which data it could use and what could be done with that data.

So if you buy the premise of the article, it wasn't NoSQL, so it must have been secure. In reality we were very secure from external breaches, at least as secure as you can reasonably be.

Personal information was secured to the maximum extent possible. Encrypted, backed up to multiple independent tapes stored in separate locations etc. On the other hand, the overall complexity of the system was very high and if someone breached our network there would have been a very large surface area to attack. I didn't feel like the article addressed this aspect of security in the least. Security just isn't one of those things that are easy or are ever done. It takes constant vigilance and continuous adjustment and improvement.

NyxWulf··on Why NoSQL Equals NoSecurity
We use hadoop, cassandra, and redis in production. I've never found Information Week to be a high quality source of information, and unfortunately this is no exception. Data systems almost always rely on being in segmented and protected portions of a network with access being controlled by applications and firewalls.

Overall this article struck me as having an axe to grind. I wouldn't put, SQL Server, Oracle, MySQL or any other database on the web with open access. Just like we don't do that with any of our NoSQL solutions. Generally, I've found that security granularity is not as robust in the newer Big Data tools, but that was the same argument against MySQL and Postgres a decade ago.

* Edit: Since we are playing on names, apparently Information Week = Information Weak.

NyxWulf··on Amazon S3 - 905 Billion Objects and 650,000 Requests/Second
They are all for 1/4 of the year, it's just that the other segments are for Q4 and that segment is for Q1.
NyxWulf··on The "Hello World" of genetic algorithms
I've read through a large portion of the PDF version of your book quite a few times. It is very well written and helped me understand a number of different techniques very well.

I definitely recommend Sean's book as a starting point if you are interested in this field.

(I don't know Sean, and don't get anything for this recommendation other than the happy glow of helping a well deserved author get more recognition)

NyxWulf··on Tesseract is now Crossfilter
From a branding point of view I like this name much better. It's much easier to communicate, two simple words with obvious spellings. If I say hey you should checkout Crossfilter, they can easily look that up. With Tesseract, not so much. Crossfilter is also suggestive of what it does, and the name helps to reinforce what it's for. Overall, very large improvement, imo.
NyxWulf··on Go to Trial - Crash the Justice System
This is an interesting idea. It's hard to say how probable it is, because that requires certain tipping point actions to occur like we saw in the Arab Spring to get a mass movement going.

The reality is though that our current system is based on the compliance of the people within the system. When you find a pressure point in the system like this, it's very possible that something will come along and really start to push on it.

Maybe the most realistic way to increase capacity if they did this would be to more efficiently process things going to trial and those who deal. From an efficiency point of view there are many things that could be done to make this process happen much faster, the question is whether or not the Justice System would go for it.

Little's Law shows the overall behavior of a System: I = RT

Where inventory is equal to the arrival rate x time in the system. If the Arrival Rate suddenly doubles, inventory or time in the system must go up, often in non-linear ways. The question is always probability, but if the arrival rate went up, it would certainly shock the system. Whether or not it would crash it is a different question that is much harder to predict, it all depends on the scale and suddenness of the system shock.

NyxWulf··on The iPad Is Unbeatable
This article was focused on the business landscape and Apple's competitors. Saying that some new technology may emerge in the next decade or two to unseat it is missing the point. If the Ipad turns out to be unbeatable for the next decade or two, some very big companies will be going out of business I suspect.
NyxWulf··on How to find a short domain name
I just checked out lean domain search, it actually looks pretty handy. I like how fast it is at generating domain names and checking for availability. One feature that would make it much more useful is the ability to specify multiple words and have it do a combinatorial check on them against your other words.

One other thing, the "LDS" favicon took me by surprise. I had tabbed away for a second, then coming back I saw LDS and thought "now how the hell did I get on Mormon church site?". Only after I clicked back to it did I realize it was your site. I live in Utah, and the Mormon church is very prevalent here, so I don't know if others would have the same thought or reaction. Just thought I'd mention it. Good luck with leandomainsearch

NyxWulf··on Big Data's invisible open source community
I'm at the airport heading home from Strata right now. I have to say the conference was very disappointing. There were far too many commercially "sponsored" sessions. There seemed to be no quality or content guidelines for the talks and most of them were either sales pitches, vague and fluffy but full of hype and buzzwords, or they were entry level.

This was my first time at Strata, but compared to other technical conferences, I walked away thinking it was a serious waste of time.

It's possible that's just a sign of the gold rushy nature of Big Data right now, but as a serious practitioner, I can't recommend this conference.

-- Edit I will add that it's possible my selection turned out to be very poor, and there were good sessions I didn't make it to. I really hope that was the case actually. If anyone was there, did you find any sessions particularly valuable? I would be interested in input from people who are doing Big Data now, not those thinking about it.

NyxWulf··on Stochastic Pattern Recognition Dramatically Outperforms Conventional Techniques
I believe that was in reference to the construction of logic gates. In that context I don't think probabilistic gates are considered conventional.
NyxWulf··on Cogs bad
The systems I work on are pretty sizable in scale, right now we are doing about 120 million calls a day on one of core clusters. I don't claim to be the end all authority on scaling, but I do have some observations based on what I've dealt with.

First off, that type of architecture is brittle to logic changes. If you have a completely static architecture that you don't need to change it may be fine, but deploying those changes to every machine in the cluster is problematic.

Second, not all components have the same underlying machine requirements. For instance, our nginx servers don't need much ram, but the HAProxy load balancers with nginx that terminate SSL need a lot of Ram and a good chunk of CPU. Hadoop works better with JBOD (Just a bunch of disks), whereas cassandra seems to work better with a raid 0 configuration. Certain layers like the Nginx through certain paths have real time requirements which means ultra low latency. Other things need to operate against massive data sets and compute answers in a few hours.

So, not every machine in your architecture can have all of the services required by every other part of your architecture. A lot of it depends on workload types and what the underlying requirements are for your system. There are many more reasons, but I'll leave it at those for now. Ultimately the post is right that a single machine can work very well, but it's also misguided in just dismissing the HN commenter. There are many other cases where distributed architectures are required. Guaranteeing robustness and performance in the face of service and machine failures is very difficult, and is essentially impossible on a single machine.

Which approach you apply depends, it depends on the unique situation and the unique constraints you have. Using a single model to solve all problems seems to be worse than using no model. Learn multiple models, learn how and when to apply each (yes - for those of you in the class, I'm taking the model thinking class :) ).

NyxWulf··on Tumblr Architecture - 15B Page Views a Month and Harder to Scale than Twitter
It all depends on the nature of the request. Our nginx servers are tuned to burst to 7000 requests/second and run stable at 4800 requests/second for simple content. Our overall scale is about 1/5th of what Tumblr is doing daily and we have a much smaller infrastructure footprint. Although in fairness our content is no where near as dynamic as theirs is.
NyxWulf··on Judge: Americans can be forced to decrypt their laptops
In support of the notion that Under a microscope, everyone is a criminal I'll add this data point.

The police raided the house of a veteran here in town a few months back for possession of small amounts of marijuana. The veteran opened fire and during the ensuing shoot out one officer died and five more were sent to the hospital. The news continuously talked about this veterans limited criminal history.

Ultimately he had a speeding ticket and had been charged with something like jay walking which was dismissed. That was his limited criminal history.

I completely agree that if you look at someone close enough, almost all will have limited criminal histories.

NyxWulf··on Alexis Ohanian vs NBC - Debating SOPA
I agree. In particular I don't find the efficacy argument very strong. Making murder against the law doesn't stop it either, does that mean we should just let it go?

China also knows that their censoring doesn't stop everyone, but it stops an awful lot of people. So saying it's not going to work and then criticizing China for censorship feels like talking out of both sides of our mouth.

The broader implications are that his has a chilling effect on any type of user generated content and free speech in general.

It turns international business entities into enforcement arms for the government. Particularly problematic is that it's an attempt to extend the reach of our laws into jurisdictions with different laws and to do so through a business. People get all fired up when someone violates one of our laws, but they don't like it when China or Muslims attempt to push their laws into our jurisdictions.

NyxWulf··on A Ruby-based lightweight publish-subscribe messaging system
I haven't seen any one product that does all of that, but generally I want to separate heartbeating and status updates from the other queue functionality. ZooKeeper will do a lot of the heartbeating, presence, status changes, list clients, etc. I don't think it's designed to function as a queue though. I think with a combination of Redis and ZooKeeper you could get what you want.
NyxWulf··on [dead]
I have to say that this type of behavior is the exact thing that moved me from Reddit to HN. Something one of organizational behavior professors said stuck with me: anonymous online discussions have a disinhibiting effect. The point was clearly along the lines of what the post discusses.

Unfortunately Clay Shirky's post about a group being it's own worst enemy (http://www.shirky.com/writings/group_enemy.html) appears to be startlingly accurate.

NyxWulf··on Don't Outsource
Bribing the foreign officials would definitely violate the foreign corrupt practices act. Paying the developer 100k to sever the relationship would not have been a bribe. They may have been outraged by that, but depending on the nature of the relationship that may have been a reasonable fee. It also sounds like it would have been a lot cheaper in the long run.
NyxWulf··on Don't Outsource
Criminality is related to breaking laws. Doing business internationally is difficult because the laws are not consistent and cross-border enforcement is difficult in the best of cases. When entering into a business relationship you often don't know much about the people you are doing business with. People make similar mistakes in the U.S. by giving full and complete access to their source code to unknown and untrusted agents and then get screwed.

To be fair though, the developer probably has a different viewpoint. I've been screwed out of ridiculous sums of money on different deals through unscrupulous business owners thinking they had me over a barrel. Ultimately I always found it cheaper to walk away rather than trying to resolve it even in the U.S. court system. I have to wonder what would cause this person to react so hostilely for such a sustained period of time. This is personal somehow, there is no profit in this for TSC according to the story (100k wouldn't justify this).

There are always two sides to every story, this is suspiciously one sided and perhaps more disturbing it paints the entire indian culture and economy with the same broad brush.

The lesson here should be that development in general is hard. Outsourcing is even harder. Business disputes often get very messy, cross border disputes even more so. When engaging in any business transaction give serious consideration to what happens if this transaction goes south. Do they have access to your core systems? How bad can they hurt you? These exercises are just as valid with internal developers as they are with any external agent. Once you grow beyond the startup stage, building a sustainable and protectable business should be one of the main focus points of any management team. These guys made mistakes, who doesn't. This developer is acting childishly, it happens. Don't over-generalize from this single example.

NyxWulf··on A single line to 3 cashiers is ~3x faster than a separate line for each cashier
Queueing theory is a prominent topic in Operations Research. If you want to read a detailed book on it, Fundamentals of Queueing theory by Thompson et al is the bible.

A couple of additional thoughts, pooling queues like this only works if the changeover time to process different types of things is low. So in the case of a grocery store where there is no separate changeover or batch setup time it works well. If you are pooling work to people though it breaks down if only certain people can do certain types of work or the change over time to switch to a different task exceeds a certain threshold.

The other thing is the title of this post is almost certainly incorrect. Switching to a single queue reduces average queue time, but it does not reduce work time. In order to be 3x faster you would have to reduce both. The other factor at play is that people continuously rebalance from slow lines to fast lines, which also helps mitigate some of the problems.

* some grammatical edits

NyxWulf··on Clayton Christensen: How MBA-driven Profit focus is Killing the U.S. Economy
My experience was more like yours as well. Here are some metrics you can use to evaluate certain things, but here are the 50 problems with using this metric in isolation.

Generally I think the curriculum taught in my school was fantastic, but I think there is a segment of my peers who failed to grasp the underlying meaning and walked away with simple metrics and formulas. Just like cut and paste programmers or those in my CS curricula who learned a few things but did not grok fully, they give everyone a bad name when they egregiously misapply concepts.

Just like those people who shallowly apply metrics, labeling an MBA as basing all decisions on IRR and then painting all MBA's as alike is shallow and overly broad. For example while accounting and finance were covered, there is also economics, operations, strategy, negotiation, leadership, communications, marketing, managerial accounting, and more. All of which look at things as much more than a collection of financial metrics.

The world is a very complex place. Simple sentiments, arguments or sound bytes rarely capture the essence of even the simplest underlying systems. Sweeping generalizations claiming to explain all of the problems of an extremely complex dynamic environment set all kinds of alarm bells ringing in my head.

NyxWulf··on High IQ linked to drug use
My experience with pharma was similar to the GP sans the weed. I take Adderall.
NyxWulf··on Redis 2.4.0 final was just released
@antirez and the Redis team.

Thank you for the wonderful work you do. Redis is an integral part of our infrastructure, and we have truly enjoyed working with it.

NyxWulf··on Redis 2.4.0 final was just released
The repos update too slowly for them to be up to date on Redis, at least at the current pace Redis is moving at. You can find the latest build here: http://redis.io/download

it's pretty simple to build and install. Setting up a production environment takes a few more steps. Here are a couple of posts that will get you there: http://www.denofubiquity.com/nosql/412/ http://jramirez.tumblr.com/post/2589232577/prime-time-redis-...

Those are both written for 2.2, but my config still works fine with 2.4 using settings derived from those posts.

NyxWulf··on In response to This*: Stop letting engineers get away with social ineptitude
Sociopath is a tough word because it means a lot of things and it really isn't used in clinical diagnosis. Based on the historical patterns though a sociopath was typically someone lacking any type of empathy, so they would do things hurt other people just to amuse themselves by watching the results. There is no where near enough information to diagnose sociopathy presented.

If you look at the Anti-Social Personality disorders, this may fall closer to a Cluster B - Histrionic. Which is what many people consider someone with "anger management" issues.

I agree with what I think is the intent of your post, that people with a personality disorder are suffering from something they may not have control of. However, I do think anger management issues are a disorder and are not of a kind with being "black" as there is no inherent disorder associated with being black.

I think that employers have a very real, and legitimate interest in not employing people with severe personality disorders, particularly if they are not under control or in treatment for the disorder. Even then, it doesn't make sense to put some people in a position where they can cripple your company if they are very upset over something.

← PreviousPage 4 of 7Next →