HNHacker News
TopNewBestAskShowJobs

NyxWulf

933 karma · joined November 7, 2008

submissionscomments
NyxWulf··on The Evidence Supports Artificial Sweeteners Over Sugar
I learned this from my doctor (who is a diabetic specialist), some years ago we were talking about how to form good habits for kids and my wife and I were avoiding caffeine beverages for the kids, but letting them have apple juice etc. His opinion based on the evidence is that he would much rather have our kids drink caffeine every day but stay away from the sugar. His specific point was that people on the internet point to unknown risks with aspartame or caffeine, but ignore the mountain of known risks associated with sugar.

It was an eye opening discussion to say the least.

NyxWulf··on I Fooled Millions Into Thinking Chocolate Helps Weight Loss
For me the take away is that non-trivial independent replication of results still stands as the gold standard for experimentation. P-hacking shows up clearly when the experiment is replicated. P-values in my experience are only a starting point for further experiments.

Take the cold-fusion debacle. When independent groups attempted to replicate the results they were unable to do so.

Of course the other big take away is that people really don't understand statistics at a basic level, even the "scientific reporters".

NyxWulf··on Command-line tools can be faster than your Hadoop cluster
This also isn't a straight either or proposition. I build local command line pipelines and do testing and/or processing. When either the amount of data needed to be processed passes into the range where memory or network bandwidth makes the processing more efficient on a Hadoop cluster I make some fairly minimal conversions and run the stream processing on the Hadoop cluster in streaming mode. It hasn't been uncommon for my jobs to be much faster than the same jobs run on the cluster with Hive or some other framework. Much of the speed boils down to the optimizer and the planner.

Overall I find it very efficient to use the same toolset locally and then scale it up to a cluster when and if I need to.

NyxWulf··on Giving Away Our Recommendation Engine
You can use Amazon Elastic MapReduce to provision your own cluster. I'm guessing the value prop they bring is the ease of handling that part for you.
NyxWulf··on Variance
You are loading the JavaScript library from the CDN into a browser. The data also goes to the browser, not to their CDN. So if sending your data to them is the show stopper for you, that shouldn't stop the show since that's not what happens.
NyxWulf··on Why Puppet, Chef, Ansible aren't good enough
Yeah that xkcd came to my mind as well. I'm amazed at the timeless truths he captures in those comics. Creating another version which may be better isn't the hard part. Gaining consensus and getting people to give up the other ones is the hard part.
NyxWulf··on Unix Command Lines and Relations
wc -l is faster however it only works on an uncompressed file. The pipeline in the article was doing much more complicated work on the stream of output. I will often separate my pipeline logic from my "read this data and feed it into the pipeline" logic, since the logic around what to read in and where to write it can often change even when using the same logical pipeline.

As a simple example: cat somefiles.txt | pipeline_script > stored_output.txt

Other times I'm reading gzipped files or remote log files, I don't want that data mixed in with the pipeline logic. If I want to move the output files to a different directory on one set of servers, that may not impact my generalized pipeline.

I work with trillions of lines of log files and there are many ways to scale up pipelines. I wouldn't start optimizing the difference between 1.6s and 1.275s unless it made an economic impact on the problem. How often is it run? Can you process more lines in an economic unit of time with the faster version? If this is a job that runs once an hour or even once a minute how many lines are collected during that time?

Intermixing the grep or wc -l logic into the pipeline logic can have adverse support and maintenance costs and many times saves machine time while spending programmer time. Which one dominates in this scenario?

It's like the old saying...there is more than one way to skin a "cat". :)

NyxWulf··on Your top 100 unix commands, for science
This works for me in zsh on OSX

history \ | sed "s/^[0-9 ]//" \ | perl -pe 's/ \| */\n/g' \ | gawk '{counts[$1] += 1} END { for (x in counts) { print counts[x],x}}' OFS="\t" \ | column -t \ | sort -k 1,1nr \ | head -100

Ok not sure how to format this on HN https://gist.github.com/nyxwulf/5608955#file-gistfile1-sh

NyxWulf··on The Beauty of Concurrency in Go
Observing other channels, or seeing when a channel does fire is what the select structure is for. It is one of the features that makes channels and goroutines very nice in go.

http://golang.org/ref/spec#Select_statements

http://blog.golang.org/2010/09/go-concurrency-patterns-timin...

http://talks.golang.org/2012/concurrency.slide#1

Video for the slides: http://www.youtube.com/watch?v=f6kdp27TYZs

NyxWulf··on General purpose programming languages' speed of light
Interesting concept, but the flaw at the heart is the presupposition that lack of progress right now means we are at the ultimate limit of what can be accomplished. Imagine if cavemen learning to paint on walls said, well we haven't improved in a few millennia, so this is probably the most complex thing that can be represented by drawings. Or what about math stopping with Euclid? It was thousands of years later that progress happened.

Technology comes in fits and starts. A lot of new things happened in the 50s and 60s, we are still trying to figure out ways to use and apply them. Just because someone thought about and prototyped something then doesn't mean it's not new when that feature goes mainstream (e.g. garbage collection in Java, channels for concurrency in Go, etc).

For a long time we couldn't break the sound barrier, which is a limit, but not the speed of light limit. Because progress is stalled now doesn't mean there will never be progress in the future.

NyxWulf··on Ecomom's Founder and CEO Jody Sherman Passes Away
When I was in high school one of the students committed suicide. There was a lot of discussion and outreach at the school at the time. One of the things they talked about was that once one student commits suicide it greatly increases the chances that others will follow.

My mother is also a social worker at the VA Hospital, and she's had some of her patients opt out so to speak. It does appear to have a very strong pull on other people who are suicidal to choose that path.

So if anyone reading this is having those thoughts, please, please talk to someone professionally. Call the suicide hotline in your area. There are very real things that can be done to help you through this.

NyxWulf··on Why political journalists can’t stand Nate Silver
It seems like you are confusing predictions about this election with overall model accuracy.

The state by state predictions are the primary method by which you can judge the accuracy of his model. If his model forecasts a state as 60% for one candidate, you can assess that level of accuracy by looking at all 60% predictions.

Let's take some examples from todays date on FiveThirtyEight.

* Florida 54.8% chance of Romney win

* Virginia 67.0% chance of Obama win

* Nevada 67.9% chance of Obama win

* North Carolina 79.6% chance of Romney win

* New Hampshire 80.4% chance of Obama win

* Iowa 80.7% chance of Obama win

* Nevada 88.7% chance of Obama win

Other states like Texas, Utah, Idaho, Wyoming are projected at 100% for Romney and New York, California, Oregon, and Illinois at 100% for Obama.

If any of the 100% states go to the opposite candidate, that is a model problem. If some of the ones specified with extremely high percentages 95% go against his predictions with a high margin of victory, again that is a model problem.

Finally, some of those close races should go against the models prediction.

Let's take the 7 states listed above. 5 of the 7 are projected for Obama and 2 for Romney. However there is only about a 37.68% chance of that exact distribution happening. I break it down as follows:

Obama-Romney * 0-7 0.02%

* 1-7 0.14%

* 2-5 2.26%

* 3-4 12.14%

* 4-3 31.48%

* 5-2 37.68%

* 6-1 16.00%

* 7-0 0.28%

Each of these numbers are probabilistic statements about the likelihood of the overall event occurring based on the probabilities. They are from a single simulation of the 7 state probabilities run 10,000 times. Each of them has it's own distribution, eg 5-2 was 37.68% in the first run, 37.38% in the next, then 38.03%, then 36.85%, etc.

This is a very simple model prediction, but by taking all 50 states into account you can get a very clear assessment of how well his model is actually predicting the outcome of the election. Complicating matters is the time series nature of the predictions.

This type of model is precisely the way you get away from "Fooled by Randomness and all that". His model is clearly articulating the amount of uncertainty in the forecast.

NyxWulf··on Python for Data Analysis
You can get the kindle version now. John Cook may have gotten an advance copy of the book.
NyxWulf··on An Operations Mindset is at Odds with Innovation
The arguments advanced against an "Operations mindset" are mostly strawmen. Operations is primarily about efficiently operating an organization towards a goal. The type of thinking advanced in the article however is more likely to be pushed by someone with no operations management training.

For instance, putting people on multiple projects versus single projects. Multi-tasking and context switching cause known losses in efficiency even under the perfect scenario with no startup time. So if you want to emphasize getting projects completed, multi-tasking must go. This is an operations management mindset, though it runs against much current practice.

The points about efficiency are similarly misguided. You don't focus on worker efficiency, operations points you at throughput and global optimization rather than local optimization.

Those however are about focusing on improvements and operations of the current systems. It is true that innovation is different than operations. In the same way that learning to program efficiently in a language is different than inventing a programming language. They are different kinds of things. Many operations principles however can be successfully applied to innovation. For an excellent read on the topic, read "The Principles of Product Development Flow" by Reinertsen.

To get a better understanding of operations in general, and not the hyperbole advanced in the article a good starting place is "The Goal" by Goldratt.

One big revelation I learned while studying operations is that the things that largely drive developers crazy aren't good management practices. They aren't advanced by operations researchers. They are in fact what I always thought they were, bad management practices.

NyxWulf··on Tomorrow Theme
I use vividchalk, which I was happy to see there on the list. It's amazing how many variations on color schemes there are.
NyxWulf··on Show HN: GoScale Cloud Scaling in Milliseconds
That's the difference between guaranteed capacity and most services. It's very common to use the properties of normal usage patterns and pool the excess capacity. The phone system has done that for ages. Ever try to make a call during an emergency?

Just because their service doesn't cover certain extreme situations does not mean it's worthless. It would worth knowing how much excess capacity they are pooling together, but vendors generally won't release those types of details.

NyxWulf··on Your car, tracked: the rapid rise of license plate readers
In Utah the year is the only one that changes color, all of the months are white. That's why I was looking at his plates, to see if I could see even the letters on his plate. Granted I'm not a trained observer of registration violations and maybe it was observable to someone who is. Given the distance and the visibility during the night I'm pretty skeptical. It's just a sneaking suspicion I've had - because I know I'd want to use those license plate readers in exactly that why if I was in their place.
NyxWulf··on Your car, tracked: the rapid rise of license plate readers
I knew it! Roughly a year ago I was driving home from work at night. My registration was expired by a few days, and I drove past an officer in a parking lot. As I was passing him, I knew my plate was expired, so I was looking at his plate to see if I would be able to determine from that distance if the plates were expired. So I was scrutinizing his plates to see how much detail I could see. I have better than 20/20 eyesight, and I knew there was no way I could see that level of detail at the distance he was at. So imagine my surprise when he comes after me immediately and pulls me over.

More out of curiosity than anything I asked him how he knew my plate was expired, his response was "I have really good eyes". I've thought back on that experience over the last year, and I knew the technology for reading a license plate wasn't hard, but I didn't know how widespread the use was. After reading this piece and seeing the picture of those cameras, which are mounted in that area, I'd bet they are using it to give expired registration tickets too.

NyxWulf··on Is Algebra Necessary?
Even if people don't learn to "properly" understand the Gaussian distribution, basic understanding of the nature of variation and the difference between random events and something that indicates an actual change in an underlying system would be immensely useful to people.

You don't have to know all of the fundamental underlying ideas of any concept before it has practical utility.

NyxWulf··on The Value of Time
Let's take the salesman example. A salesman isn't literally making $150/hour. As a hypothetical the value might be $0 in hour 0, $450 in hour 1 and $0 in hour 2 for an average of $150/hour. Much of sales work is this way, their is preparation time, and sales time. There may be prospecting, etc. So even though they may not make the money during that hour, if you are burning their motivation to prep or prospect you are burning their ability to get through hour 0 and into hour 1.

Part of the reason to use an average is that it gives you a very simple, but generally applicable metric. There are always exceptions and the rate may change frequently or slowly. Those are nitpicks though. The general point is that your time should be valued against the best use of your time. For instance the hotel example only makes sense if the alternative to walking to the convention is working or more time networking to generate revenue. If the alternative is sitting in your hotel room longer watching TV then $160/hour is not the replacement rate.

NyxWulf··on AWS is down due to an electrical storm in the US
That's our next step is automating our fail over. We currently use DNS Made Easy and while you can do that with them it's tougher. We are switching to Dyn, and their Layer 2 failure detection will put our failover in that 6 second range.
NyxWulf··on AWS is down due to an electrical storm in the US
If you operate a service where uptime is critical, two of everything isn't enough. You have to ensure they will be separated geographically and logically so that when things like this happen, you don't lose your systems.

We operate systems that sit on the pages of the top e-commerce companies in the world. We have 10 separate segments of clusters. Operating in four AZ's in East, three AZ's in West-1 and two AZ's in West-2. When this outage happened, the servers that were impacted in east were removed from our DNS and within 9 minutes the impact of this event on their sites was eliminated.

NyxWulf··on Rogue Amoeba responds to Phil Schiller
heh, I was about to, but then I saw your response. Thanks.
NyxWulf··on The danger of using averages alone in web analytics
I agree with many of the points in the article, however it is conflating two unrelated things. Using simple averages in isolation is almost always a mistake. The real story in a system almost always lies in it's variation and not in the average. There are many ways to look at that, one of them is a histogram, although I tend to prefer time series graphs with moving ranges. One of my favorites is the Xmr chart or any of a number of statistical process control charts. Using those you can get a much better understanding of systems over time.

The transition to Real User metrics is not at all related to using an average. All recording and instrumentation processes will benefit through using more sophisticated and nuanced tools than a simple average. Real user metrics are an important data point, but they lack certain information you can get from external monitoring systems. We use both pingdom and catchpoint, by far my favorite is Catchpoint because I can see things like what ISP is involved with a slow request, what geographic region, etc. I can also get scatter plots and nice statistical graphs around median, geometric mean, 75th, 95th, 99th percentile.

So in short the main points are good, simple averages are misleading. Capturing end user performance data is good. Not using external monitoring though isn't a good idea because there a number of things that you can identify if you have that insight.

NyxWulf··on Disposable Email address
For this type of service spamgourmet is my long time favorite.

Create disposable e-mail addresses that forward to your real e-mail address. You get a name stub and then you can create e-mail addresses on the fly. For instance you can have something like "hacker" as your name stub, and they have 20ish domains to use. So it works like this, say you are signing up for a subscription with inc you can then use inc.hacker@recursor.net or inc.hacker@neverbox.com or inc.hacker@spamgourmet.com. Whichever you prefer as the domain. In this case it would use my default burnout rate which is set at 3 when you signup. I bumped mine up some, but you can also specify it. So if you wanted say 10 e-mails before it burned out you could use inc.10.hacker@recursor.net. If you don't do anything else, once you receive 10 e-mails at that mailbox it will burnout and start eating them. You can also go whitelist the domain so if you get an e-mail from inc.com it will come through, but it will use the normal burnout settings for e-mail from anyone else. There are also much more advanced features for when/if someone figures out how this works and tries to spam you like setting watchwords. So if an e-mail doesn't contain the current watchword it will not allow the creation of a new e-mail address, but old ones continue to work.

One thing that is nice is the interface shows you how many e-mails have been received on a given address versus how many have been eaten. For instance when I signed up with inc, they apparently gave my e-mail away and I started getting e-mail on that inbox, so I let it burn out. Now they can no longer reach me. Market Samurai has been incredibly bad, they've had my e-mail address for almost 2 years and I have already received 2500 spam messages on that inbox.

The great thing is this is all free and open source. You can donate, and if you use the service a lot, you probably should. You can also download all the source and set one up yourself. Overall the current setup is not user friendly by design. In the FAQ it specifically discusses they don't want to support end users who can't take the time to read the FAQ. Overall I cannot overstate how great this has been. It solves such a major source of pain and does it in such an elegant way. If you haven't found a service you are happy with, give spamgourmet a try.

NyxWulf··on Big Data's Big Problem: Little Talent
Most business people aren't interested in model accuracy as a term. They want something that provides benefit, e.g. cost savings, increased revenue, increased profits, etc.

The sales process of convincing someone they need an accurate model is tough, especially because robust models are time consuming and expensive to build.

If you can come up with a model that shows good results, and people know they need those results, then you can start a company selling either a service or product to get those results. If people don't know they need your results - then you have to educate them, in which case it's a much more difficult business to start.

I don't know many business people with the temperament, understanding, or the pocket book to deal with general research type problems.

NyxWulf··on Big Data's Big Problem: Little Talent
I couldn't agree more. Accuracy is a problem, variation is another problem. Dealing with layers in the business who have no math or statistics background but very strong opinions is yet another complication.

These types of conversations aren't uncommon.

Other - "I need you to prove our stuff does X, Y, and Z".

Me - "Ok.."

<time elapses>

Me - "Ok the data shows our stuff does X but Y and Z are just random noise"

Other - "We ran it once before with this other guy and it showed our stuff did X,Y and Z. We've been promising it to our clients for a year. He gave us several examples, but when the clients asked to see the underlying data he couldn't produce it. So we just need you to prove it does X,Y, and Z."

Me - "The data only shows it does X. Y and Z are impacted positively through X, but once you condition on X, Y and Z are not causally affected by our stuff"

Other - "Yeah...well I promised client we would give them a report by {{insert random ridiculous date here}} proving it did X, Y and Z. We are going to lose them if we don't deliver a report saying that"

Me - trying for the 50th time to explain they shouldn't promise a positive result when we've never looked at the data.

There are hundreds of variations on this conversation. Your code is wrong is one variant (which depending on the timeline is hard to dispute). Of course if you take long enough that your code is correct, then you are going to slow. This isn't a science experiment, just make it work is another. Watching someone go slack jawed and start drooling because you accidentally used a math term is always interesting.

I have a whole new perspective of being on the cutting edge. It seems like it mostly means you are on the cutting edge of comments from people who don't know how ridiculously hard what you are doing is.

NyxWulf··on Big Data's Big Problem: Little Talent
I agree with Paul's comment. The 250 number feels low to me, but that is applying a specific model. Typically people come with some set of favorite models, and many of them provide that vast majority of the benefit a business needs. Especially when the current model in use is slipshod and busted at best.
NyxWulf··on Big Data's Big Problem: Little Talent
As someone in the big data field on the ground (VP of Engineering). Let me give you my thoughts on it.

Your impression about the hype is correct. There are a lot of vendors offering BIG solutions, if you pay them BIGGER money. Where I used to translate the word enterprise to $$, now I translate Big Data to $$$$$$$.

When I'm hiring, I don't go looking for Big Data people, because generally they don't exist. Statistics is a really great general addition to a programmers toolkit. Machine Learning is valuable as well, although in my experience the application is more limited. What this article doesn't mention is a whole host of other skills required.

Modeling, and not just a formal mathematical model, but applying any type of model to your data to get insight. Check out the model-thinking class on coursera.

Exploratory Data Analysis, much different skill than confirmatory statistics.

Design of Experiments, specialized subfield within statistics.

Logistics, how to setup, maintain, and maximally utilize an efficient distributed cluster and build a pipeline getting your data to the cluster, cleaning it, building it into a model, and then extracting insight and delivering that end value.

Those are a couple of the skills at a high level. At a more nuts and bolts level, Hadoop is the defacto standard for Big Data. Learning how to build a data pipeline out of the Linux tool chain is very common in the data science world.

The overall value stream for Big Data is deep and wide. Most companies don't have expertise in much of these, and so at the current time you have to learn them yourself or find a company focused on building a team around it.

If you are just learning this yourself, you'll probably get an academic knowledge. If you want to make yourself valuable in the marketplace, you'll really want to get hands on experience. Knowing a z-score is one thing, building a process to gather data and compute a model against it is a whole different ball game. As the article mentions, if you have nice clean data it's easy to apply a model. If you have messy ugly data from 20 different vendors and 200 clients with various failures, anomalies, and you have to figure out what type of model is helpful, oh and you have a deadline because for 500th time someone promised something impossible to the client, then you have something closer to what Big Data is today.

* grammar edits

NyxWulf··on Big Data's Big Problem: Little Talent
It's not that you can solve every problem, but supply and demand are very real. If the wage rates for Big Data get high enough more and more people will try out the field, which will generate a larger supply.

The reasons companies don't just throw out huge salaries though has to do with the demand side. The salaries companies are willing to pay is related to the marginal advantage they can gain from hiring someone with that skillset. If for example a company will gain say 200k per year in total advantage, that would place a hard cap on how much they would be willing to pay in salary.

So if the advantage is very high, companies will pay more. If the supply increases sufficiently wage rates will drop because there is over supply. If the supply doesn't increase enough, wages will increase more - however each company will drop out at it's own value point. This provides the natural limit to where most salaries cap out.

← PreviousPage 3 of 7Next →