It was an eye opening discussion to say the least.
933 karma · joined November 7, 2008
It was an eye opening discussion to say the least.
Take the cold-fusion debacle. When independent groups attempted to replicate the results they were unable to do so.
Of course the other big take away is that people really don't understand statistics at a basic level, even the "scientific reporters".
Overall I find it very efficient to use the same toolset locally and then scale it up to a cluster when and if I need to.
As a simple example: cat somefiles.txt | pipeline_script > stored_output.txt
Other times I'm reading gzipped files or remote log files, I don't want that data mixed in with the pipeline logic. If I want to move the output files to a different directory on one set of servers, that may not impact my generalized pipeline.
I work with trillions of lines of log files and there are many ways to scale up pipelines. I wouldn't start optimizing the difference between 1.6s and 1.275s unless it made an economic impact on the problem. How often is it run? Can you process more lines in an economic unit of time with the faster version? If this is a job that runs once an hour or even once a minute how many lines are collected during that time?
Intermixing the grep or wc -l logic into the pipeline logic can have adverse support and maintenance costs and many times saves machine time while spending programmer time. Which one dominates in this scenario?
It's like the old saying...there is more than one way to skin a "cat". :)
history \ | sed "s/^[0-9 ]//" \ | perl -pe 's/ \| */\n/g' \ | gawk '{counts[$1] += 1} END { for (x in counts) { print counts[x],x}}' OFS="\t" \ | column -t \ | sort -k 1,1nr \ | head -100
Ok not sure how to format this on HN https://gist.github.com/nyxwulf/5608955#file-gistfile1-sh
http://golang.org/ref/spec#Select_statements
http://blog.golang.org/2010/09/go-concurrency-patterns-timin...
http://talks.golang.org/2012/concurrency.slide#1
Video for the slides: http://www.youtube.com/watch?v=f6kdp27TYZs
Technology comes in fits and starts. A lot of new things happened in the 50s and 60s, we are still trying to figure out ways to use and apply them. Just because someone thought about and prototyped something then doesn't mean it's not new when that feature goes mainstream (e.g. garbage collection in Java, channels for concurrency in Go, etc).
For a long time we couldn't break the sound barrier, which is a limit, but not the speed of light limit. Because progress is stalled now doesn't mean there will never be progress in the future.
My mother is also a social worker at the VA Hospital, and she's had some of her patients opt out so to speak. It does appear to have a very strong pull on other people who are suicidal to choose that path.
So if anyone reading this is having those thoughts, please, please talk to someone professionally. Call the suicide hotline in your area. There are very real things that can be done to help you through this.
The state by state predictions are the primary method by which you can judge the accuracy of his model. If his model forecasts a state as 60% for one candidate, you can assess that level of accuracy by looking at all 60% predictions.
Let's take some examples from todays date on FiveThirtyEight.
* Florida 54.8% chance of Romney win
* Virginia 67.0% chance of Obama win
* Nevada 67.9% chance of Obama win
* North Carolina 79.6% chance of Romney win
* New Hampshire 80.4% chance of Obama win
* Iowa 80.7% chance of Obama win
* Nevada 88.7% chance of Obama win
Other states like Texas, Utah, Idaho, Wyoming are projected at 100% for Romney and New York, California, Oregon, and Illinois at 100% for Obama.
If any of the 100% states go to the opposite candidate, that is a model problem. If some of the ones specified with extremely high percentages 95% go against his predictions with a high margin of victory, again that is a model problem.
Finally, some of those close races should go against the models prediction.
Let's take the 7 states listed above. 5 of the 7 are projected for Obama and 2 for Romney. However there is only about a 37.68% chance of that exact distribution happening. I break it down as follows:
Obama-Romney * 0-7 0.02%
* 1-7 0.14%
* 2-5 2.26%
* 3-4 12.14%
* 4-3 31.48%
* 5-2 37.68%
* 6-1 16.00%
* 7-0 0.28%
Each of these numbers are probabilistic statements about the likelihood of the overall event occurring based on the probabilities. They are from a single simulation of the 7 state probabilities run 10,000 times. Each of them has it's own distribution, eg 5-2 was 37.68% in the first run, 37.38% in the next, then 38.03%, then 36.85%, etc.
This is a very simple model prediction, but by taking all 50 states into account you can get a very clear assessment of how well his model is actually predicting the outcome of the election. Complicating matters is the time series nature of the predictions.
This type of model is precisely the way you get away from "Fooled by Randomness and all that". His model is clearly articulating the amount of uncertainty in the forecast.
For instance, putting people on multiple projects versus single projects. Multi-tasking and context switching cause known losses in efficiency even under the perfect scenario with no startup time. So if you want to emphasize getting projects completed, multi-tasking must go. This is an operations management mindset, though it runs against much current practice.
The points about efficiency are similarly misguided. You don't focus on worker efficiency, operations points you at throughput and global optimization rather than local optimization.
Those however are about focusing on improvements and operations of the current systems. It is true that innovation is different than operations. In the same way that learning to program efficiently in a language is different than inventing a programming language. They are different kinds of things. Many operations principles however can be successfully applied to innovation. For an excellent read on the topic, read "The Principles of Product Development Flow" by Reinertsen.
To get a better understanding of operations in general, and not the hyperbole advanced in the article a good starting place is "The Goal" by Goldratt.
One big revelation I learned while studying operations is that the things that largely drive developers crazy aren't good management practices. They aren't advanced by operations researchers. They are in fact what I always thought they were, bad management practices.
Just because their service doesn't cover certain extreme situations does not mean it's worthless. It would worth knowing how much excess capacity they are pooling together, but vendors generally won't release those types of details.
More out of curiosity than anything I asked him how he knew my plate was expired, his response was "I have really good eyes". I've thought back on that experience over the last year, and I knew the technology for reading a license plate wasn't hard, but I didn't know how widespread the use was. After reading this piece and seeing the picture of those cameras, which are mounted in that area, I'd bet they are using it to give expired registration tickets too.
You don't have to know all of the fundamental underlying ideas of any concept before it has practical utility.
Part of the reason to use an average is that it gives you a very simple, but generally applicable metric. There are always exceptions and the rate may change frequently or slowly. Those are nitpicks though. The general point is that your time should be valued against the best use of your time. For instance the hotel example only makes sense if the alternative to walking to the convention is working or more time networking to generate revenue. If the alternative is sitting in your hotel room longer watching TV then $160/hour is not the replacement rate.
We operate systems that sit on the pages of the top e-commerce companies in the world. We have 10 separate segments of clusters. Operating in four AZ's in East, three AZ's in West-1 and two AZ's in West-2. When this outage happened, the servers that were impacted in east were removed from our DNS and within 9 minutes the impact of this event on their sites was eliminated.
The transition to Real User metrics is not at all related to using an average. All recording and instrumentation processes will benefit through using more sophisticated and nuanced tools than a simple average. Real user metrics are an important data point, but they lack certain information you can get from external monitoring systems. We use both pingdom and catchpoint, by far my favorite is Catchpoint because I can see things like what ISP is involved with a slow request, what geographic region, etc. I can also get scatter plots and nice statistical graphs around median, geometric mean, 75th, 95th, 99th percentile.
So in short the main points are good, simple averages are misleading. Capturing end user performance data is good. Not using external monitoring though isn't a good idea because there a number of things that you can identify if you have that insight.
Create disposable e-mail addresses that forward to your real e-mail address. You get a name stub and then you can create e-mail addresses on the fly. For instance you can have something like "hacker" as your name stub, and they have 20ish domains to use. So it works like this, say you are signing up for a subscription with inc you can then use inc.hacker@recursor.net or inc.hacker@neverbox.com or inc.hacker@spamgourmet.com. Whichever you prefer as the domain. In this case it would use my default burnout rate which is set at 3 when you signup. I bumped mine up some, but you can also specify it. So if you wanted say 10 e-mails before it burned out you could use inc.10.hacker@recursor.net. If you don't do anything else, once you receive 10 e-mails at that mailbox it will burnout and start eating them. You can also go whitelist the domain so if you get an e-mail from inc.com it will come through, but it will use the normal burnout settings for e-mail from anyone else. There are also much more advanced features for when/if someone figures out how this works and tries to spam you like setting watchwords. So if an e-mail doesn't contain the current watchword it will not allow the creation of a new e-mail address, but old ones continue to work.
One thing that is nice is the interface shows you how many e-mails have been received on a given address versus how many have been eaten. For instance when I signed up with inc, they apparently gave my e-mail away and I started getting e-mail on that inbox, so I let it burn out. Now they can no longer reach me. Market Samurai has been incredibly bad, they've had my e-mail address for almost 2 years and I have already received 2500 spam messages on that inbox.
The great thing is this is all free and open source. You can donate, and if you use the service a lot, you probably should. You can also download all the source and set one up yourself. Overall the current setup is not user friendly by design. In the FAQ it specifically discusses they don't want to support end users who can't take the time to read the FAQ. Overall I cannot overstate how great this has been. It solves such a major source of pain and does it in such an elegant way. If you haven't found a service you are happy with, give spamgourmet a try.
The sales process of convincing someone they need an accurate model is tough, especially because robust models are time consuming and expensive to build.
If you can come up with a model that shows good results, and people know they need those results, then you can start a company selling either a service or product to get those results. If people don't know they need your results - then you have to educate them, in which case it's a much more difficult business to start.
I don't know many business people with the temperament, understanding, or the pocket book to deal with general research type problems.
These types of conversations aren't uncommon.
Other - "I need you to prove our stuff does X, Y, and Z".
Me - "Ok.."
<time elapses>
Me - "Ok the data shows our stuff does X but Y and Z are just random noise"
Other - "We ran it once before with this other guy and it showed our stuff did X,Y and Z. We've been promising it to our clients for a year. He gave us several examples, but when the clients asked to see the underlying data he couldn't produce it. So we just need you to prove it does X,Y, and Z."
Me - "The data only shows it does X. Y and Z are impacted positively through X, but once you condition on X, Y and Z are not causally affected by our stuff"
Other - "Yeah...well I promised client we would give them a report by {{insert random ridiculous date here}} proving it did X, Y and Z. We are going to lose them if we don't deliver a report saying that"
Me - trying for the 50th time to explain they shouldn't promise a positive result when we've never looked at the data.
There are hundreds of variations on this conversation. Your code is wrong is one variant (which depending on the timeline is hard to dispute). Of course if you take long enough that your code is correct, then you are going to slow. This isn't a science experiment, just make it work is another. Watching someone go slack jawed and start drooling because you accidentally used a math term is always interesting.
I have a whole new perspective of being on the cutting edge. It seems like it mostly means you are on the cutting edge of comments from people who don't know how ridiculously hard what you are doing is.
Your impression about the hype is correct. There are a lot of vendors offering BIG solutions, if you pay them BIGGER money. Where I used to translate the word enterprise to $$, now I translate Big Data to $$$$$$$.
When I'm hiring, I don't go looking for Big Data people, because generally they don't exist. Statistics is a really great general addition to a programmers toolkit. Machine Learning is valuable as well, although in my experience the application is more limited. What this article doesn't mention is a whole host of other skills required.
Modeling, and not just a formal mathematical model, but applying any type of model to your data to get insight. Check out the model-thinking class on coursera.
Exploratory Data Analysis, much different skill than confirmatory statistics.
Design of Experiments, specialized subfield within statistics.
Logistics, how to setup, maintain, and maximally utilize an efficient distributed cluster and build a pipeline getting your data to the cluster, cleaning it, building it into a model, and then extracting insight and delivering that end value.
Those are a couple of the skills at a high level. At a more nuts and bolts level, Hadoop is the defacto standard for Big Data. Learning how to build a data pipeline out of the Linux tool chain is very common in the data science world.
The overall value stream for Big Data is deep and wide. Most companies don't have expertise in much of these, and so at the current time you have to learn them yourself or find a company focused on building a team around it.
If you are just learning this yourself, you'll probably get an academic knowledge. If you want to make yourself valuable in the marketplace, you'll really want to get hands on experience. Knowing a z-score is one thing, building a process to gather data and compute a model against it is a whole different ball game. As the article mentions, if you have nice clean data it's easy to apply a model. If you have messy ugly data from 20 different vendors and 200 clients with various failures, anomalies, and you have to figure out what type of model is helpful, oh and you have a deadline because for 500th time someone promised something impossible to the client, then you have something closer to what Big Data is today.
* grammar edits
The reasons companies don't just throw out huge salaries though has to do with the demand side. The salaries companies are willing to pay is related to the marginal advantage they can gain from hiring someone with that skillset. If for example a company will gain say 200k per year in total advantage, that would place a hard cap on how much they would be willing to pay in salary.
So if the advantage is very high, companies will pay more. If the supply increases sufficiently wage rates will drop because there is over supply. If the supply doesn't increase enough, wages will increase more - however each company will drop out at it's own value point. This provides the natural limit to where most salaries cap out.