EvanMiller

It's time everyone understood a fundamental precept of startup acquisition rumors:

If an acquisition price is leaked before the deal is completed, the acquiree has no intention of selling.

In a private acquisition, there are only two parties in a position to leak: the acquirer and the acquiree. The acquirer, which has shareholders, competitors, and lots of money, has no incentive to leak the price on the table, or even the fact that there is a table. It gives away too much information to competitors and creates an unwanted news cycle. So in fact they'll be pissed if information about the negotiation is leaked.

If the acquiree actually wants to do a deal, they don't want to piss off the potential acquirer, so they'll keep their mouths shut. But they might have plenty of reasons to leak the negotation price, particularly if they're a company that is losing money and wants to inflate the company's valuation before another round of private financing. Going before investors saying "We turned down a billion dollars from Google, maybe you heard about it" creates a nice focal point for the ensuing discussion about the company's valuation.

Negotiating in bad faith like this happens all the time, most famously with Facebook/Yahoo! (2006) and Groupon/Google (2010). So the next time you read another news story about an acquisition rumor that includes a price tag, ignore it, because the story's not for you; it's for a future investor.

(Note that the foregoing logic does not apply if the acquiree is publicly traded, because the deal must be approved by shareholders, so the acquisition price is always "leaked" before the papers are signed.)

Well, technically there is a difference between separate heaps and no shared memory. Data in Erlang is immutable, so processes could use a combined heap, but they still wouldn't have read-write shared memory.

However, in my opinion separate heaps was absolutely 100% the correct design decision. The main benefit is not having to worry about the effects of a long-running, stop-the-world garbage collection, which can have catastrophic effects on user interaction, server response times, request queue lengths, etc. An additional benefit is that the language implementers can use "dumb" algorithms for garbage collection and avoid a large class of difficult-to-track bugs.

Robert Virding talked about these issues at some length at this year's Erlang Factory; hopefully the video will be posted soon.

Man. This article, and the business-book morality that underpins it, is utter tripe.

I heartily advocate that when it comes to managing their own time, employees do whatever they can get away with without getting fired. When it comes to negotiating compensation, companies exercise an enormous strategic advantage over employees, which they've proven completely willing to exercise in dishonest ways (cf. Apple/Google "non-poach" agreements). So when it comes to time-management, in the general case, I say fuck 'em. Get your work done, but when the door's closed, get out the clown suit and practice your torch-juggling routine if that's what you feel like doing.

Of course if you actually like your boss, or you don't think your company is run by dicks, and you'd feel bad about getting clown makeup on the company furniture, then don't do it. But only YOU can decide what's right in a given situation. No one needs employment morality lessons from business book authors.

Why not return the favor and call it "Jobs"?

Julia's math library is starting to shape up:

http://docs.julialang.org/en/latest/stdlib/base/#mathematica...

It even has a digamma, Riemann zeta, and Bessel functions with fractional orders, which you won't find in Excel.

The statistics built-ins are a bit weak, however.

What, pray tell, is the hypothesis to be tested? Whether "my" or "your" results in more clicks? How do we know they aren't clicks borne of confusion? Whether "my" or "your" users visit the site more often? Over what time frame? A day? A week? A year? A decade?

A/B testing isn't magic fairy dust that settles design questions. You need to have a clear hypothesis to test. It's especially difficult if the design question concerns long-term perceptions and psychology. An A/B test in this situation is anything but straightforward.

I'd be interested to learn about how this design decision was made in the companies that have had to face it.

If I had one piece of advice it would be: Descend the rabbit holes!

Rabbit Hole Syndrome is a symptom of having a curious, persistent, independent mind. If there were a drug that cured Rabbit Hole Syndrome, and everyone just worked on Shipping Their Products and Not Asking Questions, there'd be no one left to make the interesting discoveries that make the world better in the long run.

There are very famous quotes from Richard Hamming (see "You and Your Research") and Richard Feynman (see "Surely You're Joking!") about the importance of working on problems that seem trivial at first. Not only do they help you enjoy problem-solving for its own sake, but if you work on enough silly problems, then eventually the odds are good that you'll stumble upon something that other people will later think is really important. Incidentally if you read Thomas Kuhn you'll find that this tends to be how scientific revolutions happen: someone is bothered by some tiny little thing that doesn't quite make sense according to the prevailing theory, starts digging at the little cracks, and finally the whole system comes crashing down.

The major benefit of going down rabbit holes is that is opens yourself to serendipity: sometimes you'll turn out to be right about something for the wrong reason, or discover something that you later realize contains a solution to a seemingly unrelated problem. The more rabbit holes you've gone down, the more they start to connect.

In addition, frivolous research helps you develop a very "bottom-up" view of the world. If you know the details cold, then you are better able to see through high-level BS ("conventional wisdom") and evaluate things on their own terms. You'll find that regardless of what's optimal, most things are done a certain way only because they've always been done that way, and that regardless of what's true, most people believe things only because other people believe them.

Of course there are costs to all this -- in particular, "schleppers" will resent you for being irresponsible, and most people will think you're crazy if you ever explain what stupid little you've been working on lately. Which is why Rabbit Holers should try to should avoid actual responsibilities to the greatest possible extent and absolutely not care about what most people think about their work.

Anyway, to sum up, if you're lucky enough to be in a position to descend rabbit holes without impoverishing your family or bankrupting your company, I say go for it. It strengthens your most valuable asset (your mind), and who knows, maybe one day you'll discover something you can teach the rest of us.

> if you don't do proofs you won't get any better at understanding mathematics

There is a vast world of mathematical understanding available to programmers who don't like proofs. I think there is some miscommunication here about what constitutes mathematics. Mathematicians tend to think of the art of proving things, whereas engineers (and programmers, and scientists) think of equations and how to solve them. I think mathematicians actually do a disservice to engineers by trying to beat them over the head with real analysis and set theory (which is why most physics and engineering departments end up having to teach their own mathematical methods courses).

The "foundational" knowledge of mathematics -- epsilons, deltas, Cantor sets, Banach spaces, etc. -- is irrelevant to almost all practitioners. Newton, Taylor, Euler, Gauss, Laplace, Bessel, Maxwell and the gang got pretty far without it. When programmers say they want to learn math, they mean they want practical knowledge: interesting functions, how to compute them, and their properties that can be applied with utility. They don't want -- or need -- to spend time poring over proofs.

5) Take a random sample of 50,000 records and get 90% of the business intelligence value in about 2 seconds.

If you're fitting a model with only 70 degrees of freedom then analyzing 150M records is a complete waste of time.

That's good to hear, because I am constantly running out of doorstops in my home. With current technology I have to go the hardware store 2-3 times a week just to pick up new doorstops. If I had a 3D printer in my home, I could simply print out new doorstops every morning, and never have to worry again about needing a doorstop at 2 AM when the hardware store is closed.

I'm not sure where I'd get the raw plastic, though. Do they sell that at the hardware store?

The (1) next to your name is your "karma", which is designed to make the primitive hoarding and status-seeking parts of your monkey-brain feel inadequate when no one upvotes your comments.

Looks nice, but arranging the elements as a sphere, spiral, etc. completely defeats the purpose of the periodic table. Also, your atomic numbers are off by one.

There is more information on Michael Minovitch's website:

http://www.gravityassist.com/

He sounds rather bitter that his story has not made him more famous, and comes off as something of a crank at that ("How Did Minovitch Discover (Create) His New Theory Of Space Travel?"). I think Minovitch is basically an engineer at heart, and is resentful that all of the attention seems to go to the scientists, i.e. people who come up with the actual physical theories about mass and energy and so forth.

Our culture does value scientists more than engineers. You could argue that this is good because theories are universally applicable and represent fundamental understanding, or that this is bad because many theories have absolutely no applications and that it's the engineers who deliver value to society. You could also argue that the amount of prestige that we heap on scientists tends to go to their heads and they stop doing good work. But whatever. Stay sharp, do what you're best at, and don't pay attention to other people's benefits packages.

The money quote of this article is:

"I propose to term such correlations... the serial correlations for the given series."

The basic idea here is that observations in a time-series are not actually independent because each observation is highly correlated with the previous observation, and so the usual significance tests and standard errors do not apply.

Some links if you're interested in learning more about analyzing serial correlation and time-series data:

http://en.wikipedia.org/wiki/Autoregressive_model

http://en.wikipedia.org/wiki/Newey–West_estimator

http://en.wikipedia.org/wiki/Prais-Winsten_transformation

MYTH: 3D printers will never be able to print underwear

FACT: Wrong. Hard plastic underwear provides amazing protection and support

MYTH: 3D printers will never be able to print the Mona Lisa

FACT: Wrong again. Today you can download a 3D model for "rectangular hard plastic canvas" from the Internet, print it out using your Makerbot, and then paint the Mona Lisa on it

MYTH: 3D printers will never be able to print small, wobbly, leaky coffee cups that warp when hot beverages are poured into them, and that lacerate your lips when you attempt to drink from them

FACT: This is already possible on some of today's higher-end 3D printers

MYTH: 3D printers are useful in any way

FACT: It turns out this myth is true. Watching a 3D printer emit a hard plastic object in a color that nobody wants keeps nerds in a state of rapture that prevents them from engaging in more dangerous activities such as shooting heroin or starting actual businesses

Igor has been one of my programmer heroes for many years now, and I'm glad to learn more about his background through this article. The reading's a bit tough, as many verbs appear to be missing and a few Russian words remain untranslated. Some interesting takeaways:

* Igor wrote Nginx while working as a sysadmin for Rambler. Classic bottom-up innovation: he could see Apache's shortcomings in a very up-close and personal way. This also explains why Rambler does not hold the copyright on Nginx (see the end of the interview).

* He is quite modest when comparing Nginx to other web servers. This reflects well on his character, but perhaps has "slowed" the adoption of Nginx. It's clear from this article that Igor is an engineer, not a salesman. I find this to be refreshing in an era when many open-source projects seem to achieve adoption by being the loudest, rather than by being the best.

* Igor put off starting a company until there was just too much work for him to do alone. I liked this quote: "I rarely change my life direction: for example, [before] Rambler, I spent seven years working for a company, [and at] Rambler, I also worked for ten years. Change is hard for me. But, nevertheless, by the spring of this year, I did finally decide to found a company that would help the further development of the project."

* Despite having U.S. investors, Nginx's engineering team is based in Russia. One interpretation is that you don't need to be in the Valley to develop a first-rate product; you just need to be in the Valley when it's time to hustle it.

Anyway, an interesting read. Apparently this article is his first public interview. I know Igor doesn't like publicity, but it's inspiring to read his story.

This is data visualization gone wrong. I am afraid you have drawn a wildly incorrect conclusion because the programmers were too lazy to deal with the political and historical inconvenience of Gov. George C. Wallace, the Southern Dixiecrat and presidential nominee of the American Independent Party. Alabama didn't suddenly take 73% a left turn to vote for LBJ's vice president in 1968; they, along with Mississippi, Louisiana, Georgia, and Arkansas, voted for George Wallace, a candidate who made Richard Nixon look like Bob Dylan in comparison. Get the facts:

http://uselectionatlas.org/RESULTS/national.php?year=1968...

The moral of the story (in my mind) is that while it's tempting to throw a big dataset into a cool visualization algorithm, if you're not careful about what each individual data point means, you might end up imparting more confusion and misconceptions than actual knowledge.

Thanks. I've been testing on 3-4M records on a quad-core laptop, and it works great. I would guess that 10M records will be doable on an 8-12 core Mac Pro, but I don't have access to one to test this particular hypothesis.

The program uses the Objective-C run-time, but the core routines are written in C.

Thanks. I have updated the copy to better reflect the distinction between the two versions.

Sorry. It's back up.

For future reference, uploading a video to Dropbox when you are on the HN front page is a bad idea. My account was suspended almost immediately. Switching to YouTube...

Sorry about that. The HN traffic was unexpected. I will upload to Dropbox ASAP.

Sorry for the poor audio quality in the video. Some random thoughts that may be of interest:

1. I've been in grad school for economics the last four years. The program basically embodies all the things I've learned about econometrics, i.e. applied statistics. Wizard does about 1% of the things that R does, but it does them very, very well.

2. Wizard is fast. All the tight loops are written in C. Wizard will use all the cores on your Mac as well as the CPU's vector capabilities. This was achieved with a combination of Grand Central Dispatch and OpenCL. I highly recommend both technologies. For my research I am using Wizard on data sets with ~3 million rows, and it works like a champ.

3. The main weaknesses of the program is that it does not deal with time-series data particularly well, and you can't manually enter data. These shortcomings will be addressed in future releases.

4. The program is closed-source. I might make some of the libraries open-source. Or I might not. I haven't decided.

5. The non-Pro version will be released sometime next week.

6. I have absolutely no regrets about not writing Wizard using JavaScript and HTML5.

7. [Edit] You can read a review of the pre-release version of the program here: http://www.macstats.org/reviews/wizard.html

An event horizon means there is no going back, not that you are actually in the black hole. Although Microsoft may have continued to put out innovative products and create shareholder value after 1992, the company is now run by a textbook bozo. It must be depressing for people who actually know a thing or two about computer science.

Some neat Erlang projects:

* ErlGuten, PDF generator: https://github.com/hwatkins/erlguten

* OpenCL binding: https://github.com/tonyrog/cl

* gen_smtp, SMTP library: https://github.com/Vagabond/gen_smtp

* erl_img, image manipulator: https://github.com/evanmiller/erl_img

* ErlyDTL, Django template compiler: https://github.com/evanmiller/erlydtl

* BossDB, evented database abstraction layer: https://github.com/evanmiller/boss_db

* Jerome, RTF/HTML/BBCode/Textile processor: https://github.com/evanmiller/jerome

* Cowboy, web server with WebSockets: https://github.com/extend/cowboy

* Chicago Boss, full-featured web framework: http://www.chicagoboss.org/

Those are just my favorites, there are many more listed here: http://erlagner.org/

Things I would use Erlang for:

* Network programming

* Web programming

* Sending a lot of emails

Things I would not use Erlang for:

* Numerical programming

* Video decoding

* Desktop or mobile applications

Basically Erlang is really good at receiving, parsing, and sending data, and bad at anything remotely processor-intensive (or graphical). I think this situation might change with Erlang's OpenCL binding (second link above), which lets the programmer skip the overhead of Erlang's run-time and go straight to the CPU or GPU for numerical tasks.

Dalton Caldwell sends email the hard way.

Well, if all you care about is the space complexity you could do the same thing with 8 bytes of memory, an integer and a floating point. Start the floating point at zero. For each item, set the integer to 0, then compare the item to every other item (including itself) and add one to the integer if they are equal. After each iteration, add one divided by the integer to the floating point, and repeat for the next item. When you're done the answer will be stored in the floating point.

Of course this method requires O(N^2) time (that is, 1,000,000,000,000,000,000 compare operations when counting a billion objects), but who cares? It only uses 8 bytes!

Original author here. For the academically inclined, there is a critique of this approach in this paper:

http://www.dcs.bbk.ac.uk/~dell/publications/dellzhang_ictir2...

Of course, I think the authors miss the point of the algorithm, since I basically wanted a system that is one-sided (i.e. false negatives are OK but false positives are bad).

Also, if you deal with more than two outcomes you might be interested in multinomial confidence intervals, described here:

http://www.math.wsu.edu/faculty/genz/papers/mvnsing/node8.ht...

The application to 5-star systems is not straightforward, since it's not clear to me how stars relate to each other. Is it a linear scale? Are they discrete buckets? Or maybe we want to use Tukey's froots and flogs? I'm not sure.

By the way, I'm coming out with a stats app for Mac soon that implements this algorithm and much more. Drop me your email address if interested:

http://wizard.evanmiller.org/

Not to poop in your cocoa puffs, but I wrote an Nginx module to do the same thing in 2007.

https://github.com/evanmiller/mod_zip

The module is quite mature at this point, and is used in production on many websites (including Box.net, which commissioned the initial work). The module supports the Content-Length header, Range and If-Range requests, ZIP-64 for large archives, and filename transcoding with iconv. Being written in C, it will probably use much less RAM than an equivalent Node.js module.

I have found that the hardest part of generating ZIP files on the fly has nothing to do with network programming; it's producing files that open correctly on all platforms, including Mac OS X's busted-ass BOMArchiveHelper.app.