HNHacker News
TopNewBestAskShowJobs

neilc

4,830 karma · joined February 26, 2007

CTO / Co-founder, Determined AI http://neilconway.org
submissionscomments
neilc··on Hedge Funds Faced Choppy Waters in 2015, but Chiefs Cashed In
> Pensions could've been higher if they didn't take that much out

Pensions would also be lower if they were invested in funds that made lower or negative returns. The theory (right or wrong) is that the high fees charged by the hedge funds are justified because they provide better risk-adjusted returns than other investment vehicles. Hedge funds may be bad and their CEOs may be overcompensated, but it is more complicated than just "every dollar paid to Kenneth Griffin is a dollar taken from a pension fund."

neilc··on What Happened to Google Maps?
Personally I like this UX. I often pullup the directions to a destination without actually starting turn-by-turn navigation: I'd like to see whether Google recommends taking highway X or Y, and then I can drive the rest of the route myself.
neilc··on A Billion Taxi Rides in PostgreSQL
> PostgreSQL stores each record, regardless of attributes, in an 8K block of data (excluding TOAST fields and indices) on the disk. If you have a table with 5 columns of integers they will use 8K of disk space for each row, not 20 bytes.

This is wrong. Postgres uses 8KB blocks for heap data by default, but you can (and typically will) have multiple rows in a single page.

neilc··on Psychology study that induced "reproducibility crisis" was wrong: researchers
From the original paper:

"We constructed a sampling frame and selection process to minimize selection biases and maximize generalizability of the accumulated evidence. Simultaneously, to maintain high quality, within this sampling frame we matched individual replication projects with teams that had relevant interests and expertise. We pursued a quasi-random sample by defining the sampling frame as 2008 articles of three important psychology journals: Psychological Science (PSCI), Journal of Personality and Social Psychology (JPSP), and Journal of Experimental Psychology: Learning, Memory, and Cognition (JEP: LMC)...The first replication teams could select from a pool of the first 20 articles from each journal, starting with the first article published in the first 2008 issue. Project coordinators facilitated matching articles with replication teams by interests and expertise until the remaining articles were difficult to match. If there were still interested teams, then another 10 articles from one or more of the three journals were made available from the sampling frame. Further, project coordinators actively recruited teams from the community with relevant experience for particular articles."

neilc··on GCC 6: -Wmisleading-indentation vs. “goto fail;”
> I thought that the go compiler would throw an error of your code is not in the format produced by gofmt.

This is not the case.

neilc··on Padding is hard
> why the compiler shouldn't reorder structs so that they take the least space possible

I think you'd get most of the benefit by writing a static analysis tool (a la "go vet") that suggests when reordering a struct's fields would reduce memory consumption.

neilc··on “Fashion Is Hard. PostgreSQL Is Easy” [video]
PG 9.5 will improve this situation somewhat, e.g., builtin functions like jsonb_set(), and overloading the '-' operator for jsonb values:

http://www.postgresql.org/docs/devel/static/functions-json.h...

That said, if you're doing a lot of mutation of large JSON documents stored in a single Postgres row value, the storage/concurrency control behavior still won't be ideal.

neilc··on Michael Stonebraker wins Turing Award
"Architecture of Database Systems" by Hellerstein, Stonebraker, and Hamilton is a pretty good starting place: http://db.cs.berkeley.edu/papers/fntdb07-architecture.pdf
neilc··on A Note to My Readers
> 50 years everything we've done will be superceded by something vastly more cool and interesting. Within 100 years it'll all be gone.

Which professions don't have that property? Not many. Science and writing/art give you a chance for that kind of longevity, I suppose -- although even then, doing something that has any real impact 50+ years later is difficult and very rare.

neilc··on Apache Flink – Fast and reliable large-scale data processing engine
Publications from the academic project that spawned this work can apparently be found here: http://stratosphere.eu/project/publications/
neilc··on Neat Algorithms: Paxos
> production distributed systems should handle Byzantine failures

I don't think this is true, at least not as stated. You should certainly be thinking about more than crash-stop failures, but full-blown Byzantine fault tolerance is rarely warranted in practice. Empirically, the number of systems that use non-Byzantine vs. Byzantine agreement is probably on the order of 100:1.

neilc··on Bolt: A low-level key/value database for Go
Note that using a sqlite wrapper like https://github.com/mattn/go-sqlite3, you also don't have any external dependencies and your application still ships as a single binary. However, there are other advantages of a Go-native storage library (e.g., doesn't require cgo, better integration with tools like pprof, etc.)
neilc··on If Eventual Consistency Seems Hard, Wait Till You Try MVCC
> Serializing access to shared data items fundamentally requires some serial execution.

Sure, but typical MVCC systems that provide SI will serialize transactions in some cases anyway. For example, if two transactions try to update the same row in Postgres, the second will block until the first one commits/aborts. This is true under RC, RR, or SERIALIZABLE.

> This fundamental overhead is a key reason why almost every commodity RDBMS defaults to one of these weaker isolation levels and why databases like Oracle don't support serializable isolation at all.

I don't think that's true: if you already have an MVCC implementation that provides SI, the additional overhead of providing SSI is relatively modest. I'd say the main reason Oracle doesn't provide SERIALIZABLE is historical: their MVCC implementation predated Cahill's work by many years, and people literally didn't know how to use MVCC to provide SSI at the time.

neilc··on Mean People Fail
Just neglecting to continue to develop a piece of software isn't exactly "inflicting billions of damage" on "humanity". Letting IE stagnate was a dumb move, but it's not like MS had a moral imperative to constantly improve it. Shame on the marketplace if it took a while for a feasible competitor to emerge.
neilc··on Following a Select Statement Through Postgres Internals
> Postgres did an awful job on that query. Small values of LIMIT should not require a full sort, even when there's no index.

Postgres will not necessarily do a full external merge sort for the query plan shown in the article: when there is a LIMIT k clause, the Postgres sorting code has optimizations to only keep the top k values in memory and do an in-memory sort (obviously, a full seqscan of the input is still needed without an index). Search for TSS_BOUNDED and tuplesort_set_bound() here:

https://github.com/postgres/postgres/blob/master/src/backend...

neilc··on Readings in Databases
perhaps the deprecation of spinning disks has reduced the complexity of the issue

SSDs are much more complicated than spinning disks, so if anything the problem has gotten worse.

neilc··on Tim Hortons agrees to Burger King offer for $94 a share
> there aren't many countries where corporation tax is above 35%... Japan comes to mind

Japan's is only slightly higher, at 35.64%:

http://www.kpmg.com/global/en/services/tax/tax-tools-and-res...

but that will actually be cut below 30% in the near future:

http://money.cnn.com/2014/06/24/news/economy/japan-abenomics...

neilc··on Window Functions in PostgreSQL
UPSERT is not a simple feature to implement correctly:

http://www.pgcon.org/2014/schedule/events/661.en.html

neilc··on Clark's Curious Corollary of Concurrency
Well, in practice the latency that normaly matters is between "start doing activity" and "finish doing activity", and the time actualy working (that depends on bandwidth) is normaly orders of magnitude bigger than the time waiting for data (that depends on latency).

I don't think that's true; moreover, it is likely to become even less true in the future. Once the data needed for a computation has arrived at the CPU, for most applications the required computation is pretty cheap -- the time spent waiting for the data often dominates the time spent computing with the data. Much of modern CPU design has been devoted to trying to hide the high latency of memory accesses.

neilc··on Clark's Curious Corollary of Concurrency
Another factor is that technology improvements tend to favor increases in bandwidth over increases in latency (and there are hard limitations on latency, like the speed of light in distributed systems). This short paper by David Patterson is a great read on the subject:

http://www.cis.gvsu.edu/~kurmasz/Teaching/OldCourses/CS451/G...

neilc··on What will health nuts eat on Super Bowl Sunday? Nuts
Uh, what? Unsaturated fats are generally regarded as healthier than saturated fats. Which specific unsaturated fats do you think have "many negative health associations"?
neilc··on The best Postgres feature you're not using – CTEs aka WITH clauses
I hate joins in the where clause because it is semantically wrong and confuses the building the set part with the filtering the result part.

It is not "semantically wrong": conceptually, you can compute a join by taking the Cartesian product and then applying the join predicates to the result, which is actually what the implicit join syntax suggests. Semantically, there is no difference between applying an (inner) join predicate and applying any other predicate. There is also no actual division between "building the set" and "filtering the result": different predicates might be applied in different orders, before or after different relations in the query have been accessed.

neilc··on The Saddest Moment: Byzantine fault tolerance [pdf]
Probably http://pmg.csail.mit.edu/papers/osdi99.pdf
neilc··on Facebook
On Facebook, you have no control when others are sharing your photograph.

The same is true of the Internet in general.

neilc··on ZeroMQ is just sockets
Although that Danish phrase actually refers to a dessert (https://en.wikipedia.org/wiki/R%C3%B8dgr%C3%B8d)
neilc··on Ubuntu 13.10 (Saucy Salamander)
There have been countless research papers on this topic, so I doubt it -- http://scholar.google.com/scholar?hl=en&q=dynamic+memory+com.... Not to mention the fact that RAM compression was quite common back in the DOS/Win95 days: http://oreilly.com/centers/windows/feat/softram/dblscan.html
neilc··on Square Cash
I believe Amazon Payments charges 2.9% + $0.30 per transaction, which would not make it an effective way to rack up reward points.
neilc··on Inside 23andMe founder Anne Wojcicki's $99 DNA Revolution
The amount of actual actionable information you can get from a gene map like that is.... a LOT more limited than people such as 23AndMe customers think.

I wouldn't be so sure that 23andMe's business model depends on their customers being uninformed -- for $99, you don't need a vast amount of actionable investment to justify the purchase. Hell, getting your teeth cleaned without insurance can easily cost you more than $100. I wasn't expecting a ton of actionable insight from 23andme, but I don't regret spending the $99.

neilc··on Facebook friends could change your credit score
If gender is predictive of higher/lower risk for a particular policy, then banning its use seems bizarre. It makes even less sense if the the insurance company is allowed to consider other factors that are correlated with gender (e.g., age, profession).
neilc··on Snowden Leaves Hong Kong on Commercial Flight to Moscow
20,000-40,000 people per year seek asylum in the US; we're discussing one person who has fled the US to avoid prosecution. Let's not overreact.
← PreviousPage 3 of 34Next →