bkirwi

FWIW, this is addressed in the article and the paper it covers: the slowdown involved reduced "proactive policing" -- "systematic and aggressive enforcement of low-level violations" -- but the observed drop is in major crimes like burglary and assault. The authors also do some work to control for the overall crime reporting rate.

Worth mentioning that, while the `git bisect` algorithm is pretty easy to run by hand if you have a linear history, `bisect` also handles very complex branching and merging histories as well. Very clever and worth using!

The issue being pointed out here isn't that the computation is out of sync with the outside world... it's that it's our of sync with _itself_! It will return answers that are not just stale, but inaccurate for any time in history.

This might still be fine, depending on your needs, but IMO a legitimate distinction.

Based on that description of Pijul, it does seem to be a semilattice: it has a merge operation that's associative and commutative, and while it doesn't explicitly state that it's idempotent it's hard to imagine merging two states and getting anything else back than the same state.

Not super familiar with Pijul though!

Of course, there really is something special about 0, 1, and Inf -- they're the kind of numbers that tend to arise from theory, not just from tuning. Tuning is hugely important for real systems, so an allergy to 'arbitrary' numbers is not useful; on the other hand, this rule is often a nice heuristic to keep in mind when you're picking which numbers to expose in your config files.

It's worth noting that green threads are not 'enough' -- in any language with concurrency, you still need to be explicit about what work can run concurrently and what needs to be sequential. In Go, for example, the default is sequential operation; if you want to run a bunch of code concurrently, you need to do extra work to launch a bunch of new goroutines / wait for everyone to finish. In Haskell, which also (typically) has green threads, there's a popular `async` package that supplies a promise-like construct -- it's quite useful when you want to be explicit about concurrency and ordering.

Yeah, it makes sense to mimic the D3 style: the only thing worse than a bad convention is two conventions.

Money, capital, and Marx.

It's an odd definition of personal privilege you have, where the privileged person suffers two years of harassment and abuse, and the person without privilege immediately returns to a lucrative career.

It's also an odd definition of professional victim, where publicly being a 'victim' results in you getting booted out of the profession.

I don't want to comment on the event itself here -- but I think that the aftermath, and the continued reaction to it, say a lot of sad things about the tech community.

Sure! Here's a couple concrete applications:

- Any monoid operation is trivially parallelizable: take the dataset, split it into chunks, combine all the elements in each chunk together, then combine those results together as a final step.

- If I'm updating a row in my database with a monoid operation, I can always 'pre-aggregate' a batch of values together on the client side before taking that result and combining it with the stored value.

- If I store some statistics for every day of the year, I can calculate the monthly statistics very cheaply -- as long as those stats are a monoid.

The monoid abstraction seems weird at first, because it's so dang general, but it ends up hitting a bit of a sweet spot: the rules are just strong enough to be useful for a bunch of things, but simple enough that be applied to all kinds of different situations. You can think of it kind of like a hub-and-spokes thing -- this interface connects all kinds of different data types to all kinds of different cool situations, so you get a lot of functionality with a lot less typing and thinking.

Here, try this...

A monoid is made up of:

- Some arbitrary type of thing. (Lists, sets, strings, numbers, whatever.) Let's call it 'A'.

- A value of type A. Let's call it the 'empty' value.

- A function that takes two A values as arguments, and returns a single one. Let's call it 'combine' function.

Here are some examples of monoids:

- String concatenation: 'combine' is string concatenation, and 'empty' is the empty string.

- Integer plus: 'combine' is the usual + operation, and 'empty' is the value 0.

- Set union: 'combine' is the union of two sets, and 'empty' is the empty set.

You can come up with monoids for all sorts of things: maps, lists, functions, bloom filters, futures / promises, ...

There are, however, a couple rules:

- It doesn't matter which order you combine things in: `("hello" + "world") + "!"` has the same result as `"hello " + ("world" + "!")`; and `1 + (2 + 3)` has the same result as `(1 + 2) + 3`.

- You can add the 'empty' value to either end without changing anything. `"" + "hello"` is the same as `"hello" + ""` and plain old `"hello"`. Likewise, `3 + 0 == 0 + 3 == 3`.

That's it! To get an intuition, you might want to pick a couple more types from the list above and pick a good 'empty' / 'combine' function that satisfies the rules.

Of course, why you'd want to do this is another question. (And one I'd be happy to answer, if you're curious.)

Here you go, then:

    -- little-endian sequence of bytes to arbitrary integral type
    integerWithBytes :: (Bits a, Integral a) => [Word8] -> a
    integerWithBytes = foldr (\byte acc -> (acc `shiftL` 8) + fromIntegral byte) 0

At least, I'm pretty sure that's what the article was trying to do...

I don't think this is some amazing showcase of Haskell; after all, the article includes a (working?) implementation in C++. But it does serve as a nice counterexample to the idea -- expressed elsewhere in the thread -- that being generic is necessarily a hugely painful or complicated thing.

Probably this one?

http://ferd.ca/queues-don-t-fix-overload.html

Queues and similar things help flatten out the spikes, but they don't add capacity. Like many of these tools, use with discretion.

There's a bit of writing on StackOverflow about it:

http://stackoverflow.com/questions/2982012/haskells-typeclas...

The main similarity is that 'methods' are defined 'outside' the data type they belong to. But there are a huge number of differences: Haskell's typeclasses are used for many things that don't look like methods at all, while Go's behave more like a classic OOP interface you'd find in Java.

So, perhaps superficially similar-looking, but they work quite differently in practice.

> Demanding this mind of self censorship is a sign that a society is fragile rather than robust. A robust society can take the jokes.

Sure, but you don't magically make society robust by telling racist jokes. The healthy society comes first; the fact that these jokes no longer sting is just a pleasant side effect.

In Canada, as I understand it, it's illegal to refer to yourself as an engineer / a position as an engineering position unless you have Real Engineering Credentials. So a lot of the touchiness is just the usual sensitivity you find around a legal issue.

Yes, Kafka certainly is good at those things, and I suspect Kafka will only get better at them. But it's actually quite a simple bit of infrastructure at heart[0] and this means it's useful for a large variety of other things, many of which we're only figuring out now.

Re: optimistic locking... there are a couple proposals for this floating around, and I'm not sure which / if one will get in. It certainly seems consistent with the general mission, though.

[0] http://engineering.linkedin.com/distributed-systems/log-what...

Hey, thanks for the excellent writeup.

I enjoyed and totally endorse the content of your blog post, but I did find the title / intro a bit misleading. Exactly-once is pretty much always what people want: it's the easiest model for most folks to reason about, especially when they're new to the whole distributed-systems thing. (If exactly-once delivery was easy, you certainly wouldn't find people going out of their way to build systems that dropped or duplicated messages.) Of course, you might not need exactly-once delivery to get what you want -- which is good, because like you say, that's a hard / impossible problem to solve in general.

It feels to me like people in the stream processing universe have taken that message a bit too much to heart -- things like the 'lambda architecture' treat stream processing as "too hard to get right" and relegate it to approximate, disposable calculations only. My post was partly an effort to push back on this; as a community, I don't think we should give up without a fight.

Oh hi!

The Samza ecosystem seems to be a magnet for this sort of thing these days -- the community's really committed to taking full advantage of Kafka, and not just papering over it so it looks just like any other queuing system. Really excited to see where all this will lead.

One additional note: this is a survey article, not based on a single study. This seems to be the case of NPR staff using a questionable reference, not just rewriting a press release.

Startup idea: use the integrated scheme interpreter to program an interactive fiction game, and post it here.

It always seemed to me like the academic-style sabbatical was a nice solution to this. Instead of trying to convince a bunch of people that what you're going to do is going to be worthwhile, you develop a culture where capable people are regularly given time to work on whatever they think is important.

Clojure's a nice example: it's been hugely successful, but I have a hard time imagining anyone raising two years' worth of salary up front to work on their idea for an idiosyncratic Lisp implementation. It looks like he had to fund that work himself; how much more innovation would you see if this kind of extended project was common?

Yes, that's the principal idea, I think -- separating out the implementation details from the actual conditions that are necessary for correctness. IIRC, I've seen another presentation of this where the author used that freedom to implement a work-stealing quicksort across multiple threads.

If you find yourself creating a throwaway just so you can criticize someone's coding style on a quick graphics experiment, you may want to reconsider your life choices.

Not guilty! I picked this one up on Twitter this afternoon, but it's possible the lineage traces back to your lobste.rs post. (I have no lobste.rs account, though not for lack of interest -- while sparse, the comments there seem to be of very high quality.)

There you go.[0] I believe it's called the 'arbiter problem' in that context, and your description seems basically correct.

[0]: http://research.microsoft.com/en-us/um/people/lamport/pubs/b...

    var (
            ch  = make(chan *http.Response)
            err error
    )

    ....
 
    return func() (*http.Response, error) {
        return <-ch, err
    }

It seems to me like calling that returned function is destructive -- if you block on it more than one, it would block forever. Is that right? It seems like a different contract than the usual Future / Promise shtick.

Thanks!

Yep! You can trace that thread back to Lamport's work some decades back.

The thing I find really compelling about this paper is that it's not tied to a single coordination method, though -- it pulls everything back to application level invariants. Human beings have a bad track record of reasoning about invariants in distributed systems, but computers are pretty good at that sort of thing... if we can specify our invariants explicitly, and let the machine figure out how exactly to spread that work out on the cluster, I think we'll come out far ahead.

If anyone's trying this at home, I'd suggest scoping down the problem a bit:

- Trying to get rid of a specific problematic dependency? It's easy to filter a DAG to only those paths that lead to a particular node.

- Trying to reduce the raw number of dependencies? Consider collapsing clusters of transitive dependencies down to a single node. (If one of your direct dependencies pulls in 20 unique deps of its own, you don't really need to know what they are or how they're connected to each other -- just that there are 20 of them.)