jbert

Excellent post. Your type-based approach is a very useful tool for thinking about things which I find difficult. Thanks for that.

Nice.

It looks like that's effectively using true/false (as defined via lambda) as the two constants and inlining the "if true" and "if false"?

I like the definition of cons/car/cdr in terms of lambda.

A cons cell is just a function you can call to get back the car or cdr. Once you have 'if' and any two constants, you can do:

(define mcons (lambda (a b) (lambda (method) (if (= method :car) a (if (= method :cdr) b)))))

(define (mcar p) (p :car))

(define (mcdr p) (p :cdr))

Using setf is a nice example of something which a macro can do which a function cannot (as well as being a good example of code-is-data).

Lots of posts discussing lisp macrology lack that kind of compelling use case.

The 'magic' used by setf is a 'destructuring bind': http://dunsmor.com/lisp/onlisp/onlisp_22.html

So, immutability and/or api contract is important here.

If I'm pulling in a function, I want it to do what I think I want. Sometimes I want that to change (get a bug fix), but sometimes I don't (someone introduces a bug, or makes the func more general and introduces slowdowns).

This feels like a job for a content-addressable git-like tool. How about this:

I can discover my function (via whatever means). The function is actually named 8804ea505fda087da53b799434c377f015933707 (the sha-something of it's (normalised?) textual representation).

I then import it into my codebase as "useful_fun". My code reads like:

    useful_fun("do it", "to it")

but I have some kind of dependencies/import record which says that "useful_fun" is actually 8804ea505fda087da53b799434c377f015933707. That means one and only one thing across all time, the func with that hash.

So how do we handle updates? If we want a golang-like model, the developer could run something like "update deps". This would:

- go back to the central repository, looking for updates to 8804ea505fda087da53b799434c377f015933707. It might find 5. Local policy then determines what happens. Could be "always choose the original authors update" or "choose the one with the most votes" or "always ask the dev, showing diffs".

Note that because the unique name is based on the function content, any change to it creates a new item in the db. (Content-addressability, same way git and other systems do it.)

- stuff can be grouped and batched. If I pull in 10 functions tagged with the same project ('module') and they've all been updated, I can say "and do the same with all the others".

- This kind of metadata allows all kind of good stuff. I can subscribe to alerts on the functions I've imported and get told about new versions, or security warnings. This kind of subscription information can be used as a popularity contest to solve the "which fork on github do I want to use" problem?

- people can still publish modules. They now look like a git directory or tree. A git tree is a blob which contains the hashes of the files within it. A 'module' could be a blob which specifies which (immutable) functions are in it.

If we use normalised functions, we've now got a module representation which allows arbitrary functions to be pulled together. At fetch time, we can denormalise into the user's preferred coding style. At push time, we renormalise. We aren't grouping stuff into files, so a 'project' or a 'module' consists solely of the semantic contents, nothing to do with artificial grouping for the file system.

Seems like an interesting future.

Why try and be so clever?

I'd hate to be tracing a problem and find code deep in a lib trying to do that. It's OK-ish to expose such things in explicit API, but then you put the burden of choosing (and updating) such constants on the app developer.

If you're trying to avoid too much slop (up to 50% waste), you could either:

- grow by a smaller factor (1.25x, 1.5x)

- trigger a shrinking realloc when certain usage/stability criteria are met (bonus points for exposing this as explicit API rather than implicit behaviour on access, since that could cause exactly the kind of "weird slowdown" people spend ages trying to find (http://antirez.com/news/84)

Even worrying about slop is probably wrong, since if it's a big alloc wasting space, your VM system will probably not even map any physical RAM until it's used. i.e. there would be literally no downside.

(And as to your specific idea - imagine for a sec that future systems might introduce new ways of restricting memory (e.g. linux cgroups) or adding memory (hot-swap RAM, emulated by VMs?). Then your code would be tuning with the wrong value. Not saying that that is likely, but imho you'd just making things more fragile by trying to embed magic in the code.

> You can certainly set reasonable limits.

No, not really. Not unless you plan to review them every few years or so. Which no-one will ever do.

The point about exponential allocation is that it is scale-independent. It doesn't depend on measures of the underlying size, so the same strategy works in 2000 and 2020.

Dumb question - if problem is a latency spike in the parent, any chance you can continue serving in the child and let the parent do the BGSAVE (and take the latency spike)?

(The child should be able to inherit all open fds?)

To me, the interesting thing in terms of building reliable, performant systems is what do you want to do when the request rate (incoming desired throughput) exceeds your possible throughput.

Broadly, you have to choose between quickly failing the additional requests, queueing them up yourself (have a fixed number of requests you process at a time) or you just attempt to process them as they come in.

- If you don't plan for this eventuality you'll basically end up doing the latter option.

This will overwhelm your processing step, which will then typically cause the latency of all requests you are processing to up. (e.g. because you're over your working RAM set, CPU budget etc). Worse, it will go up in unpredictable ways, often with a sharp hockey stick curve.

- If you queue up requests, you're adding latency to everything right there. Your backend is still cranking away at the same low latency, but your system latency now additionally has the queue dwell time added in. On the plus side, you'll avoid unpredictable performance "cliffs" where everything drops down to nothing. On the downside, if your queue gets too long, you'll end up in the unhappy state of very quickly processing stale requests which no-one cares about.

- Simply failing the additional requests protects your system, but exposes error conditions externally. (Of course, so does having an 10s HTTP request time...)

Hybrid approaches can work well. (e.g. running a managed queue but cull the queue (and fail the associated request) before you get into the "queue of death" scenario).

Aaaaand....every approach apart from "just try to process it" requires you to have an understanding of your backend capacity.

You can to test and measure to find a static number, but an additional problem is that the load caused by real-world requests can differ from your test load.

And your backend capacity can be affected by failing components, high load on adjacent components, code deployment, backup, cold cache restart, etc etc.

So it's best probably to have a queueing/failing strategy which takes into account the real-world health and latency of your backend.

Which appears to handle all cases, except that due to the sharp hockey-stick latency in response to overload, by the time you detect a problem it may be too late.

Fun.

Seconded. That is a great read to help improve your understanding of the C runtime environment (stack frames, longjmp etc).

It's pretty interesting to read into the original "broadcast" ethernet specs.

The "thick yellow wire" and the "coax" both had a single broadcast domain for all devices on the segment. You then linked segments with a bridge (so you could scale networks).

Some things I find interesting about them:

1) The idea is based on radio broadcast (ALOHANET). Since you're using radio broadcast there, a single collision domain makes sense. http://en.wikipedia.org/wiki/ALOHAnet. The protocol is best understood as "room full of people shouting at each other". Don't start talking if something else is, and if two people start at the same time, you both back off. Simple as that.

2) If two nodes try to transmit at the same time, you get a collision and both retry. The spec has a random retransmit timer to avoid immediate (and perpetual) recollision. Requires fair nodes...

3) The max cable length and packet size and related via the speed of light (in copper): http://www.wildpackets.com/resources/compendium/ethernet/pro...

4) It was (semi-)reasonable to use a "vampire tap" connector to dig through the insulation and into the core of a the "thick yellow" copper cable. Just don't wiggle it and cut the copper (or introduce too much of an edge and get signal reflection and hence more collisions).

5) When you had a copper connection between all the ports on your network, an etherkiller really meant something. (A cable, mains plug on one end, coax connecter on the other. Plug in and fry every network card on the segment). RJ-45 etherkillers are single-target. Pah.

I guess I would prefer that SMTP "skip over" the CA/DNSSEC model and go straight to something like HSTS+TACK to add the expectation of validity to the expected certs, rather than place trust in the DNSSEC roots.

I guess DANE (with or without DNSSEC) could also be used to cover the first connection, but I guess I'd prefer a world where the previous connection was used as the primary point of trust (the HSTS+TACK approach) rather than the "trusted root" (DNSSEC or CA).

Thanks for that. Do you have an understanding of what would happen if SMTP servers switched to demanding TLS (i.e. bouncing the message if next hop doesn't offer STARTTLS?) with validated certs?

It would hurt mail reachability, but are there any useful numbers as to how badly?

Additionally, it seems as though a solution analogous to HSTS might be better than involving DNSSEC. An SMTP server could advertise that it should be connected to only over TLS - information that could be cached by the sending system.

There is still a MITM problem on 1st connection, but that's an issue for many systems and avoids trusting DNSSEC.

Doesn't this just swap the existing set of trusted CAs (used to bootstrap TLS trust) for a new set of trusted DNS root operators (used to bootstrap DNSSEC trust)?

Surely if you're worried about state actors, this is no help?

I use this http://pastebin.com/HDJvD3GC which as I say isn't my own work (although I've tweaked it a bit).

It depends on some git output strings, so is theoretically fragile but in practice has been stable over a few years of git updates.

Slightly related, I get a lot of value out of a bash prompt which tells me:

- if my git repo is dirty (uncommitted changes)

- which branch it is on

- if (and how many) changes I have stashed

It's not originally my work but I find it massively useful, since it avoids the "committed but not pushed" failure mode, the "on wrong branch" failure more and the "stashed and forgotten about" failure mode.

I know that regularly running git status would work, but this is a useful visual reminder for me.

At a guess (after 1 min grep), this is the original code (from an old copy of the HN src). afaics, that's the code to render the entire table for all the stories (plus the 'more' link), including the html (note the inline 'tr' tags etc).

https://github.com/wting/hackernews/blob/master/news.arc#L92...

The point is that all the variable you want for rendering "page 2" are right there in scope.

A closure can capture those, so you don't have to do any extra work serialising them, sending them to the browser, validating them on return, etc.

Closures can seem odd the first time you come across them, but they're pretty mainstream these days (perl, python, ruby, C++, C#, golang all have them (some of them for a very long time :-), as well as the more traditionally-functional languages).

There's more info here and it's the sort of topic you'll get lots of answers from in HN. http://en.wikipedia.org/wiki/Closure_%28computer_programming...

No, aiui, the trust model is the same. If a peer announces routes to you, all you can do is:

- throw away "martians" (routes to known-reserved/bad parts of IP space)

- throw away stuff you know they shouldn't be announcing (your stuff basically)

other than that, I think you have to trust them when they say that they have a magic route to IP 16.0.0.1 which is shorter than any other you know about. (Since it might be true).

Thanks for that. I hadn't realised that most (large) commodity systems these days had a NUMA setup. I thought it was relegated to the more esoteric end of things (super computer setup, single-system-image clusters).

Thanks - I can see that blindly treating NUMA memory as "uniform random access" is going to hurt.

> The biggest insight was this: memory behaves like complex block storage, with everything implied with respect to optimization, if you really care about performance.

Could you describe the factors causing that? Is it just the cacheing hierarchy or are there other effects?

Would you get simpler (faster) ops if you stole the high bit instead of the low bit?

e.g. x+y becomes (x + y | (1<<63))

Or does the loss of overflow detection hurt you more?

Nice. I hadn't thought of there being a limit (due to vacuum) to sound intensity.

Reminds me of this recent comment on reddit: http://np.reddit.com/r/space/comments/2h9y9g/the_first_launc...

"the Saturn V rocket produced a SWL (Sound Power Level) of about 220 decibels, which is sufficient to melt concrete nearby and set grass aflame a mile away,"

When bytes count (and they often don't, but in this case they probably do), what would be your preferred system? And what do you see as your favoured approach's main advantage(s) over ASN.1/BER?

> Now open the compiled `main` file in a text editor and search for this text:

FYI, the unix utility 'strings' has this as it's raison d'etre.

Awesome app. I was thinking about something like this recently for a niche.

I was going to ask how you direct the SMS to the app, but taking a quick look at the src, it seems the app reads the sms inbox, looking for the specific phone number?

I guess this means that:

- the user sees alerts for the incoming sms?

- the user needs to delete (does the app/could the app do that?)

Great stuff, really interesting. Thanks for sharing it.

I was thinking about that. I think the way to do the test would be to put in a known key value, then call the platform-specific equivalent of "abort and write a core file". The test would then grovel through the core looking for the known key sequence.

Can you play the game the other way and "fail safe"?

i.e. declare the storage volatile but running your crypto code on a non-volatile ptr to it (obtained via cast) to get your performance back?

If the compiler then generates enough smarts to work out that the non-volatile ptr you've passed into your crypto code is referring to volatile storage, then you keep security but get a (noticeable in testing?) performance hit.

I guess that's not as good as your solution though.

This is amazing technology - but I'm curious as to how unusual it is? The article says "It feeds (fed) power from the Scattergood Steam Plant in El Segundo to a distribution center near Bundy and S.M. Blvd."

Are most power stations and distribution centres connected in a similar way? Or are they typically co-located?

i.e. are there loads of cables like this, and if not why did they do it in this case?

I have a pet theory as to why rubberducking works. Basically I think it uses some additional brain machinery to check the metadata on our facts - i.e. how reliable are each of the 'facts' we are using in our reasoning.

I tried to go into it in more detail here: http://www.berthels.co.uk/2013/03/29/how-do-you-know-what-yo...

I think that also works if the link is actually to the immutable commit to which the branch is currently pointing?