ajdecon

Oil and gas already does a hell of a lot of computing, and invests heavily in supercomputing facilities. [0] I'm not too familiar with the domain, but looking at abstracts of conferences like the Oil & Gas HPC Workshop [1] might give you some idea of the work already being done in those fields.

It's entirely possible that machine learning techniques could be used as well as modeling and simulation for these problems. (They may be already! But my limited expose suggests they're mostly doing traditional HPC.) But you'd probably need to be a petroleum expert already, not just someone with machine learning expertise living in an O&G hub.[2]

[0] Just an example: http://www.aiche.org/chenected/2012/12/bp-building-new-houst...

[1] http://rice2014.og-hpc.org/

[2] Apologies, btw, if you are a petroleum expert who's just frustrated by a lack of machine learning in the industry. :)

I really think there are no good reasons to include build tools in a docker image. The author lists three possible reasons:

- you need a specific version (e.g. redis is pretty old in the Debian repositories).

- you need to compile with specific options.

- you will need to npm install (or equivalent) some modules which compile to binary.

But you can avoid all of these by building your own DEB/RPM packages and installing those into the container.

This might make the container less "whitebox", in that the Dockerfile no longer contains the full steps to reproduce the built image from public sources. But having an internal package repository makes a lot of sense, and not just for building small Docker images. Keeping your own package repository helps make your server builds more reproducible in general, and provides clean mechanisms for performing updates on your own software as well as third-party packages.

(edited for formatting)

At the moment I use feedly, though I may consider newsblur at some point.

I could not live without my rss feeds. :)

The Cray brochure for this system has more information, including detailed specs: http://www.cray.com/Assets/PDF/products/cs/CrayCS-Storm.pdf

A few fun points from the brochure:

- The internal network ("interconnect") is Infiniband in either a torus or fat tree topology

- Up to 512 GB RAM per node

- Power usage is up to 63 kW per rack

- You have the option of including liquid-cooled rear door heat exchangers, if your datacenter air cooling isn't up to the task ;-)

- Compute nodes can have disks installed, but the system supports diskless booting

- Each compute node weighs 88 lbs

One thing to remember about these machines is that the internal high-bandwidth, low-latency network is a big part of why they get such good performance. Most applications that run on these machines are long-running computations consisting of many processes spread across the entire machine, with lots of inter-process communication throughout the lifetime of the job. Think weather prediction, simulations of fluid dynamics, or complex mechanical systems.

(I work on similar systems to these every day. Huge amounts of fun, but quite different needs than your typical web service, or even most Hadoop-ish data analysis problems.)

Remember you may have to deploy third-party processes as well as apps whose code you control. Most existing software uses OS-standard APIs for doing things like resolving hosts, and you can't just point it at a path in zk. That means running a DNS server or configuring /etc/hosts.

I don't have time to write a detailed post, but would a book recommendation work? :)

"The Practice of Cloud System Administration: Designing and Operating Large Distributed Systems", coming out next month, provides a good viewpoint on how distributed systems design knowledge is useful to someone in a sysadmin role. (I've read a preview via Safari Online and found it an excellent resource.)

[0] http://www.amazon.com/The-Practice-Cloud-System-Administrati...

There are a few others... Nvidia has a list at http://www.nvidia.com/object/gpu-cloud-computing-services.ht...

DevOps is a rather overloaded term at the moment. I've seen it refer to any of the following:

- Encouraging collaboration between your Dev, Ops, and QA teams, with some cross-training so they can work together better

- Merging those teams under the same manager to try to improve that collaboration

- Making your developers responsible for all those roles, and never hiring a dedicated sysadmin or QA engineer

I personally think any of those is fine. Startups will err toward having fewer people and all of them be developers, while in a larger company it probably makes sense to specialize more and make "DevOps" mean close collaboration between those teams.

Of course, I've also seen "DevOps" as a job title for what would have previously been a "system administrator" or "site reliability engineer", and I have much less patience for that. :) Occasionally I see a job posting for a role that is actually dev + ops, but most often a "DevOps" posting means "we need a sysadmin, but we don't think sysadmins are cool enough to work here."

Ha, I missed that this existed. Thank you!

I got stuck on level3 over the weekend and then had to put it down due to life stuff. But it was still a ton of fun, and I learned a lot.

If anyone from Stripe is here, do you plan to post the original problem statements and repos for the problems anywhere public now that it's done? I'd love to get a look at level 4... :)

Without presuming to speak for Evan --

Julia's target audience is technical computing, and a large fraction of software in this space is built to solve a particular problem that only might matter for 6 months or a year. You might be trying to simulate the behavior of an experiment you just designed, for example, or trying to analyze a very specific property of a data set. These codes are often very tightly coupled to the scientific problem, and are only ever used in the context of a particular short-lived project. You do the experiments, write the paper, and move on with your life.

To be clear, I don't think Julia itself encourages this pattern any more or less than another language. But it's a very common pattern for scientists, so they often don't care about long-term maintainability.

Granted, this sometimes comes back to bite them later, if they discover the old code is good for a newer experiment, or they need to go back and re-validate results. But this doesn't always happen, and it's not like they're running a live service with customers -- when they finish a given paper, it's actually not unlikely that no one will ever need to use that software again. It's not easy to argue that they should care about maintainability when there's a decent chance this is one-off code.

> Why is the valley crowd treating distributed systems engineering like it's a new fad?

There's certainly been a lot more interest lately in building distributed systems, and in certain models mostly related to databases. I think a lot of this has been pushed by a few companies with an interest in either pushing these fields forward or being known as experts. A good example might be Basho, who've generated a lot of buzz around the Ricon conferences and the Think Distributed podcast.

To be clear, I think this is a good thing. I personally think distributed systems are a really interesting topic, and a big swell in interest generates discussion and lots of interesting reading material. :) Granted it also generates a lot of faddishness and crap, but that's the price of any kind of wide interest in a topic.

And really, while there's some crap out there, a lot of it's actually really good. I've been enjoying aphyr's recent blogging about Jepsen, a lot of the recorded talks from Ricon West were really interesting, and there's been some good discussion on Twitter and lobste.rs related to these topics.

So sure, the valley might be treating this like a fad. They also do that with *.js and obscure editor plugins. :) But in this case I think that's not a bad thing.

I don't see many "I quit!" or "don't go!" essays from science, but you do see scientists expressing concerns about continuing to accept so many PhD students who won't have a clear career path.

This seems especially true in biotech, though part of their concerns seem to be about how the new students will compete with them for grant dollars... see http://scientopia.org/blogs/drugmonkey/2013/01/28/shut-off-t... for an example.

The processor is Apple's own design and its performance is a distinguishing feature. RAM is commodity tech, and Apple will never pick a number that makes everyone happy. I agree that it'd be nice to get that number, but I can also see which one a marketer would rather feature.

I contributed fairly regularly during the early days of the site, but these days I contribute rarely -- mostly to make minor factual corrections or add sources. I haven't run into much of the site's famed editorial bureaucracy but what little I have doesn't encourage me to contribute more.

I don't particularly like homebrew compared to the Linux distro package managers, but I have found it convenient for messing around on my Macbook due to the large existing library.

A couple of other interesting projects in the Linux software installation space, which I have been using recently:

* fpm ("Effing Package Management"): https://github.com/jordansissel/fpm/wiki

Can be used to generate deb or rpm packages given a source directory or existing package of various types. (For example, generate an RPM from a Rubygem.)

I use this frequently to quickly create distro packages from software which only has source available, then put them in a yum repository on S3 for my private projects. (Creating package repositories is so easy, and signing them isn't much harder. It's really worth doing.)

* EasyBuild: https://github.com/hpcugent/easybuild

EasyBuild automates building software from source, and provides a framework for automating a variety of complex build processes. It's targeted at high-performance scientific computing, where many of the applications have arcane or bizarre build processes, and includes modules for a huge variety of scientific apps.

The big advantage (IMO) is that EasyBuild integrates with Environment Modules [1], which makes it easy to work with many different versions of the same software side-by-side. EB doesn't put everything in the same tree; each package is installed into its own directory and Environment Modules is used to manage the PATH, LD_LIBRARY_PATH, etc. "module load hdf5/1.8.11" and you're good.

I use it extensively both on my laptop and on some of the compute clusters I manage, where users frequently need to be able to switch library and application versions quickly and easily.

One workflow I've been experimenting with recently: Use EasyBuild to build and install all packages into locations in /opt on a dev machine, then use fpm to package them and upload into my repo. Reproduce the system by installing the relevant packages from yum, and still get the flexibility of using Environment Modules.

[1] http://modules.sourceforge.net/

Can you provide some examples of where you think homebrew has a better UI than apt-get? I can't think of anything in homebrew which feels significantly easier than apt-get or yum.

Since you mention kexec and TFTP+NFS, are you currently using Perceus? Or is there another system out there with that combo?

As two examples: Partial trust means you trust a key for different purposes, or for different levels of validity.

Different purposes: "Do I trust that this key correctly identifies this person?" is a separate question from "Do I trust this person to do proper verification before signing others' keys?" (i.e., trusted link in the Web of Trust)

Different validity: "I've met this person and verified they own the key", is different from, "They've identified themselves with two forms of government ID", is different from "I've known them all my life".

> I think it is reasonable to observe that the current PGP WoT is designed to try to unambiguously and unimpeachably map public keys to entities (if not necessarily literal human beings).

I don't really think the PGP WoT can "unambiguously and unimpeachably" provide that mapping, or is even intended to. At best, I think that the WoT documents trust relationships between entities with keypairs.

I actually really like this feature of the WoT, because I think it does a good job of simulating the actual trust relationships between people in real life. In a private conversation, I might imagine a friend vouching for someone another person as trustworthy; a key signature from my friend does something similar, in a secure fashion. This is good because I don't want my web of trust or social network to be able to say "this key is certain to be trustworthy": after all, it can't actually guarantee that. But I don't mind seeing trust opinions from my friends, and their friends' friends.

If anything, I think the trust relationships in GPG ("unknown", "marginal", "full") are too unclear, and I know that these numbers mean different things to different people. I'd prefer the ability to add a short note to the signature so I could say something like,

"I know this person well and you can be confident that this key is theirs, but I don't think they're careful enough to trust their signatures."

It is, in fact, possible to make a process immune to the OOM killer:

echo -17 > /proc/$PID/oom_adj

where $PID is the process ID you want to protect.

oom_adj can be tuned with other values to make a process more or less likely to be killed.

http://www.oracle.com/technetwork/articles/servers-storage-d...

A simple line I've used before:

"Please feel free to CC your technical team so we can make sure we're all on the same page regarding [technical requirements X, Y, and Z]."

Or something to that effect.

If the purchasing agent is feeling over-their-head technically, they might jump at the chance to involve someone who actually understands the requirements.

Disclaimer: I don't do SaaS, but I've been involved in a few enterprise sales in the capacity of "sales engineer".

...Did you read the article, or just the title?

This article is about giving money directly to Kenyan villagers, and in a different program to families in Mexico; and about how they've used the money on productive improvements like replacing thatched roofs with metal ones.

It also acknowledges that this might not be the most effective approach, but that it's being tried in these contexts and deserves further study.

This seems like a rather different context than misused American welfare dollars... and it seems unlikely to me that "low IQ" is a sufficient explanation for third-world village poverty.

Windows at work, Mac OS X on personal laptop. But both of them are basically just SSH clients.

CentOS/RHEL on servers.

Ironically, the "we can work you harder" one was the one that I actually saw the email for. :) The others, I admit, may have been paraphrased.

> And the worst part is, it's surely conscious on nobody's part. There have even been studies showing this effect.

There are studies showing that the unconscious effect exists. But it's definitely conscious at times.

I know a few people in HR in Silicon Valley, and here are a few things they've heard from actual hiring managers:

"You don't have any kids? Great, we can work you harder!"

"If he had to reschedule the phone interview because his kid is sick, I don't want to move forward hiring him. He obviously has his priorities wrong."

"I don't know, she's at just about the age most women get pregnant. Don't we have a guy we could interview?"

:-(

* Lois Bujold * Connie Willis * Vernor Vinge * Kim Stanley Robinson * Neal Stephenson * Larry Niven * Hell, even John Scalzi

If it weren't for the "alive" requirement, I'd include Iain M. Banks. :-( Regardless, I highly recommend anything Banks has written over anything of Card's.

If we're willing to include fantasy...

* Brandon Sanderson * Patrick Rothfuss * George R. R. Martin * Terry Pratchett * Mira Grant * Jim Butcher

This just from a brief glance over my most-recent on Goodreads. There is lots and lots of good Sci-Fi and Fantasy out there, and I'd be willing to argue any of these authors over Card in terms of quality. (Though -- to be fair -- I was monumentally unimpressed with Card even before I knew his politics.)

A few of the authors on my list have politics I disagree with as well, but they at least are much less obnoxious and hateful about it.

I'm at a DevOps conference, which technically counts as work.

/etc/hosts ?

> I always assumed the GRE must have been designed to accommodate/calibrated to humanities and social science grad students (or other majors without a significant math component) more than STEM students.

This is probably true. I mean, this is why we have GRE subject tests...