Comment by jws | Hacker News Reader

Comment by jws | Hacker News Reader

Parent Full thread

I altered the program to first allocate 64G then go through and write on each page of the allocation on my 16G machine, dropped my swap partition.

After suitable screwing around to keep gcc from optimizing away the writes, it goes a bit like this…

  jim@gvdev:~$ ./fail
  Before
  Allocated 1 GB
  Allocated 2 GB
  ...
  Allocated 64 GB
  After
  Writing in 0 GB
  Writing in 1 GB
  ...
  Writing in 15 GB
  ... rather long delay here...
  Write failed: Broken pipe
  upstairs:~ jim$    <----- bad news here, notice the machine.
  upstairs:~ jim$ slogin gvdev.XXXXX.net
  ssh: connect to host gvdev.XXXXX.net port 22: Network is unreachable

… so now I have to get back in "going out" clothes, go downstairs, and drive over to gvdev and reboot it.

This is a problem with virtual machines vendors such as Linode where the configured swap space is usually tiny by default... It's much better to provide more swap (At least as much as RAM size) and see the performances start to deteriorate as a sign of alarm of overcommit, instead of having random processes killed.

Always good to have your sshd on auto-respawn in case OOM makes a bad choice ;) Learned this in a similar scenario.

Kinda OT, but out of interest, why do you use 'slogin' instead of 'ssh'? Just old habits dieing hard from 'rlogin'?

Yes.

So, when there is no memory and no swap space available, instead of crashing, it zombifies whole system? This is weird.

The linux Out Of Memory killer would have activated (if not disabled). This program derives a 'score' for each program, based on a number of statistics, and kills the process with the highest score. Although it usually makes the right choice, sometimes it can kill innocent programs.

It was worse than a dead sshd. Looks like something kernel-ish died badly. There was one more cron message in the syslog after this, but then nothing until I rebooted. Console unresponsive, no ping answering.

  Nov  2 21:29:33 gvdev kernel: [15115318.546376] fail[17150]: 
      segfault at 7fffe9cbc1e0 ip 00000000004006fe sp 00007fffe9cbb1d0 
      error 4 in fail[400000+1000]

Reply on news.ycombinator.com