Hacker News Reader

Comment by chris_st | Hacker News Reader

I'm curious if anyone knows the test tool they're using here, in particular for lines like:

  $machine->waitForWindow("^Desktop ");

It looks incredibly useful.

Thanks!

View on HN

It's our custom tool. NixOS has testing infrastructure built on top of QEMU, and you can use Perl scripts like this in order to automate aspects of testing inside that environment by talking to the QEMU monitor -- for example, you can not only wait for a window, but click a button in that window, then also wait for a service to start afterwords, for example. (If you were testing GUI functionality).

Tests are written in a mix of Nix and Perl automation scripts like this; the Perl scripts control QEMU, but the Nix code describes what is put into the disk images. It's fairly bespoke and tuned to our own needs. The source code for the Perl driver is here: https://github.com/NixOS/nixpkgs/blob/f148c5c4a1ffa353836bfd...

It can even do multi-machine test setups as another comment pointed out, and it's pretty fun for writing functional demonstrations; for example I've been meaning to use these tools to write examples of doing PostgreSQL replication and failover, etc.

Thanks very much for this information (you and @grhmc).

PLEASE consider releasing some form of this tool, or at least documenting the snot out of it... it looks incredible. We'd really use this at work.

I think it would be really difficult to get such a good experience with something that isn't NixOS. You might look at things like:

- http://open.qa/ - TestKitchen

But I must recommend, indeed, taking a good hard took at NixOS! :)

The internals of the test framework are pretty heavily tied to Nix and NixOS, so a release independent of Nix(OS) isn't probably going to happen. With that said, the test infrastructure itself is exposed as library functions that any Nix user can use. And even though the test VMs itself will run NixOS, the host machine can be any Linux distro with Nix installed.

Well there seems to be something called nix-shell, but I never heard of it before. It seams pretty useful. The link to plasma tests is provided in the article.

https://github.com/NixOS/nixpkgs/blob/49a6964a4250d98644da61...

Thanks!

This isn't nix-shell. nix-shell is a tool for spawning ephemeral software environments composed of nix packages, something a little like python's virtualenvs.

Digging a little bit, nixos' tests appear to be written in perl and are executed in a context that has access to a bunch of useful primitives for asserting these sorts of things. https://nixos.org/nixos/manual/#sec-writing-nixos-tests

The tooling used by NixOS is custom to NixOS. You can check out the test directory for many more examples.

I'll be writing more soon about how these tests work and how neat they are. They really help NixOS quickly release really high quality updates!

The tests are written in Perl and are invoked by Nix's build tools. You can skim the API at https://nixos.org/nixos/manual/#sec-nixos-tests

In this case, the test makes use of both xdotool and Nix's own testing library.

Reply on news.ycombinator.com