Comment by shanemhansen | Hacker News Reader

Comment by shanemhansen | Hacker News Reader

So basically their biggest problem was that node.js machine-gun hammers the filesystem with stat calls when looking for modules. This was made worse by running on a shared folder.

Vagrant has a mode called rsync-auto which will synchronize directories into a local vm. Or you could just always have the code local to the node process. It makes more sense to have your editor work with remote files.

Next biggest problem was relational database. I think that tests requiring a SQL database are usually a bad sign, and the statement "we'd like to use transactions but our orm doesn't support them" makes me sad. To paraphrase Ben Franklin, "those who would give up transactions for a framework deserve neither". Here's some ideas for speeding up your tests, in order from best to worst.

  - Make most business code rely on a service or interface. Swap that our during tests for a trivial in-memory implementation.
  - Use sqlite :memory: for tests
  - TRUNCATE TABLE
  - Set your database to running with scissors mode (disable fsync)

[edit] I just took a look at the ORM used, waterline https://github.com/balderdashy/waterline . Why wasn't step one moving to sails-memory for running tests?

Disagree about DB tests. If you want to test queries, then it's bad idea to mock DB. If you don't want to test queries, don't call your tests coverage full.

The best way to do db testing is with an empty db and in a specialized dao test. Try to make these dao tests non-overlapping so that the dao test is the only test that hits that part of the schema. Also, if your dao tests are the only thing that hit I/O you can safely do all the other non-integration tests concurrently.

Service layer tests should have mocks of the dao and web service calls. You should do most of your testing here and with unit tests for tricky algorithms.

Integration tests should hit the full stack, but those should run on a continuous integration server asynchronously and also take place in their own empty database.

The key to dao and integration testing is the empty database. Don't let it fill up with test crud between tests. PostgreSQL is pretty fast when your test db only has a couple of rows.

This is by far my favorite way to test; not something I invented but something I picked up start with an EMPTY DATABASE (with tables setup), then every test: begins a transaction,asserts the db is empty, sets up the data (via factories and DAOs), performs its queries (via DAOs) rolls back the transaction for the next test.

It's a fast approach that gives some test coverage, but some of the most important things to test are around the interaction of multiple transactions, which you can't do if you're using a single big transaction to run your test.

An empty DB?

Some major bugs are caused by people not testing their code with enough data.

Not production DB, obviously :) Generate data before tests, collect list of created records during tests, remove collected items after all tests - if DB is not empty after it, you can see something unexpected has happened. If it's not possible to collect records (because of too opaque ORM), just flush DB.

Same. Not only to ensure I wrote queries or scopes correctly, but also because I like to add extra validation constraints to be sure my application code can't sneak junk into the database.

How can I really ensure my CHECK constraints / DOMAINs, unique indexes, RAISE EXCEPTION in insert/update triggers, etc. are correctly implemented if I'm not testing against a real Postgres database?

I would recommend writing tests specifically for testing the PostgreSql changes, ideally you can write them purely in PostgreSql, maintaining a nice separation of your code and your SQL. Also, the only thing that should really need proper testing are stored procedures, your check constraints can't possible be so complicated that they warrant tests.

The unit tests that test out your business logic should use mock data - it's just much more reliable, reproducible, and faster.

That doesn't mean your data/db access needs to be in the same project, or tests as the rest of the application... if you haven't touched the db bits, don't run the db tests.

It's about creating appropriately sized discrete modules.

One of things unit-tests gives is fearless refactoring. When you can easily change behavior in one module and tests shows other modules are not affected (or affected). When you test only one module/layer you don't see the whole picture and such tests are much less representative and useful.

Only fearless if you're not using mocks/stubs/etc. With mocks and stubs in a dynamically typed language it's not quite as fearless.

Dynamically typed languages are an anti-pattern, where they are not unavoidable - i.e. web front-ends.

Bollocks. If your database schema changes right at the heart of the data model, a load of unit tests are not going to save you from some serious hassle.

> Use sqlite :memory: for tests

Then you run into issues where the functionality works on SQLite, but not on your real database.

May not work. Given the use of ORMs these days, this is a smaller issue. Though it still requires attention. Having the tests run quicker allows for better use of the programmers time

> Given the use of ORMs these days, this is a smaller issue

The ORM can't abstract away all differences between databases. For example, SQLite doesn't have Numeric type, so all of the Numeric(10, 2) fields on (e.g.) my SQLAlchemy (a Python ORM) models will trigger warnings when saving to a SQLite database.

Hence the sentence that follows after the one quoted. It does require attention. Just a bit less. In a sea of issues, reducing the importance of any might probably ease development in some way.

Where I work, tests have two modes: quick and long. Quick mode gets in memory treatment. Long mode doesn't. Anything that goes live will undergo a long mode test beforehand.

We would run those tests a couple times a week, and the memory tests idealy on every successful build (we chained the builds so we never bothered to run the DB tests if the unit tests had already failed).

Strictly speaking it's also possible to get subtle differences in behavior across OSes. Especially if you have a race condition you don't know about yet. I always tend to encourage more diversity (carefully spent).

I run a script that mounts a RAM drive and copies my postgresql database into the drive. I use tablespace to force Postgresql to look at the RAM drive for the database data. That way I am testing my queries against the same database engine but get the benefit of avoiding disk IO.

SQLite might be the easiest to configure, but some other RDBMSs have an option to run in memory too (e.g. MySQL's MEMORY storage engine). Also, as a last resort, you can always configure the DB to write to an in-memory file system.

This strategy is, in essence, mocking the FS layer in DB tests.

Often those modes don't have the same semantics, which is dangerous for testing.

After some point it's diminishing returns though. Obviously nothing can be 100% tested.

you assume that SQLite supports all the querying features I'm using (hint: it doesn't).

Well, those sorts of things cause the tests to immediately fail, and you realize the error of your ways. It's more deadly when you tests pass, but behaviour on production is different.

Hi Shane, thanks for the tip about sails-memory. We've seen a lot of inconsistencies in the way Waterline behaves, so we're hesitant to run it against a different backend than the one we use in production. We'd also lose all of the uniqueness constraints that we use in Postgres.

In general we're trying to write and run more tests that only create and manipulate objects in memory, especially for testing business logic. Unfortunately we still have to have some level of integration with the database to test things at a higher system level.

Re: transactions, most of our database code uses the ORM, so we thought other test speedups would give a bigger benefit. We recently added an interface for getting/querying with a `pg` connection directly and we're using that to implement transactions in a few critical places. I'm planning to release the code soon - hopefully in a blog post in a few weeks.

There should be some optimizer for npm modules. The current system is silly. Popcorn Time installs around 5000 files as part of its node_modules. How is that useful?

There should be some way to combine things into a single module/file or otherwise glob it all together, while removing pointless test, docs, and Adventure Time animated gifs from your deps folder.

Apparently imminent npm v3 "flattens" node_modules/, which should help with this problem. At the same time I worry that version compatibility might become undecidable...

Although I'm sure there are module maintainers shipping AT gifs, we can't really blame npm for that.

By its design _browserify_ can be very powerfull on the node space (there are dedicated options in this way). We use it a lot (with grunt & co) to pack & deploy production code (with excellent result)

Should add, a problem with `rsync-auto` is some files in node_modules (bcrypt, among others) are compiled at runtime, and produce different outputs for Mac/Linux - you can't just copy them from Mac to Linux and expect it to work. Probably some things we could do to hack around this but it didn't seem worth it.

Running against a local mysql instance running on a ramdisk and emptying the DB tables between tests yields "fast enough" performance while avoiding test pollution: ~300 full-stack scenarios in ~5sec start to finish.

In an alternate history, what would have been a good way for node.js to reduce the number of stat calls? What would the tradeoff be?

May want to check out the patch - after a syscall in a directory failed because the directory didn't exist, node would issue 4-5 additional stat() calls for the other filetypes (.js, .index.json, .json, .coffee, etc) even though the directory didn't exist. Solution was to abort after the first failed stat(). https://github.com/nodejs/io.js/pull/1920

Django also issues lots to stat()s when starting up their test framework.

Other options for databases:

    - Run all your tests in a transaction, roll it back in the end (also avoids writes to disk in most scenarios)
    - Put the DB file system on a RAM disk

Reply on news.ycombinator.com