What not to test

Tests aren't free, and when it comes to getting things done, testing too much can be as bad as not testing at all.

Automated tests are only valuable on applications that will change and get complex over time. Manual testing is generally more efficient for simple programs that won't change or require maintenance (but how often does that happen?).

As that complexity arises, your test suite will take longer to run and become more painful to maintain if you test too much.

Once you know what you can test and how, learning what not to test is the next most valuable lesson, as it also negatively defines what to test.

As always in CS, there are no absolute rules. But here are the guidelines I use for myself at the moment.

Unit tests

Unit tests are called unit for a reason. They're about testing one given class (at least in OOP, as the definition of a unit may vary over paradigms). They should also be an indicator of complexity to help you reduce your dependencies and achieve loose coupling.

They could also be called interface tests, as they should only test the public interface of an object, and how it interacts with the public interface of other objects by sending them messages (method calls).

  • Don't test private methods. They're only important for the effect they produce during a public method call, and should only be tested as a side-effect of testing a public method.

  • Don't test internal states. Don't add accessors just for testing purposes. If a method is never called anywhere in your app, it's dead code, just remove it (and the tests for it). If it's called only inside its class, it probably doesn't need to be public, hence have its own tests.

  • Only test non-trivial methods. Getters, setters, simple validations (a la validate_unique) or any kind of simple delegation are too trivial to bother testing. Non-trivial is basically any case where different contexts may return different results:

    • any kind of treatment: calculation, concatenation, regexes...
    • any branch: if, case, &&, ||, etc.
  • Don't test things you don't care about. Don't couple your test too deeply to things that may change without breaking your app (copy, unused attributes of a complex API data structure, etc.) On exceptions, assess the exception type, not the message. If your app supports it, test the copy against constants in i18n files.

  • Stub all the things. Which really means don't test more than one class in a unit test. If you need factories or complex context setups, you probably have too many dependencies. Refactor. Helping you notice that is arguably the most valuable benefit of unit-testing. There are a few exceptions, like when you're testing something that directly queries the DB in a non-trivial way (don't test if trivial). In that case you should use factories to populate the DB with just the data you need.

  • Consider private nested classes as private methods. Sometimes you're not sure you need a new class, so you start creating one but keeping it nested inside the definition of another. I consider those classes private and only test them through the nesting class. Once I'm ready to make them a reusable first-class citizen of the app, I extract them and unit-test them.

Integration and Acceptance tests

Integration tests assess how some actions (any kind of action: clicks, methods calls, etc.) take your system from a state A to a state B. They test a whole process: workflow, algorithm... They don't care about how our your objects interact behind the scene. They just care about results.

Definitions may vary so here, what I call Acceptance tests is a subset of integration tests that run over the end-user's interface: browser, API calls, UI, etc.

  • Don't try to test all the combinations. As you're testing higher level things, the complexity and number of cases grows exponentially with the number of components involved. That's what unit-tests are for: testing the low-level combinations for each component. Don't test exception cases, especially in acceptance tests, that are more costly. Only test happy-paths, i.e. the most common paths.

  • Only assess what's critical to a given workflow. Don't test the views too extensively: assess a couple of words that tell you're on the right page, a clickable call-to-action or the presence of a business-critical component. Don't go out of your way to assess anything that's not essential to accomplish the task you're testing.

  • Don't use Cucumber unless you really BDD. The mapping from english to code brings in complexity. It's valuable only if non-technical people write or validates the tests, or if the team takes the steps to agree on the domain language when writing the tests.

  • Integration tests don't need to run in the browser. You can BDD your way into complicated logic (for instance tax or financial calculation) by manipulating high-level model objects. Especially to test more cases, as acceptance tests are more expansive.

  • Don't test what you don't own. Your integration tests should expect third-party dependencies to do their job, but not substitute to their own test suite by testing their behavior extensively.

Controller tests

This one is very specific to Web applications using the MVC pattern.

Controller tests have a mixed nature. They take a bit from unit tests, as they should stub the interactions with model objects.

But also from integration tests, as they're tested with HTTP requests and responses as inputs and outputs, and rely on your framework to process those before and after calling the controller action you're actually testing.

  • Only test controller logic: Redirections, authentication, permissions, HTTP status. The actual application logic should happen through model objects that should be stubbed, and tested in their own unit tests.

  • Only test actions. Consider helpers and filters like private methods in a unit test, tested only through the public actions. Unless they have their own class, in which case it should be unit-tested too.

Other kinds of tests

  • Non-regression tests: arguably the most important. No one wants to explain the management why the same bug happened twice. If something did end up causing a bug, there goes your best proof that it should have been tested (expect maybe for copy and view-specific stuff). A non-reg test should still only be written at the necessary level (unit, controller, integration...) to catch the bug next time.

  • Routes tests: mostly useless. Unless you're writing an API, in which case you might want to test your end points when they're not covered by integration tests.

  • View tests: mostly harmful. Nobody wants their tests to break every time they change the copy, or a class name. Superficial view tests like that call-to-action is clickable or that business-critical sidebar hasn't entirely vanished should already be part of your acceptance tests anyway.

  • Browser JS tests: as useful as any. There's no reason JS shouldn't be tested as well as the rest, especially as it can handle a lot of the logic in modern apps. Both unit and integration (sometimes even controller) tests should apply. When practical, the tests should run in every supported browser.