Web development on fire? Smoke testing a Drupal Website

Then I found out about coding standards and profilers! Guidelines that told me how to document my code, inline with my code. No more maintaining class diagrams, flowcharts and what not. It could all be created, based on the little blocks of text above my classes and methods.

To start with, creating and maintaining those blocks of text could prolong the process a bit. But today, I don't really realise that I do it. I just happen to start every file, function, class and method by writing a little one-liner about what I expect from this piece of code.

Sometimes I don't even write the code. I just write a DocBlock, as it's called, making the process of writing the code much easier. The next person who has to fix that infinite redirect, or what ever silly bug I might have introduced, can quickly – by using the profiler – see where the code is stalling, and by looking at the DocBlock see what the code expects as input, how it processes the input, and what it is supposed to do. The clients were happy. Developers were happy. Everyone was happy. And we all went off to have cake.

Three years ago, this blog post would most likely have ended here.

But then someone introduced me to automated software testing. And boy was that annoying. Back to square one, with another "you should really do this" voice in my head, that only made sense to me, and not the client, and was therefore hard to sell.

Clients didn't like this because it would add extra cost to the development, and they don't see a tangible benefit.

What do we get from automated software testing then?

I'm glad you asked. Automated testing is capable of running all sorts of tests (called scenarios) against your code, making sure that your code responds in the way it's supposed to. The great advantage is that you can check what code is passing and what is failing very quickly (and then fix it). If new functionality is created for your website, you can quickly see if it breaks any other functionality.

The disadvantage is that it does take time to create the tests in the first place and then to maintain them. Because of this, we need to be careful about what tests we write, in what manner, and if the cost is justified.

You don't sell it very well. I think I'll skip on this line item.

Hold on a second. There are several types of automated tests. They all have their special purposes, pros and cons. One of them sticks out for me. It's called "smoke testing". It sounds awesome but, from the name alone, no one has any idea what it does. Though once you know what it does, you'll want it!

Why name something that does not explain what it does?

Well, I've tried to figure out where the name comes from, since it's pretty clear that it does not originate from the software development industry. Several suggestions turned up, but the one I like the best is the "plumbing" version. This version describes how plumbers use smoke testing to test if a pipe is leaking by pumping it full of smoke. If you could smell/see the smoke, there must be a leak somewhere.

It tests all aspects of the small pieces of work completed on the pipe. And that is exactly what it's supposed to do here too. It tests all the different layers that an application is depending upon. For example, if a URL fails, it will tell you that it failed (it doesn't tell where or why).

What does it do then?

It tests a URL and sees if it returns a reasonable response. That's all. Nothing more, nothing less.

In the most simple form the test-code pretends it's a user, browses to a specific URL, and sees if the server returns a status code 200, meaning that the request was processed without any problems.

Wait, I can do that. I do it all the time when I develop a website.

Of course you do. But I'm guessing that you don't visit all the other key feature pages on the web site. And, from experience, I can tell you a bug fix in a donation report for example might mess up the payment page, costing you lots of lost revenue.

This is where smoke testing might be able to assist you!

Every time you create a page on a website that contains key (or new) features, you add that page to a list of URLs and let the test framework attempt to visit it. Whenever you want, you can run a test and see if you unintentionally broke something elsewhere.

But all sort of issues could turn up that makes the web server return a status code 200.

Yes, and in case you need to check if different elements on the page are working as designed, you need another type of test.

This may not sound that useful, but given how little effort this takes, it's worth having it in your application. And in Annertech, we have smoke testing setup as part of our installation profile, so from there it's just a matter of adding another URL to a list, to get that URL tested.

In effect, we smoke test as a matter of course with almost no cost to our clients.

That sounds fair enough. How do I use it?

Well, that's something that might not be as simple to answer, because it very much depends on how the rest of your application works. And in Drupal 7 it's even less simple. Sorry for the promising introduction, but Drupal 7 has some issues regarding this, that we will cover later on. In the meantime, let's dig into some actual examples.

First, we need a testing framework PHPUnit happens to be the one that we use in this example. We get the framework via composer, to make things easier to set up. So we'll start by creating the composer.json file.

#composer.json
{
  "require-dev": {
    "phpunit/phpunit": "4.6.*",
    "guzzle/guzzle": "3.9.3"
  }
}

By simply running composer install, we should get all we need for this test to work.

Next, we create a new folder called tests. This is where we put all our relevant test code. Let set up PHPUnit, by creating phpunit.xml.dist (dist extension of the file name makes it possible to have it in a VCS repository, while still being able to be overridden by individual developers).

#tests/phpunit.xml.dist
<?xml version="1.0" encoding="UTF-8"?>
<!-- http://phpunit.de/manual/4.1/en/appendixes.configuration.html -->
<phpunit xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="http://schema.phpunit.de/4.1/phpunit.xsd"
        backupGlobals="false"
        colors="true"
        bootstrap="src/bootstrap.php">
    <testsuites>
        <testsuite name="Project Test Suite">
            <directory>src/*</directory>
        </testsuite>
    </testsuites>
</phpunit>

This file tells PHPUnit to look for test cases in the src folder, and that it should load tests/src/bootstrap.php before running the tests. bootstrap.php is where we can prepare the environment if we need anything special done prior to a test. So, let's write that bootstrap.php next. Create a src folder, inside the test folder, and add the following content:

#tests/src/bootstrap.php
/**
 * @file
 * Bootstrap file for unit tests.
 */

require_once __DIR__ . '/../settings.php';
// In this example this file resides in a Drupal installation profile, and if we
// want access to the Drupal environment, we include this file. Change, or
// remove, base on your needs.
require_once __DIR__ . '/../../../../includes/bootstrap.inc';
require_once __DIR__ . '/../../vendor/autoload.php';

We also need a file that can contain all the variables that are most likely to change, not only in different projects, but also in different environments. We call this settings.php

#tests/settings.php

/**
 * @file
 * Testing variables.
 */

// The HTTP address for the webserver to contact.
$GLOBALS['webserver'] = 'http://localhost/smoke-testing';

// Credentials, if needed for any of the URLs.
$GLOBALS['username'] = 'admin';
$GLOBALS['password'] = 'admin';
$GLOBALS['test_urls_for_anonymous_users'] = array(
  array('user'),
  array('user/password'),
  array('user/register'),
);
$GLOBALS['test_urls_for_authenticated_users'] = array(
  array('admin'),
  array('admin/content'),
);

And now we are ready to write the actual test. Inside tests/src, create new folder called smoketest. Inside that, create a new file called SmokeTest.php

#tests/src/smoketest/SmokeTest.php
/**
 * @file
 * Test if the user URLS are available.
 *
 * This is a demo of how smoke testing of URLs could work in Drupal.
 */

use Guzzle\Http\Client;
use Guzzle\Plugin\Cookie\CookiePlugin;
use Guzzle\Plugin\Cookie\CookieJar\ArrayCookieJar;

/**
 * Smoke testing.
 */
class SmokeTest extends \PHPUnit_Framework_TestCase {

  /**
   * A list of URLs to test, that doesn't require the user to log in.
   */
  public function urlProvider() {
    return $GLOBALS['test_urls_for_anonymous_users'];
  }

  /**
   * A list of URLs to test on pages that requires the user to be logged in.
   */
  public function urlProviderWithCredentials() {
    return $GLOBALS['test_urls_for_authenticated_users'];
  }

  /**
   * Test URLs without authentication.
   *
   * @dataProvider urlProvider
   */
  public function testPageIsSuccessful($url) {
    $webserver = $GLOBALS['webserver'];

    $client = new Client($webserver, array('request.options' => array('exceptions' => FALSE)));
    $request = $client->get($url);
    $response = $request->send();
    $this->assertEquals(200, $response->getStatusCode());
  }

  /**
   * Test URLs that requires the user to login.
   *
   * @dataProvider urlProviderWithCredentials
   */
  public function testPageIsSuccessfulWithCredentials($url) {
    $webserver = $GLOBALS['webserver'];

    $client = $this->getAuthenticatedClient();
    $request = $client->get($url);
    $response = $request->send();
    $this->assertEquals(200, $response->getStatusCode());
  }

  /**
   * Authenticates the user and attach the cookie to the $client object.
   */
  private function getAuthenticatedClient() {
    $webserver = $GLOBALS['webserver'];
    $username = $GLOBALS['username'];
    $password = $GLOBALS['password'];

    $client = new Client($webserver, array('request.options' => array('exceptions' => FALSE, 'allow_redirects' => FALSE)));

    // The CookieJar plugin helps us attache the Drupal cookie to the client
    // object.
    $cookie_plugin = new CookiePlugin(new ArrayCookieJar());
    $client->addSubscriber($cookie_plugin);

    // Send a request to the login page to get the required form_build_id.
    // After we receive the HTML, we traverse the DOM for the input field that
    // has the name 'form_build_id'.
    $body = (string) $client->get('user/login')->send()->getBody();
    $dom = new DOMDocument();
    $dom->loadHTML($body);
    $input_elements = $dom->getElementsByTagName('input');
    $form_build_id = '';
    foreach ($input_elements as $element) {
      if ($element->getAttribute('name') == 'form_build_id') {
        $form_build_id = $element->getAttribute('value');
      }
    }

    // These are the required values that we need to send to be able to log in.
    $post_data = array(
      'name' => $username,
      'pass' => $password,
      'form_id' => 'user_login',
      'form_build_id' => $form_build_id,
    );
    $client->post('user/login', array(), $post_data)->Send();
    return $client;
  }
}

In this file, two methods are worth some extra attention. testPageIsSuccessful() and testPageIsSuccessfulWithCredentials() creates the request, using Guzzle, fires the requests, and then we use PHPUnit to check if we are happy with the response. testPageIsSuccessfulWithCredentials() tests the URL as an authenticated user, which is why it's calling the getAuthenticatedClient() method, that takes care of logging the user in, before sending the request.

So far, so good. Let's try to run a test. Go back to the folder where your composer.json is and run this command:

vendor/bin/phpunit -c tests

If everything is set up correctly you should see something like this:

PHPUnit 4.6.9 by Sebastian Bergmann and contributors. Configuration read from /home/user/websites/smoketesting/profiles/smokeprofile/tests/phpunit.xml.dist ..... Time: 1.02 seconds, Memory: 7.75Mb OK (5 tests, 5 assertions)

In that case, you are free to add more URLs to the URL arrays in tests/settings.php.

You said something about Drupal 7 being a special flower in the smoke testing garden?

Yes, in Drupal 7 there are some special cases that you must think through, before you start relying on these tests. You might get what is called false positives on tests that you run, meaning that even if a test fails, Drupal 7 might return a response that makes the webserver think that everything is fine and dandy, and returns a status code 200. I have yet to figure out how to get around this.

Consider this case. You add a page at admin/reports/donations and add the URL into the array, so that it will get tested. Next week, another developer merges his changes with yours, gets a merge conflict, and, by accident, removes your URL from the menu hook. Now, the URL no longer exists. But Drupal 7 will still return a status code 200, because if Drupal cannot find the last piece of the URL (/donations), it tries to find a URL that matches, without that piece. This means it now tries to load admin/reports which exists. It calls the function that corresponds to that URL, and sends the last bit (donations) as a parameter. So, if the URL is missing, the test will still return positive.

So whenever you add a new URL, test the different cases, where it should fail, by hand, to make sure that it actually does.

That's all for today. Now, about that cake...