Undocumenting PHP: Harnessing Simpletest

Simpletest is quite possibly my all time favorite PHP library. The way it's put together greatly appeals to my sense of object oriented design symmetry and the high level API is absolutely fantastic to work with, doing exactly what I want it to do, and almost never getting in the way. With that said, Simpletest does suffer from a dearth of high level documentation, blog posts, and tutorials online, especially in terms of getting started with some of the newer features. What I wanted to do here was focus on a very specific aspect of getting started with Simpletest, comparing and contrasting the various ways of running PHP tests. Note that if you use Drupal or Silverstripe you won't need to care - all of this is taken care of for you. All you have to do is write the test cases and place them in the right folder.

The traditional approach for running test groups was borrowed from JUnit, using the TestSuite API. This approach binds a test case to a parent test suite, either by filename or by name of the test class.

Running test cases from a file:

require_once 'simpletest/unit_tester.php';

$test = new TestSuite('My tests');
$test->addTestFile('tests/my_test_case.php');
$test->run(new HtmlReporter);

Running test cases from a class:

require_once 'simpletest/unit_tester.php';
require_once 'tests/my_test_case.php';

$test = new TestSuite('My tests');
$test->addTestCase(new MyTestCase());
$test->run(new HtmlReporter);

The test case and test suite API is based on the composite pattern, so you can also run test cases on their own in the same way:

require_once 'simpletest/unit_tester.php';
require_once 'tests/my_test_case.php';

$test = new MyTestCase();
$test->run(new HtmlReporter);

As you can imagine, having to manually bind each test case would become fairly inconvenient once you start to build up a larger collection of tests. You would have to edit the test suite runner every time you added a new test case. The more convenient way to run a group of tests is to use a collector to scan through directories and automatically add test case files.

By default, the SimpleCollector will pull in every PHP file and test case in a directory:

require_once 'simpletest/unit_tester.php';
require_once 'simpletest/collector.php';

$test = new TestSuite('My tests');
$test->collect('tests', new SimpleCollector);
$test->run(new HtmlReporter);

The PatternCollector allows you to filter files based on a given regex pattern. This example will collect all the files in a directory that have an extension like .test.php:

require_once 'simpletest/unit_tester.php';
require_once 'simpletest/collector.php';

$test = new TestSuite('My tests');
$test->collect('tests', new SimplePatternCollector('/.test.php$/'));
$test->run(new HtmlReporter);

But it's always convenient to run individual test cases on their own. One of the more recent features in Simpletest is the autorun feature, which allows you to execute a test case directly from the file where it is defined. Autorun is used simply by including the file, and executing it via web request or the command line:

require_once 'simpletest/autorun.php';

class MyTestCase extends UnitTestCase {

    function testTrueAsTrueCanBe() {
        $this->assertTrue(true);
    }

}

Simpletest is smart enough to select a default reporter to suit the environment, so if you ran this file as http://mysite/tests/my_test_case.php it would use the HtmlReporter but if you ran it via cli as $> php -f tests/my_test_case.php it would use the command line friendly TextReporter. To override the default reporter selection, you can configure a preferred reporter:

require_once 'simpletest/autorun.php';
SimpleTest :: prefer(new CustomReporter());

class MyTestCase extends UnitTestCase {

    function testTrueAsTrueCanBe() {
        $this->assertTrue(true);
    }

}

If you want more control or convenience you could instrument your own procedural test runner. The SimpleReporter has a static helper method you can use to check the environment under which the test is running:

require_once 'simpletest/unit_tester.php';
require_once 'simpletest/collector.php';

$test_name = (SimpleReporter::inCli()) ? @$_SERVER['argv'][1] : @$_GET['test'];

if ($test_name) {
	require_once "tests/$test_name.php";
    $test = new $test_name();
} else {
    $test = new TestSuite('My tests');
    $test->collect('tests', new SimpleCollector);
}

$test->run(new DefaultReporter);

Of course, this is just the bare minimum. Error checking is a must if you don't want to annoy the hell out of yourself when running this code on repeat. Note that the DefaultReporter essentially offers the same behavior as the following snippet of code would otherwise do:

$reporter = (SimpleReporter::inCli()) ? new TextReporter : new HtmlReporter;

There is yet another approach to implementing behavior like this without needing to create a procedural runner. The SelectiveReporter allows you to target a specific branch or leaf node in the test case hierarchy and only run what you select:

require_once 'simpletest/unit_tester.php';
require_once 'simpletest/collector.php';

$test = new TestSuite('My tests');
$test->collect('tests', new SimpleCollector);

if (SimpleReporter::inCli()) {
    $result = $test->run(new SelectiveReporter(new TextReporter, @$argv[1], @$argv[2]));
    return ($result ? 0 : 1);
} else {
    $test->run(new SelectiveReporter(new HtmlReporter, @$_GET['c'], @$_GET['t']));
}

What the selective reporter enables, which the previous methods don't is the ability to execute just a single method from the test case rather than all the methods, which is the default behavior. You can run this by passing in the test case and test method to run via command line:

$> php -f my_tests.php MyTestCase testMyTest

Or via the web browser:

http://mysite/my_tests.php?c=MyTestCase&t=testMyTest

Another thing to note from this example (the previous examples skipped this minor detail) is the return value provided to the CLI script branch. The run method returns a boolean result which can be passed as an exit value to the shell script executing the test. This plays nice with the general paradigm of the shell environment and might come in handy if you were running an external script to do something on test failure.

This is just the tip of the iceberg as far as customizing Simpletest goes. The best place to start is by looking at the code of existing extensions. Writing your own reporters is quite useful in a lot of scenarios. The documentation for SimpleReporter and SimpleReporterDecorator is a good place to start. For a deeper exploration, check out the previous work I did to implement a <a href=/simpletest-treemap-reporter">Treemap Reporter</a>, which illustrates how to change the internal data structures for capturing test results from an array model to a graph based model.

If you haven't already, go start testing! It really does work.