Black-box command tests with TestHarness

Overview

This page gives some guidelines for specifying "black-box tests" to be run using the TestHarness class; see Running black-box tests".

A typical black-box test runs a JNode command or script with specified inputs, and tests that its outputs match the outputs set down by the test specification. Examples set specification may be found in the "Shell" and "CLI" projects in the respective "src/test" tree; look for files named "*-tests.xml".

Syntax for test specifications

Let's start with a simple example. This test runs "ExprCommand" command class with the arguments "1 + 1", and checks that it writes "2" to standard output and sets the return code to "0".

<testSpec title="expr 1 + 1" command="org.jnode.shell.command.posix.ExprCommand"
             runMode="AS_ALIAS" rc="0">
  <arg>1</arg>
  <arg>+</arg>
  <arg>1</arg>
  <output>2
</output>
</testSpec>

Notes:

The odd indentation of the closing "output" tag is is not a typo. This element is specifying that the output should consists of a "2" followed by a newline. If the closing tag was indented, the test would "expect" a couple of extra space characters after the newline, and this would cause a spurious test failure.
Any literal '<', '>' and '&' characters in the XML file must be suitably "escaped"; e.g. using XML character entities;

An "testSpec" element and its nested elements specifies a single test. The elements and attributes are as follows:

"title" (mandatory attribute): gives a title for the test to identify it in test reports.
"command" (mandatory attribute): gives the command name or class name to be used for the command / script.
"runMode" (optional attribute): says whether the test involves running a command alias or class ("AS_ALIAS") in the same way as the JNode CommandShell would do, running a script ("AS_SCRIPT"), or executing a class via its 'main' entry point ("AS_CLASS"). The default for "runMode" is "AS_ALIAS".
"rc" (optional attribute): gives the expected return code for the command. The default value is "0". Note that the return code cannot be checked when the "runMode" is "AS_CLASS".
"trapException" (optional attribute): if present, this is the fully qualified classname of an exception. If the test throws this exception or a subtype, the exception will be trapped, and the harness will proceed to check the test's post-conditions.".
"arg" (optional repeated elements): these elements gives the "command line" arguments for the command. If they are omitted, no arguments are passed.
"script" (conditional element): if "runMode" is "AS_SCRIPT", this element should contain the text of the script to be executed. The first line should probably be "#!<interpreter-name>".
"input" (optional element): this gives the character sequence that will be available as the input stream for the command. If this element is absent, the command will be given an empty input stream.
"output" (optional element): this gives the expected standard output contents for the command. If this element is absent, nothing is expected to be written to standard output.
"error" (optional element): this gives the expected standard error contents for the command. If this element is absent, nothing is expected to be written to standard error.
"file" (optional repeating element): this gives an input or output file for the test, as described below.

Syntax for "file" elements

A "file" element specifies an input or output file for a test. The attributes and content
are as follows:

"name" (mandatory attribute): gives the file name. This must be relative, and will be resolved relative to the test's temporary directory.
"input" (optional attribute): if "true", the element's contents will be written to a file, then made available for the test or harness to read, If "false", the element's contents will be checked against the contents of the file after the test has run.
"directory" (optional attribute): if "true", the file denotes a directory to be created or checked. In this case, the "file" element should have no content.

Script expansion

Before a script is executed, it is written to a temporary directory. Any @TEMP_DIR@ sequence in the script will be replaced with the name of the directory where input files are created and where output files are expected to appear.

Syntax for test sets

While the test harness can handle XML files containing a single <testSpec> element, it is more convenient to assemble multiple tests into a test set. Here is a simple example:

<testSet title="expr tests"">
  <include setName="../somewhere/more-tests.xml"/>
  <testSpec title="expr 1 + 1" ...>
...
  </testSpec>
  <testSpec title="expr 2 * 2" ...>
...
  </testSpec>
</testSet>

The "include" element declares that the tests in another test set should be run as part of this one. If the "setName" is relative, it will be resolved relative to this testSet's parent directory. The "testSpec" elements specify tests that are part of this test set.

Plugin handling

As a general rule, JNode command classes amd aliases are defined in plugins. When the test harness is run, it needs to know which plugins need to be loaded or the equivalent if we are running on the development platform. This is done using "plugin" elements; for example:

  ...
  <plugin id="org.jnode.shell.bjorne"
             class="org.jnode.test.shell.bjorne.BjornePseudoPlugin"/>
  <plugin id="org.jnode.shell.command.posix"/>
  ...

These elements may be child elements of both "testSpec" or "testSet" elements. A given plugin may be specified in more than one place, though if a plugin is specified differently in different places, the results are undefined. If a "plugin" element is in a "testSpec", the plugin will be loaded before the test is run. If a "plugin" element is in a "testSet", the plugin will be loaded before any test in the set, as well as any test that are "included".

The "plugin" element has the following attributes:

"id" (mandatory attribute): gives the identifier of the plugin to be loaded.
"version" optional attribute): gives the version string for the plugin to be loaded. This defaults to JNode's default plugin version string.
"class" optional attribute): gives the fully qualified class name for a "pseudo-plugin" class; see below.

When the test harness is run on JNode, a "plugin" element causes the relevant Plugin to be loaded via the JNode plugin manager, using the supplied plugin id and the supplied (or default) version string.

When the test harness is run outside of JNode, the Emu is used to provide a minimal set of services. Currently, this does not include a plugin manager, so JNode plugins cannot be loaded in the normal way. Instead, a "plugin" element triggers the following:

The plugin descriptor file is located and read to extract any aliases and command syntaxes. This information is added to Emu's alias and syntax managers.
If the "plugin" element includes a "class" attribute, the corresponding class is loaded and the default constructor is called. This provides a "hook" for doing some initialization that would normally be done by the real Plugin. For example, the "plugin" in the bjorne tests use the "BjornePseudoPlugin" class to registers the interpreter with the shell services. (This is normally done by "BjornePlugin".)