Running black-box tests

The JNode-shell project includes a test harness for running "black-box" tests on JNode commands and scripts. The harness is designed to allow tests to be run both in a development sandbox and on a running JNode system.

The test methodology is very straight-forward. Each testcase consists of a JNode alias, Java classname or inline script, together with an optional set of arguments and an optional inline input file. The command (alias or class) or script it run with the prescribed arguments and input, and the resulting output stream, error stream and return code are compared with expected results in the testcase. If there are any discrepancies between the expected and actual results, the testcase is counted as "failed". If there are any uncaught exceptions, this counts as a test "error".

The testcases are specified in XML files. Examples may be found in the JNode-Shell project in the "src/test" tree; e.g. "src/test/org/jnode/test/shell/bjorne/bjorne-shell-tests.xml". Each file typically specifies a number of distinct testcases, as outlined above. The "all-tests.xml" file (i.e. "shell/src/test/org/jnode/test/shell/all-tests.xml") should include all tests in the shell tree.

Running tests from Eclipse
The following steps can be used to run a set of tests from Eclipse.

  1. Launch Eclipse, switch to your JNode workspace, and open the JNode-shell project.
  2. Navigate to the class "org.jnode.test.shell.harness.TestHarness" ... it is in the "src/test" tree.
  3. Use "Run as>Open Run dialog" to create an app launcher for the TestHarness class.
  4. Set the full classname as above and set the Arguments to the name of a "...-tests.xml" file.
  5. If you are testing commands in a different project from "shell", make sure that the project is on the launcher's classpath.
  6. Use the launcher to run the TestHarness.

You should now see a Console view displaying the output from running the tests. The last line should give a count of tests run, failures and errors.

Running tests from the Linux shell
Running the tests from the Linux shell is simply a matter of doing what the Eclipse launcher is doing. Put the relevant "JNode-*/classes" directories on the classpath, then run:

    java org.jnode.test.shell.harness.TestHarness <xxx-tests.xml>

Running tests from within JNode.
In order to run the tests on the JNode platform, you must first boot JNode with the all plugins and tests loaded. Then, run the following command at the JNode command prompt:

    org.jnode.test.shell.harness.TestHarness -r /org/jnode/test/shell/all-tests.xml

Notes:

  1. The "-r" option tells the test harness to locate the test suite specification as a resource using the classloader. So you need to make sure that the specification and dependent specifications are included in the relevant test plugins.
  2. When using "-r", the resource path should be specified using "/" as the path separator, not ".". The resource path will be treated as absolute whether or not it starts with a "/".
  3. An alternative to "-r" might be to arrange that the test suite / test specification are accessible through JNode's file system; e.g. by uploading them to JNode before running the test harness.

TestHarness command syntax.

The TestHarness command has the following syntax:

    command [ <opt> ...] <spec-file> ...

where <opt> is one of:

        --verbose | - v       # output more information about tests run
        --debug | - d         # enable extra debugging
        --stopOnError | -E    # stop test harness on the first 'error'
        --stopOnFailure | -F  # stop test harness on the first 'failure'
        --sandbox | -s <dir-name> # specifies the dev't sandbox root directory
        --resource | -r       # looks for <spec-file> as a resource on the classpath

The first two options enable more output. In particular, "-d" causes all output captured by the harness to be echoed to the console as it is captured.

The "-s" option can be used when the running the test harness outside of JNode when the root of the development sandbox is not "..".

The "-r" option tells the harness to try to find the test specification files on TestHarness's classpath rather in the file system. This allows the test harness to be run on JNode without the hassle of copying test and test suite specification files into the JNode filesystem.

Note that the TestHarness command class does not implement JNode's "Command" API, so command completion is not available when running on JNode. This is a deliberate design decision, not an oversight.

Reading and writing test specifications.

If a test fails, you will probably need to read and understand the test's specification as a first step in diagnosing the problem. For a description of specification file syntax and what they mean, please refer to the Black-box command tests with TestHarness page.