Running tests in a dev't sandbox

Project:JNode All
Component:Miscellaneous
Category:task
Priority:normal
Assigned:Stephen Crawley
Status:active
Description

We need a simple way to run JNode tests in a development sandbox that does not require the developer (tester) to know about all of the different kinds of tests, where they are, and how to setup and run them.

To this end, I'm creating a "test.sh" script that will live in the JNode root directory. This will run tests in a project (or all projects) via "build-test.xml" Ant files in each project directory. To start with, the Ant files will launch the "blackbox" harness to run various blackbox tests. Later, we can add JUnit tests, and indeed any kind of test that can be run as a Java command or Ant task.

As part of setting this up, I will be making some changes to "all/build.xml" to do allow it to be "imported" by the "build-test.xml" files. These changes include:

  • a "cp-test" path that specifies a suitable classpath for running sandbox tests, and
  • changing some of the path variables to work when "all/build.xml" is imported from a different base directory.

I've tested normal builds, and can see no ill effects. But there is always the possibility of regressions. Fortunately, any such regressions will be isolated to one (maybe two) files, and will be simple to revert. I'll post a comment (below) with details when I've committed the changes.

I will also reinstate the small number of "command" blackbox tests that are not working after the CLI reorganisation.

#1

Submitted by Stephen Crawley on Sun, 05/03/2009 - 07:08.

The changes have been committed. If there are any regressions, reverting revision 5389 should fix them.

This issue should remain open because there is more work to do.

#2

Submitted by Stephen Crawley on Mon, 05/04/2009 - 04:44.

I've wired all of the existing Shell (blackbox and junit) and CLI (blackbox) tests into the build-test.xml files.

There few other JUnit tests in the codebase, but I don't know if they are suitable to be wired up. (Do they currently run work in the sandbox world?) Could someone who knows about the FS and other junit tests let me know which ones should be wired up?

#3

Submitted by lsantha on Mon, 05/04/2009 - 06:30.

There is a toplevel target of the build: tests, which currently runs the FS tests outside JNode but it was intended as the main target for running all tests which are possible to run outside JNode.

#4

Submitted by Stephen Crawley on Mon, 05/04/2009 - 07:02.

Thanks for the hint. I'll look to see what that target is currently wired to and translate it into build-test.xml files.

BTW, if you think that I'm doing this wrong (e.g. that it should be done via "./build.sh tests"), you are free to spend the time to do a better job.

#5

Submitted by lsantha on Mon, 05/04/2009 - 07:36.

The future usacases will tell what we need, then we can do the necessary adjustments. In principle having a single entry point (ant target) for running the whole testsuite make sense to me but running the tests selectively too for various subprojects can also be very useful for those working on the subproject.

#6

Submitted by Stephen Crawley on Mon, 05/04/2009 - 08:08.

How about I change "./build.sh test" to run "./test.sh all". Would that address your concern?

#7

Submitted by lsantha on Mon, 05/04/2009 - 15:32.

OK, so you prefer to get it rights from the start. I agree because that's more economical. Could we have a discussion about this on IRC? It might be more efficient...