Regression test support for mauve

Project:	JNode Core
Component:	Code
Category:	feature request
Priority:	critical
Assigned:	Fabien D
Status:	closed

Create an extension to the Mauve test suite which makes it possible to compare the reports of successive runs of the testsuite for providing information on regressions.
One possible solution is to save the report in a compact XML format and create a tool which can compare such XMLs to find regressions.
The introduction of regression testing is very important and urgent for the future development of JNode, therefor I mark this task with critical priority.

#1

Submitted by Stephen Crawley on Wed, 10/08/2008 - 10:58.

A generic 'diff' program be a better (more general) solution than an XML comparison tool, IMO.

#2

Submitted by lsantha on Wed, 10/08/2008 - 11:16.

I mean custom comaprasion tool, tailored to our speciffic needs, for the speciffic XML structure which will be generated by the mauve extension.
So this is a tool for comapring the data generated by two successive runs of the tests.

#3

Submitted by Stephen Crawley on Wed, 10/08/2008 - 13:41.

I understood that when I made my suggestion.

#4

Submitted by Fabien D on Wed, 10/08/2008 - 19:30.

Assigned to:

Anonymous

» Fabien D

I am taking this issue.

The output of one test run could be in xml format and looks like that :

       <mauve-test-run>
          <test name="java.lang.String#toString()" result="ok"/>
          <test name="java.lang.String#hashCode()" result="ok"/>
          ...
       </mauve-test-run>

About the evolution of the result in the time, I think we should integrate with the qalab tool : if we create a plugin for, it can generate (among others) a report like that.

#5

Submitted by Fabien D on Sat, 10/11/2008 - 15:14.

I've commited a version that implement the writer & parser for xml reports.

Here is an example of the resulting xml file (with fake data) :

       <run name="run">
         <package name="package">
           <class name="class">
             <test name="test">
               <error>
java.lang.Exception: error
at org.jnode.test.mauve.XMLReportConstants.main(XMLReportConstants.java:44)
Caused by: java.lang.Exception: nested error
... 1 more
               </error>
               <check number="1" check-point="" passed="true" expected="n/a" actual="n/a">
                 <log>
a log with
multiple lines
                 </log>
               </check>
             </test>
           </class>
         </package>
       </run>

For now, the 'testCommand' (which is an alias to org.jnode.test.mauve.MauveTestCommand) output the report both to html and xml.