SVN Usage (Deprecated : we have moved to GitHub)

This page is deprecated since we have moved to GitHub

This is a slight overview of SVN and how to use it. First of all there are three ways to access SVN: svn, svn+ssh and via https. Sourceforge uses WebDAV, that means you can also browse the repository online with your favorite browser, just click on this link.

Subversion uses three toplevel directories named trunk, branches and tags. Trunk can be compared to CVS Head, branches and tags are self-explanatory, I think

To checkout the source simply type:
svn co https://jnode.svn.sourceforge.net/svnroot/jnode/trunk/ jnode
which creates a new directory called jnode (Mind the space between "trunk/" and "jnode"!) . In this directory all stuff of the repository in /jnode/trunk will be copied to jnode/ on your local computer.

svn up|add|commit as expected.

New to subversion is copy, move and delete. If you copy or move a file, the history is also copied or moved, if you e.g. delete a directory it will not show up anymore if you do a new checkout.

If you want to make a branch from a version currently in trunk you can simply copy the content from trunk/ to branches/, e.g. by:
svn copy https://jnode.svn.sourceforge.net/svnroot/jnode/trunk/ https://jnode.svn.sourceforge.net/svnroot/jnode/branches/my-big-change-branch/

I think that's the most important for the moment, for more information have a look at the SVN Handbook located here.

Btw, for using SVN within eclipse you have to install subclipse located here.

Using Git (at GitHub)

The URLs for official git repository at GitHub are listed below :

Site: https://github.com/jnode/jnode
Https: https://github.com/jnode/jnode.git
SSH: git@github.com:jnode/jnode.git

For those who know what they are doing already and simply want push access, refer to the page on setting up push access. For those that are unfamiliar with git, there are a few git pages below that explain some of the common tasks of setting up and using git. This of course is not meant to be a replacement for the git manual.

Git Manual: http://www.kernel.org/pub/software/scm/git/docs/user-manual.html
Git Crash Course for SVN users: http://git.or.cz/course/svn.html

ssh-keygen

git config remote.origin.url git@github.com:[user]/jnode.git

cd ~/git
git clone git@github.com:jnode/jnode.git jnode

git rebase origin master

git rebase master [branch]

git push origin [branch]

git config user.name [name]
git config user.email [email]

The Configure tool

The Configure tool is a Java application that is designed to be run in the build environment to capture and record JNode's build-time configuration settings. The first generation of this tool is a simple command-line application that asks the user questions according to an XML "script" file and captures and checks the responses, and records them in property files and other kinds of file.

The Configuration tool supports the following features:

• Property types are specified in terms of regexes or value enumerations.
• Properties are specified in terms of a property name and type, with an optional default value.
• Property sets are collections of properties associated with files:
  • They are typically loaded from the file, updated by the tool and written back to the file.
  • Properties can be expanded into templates XML and Java source files as well as classic Java properties files.
  • A "FileAdapter" API allows new file formats to be added as plugin classes.
• Property values are captured in "screens" consists of a sequence of "items" which define the questions that are presented to the user.
  • Each "item" consists of a property name and a multi-line "text" that explains the property to the user.
  • A screen can be made conditional, with a "guard" property that determines whether or not the properties in the screen are captured.
• A configuration script file can "import" other script files, allowing the configuration process to modularised. All relative pathnames in scripts are resolved relative to the script file that specifies them.

The configuration tool is launched using the "configure.sh" script:

   $ ./configure.sh

When run with no command arguments as above, the script launches the tool using the configuration script at "all/conf-source/script.xml". The full command-line syntax is as follows:

   ./configure.sh
   ./configure.sh --help
   ./configure.sh [--verbose] [--debug] <script-file>

The command creates and/or updates various configuration files, depending on what the script says. Before a file is updated, a backup copy is created by renaming the existing file with a ".bak" suffix.

Configuration script files

The Configure tool uses a "script" to tell it what configuration options to capture, how to capture them and where to put them. Here is a simple example illustrating the basic structure of a script file:

<configureScript>
  <type name="integer.type" pattern="[0-9]+"/>
  <type name="yesno.type">
    <alt value="yes"/>
    <alt value="no"/>
  </type>

  <propFile name="test.properties">
    <property name="prop1" type="integer.type"
              description="Enter an integer"
              default="0"/>
    <property name="prop2" type="yesno.type"
              description="Do you want to?"
              default="no"/>
  </propFile>

  <screen title="Testing set 1">
    <item property="prop1"/>
    <item property="prop2"/>
  </screen>
</configureScript>

The main elements of a script are "types", "property sets" and "screens". Lets describe these in that order.

A "type" element introduces a property type which defines a set of allowed values for properties specified later in the script file. A property type's value set can be defined using a regular expression (pattern) or by listing the value set. For more details refer to the "Specifying property types" page.

A "propFile" element introduces a property set consisting of the properties to be written to a given property file. Each property in the property set is specified in terms of a property name and a previously defined type, together with a (one line) description and an optional default value. For more details refer to the "Specifying property files" page.

A "screen" element defines the dialog sequence that is used to request configuration properties from the user. The screen consists of a list of properties, together with (multi-line) explanations to be displayed to the user. For more details refer to the "Specifying property screens" page.

Finally, the "Advanced features" page describes the control properties and the import mechanism.

  <type name="integer.type" pattern="[0-9]+"/>
  <type name="yesno.type">
    <alt value="yes"/>
    <alt value="no"/>
  </type>

  <type name="optinteger.type" 
        pattern="[0-9]*" emptyToken="none"/>

  <type name="yesno.type">
    <alt token="oui" value="yes"/>
    <alt token="non" value="no"/>
  </type>

  <propFile fileName="jnode.properties">
    <property name="jnode.vm.size"
                 type="integer.type"
                 description="Enter VM size in Mbytes"
                 default="512"/>
    <property name="jnode.vdisk.enabled" type="yesNo.type"
              description="Configure a virtual disk"
              default="no"/>
  </propFile<

# This file contains some interesting properties

# The following property is interesting
interesting=@interesting@

# The following property is not at all interesting
boring=@boring@

    at_sequence ::= '@' name [ '/' modifiers ] '@'
    name        ::= ... # any valid property name
    modifiers   ::= ... # one or more modifier chars

  <screen title="Main JNode Build Settings">
    <item property="jnode.virt.platform">
The JNode build can generate config files for use with
various virtualization products.
    </item>
    <item property="expert.mode">
Some JNode build settings should only be used by experts.
    </item>
  </screen>

  <screen title="Virtualization Platform Settings"
          guardProp="jnode.virt.platform" valueIsNot="none">
    <item property="jnode.vm.size">
You can specify the memory size for the virtual PC.  
We recommended a memory size of least 512 Mbytes.
    </item>
    <item property="jnode.virtual.disk">
Select a disk image to be mounted as a virtual hard drive.
    </item>
  </screen>

Configuration files

The primary JNode build configuration file is the "jnode.properties" file in the project root directory.

Other important configuration files are the plugin lists. These specify the list plugins that make up the JNode boot image and the lists that are available for demand loading in various the Grub boot configurations.

Build & development environment

This chapter details the environment needed to setup a JNode development environment.

Sub-Projects

JNode has been divided into several sub-projects in order to keep it "accessible". These sub-projects are:

Each sub-project has the same directory structure:

Eclipse

JNode is usually developed in Eclipse. (It can be done without)
The various sub-projects must be imported into eclipse. Since they reference each other, it is advisably to import them in the following order:

1. core
2. shell
3. fs
4. gui
5. net
6. builder
7. distr
8. all
9. sound
10. textui
11. cli

For a more details please have a look at this Howto.

IntelliJ IDEA

JetBrains Inc has donated a Open Source License for Intellij IDEA to the dedicated developers working on JNode.

Developers can get a license by contacting Martin.
Setup of the sub-projects is done with using the modules feature like with Eclipse.

One should increase the max memory used in the bin/idea.exe.vmoptions or bin/idea.sh.vmoptions file, edit the -Xmx line to about 350mb. IntelliJ can be downloaded at http://www.jetbrains.com/idea/download/ Use at least version 5.1.1. Note that this version can import Eclipse projects.

Requirements for building under Windows

1. Make sure that you have a Sun JDK for Java 1.6.0 at or near the most recent patch level. (Some older patch levels are known to cause obscure problems with JNode builds.)
2. Make sure that the pathname for the root directory your JNode tree contains no spaces. (Spaces in the pathname are likely to break the build.)
3. Create a "bin" directory to holds some utilities; see below.
4. Use the "System" control panel to add the "bin" directory to your windows %PATH%.
5. Download the "nasm" assembler from http://nasm.sourceforge.net. (Make sure that you get the Win32 version not the DOS32 version!)
6. Open the downloaded ZIP file, and copy the "nasm.exe" file to your "bin" directory. Then rename it to "nasmw.exe".

Now, can start a Windows command prompt, change directory to the JNode root, and build JNode as explained the next section.

Requirements for building under Linux

1. Make sure that you have a Sun JDK for Java 1.6.0 at or near the most recent patch level. (Some older patch levels are known to cause obscure problems with JNode builds.)
2. Make sure that the 'nasm' assembler is installed. If not, use "System>Add/Remove Software" (or your system's equivalent) to install it.

Building

Running "build.sh" or "build.bat" with no arguments to list the available build targets. Then choose the target that best matches your target environment / platform.

Alternatively, from within Eclipse, execute the "all" target of all/build.xml. Building in Eclipse is not advised for Eclipse version 2.x because of the amount of memory the build process takes. From Eclipse 3.x make sure to use Ant in an external process.

A JNode build will typically generate in the following files:

Some builds also generate an ISO image which you can burn to disk, and then use to boot into JNode from a CD / DVD drive.

Building on the Mac OSX / Intel platform

(These instructions were contributed by "jarrah".)

I've successfully built jnode on MacOS X from the trunk and the 2.6 sources. Here's what I needed to do:

Using OSX and PowerPC for JNode development and testing

Using OSX and PPC for JNode development and testing

What we want is:
1. CVS tool
2. IDE for development
3. A way to build JNode
4. A way to boot JNode for testing

First of all we need to install the XCode tools from apple. Usually it is shipped with your OSX, look in /Applications/Installers/. If it is not there you, you can download it from apple’s site.

1. CVS tool
Well cvs is already in the OSX installation. There are some GUI tools to make the use of cvs easier. SmartCVS is a good one, which you can use it in your windows/PC computer, or linux etc.

2. IDE
Eclipse.

3. How to build JNode with a ppc machine (not FOR, WITH ppc)
Good for us, JNode build process is based on apache ant, which as a java tool runs everywhere. The only problem is the native assembly parts of JNode. For them JNode build process uses nasm and yasm.

So the only thing we need is to build them for ppc and use them. They will still make x86 binaries as they are written to do.

First of all we have to get the nasm and yasm sources. The first one is on
http://nasm.sourceforge.net
and the other is on
http://www.tortall.net/projects/yasm/

After that we unzip them and start the compile.

NASM
Open a terminal window and go inside the directory with the nasm sources

Run ./configure to create the Makefile for nasm

If everything is ok you now are ready to compile nasm. Just run ‘’make nasm‘’. Maybe there will be a problem if you try to compile all the nasm tools by running ‘’make’’ (I had), but you dont need them. Nasm is enought.

Now copy nasm in your path. /usr/bin is a good place.

YASM
The same as for nasm open a terminal window and go to the directory with yasm sources.

Run ‘’./configure’’

Run ‘’make’’

Now you can either copy yasm to /usr/bin or run ‘’make install’’ which will install the yasm tools under /usr/local/bin.

That’s all with nasm and yasm. You are ready to build JNode. You may have problems using the buildl.sh script, but you can always run the build command manually ‘’java -Xmx512M -Xms128M -jar core/lib/ant-launcher.jar -lib core/lib/ -lib /usr/lib/java/lib -f all/build.xml cd-x86’’

4. Booting JNode
Well there is only one way to do that. Emulation.

There is VirtualPC for OSX, which is pretty good and fast. To use it just create a new virtual PC and start it. When the virtual PC is started right click on the CD-Rom icon at the bottom of the window (hmm I know there is no right click on macs I assume you know to press ctrl+click). Now tell the VirtualPC to use the JNode iso image as cdrom drive and boot from it. There you are!

I think there is also qemu for ppc. I have not ever used it, so I don’t know how you can configure it.

Directory structure

The JNode sources are divided into the following groups:

• all
  Contains the global libraries, the (Ant) build files and some configuration files. This group does not contain java sources.
• builder
  Contains the java source code used to build JNode. This includes several Ant tasks, but also code used to link Elf files and to write the JNode bootimage.
• core
  Contains the JNode virtual machine code (both java and assembler), the classpath java library sources and the core of the JNode operating system, including the plugin manager, the driver framework, the resource manager and the security manager. This is by far the largest and most complex group.
• distr
  Contains the first parts of the JNode distribution. This includes an installation program and various applications.
• fs
  Contains the file system framework, the various file system implementation and the block drivers such as the IDE driver, harddisk driver, CD-ROM etc.
• gui
  Contains the JNode gui implementation. This includes the graphics drivers, the AWT peer implementation, font renderers and the JNode desktop.
• net
  Contains the JNode network layer. This includes the network drivers, the network framework, the TCP/IP stack and the connection between the network layer and the java.net package.
• shell
  Contains the JNode command shell and several system shell commands.
• textui
  Contains a copy of the charva text based AWT implementation.
• cli
  Contains the bulk of JNode's commands.

Every group is a directory below the root of the JNode CVS archive. Every group contains one or more standard directories.

• build
  This directory is created during the build and contains the intermediate build results.
• descriptors
  This directory contains the plugin descriptors of the plugins defined in this group.
• lib
  This directory contains libraries (jar files) required only by this group. An exception is the All group, for which the lib directory contains libraries used by all groups.
• src
  This directory contains the source files. Below this directory there are one or more source directories (source folders in Eclipse) containing the actual source trees.

JNode coding DOs and DONTs

This page lists some tips on how to write good JNode code.

Please add other tips as required.

Avoid using System.in, System.out, System.err

Where possible, avoid using these three variables. The problem is that they are global to the current isolate, and are not necessarily connected to the place that you expect them to be.

In a user-level command should use streams provided by the Command API; e.g. by calling the 'getInput()' method from within the command's 'execute' method. Device drivers, services and so on that do not have access to these streams should use log4j logging.

Avoid cluttering up the console with obscure or unnecessary logging

If a message is important enough to be written to the console, it should be self explanatory. If it is unimportant, it should be logged using log4j at an appropriate level. We really do not need to see console messages left over from someone's attempts to debug something 12 months ago ...

Avoid using Unsafe.debug(...) methods

The org.jnode.vm.Unsafe.debug(...) methods write to the VGA screen and (when kernel debug is enabled) to the serial port. This is ugly, and should be reserved for important early boot sequence logging, VM and GC debugging, and other situations where log4j logging cannot be used.

Don't call 'Throwable.printStackTrace' and friends

Commands should allow unexpected exceptions to propagate to the shell level where they are handled according to the user's setting of the 'jnode.debug' property. (The alternative of adding a "--debug" flag to each command, is a bad idea. It is a lot of work and will tend to lead to inconsistencies of implementation; e.g. commands that don't implement "--debug", send "--debug" output to unexpected places, overload other functionality on the flag, etcetera.)

Services, etc should make appropriate log4j calls, passing the offending Throwable as an argument.

Do use JNode's Command and syntax.* APIs for commands

Commands that are implemented using the Command and syntax.* APIs support completion and help, and are more likely to behave "normally"; e.g. with respect to stream redirection. There are lots of examples in the codebase.

If the APIs don't do what you want, raise an issue. Bear in mind that some requests may be deemed to be "to hard", or to application specific.

Do include a 'main' entry point.

It is a good idea to include a legacy "public static void main(String[])" entry point in each JNode command. This is currently only used by the old "default" command invoker, but in the future it may be used to run the command in a classic JVM.

Don't use '\n\r' or '\r\n" in output.

The line separator on JNode is '\n', like on UNIX / Linux. If you expect your command to only run on JNode, it is not unreasonable to hardwire '\n' in output messages, etc. But if you want your command to be portable, it should use 'System.getProperty("line.separator")', or one of the PrintWriter / PrintStream's 'println' methods.

JNode Java style rules

All code that is developed as part of the JNode project must conform to the style set out in Sun's "Java Style Guidelines" (JSG) with variations and exceptions listed below. Javadocs are also important, so please try to make an effort to make them accurate and comprehensive.

Note that we use CheckStyle 4.4 as our arbiter for Java style correctness. Run "./build.sh checkstyle" to check your code style before checking it in or submitting it as a patch. UPDATE: And also, please run "./build.sh javadoc" to make sure that you haven't introduced any new javadoc warnings.

No TAB characters.

No TAB characters (HT or VT) are allowed in JNode Java source code.
Whitespace TABs should be replaced with the requisite number of spaces or newlines. Non-whitespace TABs (i.e. in Java strings) should be replaced with "\t" or "\f" escape sequences.

Use 4 space indentation.

Two or three characters is too little, eight is too much.

Maximum line width 120.

The JSG recommends 80 characters, but most people use tools that can cope with much wider.

Put following keywords on same line as }

For example:

    try {
        if (condition) {
            something();
        } else {
            somethingElse();
        }
    } catch (Exception ex) {
        return 42;
    }

Note that the else is on the same line as the preceding }, as is the catch.

Indent labels by -4.

For example:

public void loopy() {
    int i;
LOOP:
    for (i = 100; i < 1000000; i++) {
        if (isPrime(i) && isPrime(i + 3) && isPrime(i + 5) {
            break LOOP;
        }
    }
}

No empty { } blocks

A { } block with no code should contain a comment to say why the block is empty. For example,

    try {
        myStream.close();
    } catch (IOException ex) {
        // we can safely ignore this
    }

No marker comments

It is generally accepted that marker comments (like the following) add little to the readability of a program. In fact, most programmers think they are an eyesore and a waste of space.

    //*******************************************************
    //
    // Start of private methods
    //
    //*******************************************************

Avoid copying javadoc

Instead of copying the javadoc from the parent class or method, use the {@inheritDoc} tag and if needed add some specific javadoc.

    /**
     * {@inheritDoc}
     */    
    public void myMethod(String param1, int param2) {
    }

Give your references in the javadoc

When you are implementing a class from a reference document, add a link in the javadoc.

Packages

The java classes of JNode are organized using the following package structure.
Note that not all packages are listed, but only the most important. For a full list, refer to the javadoc documentation.

All packages start with org.jnode.

Common packages

• org.jnode.boot
  Contains the first classes that run once JNode is booted. These classes initialize the virtual machine and start the operating system.
• org.jnode.plugin
  Contains the interfaces of the plugin manager.
• org.jnode.util
  Contains frequently used utility classes.
• org.jnode.protocol
  Contains the protocol handlers for the various (URL) protocols implemented by JNode. Every protocol maps onto a package below this package, e.g. the plugin protocol handler is implemented in org.jnode.protocol.plugin.

JNode virtual machine

• org.jnode.vm
  Contains the core classes of the JNode virtual machine.
• org.jnode.vm.classmgr
  Contains the internal classes that represent java classes, methods & field. It also contains the classfile decoder.
• org.jnode.vm.compiler
  Contains the base classes for the native code compilers that convert java bytecodes into native code for a specific platform.
• org.jnode.vm.memmgr
  Contains the java heap manager, including the object allocator and the garbage collector.
• org.jnode.vm.<arch>
  For every architecture that is supported by JNode a seperate package exists, that contains the architecture dependent classes, including classes for threads and processors and classes for the native code compilation.

JNode operating system

• org.jnode.driver
  Contains the driver framework.
  All drivers and driver API's have a seperate package below this package. Drivers of a similar type are grouped, e.g. all video drivers have a package below org.jnode.driver.video.

• org.jnode.system
  Contains the interfaces for the various low level resources in the system, such as memory regions, I/O port regions, DMA access.
• org.jnode.fs
  Contains the filesystem framework.
  All file systems have a seperate package below this package, e.g. the EXT2 filesystem implementation is contained in the org.jnode.fs.ext2 package and its sub-packages.
• org.jnode.net
  Contains the network layer.
  All network protocols have a seperate package below this package, e.g. the IPv4 protocol and its sub-protocols is contained in the org.jnode.net.ipv4 package and its sub-packages.
• org.jnode.shell
  Contains the command shell.
  All system commands are grouped in packages below this package.

Special packages

There are some packages that do not comply to the rule that all packages start with org.jnode. These are:

• java.*, javax.*
  Contains the classpath implementation of the standard java libraries.

• gnu.*
  Contains implementation classes of the the classpath library.
• org.vmmagic.pragma
  Contains exception classes and interfaces that have special meaning to the virtual machine and especially the native code compilers. These classes are mostly shared with the Jikes RVM
• org.vmmagic.unboxed
  Contains non-normal classes that are used as pointers to raw memory, object references and architecture dependent integers (words). These classes have a special meaning to the virtual machine and especially the native code compilers and should never be instantiated or used without a good knowledge of their meaning. These classes are mostly shared with the Jikes RVM

Source file requirements

Header

All java source files must contain the standard JNode header found in <jnode>/all/template/header.txt.

Do not add extra information to the header, since this header is updated automatically, at which time these extra pieces of information are lots.

Add any extra information about the class to the classes javadoc comment. If you make significant contribution to a class, feel free to add yourself as an @author. However, adding a personal copyright notice is "bad form", and unnecessary from the standpoint of copyright law. (If you are not comfortable with this, please don't contribute code to the project.)

Encoding

All Java source files and other text-based files in the JNode project must be US-ASCII encoded. This means that extended characters in Java code must be encoded in the '\uxxxx' form. Lines should end with an ASCII linefeed (LF) character, not CR LF or LF CR, and hard tab (HT) characters should not be used.

If there is a pressing need to break these rules in some configuration or regression data file, we can make an exception. However, it is advisable to highlight the use of "nasty" characters (e.g. as comments in the file) so that someone doesn't accidentally "fix" them.

How to add a plugin to JNode

This page will describe how to add a java program to JNode as plugin, so that it can be called via its alias.

First of all you need to set up Eclipse (or your favorit IDE) as described in the readme, so that JNode builds without errors and you can use it (e.g. use JNode in VMWare).

There are different ways of extending JNode with a plugin.
A plugin can contain a class that extends Plugin and (or) normal java programs.
Every plugin is described by a descriptor.

For our example we will develop a plugin that contains a normal java program.

We need a name for our plugin : we will use sample, wich is also the packagename of our plugin.
It belongs to one of the JNodes subprojects in our case we will use the ordername sample in the shell subproject.

Every java-file for our plugin has to be in (or in subfolders):



\shell\src\shell\org\jnode\shell\sample

(for me it is d:\jnode\shell\src\shell\org\jnode\shell\sample)



Now we will write a small HelloWorld.java wich will be one of our plugin programs.
Here is the source of the file HelloWorld.java :

package org.jnode.shell.sample;

public class HelloWorld{

        public static void main(String[] args){

            System.out.println(“HelloWorld – trickkiste@gmx.de“);

        }

}

thats ok, but it will not be build until we create a descriptor and add our plugin to the JNode full-plugin-list.xml.



The plugin descriptor (org.jnode.shell.sample.xml stored in the descriptors folder of the shell subproject) and looks like this :

<?xml version="1.0" encoding="UTF-8"? >

<!DOCTYPE plugin SYSTEM "jnode.dtd">


<plugin id="org.jnode.shell.sample"

            name="Sample Plugin"

            version="0.2"

            license-name="lgpl"

            provider-name="Trickkiste">

<requires>

            <import plugin="org.jnode.shell"/>

</requires>

<runtime>

            <library name="jnode-shell.jar">

                        <export name="org.jnode.shell.sample.*"/>

            </library>

</runtime>

<extension point="org.jnode.shell.aliases">

            <alias name="HelloWorld" class="org.jnode.shell.sample.HelloWorld"/>

</extension>

</plugin>

Now we need to add our Plugin to the JNode full-plugin-list.xml, this file is located in jnode\all\conf your entry should look like this :

[...]

            <plugin id="org.jnode.util"/>

            <plugin id="org.jnode.vm"/>

            <plugin id="org.jnode.vm.core"/>

            <plugin id="org.jnode.shell.sample"/>

</plugin-list>

thats it, you can now build JNode and test your HelloWorld plugin by typing HelloWorld.

What we can do now is add „normal“ programs to JNode via its provided Pluginstructure.

Arguments - The Basics

In JNode's command line interface, the Argument types are the Command programmers main tool for interacting with the user. The Argument provides support to the syntax mechanism to accept parameters, or reject malformed parameters, issuing a useful error message. The argument also supplies completion support, allowing a command to provide specific completions on specific domains.

Organization

At the moment, Arguments are mostly grouped into the shell project under the org.jnode.shell.syntax package. For the time being they will remain here. There is an effort being made to 'untangle' the syntax/argument APIs so this designation is subject to change in the future.

New arguments that are created should be placed into the cli project under the org.jnode.command.argument package if their only use is by the commands under the cli project.

How it works

Every command that accepts an option will require Arguments to capture the options and their associated values. The syntax parser makes use of an argument 'label' to map a syntax node to a specific argument. The parser then asks the Argument to 'accept' the given value. The argument may reject the token if it doesn't not satisfy it's requirements, and provide a suitable error message as to why. If it accepts the token, then it will be captured by the argument for later use by it's command.

Arguments also provide the ability to 'complete' a partial token. In some situations completions are not possible or do not make sense, but in many situations completions can be very helpful and save on typing, reduce errors, and even provide a little help if there are alot of options. The more characters there are in the token, the narrower the list of completions becomes. If the argument supplies only a single completion, this completion will be filled in for the user. This is a very powerful capability that can be used to great effect!

Using arguments

Before writing a command, it is important to consult the various specifications that many commands may have. Once you have an idea of the arguments you will need for the command, and you have a syntax put together, you can begin by adding your arguments to the command.

Along with the label that was discussed earlier, commands also take a set of flags. A set of flags is supplied by the Argument class, but individual Argument types may also supply their own specific flags. At the end of this document will be a list of known flags and their purpose, but for now we will discuss the common Argument flags.

SINGLE and MULTIPLE

By default arguments are 'SINGLE'. This means that the argument may only contain one value. In order to change this, and allow the argument to capture multiple values, you must set the MULTIPLE flag.

OPTIONAL and MANDATORY

By default arguments are 'OPTIONAL'. This means that if the argument is not populated with any values, it will not be considered an error. In order to have the parser fail if the argument is not populated with at least one value, set the MANDATORY flag.

EXISTING and NONEXISTENT

These flags are not used by all arguments and may be used to alter the behavior of 'accept' and 'complete' depending on the argument, and their values. As an example, the FileArgument by default will accept most any string that denotes a legal file name. In order to force it to only accept tokens that denote an existing file on the file system than set the EXISTING flag. In order to force it to only accept tokens that denote a file that does not exist already, set the NONEXISTENT flag.

Most arguments have overloaded constructors that allow you to not set any flags. If no such constructor exists, then feel free to create one! Optionally, it is safe to provide '0'(zero) for the flags parameter to mean no flags.

Once you have created the arguments that your command will need, you need to 'register' the arguments. This needs to be done in the Command constructor. Each argument needs to be passed to the registerArguments(Argument...) method. Once this is done, your arguments are ready to be populated the syntax parser.

(Note: Arguments that have been registered, but do not have a matching syntax node with its label will not cause an error at runtime. But they do make trouble for the 'help' command. For this reason it is recommended to not register arguments that have not yet been mapped in the syntax.)

Using arguments

When your command enters at the execute() method, the arguments will be populated with any values that were capture from the command line. For the most part, you will only need to be concerned with three methods supplied by Argument.

public boolean isSet()

If the argument has accepted and captured a token, then this method will return true. Commands should always check this method before querying for the captured values. If you query an argument for its values when it has none, the behavior is undefined and the return value (or possible exception) is unspecified and subject to change without notice. (The one case where this is not totally true is when an argument has the MANDATORY flag, as in this case this will _always_ return true. Though it is still considered 'good practice' to check this method before querying for values)

public V getValue()

This method returns the single value of an argument that was registered as SINGLE. If the argument has the MULTIPLE flag set, this method should not be used as it will throw an exception if there is more than one value captured by the argument. If there are no values captured, this currently returns null, but as noted earlier, this may not always be the case, and should not be relied upon.

public V[] getValues()

This method returns the captured values as an array. Calling this method when there the SINGLE flag is set is perfectly acceptable. Though it is usually more convenient to use the getValue() method.

Thats about it for arguments. Simple huh? Arguments are designed to allow for rapid development of commands and as such provide a nice simple interface for using arguments 'out of the box' so to speak. But the real power of arguments are their ability to be extended and manipulated in many ways so as to provide a more feature filled command line interface.

Basic argument types

Here are a list of the more common argument types, along with a short description on their purpose, features and usage.

AliasArgument

An argument that accepts an 'alias'. It provides completion against those aliases that have been registered via plugin descriptors, and may also include the bjorne-style aliases if the bjorne interpreter is in use. (I'm not sure if it currently does, if not it should!)

FileArgument

An argument that accepts a java.io.File. As used in an example above, this argument is affected by the EXISTING and NONEXISTENT flags of Argument. FileArgument also currently provides two of its own flags that may be set. ALLOW_DODGY_NAMES is used to override the 'accept' and 'complete' features to allow filenames that begin with a '-'(hyphen). Normally FileArgument would consider such a filename to be an error, and reject such a token. There is also the HYPHEN_IS_SPECIAL flag, which allows a single '-' to be accepted. The purpose for this is to allow '-' to exist amongst a list of files, denoting stardnard input or output should be used instead. This feature is subject to change (pending some 'better way' of handling this).

FlagArgument

This is likely to be the most used argument of all. It is used to denote a option that has no associated argument. This argument does not actually capture a token, instead it holds a single value of true if it has been found. This means that you can use isSet() to map its value to a local/instance boolean value. In some cases, a command may wish to allow a flag to be used multiple times to add advanced meaning. The command can use getValues().length in such a case to determine the number of times it has been specified.

IntegerArgument / LongArgument / DecimalArgument(TODO)

Allows an integer value to be captured on the command line. These arguments do not provide very helpful completion, as their domain of completions is generally too large. Their main purpose is to parse valid integers, rejecting those that are malformed.

StringArgument

This is one of the most 'accepting' arguments, as it will accept any token that is given to it. It also provides no completion. If your command really needs an unbounded String, then this is the right argument to use. This argument should be extended for cases where you want to accept a string, but the domain of acceptable strings is limited, and you wish to reject those tokens not within that domain and also possibly provide completion for the argument.

URLArgument

Similar in some respects to FileArgument, the URLArgument accepts valid tokens that represent a URL. This argument also respects the EXISTING and NONEXISTENT flags. Completion for parts of a url (the scheme for example), may be able to complete, but actual URL completion of domain names and the like may be nearly impossible. It should also be noted that the EXISTING and NONEXISTENT flags will likely cause a DNS lookup to be performed,

Syntax - Defining Commands

The syntax of a command is the definition of options, symbols and arguments that are accepted by commands. Each command defines its own syntax, allowing customization of flags and parameters, as well as defining order. The syntax is constructed using several different mechanisms, which when combined, allow for a great deal of control in restricting what is acceptable in the command line for a given command.

How it works

When you define a new command, you must give define a syntax bundle within a syntax extension point. When the plugin is loaded, the syntax bundle is parsed from the descriptor and loaded into the syntax manager. When the bundle is needed, when completing or when preparing for execution, the bundle is retrieved. Because a syntax bundle is immutable, it can be cached completely, and used concurrently.

Also, the help system uses the syntax to create usage statements and to map short & long flags to the description from an argument.

The puzzle pieces

See this document page for a concise description of the various syntax elements.

When setting out to define the syntax for a command, it is helpful to layout the synopsis and options that the command will need. The synopsis of a command can be used to define separate modes of operation. The syntax block itself is an implied <alternatives>, which means if parsing one fails, the next will be tried. To give an example of how breaking down a command into multiple synopsis can be helpful, we'll setup the syntax for a hypothetical 'config' command that allows listing, setting and clearing of some system configurations.

First, our synopsis...

config
    Lists all known configuration options and their values
config -l 
    Displays the value of the specified option
config -s  
    Sets the chosen option to the desired value
config -c 
    Clears the value of the chosen option

And our syntax...

<syntax alias="config">
  <empty />
  <option argLabel="list" shortName="l">
  <sequence>
    <option argLabel="set" shortName="s">
    <argument argLabel="value">
  </sequence>
  <option argLabel="clear" shortName="c">
</syntax>

To be continued...

Utility classes

The cli project contains a few utility classes to make implementation of common features across multiple commands easier. Because it is recommended that these classes be used when possible, they are quite well documented, and provide fairly specific information on their behavior, and how to use them. A brief outline will be provided here, along with links to the actual javadoc page.

AbstractDirectoryWalker

ADW is _the_ tool for doing recursive directory searches. It provides a Visitor pattern interface, with a set of specific callbacks for the implementor to use. It has many options for controlling what it returns, and with the right configuration, can be made to do very specific searching.

Control

The walker is mainly controlled by FileFilter instances. Multiple filters can be supplied, providing an implied '&&' between each filter. If any of the filters reject the file, then the extending class will not be asked to handle the file. This can be used to create very precise searches by combining multiple boolean filters with specific filter types.

The walker also provides the ability to filter files and directories based on a depth. When the minimum depth is set, files and directories below a given level will not be handled. The directories that are passed to walk() are considered to be at level 0. Therefore setting a min-depth of 0 will not pass those directories to the callbacks. When the maximum depth is set, directories that are at the maximum depth level will not be recursed into. They will however still be passed to the callbacks, pending acceptance by the filter set. Therefore setting a value of 0 to the max level may return the initial directories supplied to walk(), but it will not recurse into them.

Note: Boolean filters are not yet implemented, but they are on the short list.

Extending the walker

Although you can extend the walker to a class of it's own, the recommended design pattern is to implement the walker as a non-static inner class, or an anonymous inner class. This design gives the implemented callbacks of the walker access to the inner structure of the command it's used in. When the walker runs it will pass accepted files and directories to the appropriate callback methods. The walker also has callbacks for specific events, including the beginning and end of a walk, as well as when a SecurityException is encountered when attempting to access a file or directory.

public abstract void handleFile(File)

Tells the implementing class that a regular file has been found and accepted.

public abstract void handleDir(File)

Tells the implementing class that a directory has been found and accepted.

public void handleSpecialFile(File)

Tells the implementing class that a file has been found that is neither a directory or a regular file.

protected void handleRestrictedFile(File)

Tells the implementing class that it has found a file that triggered a SecurityException. By default, this method throws an IOException. This will cause walking to completly halt, which is likely undesired, and so it is highly recommended to override this method to provide suitable error message, and optionally continue walking.

protected void handleStartingDir(File)

Tells the implementing class that it is about to start walking the file system from the given file. This is triggered before the file itself is actually resolved. So the caller has a chance to do some initialization, like possibly changing the current working directory to make a relative path resolve with a different prefix path.

protected void lastAction(boolean)

Tells the implementing class that walking has finished. If the walker stopped walking because it was requested to do so, then the boolean parameter will be true. Otherwise if the walker finished normally, it will be false.

Kernel debugger

A very simple kernel debugger has been added to the JNode nano-kernel. This debugger is able to send all data outputted to the console (using Unsafe.debug) to another computer via a null-modem cable connected to COM1.

From the other computer you can give simple commands to the debugger, such as dump the processor thread queues and print the current thread.

The kernel debugger can be enabled by adding " kdb" to the grub kernel command line, or by activating it in JNode using a newly added command: "kdb".

Ewout

Using "remoteout" to record console / logger output

The remoteout command allows you to send a copy of console output and logger output to a remote TCP or UDP receiver. This allows you to capture console output for bug reports, and in the cases where JNode is crashing.

Before you run the command, you need to set up a receiver application on the remote host to accept and record the output. More details (including a brief note on the JNode RemoteReceiver application) may be found in the remoteout command page. Please read the Bugs section as well!

Boot and startup

During the boot process of JNode, the kernel image is loaded by Grub and booted. After the bootstrapper code, we're running plain java code. The fist code executed is in org.jnode.boot.Main#vmMain() which initializes the JVM and starts the plugin system.

Driver framework

The basic device driver design involves 3 components:

• Device: a representation of the actual hardware device
• Driver: a software driver able to control a Device
• DeviceAPI: a programming interface for a Device, usually implemented by the Driver.

There is a DeviceManager where all devices are registered. It delegates to DeviceToDriverMapper instances to find a suitable driver for a given device. Instances of this mapper interface use e.g. the PCI id of a device (in case of PCIDevice) to find a suitable driver. This is configurable via a configuration file.

For a device to operate there are the following resources available:

• Hardware interrupts. A driver can register an IRQHandler which is called on its own (normal java-) thread. The native kernel signals a hardware interrupt by incrementing a counter for that interrupts, after which the thread scheduler dispatches such events to the correct threads of the IRQHandler's.
• DMA channels. A driver can claim a DMA channel.
  This channel can be setup, enabled and disabled.
• IO port access. An Unsafe class has native methods for this. A device must first claim a range of IO ports before it can gain access to it.
• Memory access. A device can claim a range of the the memory addressspace. A MemoryResource is given to the device. The device can use the methods of the MemoryResource to actually access the memory.

Filesystem framework

The filesystem support in JNode is split up into a generic part and a filesystem specific part. The role of the generic part is:

1. Keep track of all mounted filesystems.
2. Map between path names are filesystem entries.
3. Share filesystem entries between various threads/processes.

The role of the filesystem specific part is:

1. Store and retrieve files.
2. Store and retrieve directories.

We should be more specific about what a filesystem is. JNode makes a distinction the a FileSystemType and a FileSystem. A FileSystemType has a name, can detect filesystems of its own type on a device and can create FileSystem instances for a specific device (usually a disk). A FileSystem implements storing/retrieving files and directories.

To access files in JNode, use the regular classes in the java.io package. They are connected to the JNode filesystem implementation. A direct connection to the filesystem implementation is not allowed.

FrameBuffer devices

This chapter details the FrameBuffer device design and the interfaces involved in the design.

FrameBufferAPI

All framebuffer devices must implement this API.

FrameBufferConfiguration

TODO write me.

Surface

TODO write me.

HardwareCursorAPI

TODO write me.

DisplayDataChannelAPI

TODO write me.

Network devices

This chapter details the design of network devices and describe the interfaces involved.

NetDeviceAPI

Every network device must implement this API.

The API contains methods to get the hardware address of the device, send data through the device and get/set protocol address information.

When a network deivce receives data, it must deliver that data to the NetworkLayerManager. The AbstractNetworkDriver class (which is usually the baseclass for all network drivers) contains a helper method (onReceive) for this purpose.

Network protocols

This chapter will detail the interfaces involved in the network protocol layer.

NetworkLayer

This interface must be implemented by all network protocol handlers.

TransportLayer

This interface must be implemented by OSI transport layer protocols.

LinkLayer

This interface must be implemented by OSI link layer protocols.

Registration

To register a network layer, the network layer class must be specified in an extension of the "org.jnode.net.networkLayers" extension point.
This is usually done in the descriptor of the plugin that holds the network layer.

Architecture specifics

This chapter contains the specific technical operating system details about the various architectures that JNode is operating on.

The new command line syntax mechanism

In the classical Java world, a command line application is launched by calling the "main" entry point method on a nominated class, passing the user's command arguments as an array of Strings. The command is responsible for working out which arguments represent options, which represent parameters and so on. While there are (non-Sun) libraries to help with this task (like the Java version of GNU getOpts), they are rather primitive.

In JNode, we take a more sophisticated approach to the issue of command arguments. A native JNode command specifies its formal arguments and command line syntax. The task of matching actual command line arguments is performed by JNode library classes. This approach offers a number of advantages over the classical Java approach:

• The application programmer has less work to do.
• The user sees more uniform command syntax.
• Diagnostics for incorrect command arguments can be more uniform.

In addition, this approach allows us to do some things at the Shell level that are difficult with (for example) UNIX style shells.

• The JNode shell does intelligent command line completion based on a command's declared syntax and argument types. For example, if the syntax requires a device name at the cursor position when the user hits TAB, the JNode shell will complete against the device namespace.
• The JNode help command uses a command's declared syntax to produce accurate "usage" and parameter type descriptions. These can be augmented by descriptions embedded in the syntax, or in separate files.
• In the new version of the JNode syntax mechanisms, command syntaxes are specified in XML separate from the Java source code. Users can tailor the command syntax, like UNIX aliases only better. This can be used to support portable scripting; e.g. Unix-like command syntaxes could be used with a POSIX shell compatible interpreter to run Unix shell scripts.

As the above suggests, there are two versions of JNode command syntax and associated mechanisms; i.e parsing, completion, help and so on. In the first version (the "old" mechanisms) the application class declares a static Argument object for each formal parameter, and creates a static "Help.Info" data structure containing Syntax objects that reference the Arguments. The command line parser and completer traverse the data structures, binding values to the Arguments.

The problems with the "old" mechanisms include:

• Use of statics to hold the Argument and Help.Info objects makes JNode commands non-reentrant, leading to unpredictable results when a command is executed in two threads.
• The Syntax, Argument and associated classes were never properly documented, making them hard to maintain and hard to use.
• There were numerous bugs and implementation issues; e.g. Unix-style named options didn't work, completion didn't work properly with alternative syntaxes, and so on.
• Command syntaxes could not be tailored, as described above.

The second version (the "new" mechanisms) are a ground-up redesign and reimplementation:

• Argument objects are created by the command class constructor, and registered to form an ArgumentBundle. Thus, command syntax is not an impediment to making command classes re-entrant.
• Syntax objects are created from XML that is defined in the command's plugin descriptor, and that can be overridden from the JNode shell using the "syntax" command.
• The "new" Syntax classes are much richer than the "old" versions. Each Syntax class has a "prepare" method that emits a simple BNF-like grammar; i.e. the MuSyntax classes. This grammar is used by the MuParser which performs n-level backtracking, and supports "normal" and "completion" modes. (Completion mode parsing works by capturing completions at the appropriate point and then initialing backtracking to find other alternatives.)

 cat 
 cat -u | -urls <url> ... |
 cat <file> ... 

 cat d.txt

 cat a.txt b.txt c.txt

 cat a.txt b.txt c.txt > d.txt

   39   <extension point="org.jnode.shell.syntaxes">
   40     <syntax alias="cat">
   41       <empty description="copy standard input to standard output"/>
   42       <sequence description="fetch and concatenate urls to standard output">
   43         <option argLabel="urls" shortName="u" longName="urls"/>
   44         <repeat minCount="1">
   45           <argument argLabel="url"/>
   46         </repeat>
   47       </sequence>
   48       <repeat minCount="1" description="concatenate files to standard output">
   49         <argument argLabel="file"/>
   50       </repeat>
   51     </syntax>

   54     private final FileArgument ARG_FILE =
   55         new FileArgument("file", Argument.OPTIONAL | Argument.MULTIPLE,
   56                 "the files to be concatenated");

   58     private final URLArgument ARG_URL =
   59         new URLArgument("url", Argument.OPTIONAL | Argument.MULTIPLE,
   60                 "the urls to be concatenated");

   62     private final FlagArgument FLAG_URLS =
   63         new FlagArgument("urls", Argument.OPTIONAL, "If set, arguments will be urls");

   67     public CatCommand() {
   68         super("Concatenate the contents of files, urls or standard input to standard output");
   69         registerArguments(ARG_FILE, ARG_URL, FLAG_URLS);
   70     }

   79     public void execute() throws IOException {
   80         this.err = getError().getPrintWriter();
   81         OutputStream out = getOutput().getOutputStream();
   82         File[] files = ARG_FILE.getValues();
   83         URL[] urls = ARG_URL.getValues();
   84 
   85         boolean ok = true;
   86         if (urls != null && urls.length > 0) {
   87             for (URL url : urls) {
   ...
  107         } else if (files != null && files.length > 0) {
  108             for (File file : files) {
   ...
  127         } else {
  128             process(getInput().getInputStream(), out);
  129         }
  130         out.flush();
  131         if (!ok) {
  132             exit(1);
  133         }
  134     }

 cat ( ( -u | -urls <url> ) | <file> ) ...

 cat ( <url> | <file> ) ...

    public static void main(String[] args) throws Exception {
        new XxxClass().execute(args);
    }

    <argument argLabel="foo">
    <argument label="foo" description="this controls the command's fooing" argLabel="foo">

    <empty description="dir with no arguments lists the current directory">

    <option argLabel="filename" shortName="f">
    <option argLabel="filename" longName="file">
    <option argLabel="filename" shortName="f" longName="file">

    <verb symbol="subcommand1" argLabel="someArg">

    <sequence description="the input and output files">
        <argument argLabel="input"/>
        <argument argLabel="output"/>
    </sequence>

    <alternatives description="specify an input or output file">
        <option shortName="i" argLabel="input"/>
        <option shortName="o" argLabel="output"/>
    </alternatives>

    <repeat description="zero or more files">
        <argument argLabel="file"/>
    </repeat>

    <repeat minCount="1" description="one or more files">
        <argument argLabel="file"/>
    </repeat>

    <repeat maxCount="5" eager="true" 
              description="as many files as possible, up to 5">
        <argument argLabel="file"/>
    </repeat>

    <optional description="nothing, or an input file and an output file">
        <argument argLabel="input"/>
        <argument argLabel="output"/>
    </optional>

    <optional eager="true"
                 description="an input file and an output file, or nothing">
        <argument argLabel="input"/>
        <argument argLabel="output"/>
    </optional>

    <powerSet description="any number of inputs and outputs">
        <option argLabel="input" shortName="i"/>
        <option argLabel="output" shortName="o"/>
    </powerSet>

    <optionSet description="flags and value options">
        <option argLabel="flagOne" shortName="1"/>
        <option argLabel="flagTwo" shortName="2"/>
        <option argLabel="argThree" shortName="3"/>
    </optionSet>

    <syntax alias="cpuid">
        <empty description="output the computer's id">
    </syntax>

    <syntax alias="dir">
        <empty description="list the current directory"/>
        <argument argLabel="directory" description="list the given directory"/>
    </syntax>

    <alternatives>
        <argument argLabel="number">
        <argument argLabel="file">
    </alternatives>

Using Command Line Completion (old syntax mechanism)

Note: this page describes the old syntax mechanism which is currently being phased out. Please refer to the parent menu for the pages on the new syntax mechanism.

The JNode Command Line Completion is one of the central aspects of the shell. JNode makes use of a sophisticated object model to declare command line arguments. This also provides for a standard way to extract a help-document that can be viewed by the user in different ways. Additionally, the very same object model can be used to access the arguments in a convenient manner, instead of doing the 133735th command line parsing implementation in computer history.

The following terms play an important role in this architecture:

• Parameter
  A key that can be passed to a programm. Typically, this are command line switches like "-h" or "--help", or indicators for the type of the assigned argument.
• Argument
  A value that can be passed to a program. This can be filenames, free texts, integers or whatever type of arguments the program needs.
• Syntax
  A program can define multiple Syntaxes, which provide for structurally different tasks it can handle. A Syntax is defined as a collection of mandatory and optional Parameters.

Black-box command tests with TestHarness

Overview

This page gives some guidelines for specifying "black-box tests" to be run using the TestHarness class; see Running black-box tests".

A typical black-box test runs a JNode command or script with specified inputs, and tests that its outputs match the outputs set down by the test specification. Examples set specification may be found in the "Shell" and "CLI" projects in the respective "src/test" tree; look for files named "*-tests.xml".

Syntax for test specifications

Let's start with a simple example. This test runs "ExprCommand" command class with the arguments "1 + 1", and checks that it writes "2" to standard output and sets the return code to "0".

<testSpec title="expr 1 + 1" command="org.jnode.shell.command.posix.ExprCommand"
             runMode="AS_ALIAS" rc="0">
  <arg>1</arg>
  <arg>+</arg>
  <arg>1</arg>
  <output>2
</output>
</testSpec>

Notes:

1. The odd indentation of the closing "output" tag is is not a typo. This element is specifying that the output should consists of a "2" followed by a newline. If the closing tag was indented, the test would "expect" a couple of extra space characters after the newline, and this would cause a spurious test failure.
2. Any literal '<', '>' and '&' characters in the XML file must be suitably "escaped"; e.g. using XML character entities;

An "testSpec" element and its nested elements specifies a single test. The elements and attributes are as follows:

"title" (mandatory attribute)

gives a title for the test to identify it in test reports.

"command" (mandatory attribute)

gives the command name or class name to be used for the command / script.

"runMode" (optional attribute)

says whether the test involves running a command alias or class ("AS_ALIAS") in the same way as the JNode CommandShell would do, running a script ("AS_SCRIPT"), or executing a class via its 'main' entry point ("AS_CLASS"). The default for "runMode" is "AS_ALIAS".

"rc" (optional attribute)

gives the expected return code for the command. The default value is "0". Note that the return code cannot be checked when the "runMode" is "AS_CLASS".

"trapException" (optional attribute)

if present, this is the fully qualified classname of an exception. If the test throws this exception or a subtype, the exception will be trapped, and the harness will proceed to check the test's post-conditions.".

"arg" (optional repeated elements)

these elements gives the "command line" arguments for the command. If they are omitted, no arguments are passed.

"script" (conditional element)

if "runMode" is "AS_SCRIPT", this element should contain the text of the script to be executed. The first line should probably be "#!<interpreter-name>".

"input" (optional element)

this gives the character sequence that will be available as the input stream for the command. If this element is absent, the command will be given an empty input stream.

"output" (optional element)

this gives the expected standard output contents for the command. If this element is absent, nothing is expected to be written to standard output.

"error" (optional element)

this gives the expected standard error contents for the command. If this element is absent, nothing is expected to be written to standard error.

"file" (optional repeating element)

this gives an input or output file for the test, as described below.

Syntax for "file" elements

A "file" element specifies an input or output file for a test. The attributes and content
are as follows:

"name" (mandatory attribute)

gives the file name. This must be relative, and will be resolved relative to the test's temporary directory.

"input" (optional attribute)

if "true", the element's contents will be written to a file, then made available for the test or harness to read, If "false", the element's contents will be checked against the contents of the file after the test has run.

"directory" (optional attribute)

if "true", the file denotes a directory to be created or checked. In this case, the "file" element should have no content.

Script expansion

Before a script is executed, it is written to a temporary directory. Any @TEMP_DIR@ sequence in the script will be replaced with the name of the directory where input files are created and where output files are expected to appear.

Syntax for test sets

While the test harness can handle XML files containing a single <testSpec> element, it is more convenient to assemble multiple tests into a test set. Here is a simple example:

<testSet title="expr tests"">
  <include setName="../somewhere/more-tests.xml"/>
  <testSpec title="expr 1 + 1" ...>
...
  </testSpec>
  <testSpec title="expr 2 * 2" ...>
...
  </testSpec>
</testSet>

The "include" element declares that the tests in another test set should be run as part of this one. If the "setName" is relative, it will be resolved relative to this testSet's parent directory. The "testSpec" elements specify tests that are part of this test set.

Plugin handling

As a general rule, JNode command classes amd aliases are defined in plugins. When the test harness is run, it needs to know which plugins need to be loaded or the equivalent if we are running on the development platform. This is done using "plugin" elements; for example:

  ...
  <plugin id="org.jnode.shell.bjorne"
             class="org.jnode.test.shell.bjorne.BjornePseudoPlugin"/>
  <plugin id="org.jnode.shell.command.posix"/>
  ...

These elements may be child elements of both "testSpec" or "testSet" elements. A given plugin may be specified in more than one place, though if a plugin is specified differently in different places, the results are undefined. If a "plugin" element is in a "testSpec", the plugin will be loaded before the test is run. If a "plugin" element is in a "testSet", the plugin will be loaded before any test in the set, as well as any test that are "included".

The "plugin" element has the following attributes:

"id" (mandatory attribute)

gives the identifier of the plugin to be loaded.

"version" optional attribute)

gives the version string for the plugin to be loaded. This defaults to JNode's default plugin version string.

"class" optional attribute)

gives the fully qualified class name for a "pseudo-plugin" class; see below.

When the test harness is run on JNode, a "plugin" element causes the relevant Plugin to be loaded via the JNode plugin manager, using the supplied plugin id and the supplied (or default) version string.

When the test harness is run outside of JNode, the Emu is used to provide a minimal set of services. Currently, this does not include a plugin manager, so JNode plugins cannot be loaded in the normal way. Instead, a "plugin" element triggers the following:

1. The plugin descriptor file is located and read to extract any aliases and command syntaxes. This information is added to Emu's alias and syntax managers.
2. If the "plugin" element includes a "class" attribute, the corresponding class is loaded and the default constructor is called. This provides a "hook" for doing some initialization that would normally be done by the real Plugin. For example, the "plugin" in the bjorne tests use the "BjornePseudoPlugin" class to registers the interpreter with the shell services. (This is normally done by "BjornePlugin".)

Arrays

Arrays are allocated just like normal java objects. The number of elements of the array is stored as the first (int) instance variable of the object. The actual data is located just after this length field.

Bytecodes that work on arrays are do the index checking. E.g. on the X86 this is implemented using the bound instruction.

Classes & Objects

Each class is represented by an internal structure of class information, method information and field information. All this information is stored in normal java objects.

Every object is located somewhere in the heap. It consists of an object header and space for instance variables.

Exceptions

At the start of each method invocation a frame for that method is created on the stack. This frame contains references to the calling frame and contains a magic number that is used to differentiate between compiled code invocations and interpreted invocations.
When an exception is thrown, the exception table of the current method is inspected. When an exception handler is found, the calculation stack is cleaned and code executing continues at the handler address.
When no suitable exception handler is found in the current method, the stackframe of the current method is destroyed and the process continues at the calling method.

When stacktrace of an exception is created from the frames of each method invocation. A class called VmStackFrame has exactly the same layout as a frame on the stack and is used to enumerate all method-invcocations.

Garbage collection

JNode uses a simple mark&sweep collector. You can read on the differences, used terms and some general implementation details at wikipedia. In these terms JNode uses a non-moving stop-the-world conservative garbage collector.

About the JNode memory manager you should know the following: There is org.jnode.vm.memmgr.def.VmBootHeap. This class manages all objects that got allocated during the bootimage creation. VmDefaultHeap contains objects allocated during runtime. Each Object on the heap has a header that contains some extra information about the heap. There's information about the object's type, a reference to a monitor (if present) and the object's color (see wikipedia). JNode objects can have one of 4 different colors and one extra finalization bit. The values are defined in org.jnode.vm.classmgr.ObjectFlags.

At the beginning of a gc cycle all objects are either WHITE (i.e. not visited/newly allocated) or YELLOW (this object is awaiting finalization).

The main entry point for the gc is org.jnode.vm.memmgr.def.GCManager#gc() which triggers the gc run. As you can see one of the first things in gc() is a call to "helper.stopThreadsAtSafePoint();" which stops all threads except the garbace collector. The collection then is divided into 3 phases: markHeap, sweep and cleanup. The two optional verify calls at the beginning and end are used for debugging, to watch if the heap is consistent.

The mark phase now has to mark all reachable objects. For that JNode uses org.jnode.vm.memmgr.def.GCStack (the so called mark stack) using a breadth-first search (BFS) on the reference graph. At the beginning all roots get marked where roots are all references in static variables or any reference on any Thread's stack. Using the visitor pattern the Method org.jnode.vm.memmgr.def.GCMarkVisitor#visit get called for each object. If the object is BLACK (object was visited before and all its children got visited. Mind: This does not mean that the children's children got visited!) we simply return and continue with the next reference in the 'root set'. If the object is GREY (object got visited before, but not all children) or in the 'root set' the object gets pushed on the mark stack and mark() gets called.
Let's make another step down and examine the mark() method. It first pops an Object of the mark stack and trys to get the object type. For all children (either references in Object arrays or fields of Objects) the processChild gets called and each WHITE (not visited yet) object gets modified to be GREY. After that the object gets pushed on the mark stack. It is important to understand at that point that the mark stack might overflow! If that happens the mark stack simply discards the object to push and remembers the overflow. Back at the mark method we know one thing for sure: All children of the current object are marked GREY (or even BLACK from a previous mark()) and this is even true if the mark stack had an overflow. After examining the object's Monitor and TIB it can be turned BLACK.
Back at GCManager#markHeap() we're either finished with marking the object or the mark stack had an overflow. In the case it had an overflow we have to repeat the mark phase. Since many objects now are allready BLACK it is less likly the stack will overflow again but there's one important point to consider: All roots got marked BLACK but as said above not all children's children need to be BLACK and might be GREY or even WHITE. That's why we have to walk all heaps too in the second iteration.
At the end of the mark phase all objects are either BLACK (reachable) or WHITE (not reachable) so the WHITE ones can be removed.

The sweep again walks the heap (this time without the 'root set' as they do not contain garbage by definition) and again visits each object via org.jnode.vm.memmgr.def.GCSweepVisitor#visit. As WHITE objects are not reachable anymore it first tests if the object got previously finalized. If it was it will be freed, if not and the object has a Finalizer it will be marked YELLOW (Awaiting finalization) else it will be freed too. If the object is neither WHITE nor YELLOW it will be marked WHITE for the next gc cycle.

The cleanup phase at the end sets all objects in the bootHeap to WHITE (as they will not be swept above) as they might be BLACK and afterwards calls defragment() for every heap.

Some other thoughts regarding the JNode implementation include:

It should be also noted that JNode does not know about the stack's details. I.e. if the mark phase visits all objects of a Thread's stack it never knows for a value if it is a reference or a simple int,float,.. value. This is why the JNode garbage collector can be called conservative. Every value on the stack might be a reference pointing to a valid object. So even if it is a float on the stack, as we don't know for sure we have to visit the object and run a mark() cycle. This means on the one hand that we might mark memory as reachable that in reality is garbage on the other hand it means that we might point to YELLOW objects from the stack. As YELLOW objects are awaiting finalization (and except the case the finalizer will reactivate the object) they are garbage and so they can not be in the 'root set' (except the case where we have a random value on the stack that we incorrectly consider to be a reference). This is also the reason for the current "workaround" in GCMarkVisitor#visit() where YELLOW objects in the 'root set' trigger error messages instead of killing JNode.

There is some primilary code for WriteBarrier support in JNode. This is a start to make the gc concurrent. If the WriteBarrier is enabled during build time, the JNode JIT will include some special code into the compiled native code. For each bytecode that sets a reference to any field or local the writebarrier gets called and the object gets marked GREY. So the gc will know that the heap changed during mark. It is very tricky to do all that with proper synchronization and the current code still has bugs, which is the reason why it's not activated yet.

Java Security

This chapter covers the Java security implemented in JNode. This involves the security manager, access controller and privileged actions.
It does not involve user management.

The Java security in JNode is an implementation of the standard Java security API. This means that permissions are checked against an AccessControlContext which contains ProtectionDomain's. See the Security Architecture for more information.

In JNode the security manager is always on. This ensures that permissions are always checked.
The security manager (or better the AccessController) executes the security policy implemented by JNodePolicy. This policy is an implementation of the standard java.security.Policy class.
This policy contains some static permissions (mainly for access to certain system properties) and handles dynamic (plugin) permissions.

The dynamic permissions are plugin based. Every plugin may request certain permissions. The Policy implementation decides if these permissions are granted to the plugin.

To request permissions for a plugin, add an extension to the plugin-descriptor on connected to the "org.jnode.security.permission" extension-point.
This extension has the following structure:

Multiple permission's can be added to a single extension.

If you need specific permissions, make sure to run that code in a PrivilegedAction. Besides you're own actions, the following standard PrivilegedAction's are available:

Multi-threading

Multithreading in JNode involves the scheduling of multiple java.lang.Thread instances between 1 or more physical processors. (In reality, multiprocessor support is not yet stable). The current implementation uses the yieldpoint scheduling model as described below.

Yieldpoint scheduling

Yieldpoint scheduling means that every thread checks at certain points (called "yieldpoints") in the native code to see if it should let other threads run. The native code compiler adds yieldpoints into the native code stream at the beginning and end of a method, at backward jumps, and at method invocations. The yieldpoint code checks to see if the "yield" flag has been set for the current thread, and if is has, it issues a yield (software-)interrupt. The kernel takes over and schedules a new thread.

The "yield" flag can be set by a timer interrupt, or by the (kernel) software itself, e.g. to perform an explicit yield or in case of locking synchronization methods.

The scheduler invoked by the (native code) kernel is implemented in the VmProcessor class. This class (one instance for every processor) contains a list of threads ready to run, a list of sleeping threads and a current thread. On a reschedule, the current thread is appended to the end of the ready to run thread-list. Then the sleep list is inspected first for threads that should wake-up. These threads are added to the ready to run thread-list. After that the first thread in the ready to run thread-list is removed and used as current thread. The reschedule method returns and the (native code) kernel does the actual thread switching.

The scheduler itself runs in the context of the kernel and should not be interrupted. A special flag is set to prevent yieldpoints in the scheduler methods themselves from triggering reentrant yieldpoint interrupts. The flag is only cleared when the reschedule is complete

Why use yieldpoint scheduling?

JNode uses yield point scheduling to simplify the implementation of the garbage collector and to reduce the space needed to hold GC descriptors.

When the JNode garbage collector runs, it needs to find all "live" object references so that it can work out which objects are not garbage. A bit later, it needs to update any references for objects that have been moved in memory. Most object references live either in other objects in the heap, or in local variables and parameters held on one of the thread stacks. However, when a thread is interrupted, the contents of the hardware registers are saved in a "register save" area, an this may include object references.

The garbage collector is able to find these reference because the native compiler creates descriptors giving the offsets of references. For each class, there is a descriptor giving the offsets of its reference attributes and statics in their respective frames. For each method or constructor, another descriptor gives the corresponding stack frame layout. But we still have to deal with the saved registers.

If we allowed a JNode thread to be interrupted at any point, the native compiler would need to create descriptors all possible saved register sets. In theory, we might need a different descriptor corresponding to every bytecode. By using yield points, we can guarantee that "yields" only occur at a fixed places, thereby reducing the number of descriptors that that need to be kept.

However, the obvious downside of yieldpoints is the performance penalty of repeatedly testing the "yield" flag, especially when executing a tight loop.

Thread priorities

Thread can have different priorities, ranging from Thread.LOW_PRIORITY to Thread.HIGH_PRIORITY. In JNode these priorities are implemented via the ready to run thread-list. This list is (almost) always sorted on priority, which means that the threads with the highest priority comes first.

There is one exception on this rule, which is in the case of busy-waiting in the synchronization system. Claiming access to a monitor (internals) involves a busy-waiting loop with an explicit yield. This yield ignores the thread priority to avoid starvation of lower-priority threads, which will lead to an endless waiting time for the high priority thread.

Classes involved

The following classes are involved in the scheduling system. All of these classes are in the org.jnode.vm package.

• VmProcessor
• VmThread contains the internal (JNode specific) data for a single thread. This class is extended for each specific platform

Native code compilation

All methods are compiled before being executed. At first, the method is "compiled" to a stub that calls the most basic compiler and then invokes the compiled code.

Better compilers are invoked when the VM detects that a method is invoked often. These compilers perform more optimizations.

Object allocation

All new statements used to allocate new objects are forwarded to a HeapManager. This class allocates & initializes the object. The objects are allocated from one of several heaps. Each heap contains objects of various sizes. Allocation is currently as simple as finding the next free space that is large enough to fit all instance variables of the new object and claiming it.

An object is blanked on allocation, so all instance variables are initialized to their default (null) values. Finally the object header is initialized, and the object is returned.

To directly manipulate memory at a given address, a class called Unsafe is used. This class contains native methods to get/set the various java types.

Synchronization

Synchronization involves the implementation of synchronized methods and blocks and the wait, notify, notifyAll method of java.lang.Object.

Both items are implemented using the classes Monitor and MonitorManager.

Lightweight locks

JNode implement a lightweight locking mechanism for synchronized methods and blocks. For this purpose a lockword is added to the header of each object. Depending on the state of the object on which a thread wants to synchronize a different route it taken.

This is in principle how the various states are handled.

1. The object is not locked: the lockword it set to a merge of the id of this thread and a lockcount of '1'.
2. The object is locked by this thread: the lockcount part of the lockword is incremented.
3. The object is locked by another thread: an inflated lock is installed for the object and this thread is added to the waiting list of the inflated lock.

All manipulation of the lockword is performed using atomic instructions prefixed with multiprocessor LOCK flags.


When the lockcount part of the lockword is full, an inflated lock is also installed.


Once an object has an inflated lock installed, this inflated lock will always be used.

Wait, notify

Wait and notify(all) requires that the current thread is owner of the object on which wait/notify are invoked. The wait/notify implementation will install an inflated lock on the object if the object does not already have an inflated lock installed.