New syntax mechanism - tracking issue

Project:JNode Core
Component:Code
Category:task
Priority:normal
Assigned:Stephen Crawley
Status:closed
Description

I have just committed the initial release of the new command syntax mechanisms. I've thoroughly tested that the old mechanisms (both old-style command parsing and completion) are working properly. However, if there are any show-stopper problems, please feel free to revert revision 3844 which contains all of the changes.

I would encourage people to do a 'svn up' and take a look at the new code. The guts of the new syntax mechanism is in the new org.jnode.shell.syntax package, and I've written a suite of unit tests.

So far, I have just converted one existing command: SetCommand. If you look at it, you will see that the old "static Help.Info help" is no longer there. Indeed, the command class no longer defines the command-line syntax at all. Instead, it declares a bunch of XxxArgument objects that will act as holders for argument values, and "registers" them in the constructor.

The command-line syntax is specified in XML in a new plugin extension point; see the org.jnode.shell.command.xml plugin descriptor for the syntax for the set command. Note that the syntax is associated with the alias name, rather than the command class. The syntax information is read from the plugin extension by the DefaultSyntaxManager class.

If you delve into the code, you will see that the XxxSyntax classes that have a 1-to-1 correspondence with the syntax elements in the plugin descriptor. The MuXxxxx classes represent the lower level BNF-like syntax that is used for parsing. The XxxSyntax classes "prepare" themselves into MuXxxxx classes, and the MuParser class parses the CommandLine against the MuSyntax, binding values to the XxxArgument objects, and performing full backtracking if required. The layering allows for a rich repertoire of Syntax constructs while keeping the Mu-level parsing code fairly simple.

Tasks that remain to be done include:

  • Extending the MuParser, etc to support command completion.
  • Converting more command classes, and define the corresponding syntax descriptors in the relevant plugin files.
  • Implementing more XxxArgument classes.
  • Writing JNode "book" pages ...

I would really appreciate any constructive feedback that will allow me to improve on what I've done so far.

#1

Submitted by Stephen Crawley on Fri, 03/21/2008 - 09:00.

Revision #3859 is an overhaul of the completion mechanisms. The old version of the completion code could only cope with completion alternatives coming from one Syntax. If multiple Syntaxes matched the partial command-line, one was chosen heuristically. The new version of the code gathers the completions for all matching syntaxes and offers them to the user. It also fixes a couple of minor regressions.

In order to do this, it was necessary to change the CompletionInfo data structure and the interactions between the various parties; the console, command shell, interpreter, command line, syntax and argument. Instead of returning the completed command line, the 'complete' methods take a CompletionInfo as an argument, and add any completions to it as they are found. The whole lot get returned to the console layer which edits the completion into the command line.

The end result is that the code is a lot easier to understand, and we don't have that nasty callback into the command shell to deal with the case where one argument has multiple completions.

I do not believe I've introduced any new bugs / regressions. But if I have, there is just one checkin to revert.

Next I'm going to look at / fix the completion bugs I've noticed on the way.

#2

Submitted by Stephen Crawley on Sat, 03/29/2008 - 12:08.

Since the last update, I have fixed some old completion bugs in the "redirecting" interpreter, improved completion for new-style syntaxes, and fixed a regression in the behavior of the "thread" and "proclet" invokers when the user provided command arguments that don't match an (old-style) syntax.

Next I will be tracking down a problem I noticed in the "default" invoker and changing the shell's default invoker to "proclet" so that secondary consoles behave better by default.

#3

Submitted by Stephen Crawley on Mon, 03/31/2008 - 14:31.

After a lot more testing, I've change the initial invoker for the command shell to be ProcletCommandInvoker. This should fixing some of the strange behavior of the output streams when you use a secondary console; e.g. using console -n, leed or GUI-based console.

The revision for this change is #3919. As always, if the change is problematical, please roll back to the previous revision.

#4

Submitted by Stephen Crawley on Sun, 04/06/2008 - 07:12.

Here's a summary of recent changes, in no particular order:

  • Fixed a problem in the way that the proclet mechanism was being initialized by the ProcletCommandInvoker. In some circumstances, the "global" in/out/err streams were not being initialized. This meant that non-proclet threads (like the compiler thread) had broken streams that would NPE if used.
  • Fixed a completion bug that arose in fairly complicated syntax.
  • Improved the behavior when a command with new-style arguments didn't have a defined syntax. Command parsing, completion and Help now all use a default syntax that is generated on-the-fly from the ArgumentBundle. The result is functional, though not pretty.
  • Fixed the ArgumentBundle.createDefaultSyntax method to work in the cases of zero and one Argument in the bundle.
  • Extended IntegerArgument to do completion in the case where the allowed range of values is small (less than 100 at the moment).
  • Added another unit test ... for completion using default syntaxes.
  • Internal API reorganization to make it easier to implement an isolate invoker.
  • Patched long-standing issue with the way that the async invokers run Command classes via their execute entry-point; created another issue to address this properly.

#5

Submitted by Stephen Crawley on Thu, 04/10/2008 - 14:44.

I've created a new "syntax" command for dumping and loading new-style syntax specs as XML. So it is now possible to develop and test new syntaxes while JNode is running. You can also override the syntax specs defined in the JNode plugin descriptors ... for the current shell.

I'll write a Book page for the command tomorrow. Until then, "help syntax" should give you the basics.

#6

Submitted by Stephen Crawley on Sun, 04/13/2008 - 05:06.

The Book pages are starting to take shape. Let me know what you think.

#7

Submitted by Stephen Crawley on Sun, 04/27/2008 - 11:45.

Latest bulletin:

  • I'm making steady progress on converting commands and Argument types. If anyone else would like to leap in and work on some others, that would really help.
  • I've added an OptionalSyntax class for a common idiom.
  • I've fixed a bug in IntegerArgument completion that would exhaust the heap and kill JNode. (Ouch!)
  • I've just fixed a bug where the ProcletCommandInvoker "leaked" ProcletContexts. (The leaked ProcletContexts showed up when you ran the "thread" command.)

#8

Submitted by Stephen Crawley on Sat, 05/24/2008 - 05:20.

Progress:

  • More commands converted. I've done all of the Shell and Net commands and I'm working on FS
  • Fixed OptionSetSyntax (<optionSet>) to make it "eager". This makes (for example) "cp --force file dir" work as expected. (The old version of OptionSetSyntax parsed "--force" as a filename !!)

Work to be done:

  • Finish off the remaining commands.
  • Generate better error messages for command syntax errors; i.e. better than the current "Command arguments don't match the syntax".
  • Fix the bug in FileArgument completion.
  • My idiom of defining argument objects as "private final" with an upper-case name is a violation of the Standard Java Style Guidelines. (Upper-case is reserved for static attributes, but these ones are non-static ... and they need to be.)

#1

Submitted by Stephen Crawley on Mon, 06/09/2008 - 03:23.

Progress:

  • Fixed the bugs which caused completion to behave incorrectly for pathname arguments and pathnames after '<' and '>'.
  • More commands converted to use new syntax.

#2

Submitted by Stephen Crawley on Thu, 08/21/2008 - 06:30.
Status:active» fixed

There are no immediate plans to do further development. So I'm marking this as fixed. Feel free to open new Feature Requests and Bugs.

#3

Submitted by Anonymous (not verified) on Thu, 09/04/2008 - 08:00.
Status:fixed» closed

Automatically closed -- issue fixed for two weeks with no activity.