The new command line syntax mechanism

Submitted by Bluebit on Wed, 04/16/2008 - 06:52.

It is very good to see progress in the shell part of JNode. However I have a question. Why are the arguments converted to an array of strings before it is given to the shell command? It seems to me that information is "lost" in this conversion.

Kind regards
Jacob Kofod

‹ Proposal: Mechanism for command plug-in. KeyboardListener - InputFocus ›

Good question

Submitted by Stephen Crawley on Wed, 04/16/2008 - 09:28.

First a minor correction: the arguments are initially converted to CommandLine.Token objects that have a bit more information. (This is not fully implemented, but I intend Tokens to record more information so that the completer can complete in a style that matches what the user typed; e.g. preserving quoting style.) The String versions of the arguments are only used when necessary; e.g. when calling a "main" entry point, which expects a String[].

But I think you are really asking why we don't feed the raw command line to the command line syntax system. The reason for that is that before we feed arguments to the command and/or do command syntax processing we need to deal with shell-level constructs. With the "redirecting" interpreter, this consists of dealing with escapes and quoting, and handling '<', '>' and '|' constructs. With the "bjorne" interpreter there is a whole lot more that needs to be dealt with.

Note that this stuff needs to be dealt with before we start doing any command-specific argument parsing. Indeed, this processing can even determine which command should be executed.

If the command argument handling code had to deal with the original command line text, the syntax code would get very complicated, and with little extra benefit ... as far as I can see.

Is there another entry point?

Submitted by Bluebit on Wed, 04/16/2008 - 14:25.

You write:

"The String versions of the arguments are only used when necessary; e.g. when calling a "main" entry point, which expects a String[]."

Isn't the "main" entry point called everytime the command is executed? Or is there another entry point that can be used?

Kind regards
Jacob Kofod

The 'execute' entry point is JNode's preferred one

Submitted by Stephen Crawley on Wed, 04/16/2008 - 14:45.

The command shell (actually the invokers) try to use a command class'es 'execute' entry point if it exists. The execute method takes 4 arguments: a CommandLine object and 3 Streams objects. The CommandLine API allows a program to get hold of the command name and arguments as Tokens. But most JNode command classes delegate command line parsing to either the old or new syntax mechanism, so they don't access the tokens at all.

I need to write another Book page to describe all of this.

Could we simply mark it as deprecated

Submitted by Bluebit on Wed, 04/16/2008 - 14:55.

Okay. Smiling It confused me that there are two possible entry points. Are there any reason why the "main" entry point is there? Or could we simply mark it as deprecated?

Kind regards
Jacob Kofod

Running JNode commands on classic Java

Submitted by Stephen Crawley on Thu, 04/17/2008 - 03:17.

It would be nice to be able to run JNode applications on classic Java. (I'm not talking about trivial stuff like "echo" and "dir" ... but multi-K line programs). In order to do this, the application needs to provide both 'execute' and 'main' entry points.

By implementing 'main' as "new XxxCommand().execute(args);", we hopefully will be able to support both requirements. This will require us to implement a stand-alone version of the key JNode classes that will make command syntaxes work on a classic JVM. For example:

  • We'd need a bit of infrastructure for loading syntax from the command class's JAR file.
  • The stand-alone version of the "AbstractClass.execute(String[])" method would need to do the syntax parsing itself, rather than relying on a (non-existent) JNode invoker having already done the work.
  • Etcetera.

I haven't thought everything through, but I cannot think of any major impediments at this stage.

Middle ground

Submitted by Daniel Noll on Thu, 08/14/2008 - 02:42.

Can you not have something like appletviewer where you run...

java org.jnode.util.CommandRunner com.example.commands.FooCommand [args]

I think we are saying the same thing.

Submitted by Stephen Crawley on Thu, 08/14/2008 - 03:21.

Your proposed CommandRunner could be implemented using most of the org.jnode.shell.syntax.* classes, and replacements for the org.jnode.shell.* classes that are responsible for making syntax loading, argument parsing and command invocation happen.

(Of course, a few JNode Argument classes will make no sense outside of JNode; e.g. DeviceArgument. And some JNode commands won't be usable at all or will require major surgery to wok on a class JVM; e.g. commands that talk to JNode services or use JNode specific file system extensions.)

OK ... so this would be "nice to have". But does anyone really need this functionality now / soon? If they do, maybe I should start working on it instead of some of the other things I was planning to do.

A classic JVM architecture

Submitted by Bluebit on Sat, 04/19/2008 - 09:44.

I can clearly see the benefit of running JNode applications on a classic JVM. But it should also be easy for the JNode programmer to write programs. I think it would be a better strategy to make the shell portable. I that way scripts would also run without problems. However there could be problems if the scripts try to access JNode specific objects.

If we should go further out of that road, we could make a classic JVM architecture, at the same level as X86. In that way everything would run everywhere where the Java Runtime is installed and every bindings the program has to JNode specific objects would also work. That would also be good for testing.

Kind regards
Jacob Kofod

You are touching on a number of different issues

Submitted by Stephen Crawley on Sat, 04/19/2008 - 11:28.

We have to recognize that it won't always be possible to run a JNode applications on classic Java (JVM + standard libraries). The main issue will be dependencies on "org.jnode.**" in the JNode code. In some cases it is reasonable to provide a compatibility library, as I talked about for the syntax stuff. In other cases, we might have to say that the app is too JNode specific.

There is a clear tension here between doing new things in JNode and the need for compatibility with "standard" Java. There are a couple of things that we can do about this:

  • Make the JNode implementations of standard features behave as much like the classic Java versions as we can.
  • Try to avoid creating new application-facing interfaces in JNode. For example, if we wanted to build complex apps to run in the console text screen, we could create new JNode-specific interfaces for moving the cursor, smart repainting, colors, etc. But a more portable solution would be to enhance the console driver to emulate an ANSI terminal, and have the application do the fancy stuff with ESC sequences.
  • When writing applications that should be portable, write them with minimal dependencies on JNode specific APIs. Where JNode dependencies are unavoidable, try to structure the app to have a portability layer that can be replaced when porting to some other platform.

Your idea of a classic Java environment is an interesting one, but I'm not sure if it would be practical.

Another point: many classic Java applications are actually system dependent. It is not uncommon for a Java app to use java.lang.Process to run OS specific applications and utilities, typically coded in a different language. (I just raises a 'bug' against the JNode implementation of java.lang.Process. It is a bit of a tarpit, IMO.)

And this brings me to scripting. If we want scripting and scripts to be portable we need two things. First, the scripting language itself needs to have the same syntax and semantics on JNode and on the target platform. (For example, I'm trying to make "bjorne" have the same semantics as Unix / Linux "sh" aka the Bourne shell.) Second, the commands run from the script need to have the same syntax and behavior as the commands on the target platform. IMO, this is a harder task.

Why isn't it practical?

Submitted by Bluebit on Sat, 04/19/2008 - 20:27.

I see a JNode architecture buildt on a classic Java environment as the ultimate compatibilitylayer. I think this could be done by stripping out everything that has to do with assembler. java.lang.Object in JNode should be mapped to the host OS version of java.lang.Object etc.
Then there should be made some JNode drivers that fx. maps the JNode filesystem to the host OS filesystem. This could be done in two ways.

1) A JNode harddrive driver that maps to a host OS file. This is the same approche that fx. VMware uses.

2) A JNode filesystem that maps to the host OS filesystem. This option should be used if any real work should be done on the host OS.

Why do think that it isn't practical?

Kind regards
Jacob Kofod

Too much work

Submitted by Stephen Crawley on Sun, 04/20/2008 - 01:35.

It is not practical because it is too much work. And there are not enough people working on JNode. Too many people with ideas for someone else to implement.

Which brings me to an important question: what are YOU going to do as your first JNode project?

Another thing ...

Submitted by Stephen Crawley on Thu, 04/17/2008 - 03:19.

We still have the DefaultInvoker which only understands 'main' entry points. Maybe we should make it go away, but I'm not convinced we are ready to do that, just yet.

legacy

Submitted by Peter on Wed, 04/16/2008 - 15:46.

Do not forget normal applications. "main" is the way you start a Java application Smiling

Why has the echo command both?

Submitted by Bluebit on Wed, 04/16/2008 - 19:12.

Okay. I can see that the "main" entry point always should be a posiblity. However if a class uses the "main" entry point it shouldn't be treated like a shell command. But the echo command has for example both a "main" and a "execute" entry point?

Kind regards
Jacob Kofod

See my other comments ...

Submitted by Stephen Crawley on Thu, 04/17/2008 - 03:24.

They give two reasons why having both 'execute' and 'main' is a good thing.

But as you can see from the 'echo' example (and my other comments), the 'main' entry point can be very light-weight.

I don't get it

Submitted by Bluebit on Thu, 04/17/2008 - 06:33.

The echo command will only be executed on JNode. It don't make sense to execute it in another enviroment. Therefore the main entry point will never be used and could be removed?

Kind regards
Jacob Kofod

Let me spell it out ...

Submitted by Stephen Crawley on Thu, 04/17/2008 - 09:54.

Yes. Echo is a command that you wouldn't want to execute outside of JNode. However ...

  1. If you removed the 'main' entry point from EchoCommand, it would not run under the DefaultInvoker. Try it out if you don't believe me! Comment out the main method, run "set jnode.invoker default", then try to use echo.
  2. The 'main' method is tiny. It just constructs an object and calls a one parameter method on it. I cannot see the harm in just leaving it there for now. Especially since it is useful ... see point 1.

Maybe when DefaultInvoker is gone and forgotten we can go around pruning the 'main' methods on commands that make no sense outside of JNode. But frankly, I've got far more important things to do right now.

Thanks :-)

Submitted by Bluebit on Thu, 04/17/2008 - 19:49.

That leads me to a new question. Why do we have more than one Invoker? Why not just have one Invoker that works like this:

The invoker could just look at the class that should be invoked, if it implements the Command inferface then it should be treated as a shell command and the "execute" method should be called, otherwise it should just be invoked as a normal Java program using the "main" entry point.

Kind regards
Jacob Kofod

They run commands differently

Submitted by Stephen Crawley on Thu, 04/17/2008 - 22:27.

The invokers run commands differently:

  • DefaultInvoker simply calls the 'main' method in the CommandShell's main thread.
  • ThreadInvoker runs commands in their own threads, but has issues when used with classic Java applications. (In brief, you can't implement shell-level stream redirection / pipelines if applications are accessing their respective streams via the System.{in,out,err} globals.)
  • ProcletInvoker runs each command as a separate "proclet". This is a mechanism that makes System.in,out,err go to different places in different proclets. A proclet equates to a group of threads belonging to a designated thread group. Proclets are not perfect, but they are good enough that most applications will 'just work' when they are pipelined, etc. For more info, search the JNode website for the term 'proclet'.
  • IsolateInvoker (not checked in yet) will run commands in their own isolates.

The user-visible differences between the invokers are all about how commands interact with the environment and each other, and whether not they can be killed, stopped or put into the background. (Actually, killing and stopping will be problematic until isolates are fully implemented. The Thread.kill,stop etc methods are deprecated by Sun and inherently unreliable. The JNode versions tend to crash the JVM.)

Originally, the invokers were responsible for dealing with shell syntax as well. One of the first things I did in the JNode project was to separate the shell's 'interpreting' and 'invoking' behaviors into separate classes.