Proposal: Changing the command API without rewriting commands.

Submitted by gchii on Wed, 08/13/2008 - 14:49.

Here is a basic proposal for the Command API to enable future improvements to the API without rewriting commands.

The execute() method is something like this:
public void execute( CommandLine cl, InputStream stdin, OutputStream stdout, OutputStream stderr );

This works--for as long as we do not pass a new parameter to execute method.

Let's talk about an equivalent method. Let's say that a command has-a command context. And let's say that a command context has-a command line, a standard input stream, a standard output stream, a standard error stream. The execute() method becomes something like

public void execute( CommandContext ctx );

This works, too--but it is more future-oriented.

Next week, let's say that we want to pass credentials to a command, so that permissions are based upon credentials. We rework the command context interface. We add a credentials property. The execute method becomes something like

public void execute( CommandContext ctx );

But wait! That is the same signature we already have. Commands that do not interact with credentials are unchanged.

The week after next, let's say that we want to pass a collection of attributes to a command. We rework the command context interface, again. We add an environment property. The execute method becomes something like

public void execute( CommandContext ctx );

Yet again, the signature remains the same.

‹ Proposal: spawn() Question: Running a command in separate thread ›

OutputStream vs. PrintStream vs. Writer vs. Tty

Submitted by gchii on Fri, 08/22/2008 - 03:01.

For this discussion, let's say that a CommandContext interface is passed instead of stdin, stdout, stderr in the execute() method.

Yes, it supports a raw binary output stream (tar, gz, md5sum).

  public OutputStream getOutputStream();

And yes, it supports an optional PrintStream (connected to the underlying output stream).

  public PrintStream getPrintStream();

And yes, it supports an optional Writer (connect to the underlying output stream).

  public Writer getWriter();

And yes, it supports an optional tty-style console (connected to the underlying output stream.

  public Tty getTty();

If there is no tty, the getTty() method has nothing to return.

And we go on supporting whatever anyone wants connected to their standard output. Why? Because custom implementations of CommandContext are possible.

When we close() a command context, we close all of its resources.

  public void close();

There are strong arguments for keeping the same execute method.

  public void execute( CommandLine, InputStream stdin, PrintWriter stdout, PrintWriter stderr );

And yet, it is not possible to implement certain command with this interface. Some commands require OutputStream stdout. Some commands require PrintWriter stdout. Some commands require Writer stdout. Some commands require Tty stdout.

This suggests a new StandardOutput interface (or a CommandContext).

public interface StandardOutput {
  public OutputStream getOutputStream();
  public PrintWriter getPrintWriter();
  public Writer getWriter();
  public Tty getTty();
  public void close();
}

public interface Command {
  public void execute( CommandLine cl, InputStream stdin, StandardOutput stdout, PrintWriter stderr );
}

wget

Submitted by gchii on Fri, 08/22/2008 - 03:24.

The wget command is popular. It downloads a file, for example, from an HTTP service.

Let's say that we get a zip file.

wget http://jnode.org/example.zip

Normally, it opens a file (java.io.FileOutputStream).

Is it possible to redirect the stdout and stderr of wget to the same file?

wget http://jnode.org/example.zip > ggg.txt 2&>1

It is possible with the Bjorne interpreter

Submitted by Stephen Crawley on Fri, 08/22/2008 - 06:18.

Or at least it should be. However the RedirectingInterpreter that the shell uses by default soes not support this.

I guess I need to get back to finishing off the Bjorne interpreter.

ls|grep

Submitted by gchii on Fri, 08/22/2008 - 03:16.

Let's say that we really want the ls command to behave like ls -1 when its output stream is redirected.

  public void execute(
      CommandLine cl,
      StandardInput stdin,
      StandardOutput stdout,
      StandardOutput stderr ) {
    if ( stdout.getTty() == null ) {
    :
    }
    else {
    :
    }
  }

Like output, like input

Submitted by gchii on Fri, 08/22/2008 - 03:07.

And if we are going to have a StandardOutput, we might try a StandardInput.

public interface StandardInput {
  public InputStream getInputStream();
  public DataInputStream getDataInputStream();
  public Reader getReader();
  public Tty getTty();
}

public interface Command {
  public void execute(
      CommandLine cl,
      StandardInput stdin,
      StandardOutput stdout,
      StandardOutput stderr
      );
}

Worthy to keep in mind

Submitted by lsantha on Wed, 08/13/2008 - 20:49.

This idea might make sense on the next occasion when due to exact demand we will have to change the signature of execute().

The time may be soon

Submitted by Stephen Crawley on Thu, 08/14/2008 - 03:40.

I think the time to make other changes to the execute method is not far away. For example, we need to address the issue of making the console subsystem UTF-8 (etc) capable. Among other things, that means changing the Command API to allow the input/output/error streams to implement Reader/Writer rather than InputStream and (yuk!) PrintStream.

Yes, changing this API will be a bit of a pain, but the code of the various JNode shell commands is much more consistent now, so the pain shouldn't be to great. Trust me, I've seen it all Smiling

Reader/Writer

Submitted by gchii on Thu, 08/14/2008 - 12:49.

A command and command context is intended for octets, regardless of their interpretation. It does not presume characters.

Let's say that we have an md5sum command and it reads from standard input. It reads its standard input as a binary stream of octets. It does not use reader/writer, does it?

md5sum < jnode.iso

Let's say that we have a gzip command for compression and it reads from standard input, writes to standard output. It too reads/writes as a binary stream.

tar czf myarchive.tar.gz .

becomes

tar . | gzip > myarchive.tar.gz

(Please forgive me if it isn't the exact Linux syntax.)

Therefore, an abstract command class should be designed for character commands. It should connect a reader to its standard input and writer to its standard output.

Instead of AbstractCommand, we might provide AbstractUtf8Command and AbstractBinaryCommand.

Command interface

Submitted by gchii on Wed, 08/13/2008 - 14:54. 
public interface Command {
  public void execute( CommandContext ctx );
}

Alternatively ...

Submitted by Stephen Crawley on Thu, 08/14/2008 - 03:26.

Alternatively, make the methods in your CommandContext interface part of the Command interface and implement them in AbstractCommand.

Explanation for possible implementations

Submitted by gchii on Thu, 08/14/2008 - 13:31.

Please let me explain why a command has-a command context, instead of command is-a command context.

First, this preserves an important feature of the current execute() method. The entire command context is passed to a command as parameters. We continue to use the default constructor to instantiate a command. We continue to not invoke any other method(s); a command object has a very simple life cycle. We continue to invoke the execute() method; we are passing the context as a single parameter instead of separate parameters.

Second, there is an issue of command object reuse. Should a command object be reusable, constructed once and invoked many times? (Should a command become something like a servlet?) There are two options. (1) A command interface provides these methods, requiring a new command object to be instantiated every time the command is invoked. (2) A command context interface provides these methods, enabling a command object to be optionally reused.

When a command context is passed to a command as parameters, a new command object is created once and only once. The execute method is invoked multiple times.

Should a shell create a new command object every time? Or, should it cache and reuse command objects? This is a shell implementation issue, not an API issue.

When we are writing a command class, is it safe to assume that a new instance of the class is created by the shell every time? If so, the API should expose this assumption and insist on an interface something like this:

public interface Command {
  public abstract CommandContext getCommandContext();
  public abstract void setCommandContext( CommandContext ctx );
  public void execute();
}

But this blocks a shell from implementing a cache of reusable command objects.

Thanks,

Some facts about the current Command APIs

Submitted by Stephen Crawley on Thu, 08/14/2008 - 22:17.

Things have changed a lot in the past 6 months.

  1. All command classes in the JNode codebase are subclasses of AbstractCommand. This class provides some infrastructure methods that are required to make the new Syntax mechanisms work, and to supporta simple implementation of the legacy 'main' entry point.
  2. The Syntax mechanism works by binding argument values to Argument objects created and "registered" by the command class constructor. This means that a command class instance cannot be reused. (In fact this is a good thing. The old approach using statics was not reentrant ... you could not safely launch a command two or more times in parallel.)

I think these two points somewhat invalidate your reasons for using a separate CommandContext object. This is not to say that the idea won't work. But I think it reduces to a minor style debate, and that the "implementors choice" principal applies.

Puzzled with Command API.

Submitted by gchii on Tue, 08/19/2008 - 14:21.

Please help me to understand this. I am putting a lot of effort into comprehending the Command API for 0.2.7 so that I might re-implement existing classes as JNode command classes.

1. Are you saying that, starting with 0.2.7, a command class MUST extend AbstractCommand? that it is not enough to simply implement the Command interface?

2. You have explained that a command instance CANNOT be reused. I'm sorry that I did not understand this important feature from the JavaDoc documentation.

Clearly, I do not understand. When a command class CANNOT be reused, it has a very specific life-cycle.

1. A shell MUST create an instance of a command class.
2. A shell MUST set the command line arguments.
3. A shell MUST open standard input, output and error streams.
4. A shell MUST invoke execute(...).
5. A shell MUST close standard input, output and error streams.
6. A command is abandoned to the garbage collector.

Is this a more accurate description of the most recent Command API?

The lifecycle of a command.

Submitted by Stephen Crawley on Wed, 08/20/2008 - 10:33.

There is more info on some aspects of the shell / command / syntax mechanisms here:

http://www.jnode.org/node/2546

In general though, your best bet is to read the code. (This is true for most of the JNode codebase.) A decent IDE (such as Eclipse) is a lot of help in figuring things out.

Anyway ,,, here is a rough sketch of the lifecycle of a command, assuming that we are using the ThreadInvoker or ProcletInvoker,

  1. The command shell grabs the next line from the user (or wherever).
  2. The current interpreter parses the line into separate commands, argument tokens are extracted and redirections and pipelines are analysed. The result is one or mode CommandLine objects with their streams.
  3. Assuming we have just one CommandObject, the interpreter calls the shells 'invoke' method passing the CommandLine object. The CommandShell dispatches to the current CommandInvoker.
  4. The invoker looks up the command alias named by the CommandLine to give a java class name and loads the class.
  5. The invoker examines the loaded class. If it does not implement Command, the invoker looks for a "static void main(String[])" entry point and invokes it. All done.
  6. If the command class does implement Command, we create an instance of the class.
  7. It the class is a subtype of AbstractCommand and its constructor registers Arguments to create an ArgumentBundle, the "new" syntax mechanisms are used to parse the CommandLine argument tokens against a new-style Syntax and bind values to the Arguments in the bundle. Then command.execute is called passing the CommandLine (which is typically ignored) and the in/out/err streams.
  8. If the command class does not extend AbstractCommand or no Arguments were registered by the constructor, the execute method is called immediately passing the CommandLine and the in/out/err streams. This will typically use the "old" syntax mechanisms by parsing the CommandLine against the command's static "Help.Info" object, binding values to its Argument nodes.
  9. When everything is done, the interpreter closes the streams if this is required and the command class instance (if any) is dropped on the floor for the GC.

So, the real reason that AbstractCommand instances are not reusable is that they contain state about the command arguments (in the ArgumentBundle), and potentially other state created during command execution. If we tried to clear the state and recycle command objects, we'd have the problem of command classes that behave unpredictably because they don't reset their state properly. Not to mention garbage retention and potential security issues.

It is generally a bad idea to try and recyle objects in Java. Assuming you have a good GC it is (usually) more efficient to drop old objects and create new ones as required. And if you have a poor GC, it is better to fix the GC than to "optimize" applications for the poor GC. (Not least because optimizing for the poor GC will de-optimize for a good one!!)

CommandContext interface

Submitted by gchii on Wed, 08/13/2008 - 14:55. 
public interface CommandContext {
  public abstract CommandLine getCommandLine();
  public abstract void setCommandLine( CommandLine v );
  public abstract InputStream getStandardInput();
  public abstract void setStandardInput( InputStream v );
  public abstract OutputStream getStandardOutput();
  public abstract void setStandardOutput( OutputStream v );
  public abstract OutputStream getStandardError();
  public abstract void setStandardError( OutputStream v );
}

Are you sure the commands

Submitted by lsantha on Wed, 08/13/2008 - 20:51.

Are you sure the commands shoud be able to access the setters?

Why setters?

Submitted by gchii on Thu, 08/14/2008 - 13:04.

A command context is used by a command and a shell.

A shell must be able to set the properties of a command context. Ideally, it should use the command context interface--rather than the constructor of an implementation of a command context interface. In other words, a shell is a command and it too has-a command context. It creates a command context from its own context.

A command can replace its own output stream. Are there any additional security concerns? Already, a command must have permission to open a stream. A command can be invoked. It can also invoke another command. The called command gets a clone of the calling context.

For example, let's say that I really want to use my own custom shell. At the command prompt, I type 'myshell'. The next thing I see is the prompt from myshell. The command context of myshell is a copy of the command context from the default shell.

And at the myshell prompt, I might type 'shell' to start a default shell.

Now, let's say I type 'myshell < myscript'. An input stream in the command context is no longer attached to a keyboard. The myshell command does not need to know and does not need to care that bits are coming from a file called myscript.

An API provides all the possible implementations. If just one command needs a setter, it can be part of the interface. Where else would it go?

Thanks,

Probably they mostly shouldn't ...

Submitted by Stephen Crawley on Wed, 08/13/2008 - 23:03.

... but there may be cases where they do. So it is not a bad idea to provide the APIs and (of course) protect them using Java security / permissions. (OTOH, it would be simpler to delay the provision of the setters until the actual need arises ....)