Is FileArgument pathname expansion a bad idea?
Submitted by Stephen Crawley on Fri, 11/16/2007 - 05:16.
(After posting this as a blog entry, I realised I should have made it a forum topic ...)
In traditional operating systems, there are two approaches to implementing pathname patterns, or wildcards as they are often called. The UNIX approach is to have the command shell expand the patterns and pass the resulting pathnames to the command. Thus "cat *.java" typically leads to the "cat" command being run with multiple file names as separated arguments. The "cat" command does not get to see the original pattern.
The DOS / VMS approach is for the patterns to be passed intact to the command. It is then up to the command to treat an argument containing wildcard characters literally, expand it into a list of pathnames, or interpret it some other way. (In the case of DCL, there is a command for expanding wildcards that allows you to process them in a script.)
The advantages of the UNIX approach is uniformity and simplicity. Wildcards are always treated the same in the context of a given shell. Furthermore, an application command does not need to take account of patterns.
The advantage of the DOS / VMS approach is that it is not a one-size-fits all solution. Command arguments that do not represent pathnames need not be expanded. Furthermore there is room to implement command syntaxes like the DCL "copy *.foo *.bar" syntax. This means copy all files with the extension "foo" to a corresponding file with the same base name and the extension "bar". (Incidentally, one could implement this as a UNIX command, with the caveat that the user would need to quote the arguments; e.g. "dclcopy '*.foo' '*.bar'".)
So that's the background.
In JNode, the builtin interpreter understands argument quoting and escapes, as well as redirection and pipeline syntax. But it does not perform pathname expansion. This currently happens when the application code calls "getFiles()" on a FileArgument that has been populated by the Syntax argument parser. This present a number of problems:
- If an application's Syntax doesn't define an argument as a FileArgument, or if it uses "getValues()" rather than "getFiles()", it will see the unexpanded pattern. (OK that is arguable an application bug. But it is pretty common right now. For example, neither "dir" or "cat" get this right.)
- Classic java applications which do their own argument parsing independent of Syntax and friends will no be prepared to do pathname expansion. Which means that they will behave differently from "native" JNode applications.
- There is a problem if a user needs to include a literal '*' or '?' character in a pathname. By the time the argument string reaches the pathname expansion code, the interpreter will have stripped quotes and '\' escapes. So, for "xyz '*.java'" the application would see arguments "*.java" and would dutifully expand the pattern. Double escapes or escaped quotes (e.g. "xyz \\*.java" or "xyz \'*.java\'") would probably work, but this approach is ugly ... and interpreter specific: see below.
- If we want to support a UNIX style shell, having the application perform / control pathname expansion is potentially problematic. For example, if the "foo" application did pathname expansion, then invoking "foo \*.java" from the bjorne shell (with globbing enabled) would run "foo" with one argument. But the foo command would then expand the argument into multiple pathnames ... which is not what you would expect. The user would need to use double escapes to stop this happening. Ugly.
IMO, a good summary of the above is "a horrible mess".
I've thought of two ways to address this. The easy way is to move pattern expansion out of FileArgument.getFiles() and put into the "redirecting" command interpreter ... where it belongs (IMO).
The other alternative is to try to make FileArgument (and similar) aware of how the original arguments were formed by the interpreter; i.e. whether pathname expansion would have been performed, which if any characters in the argument were quoted / escaped. This will be complicated, and it is not immediately obvious that it can be made to work without user visible anomalies.
Commants? Ideas?
- Login to post comments
Some thought
Hi Stephen,
perhaps I have some usefull comments for this "horrible mess"
I'm not sure in the end how to do it but I don't think we should move the expansion into the interpreter or at least not completely. First of all I like the principles of the Argument handling and I think the FileArgument fits very well into it. At the moment it is very basic, but just to give an impression what could possibly come next: jtar will get all tar files, when you call "jtar *" (no .java files,...). Further we could do advanced pathlevel handling. In Linux I know about "/**/*.foo", might be my ignorance, but I don't know a way of doing recursive handling (i.e. the expression wouldn't return /a/b/c.foo).
The second reason is that with FileArgument handling the Application can decide itself if it wants the expended or non-expanded version of the argument (and for sure we should fix any app that uses it incorrect).
I'm not sure about the details but at the moment we allready differentiate between Command.execute and App.main. So could it be a solution to do the expansion for "normal apps" and the "normal" Argument handling for JNode commands?
About your dclcopy: If it is a JNode Command, for that command you would probably use .getValue() instead. So the user doesn't need to escape *.foo in order to get what he expects.
Re: your thoughts ... continued
Re: "jtar *" matching only tar files.
I suppose one could do that, but I don't think that it would improve usability. On the contrary, the new user experience would be like that of someone trying to learn a new "natural" language like English; lots of irregular verbs to learn and a bazillion strange idioms.
Re: "recursive" patterns in UNIX.
UNIX has historically taken the view that wildcards should not match a "/". Instead, you typically use a utility like "find" if you want perform some action on all files in a tree. For example, "rm `find . -name \*.java`" or "find . -name \*.java | xargs rm" will remove all ".java" files in a tree. (The first version fails if there are too many files ... the second does not.)
But actually, this is NOT an argument for putting file expansion into FileArgument. If we decided to implement a richer (or just different) wildcard "language", we could do this in the shell / interpreter. Indeed it is non unknown for (UNIX) shells to have different wildcard languages. For example, the classic "sh" did not support "[...]", "{,..}" or "~". These expansions were first introduced in "csh".
However, I would suggest that a wildcard syntax that supports recursion is going to be hard to understand and error prone for the average user. The UNIX "find" approach seems to me to be a better idea.
Re: apps "deciding" whether to use the expanded or non-expanded version.
Re: differentiating between Command.execute and Command.main.
Actually, the three existing interpreters are currently NOT aware of the different entry points. The resolution of class entry points occurs in the invokers. The three existing invokers try to hide the differences between the two entry points. Unfortunately, the fact that System.* streams are global means that stream redirection cannot work for some invokers using the Command.main entry point.
Another point is that the Command.main entry point for many JNode commands is just a wrapper for the Command.execute entry point. Thus it is incorrect to assume that the Command.main entry point will not use FileArgument.
Re: "dclcopy".
I think you missed my point. My point was that "dclcopy" is in fact implementable in the context of a UNIX shell. A better example would be the (real) UNIX "find" and "grep" commands where it is common practice to escape wildcard (and other) characters to avoid premature expansion by the shell. This is no problem for an experienced UNIX user.
Re: your thoughts
Taking your points one at a time:
I agree that the Info/Syntax/Parameter/Argument classes are good in principle. Specifically, I see the following really good points:
However there are a number of important issues:
Next point:
While FileArgument fits into the Syntax/Parameter/Argument model, when you add wildcards, this starts to break down. In particular:
By contrast, a UNIX shell does wildcard (and other) expansions the same way, irrespective of the command. This is much easier for users to understand.
To be continued ...
Couldn't the interpreter
Couldn't the interpreter determine wether the command expects expanded or non-expanded files? That way I can make a version of cat, dir or whatever that's simple, and later extend it to also make use of more advanced features. This obviously has an analogy to calling functions. If the interpreter can't find a version of the command that accepts unexpanded pathnames, it could expand it for him and call the function on each pathname.
The problem is ...
... that the POSIX shell spec requires that wildcard expansion is done BEFORE the shell decides which command to execute. It usually doesn't make any difference, but it is not difficult to come up with examples where it does.
Well, as long as it is
Well, as long as it is _our_ version of cat, dir or whatever we can use the FileArgument anyway. So the command can decide itself if it wants the pattern (.getValue()) or the expanded files (.getFiles()).
We also have an easy way to determine if a command is a JNode command (it extends Command interface) or a standard application (we need to call main(..)).
The problem is in normal java apps. Should an app get the expanded file list or see the pattern? What if "foo" expects a pattern but "bar" a list of files? ...
I don't have an answer to these questions and I see that it isn't that easy to solve, but what I know, and what I wanted to express in my first post is, that I like the principles of our Syntax handling and our FileArgument and that I wouldn't drop it.
BTW. I just remembered a discussion with levente, perhaps he wants to comment on that too. Afair we said, that it would make sense to have a syntax-description for non-jnode commands too. E.g. if we include an external app our descriptor might include a BNF String to describe the commands syntax. It wouldn't be an advantage for the command but for the user, because he gets nice tabcompletion also for non-jnode apps.
So perhaps it could really be an idea to handle it via syntax too (i.e. we don't expand except it is noted in the BNF syntax string of the command's descriptor)?
BNF descriptions of syntax
(I've addressed most of your points in other posts ...)
Real BNF is a bit clunky; e.g. you can only express repetition through recursion. EBNF is a bit better, because it supports optional and repeated groups on the RHS of a production. Unfortunately both BNF and EBNF are unfamiliar to the majority of programmers, and they are not suitable for end users.
Other alternatives are XML, a formalization of the meta-syntax used in UNIX manual entries, or a formalization of the meta-syntax output by the JNode "help" command.
To my mind, the key issue is not the language (meta-syntax) we use for expressing syntaxes. Rather, the big thing that we need to do is to get (concrete) command syntax specifications out of the application classes. The application should simply provide a set of abstract parameters and the expected type and multiplicity of their values. If an application expects a parameter to be filename pattern, the parameter type should be set accordingly.
Take a look at Powershell
Powershell uses attributes (annotation in java) to define what input is expected by a command (called cmdlets in Powershell).
Take a look here:
http://msdn2.microsoft.com/en-us/library/ms714433.aspx
Comments on Powershell
The potentially worthwhile thing that Powershell adds (relative to the current JNode model) is use of annotations on get/set methods to hook them up with the command argument parsing scheme.
Interesting, but I think I prefer the JNode approach of accessing parameter values via a binding object. Certainly, using attributes (i.e. Java annotations) won't help in the (my) goal of separating the concrete command syntax(es) from the application code.
Powershell's 'ValueFromPipeline...' attributes are nasty and ad hoc IMO. The rest of the attributes are more or less mirrored by existing JNode Syntax functionality.
Command and annotations
I think that annotation help in the goal of separating the concrete command syntax from the application code. No parsing is needed in the application code when you mark the setters methodes with information about what the command expects as input. The shell should then try to parse the commandline and match that with the annotationed setters in the command. This would make the command syntax agnostic.
Example:
public class TestCommand extends AbstractCommand
{
public enum Option1Enum {a, b, c};
public enum Option2Enum {x, y, z};
private Option1Enum option1;
private Option2Enum option2;
@Parameter
public void setOption1(Option1Enum selectedOption)
{
option1 = selectedOption;
}
@Parameter
public void setOption2(Option2Enum selectedOption)
{
option2 = selectedOption;
}
public void execute() {
...
}
}
And you could then write this in the commandline:
>testcommand -a -y
or the shell could accept this variant:
>testcommand -ay
it depends only on the shell!
If the command needs a list of files it could look like this:
private File[] filesToCopy;
@Parameter
public void setFilesParameter(File[] files)
{
filesToCopy = files;
}
I the commandline:
>TestCommand *.jar
The shell interprets the pattern and give the command the list of files matching the pattern. Again it depends on the shell if it wants to accept that kind of pattern.
If a command want to interpret a pattern on its own you could do something like this:
@Parameter
public void setFilesParameter(string filesPattern)
{
// interpret filesPattern
}
It would be the same in the commandline:
>TestCommand *.jar
Annotations do not provide the separation we need
Annotations allow the application programmer to define command line syntax at a higher level of abstraction. This has clear advantages for the application programmer ... versus the classic "roll your own" approach to command line parsing.
However, the definition of the command line syntax remains in the source code ... i.e. in the FooCommand.java file. The syntax cannot be altered without altering the source code, and that impacts other users / uses of the command.
For our purposes, the concrete command syntax specification needs to be separate from the code file for reasons previously stated. I say "needs" advisedly, because without the ability to tailor the command line syntax, some things we are trying to do become a lot more difficult. Specifically creating a UNIX-like shell and command set for JNode.
Another example
I have a command that accepts a date as the first parameter.
The command would look something like this:
public class FooDateCommand extends AbstractCommand
{
private DateTime date;
@Parameter
public void setDate(DateTime Date)
{
date = Date;
}
...
}
One shell only accepts this date format (yyyy-mm-dd) so the command should be called in this way:
>FooDateCommand 2007-11-27
Another shell only accepts this date format (dd-mm-yyyy) so the commandline in this shell looks like this:
>FooDateCommand 27-11-2007
A thrid shell is maybe very smart and knows what format the user prefer and uses that.
Does it matter for the command? No as long as it gets a DateTime object it's fine.
Annotations do provide separation!
The command programmer only defines what input the command takes, he doesn't define the syntax.
The shell should use reflection to look at the command and then try to match what the user is writing on the commandline with the parameters that the command accepts.
Do you know if Annotations work on JNode yet?
Have you tried them?
Yes it should work
According to this http://www.jnode.org/node/638 it should work.