Commands TODO list

Submitted by cluster on Tue, 04/28/2009 - 14:24.

JNode-Shell

This is a bit of my current TODO list for implementing commands. Feel free to pick one of this off the list. If you have something complete and working, create an issue with a patch in the issue tracker, or push a branch into the public git repo.

Commands that need further work

mv: one note about this, File.renameTo() does not work with directories and causes files to be copied and deleted.
rm/rmdir: I dont think these are implemented properly when dealing with recursive removal.
tail: needs 'follow' support implemented. Currently it has the same features as 'head', but does not have a 'follow' mode.
archiving: lots of work to do here. Tar/zip/unzip are incomplete implementations. Also there are no implementations of cpio, lzma, compress and pack algorithms.
wget: This has a basic implementation, but has lots of room for improvement.

Commands that need implementing

patch/diff/diff3: We're going to need this sooner or later (look at java-diff (site 1) / java-diff (site 2).
sha based checksum: for checksum on files (look at Jacksum, which has 58 checksum algorithms !)
sed: This can probably be implemented with a simple command interface to a sed library.
python: command/interpreter interface to Jython
php: Interface to Quercus
gnu getopt: command line parsing for shell scripts
tee: I'm not sure any of our shell interpreters are currently able to implement this properly, but at least a basic implementation can be provided.
vi: This would make my day (look at jVi and jvim)
git: This will rely on the work of the JGit project, which is under development
svn: Need to implement a command interface to SVN Kit.
ssh/scp: Implement an interface to an ssh library. Trilead SSH for Java
xargs: This would keep someone busy for awhile
slocate/updatedb: This is the file system searcher that uses a cached index for faster searching.
logrotate: probably wont need this for awhile, but will be nice to have
crontab: might be awhile before we can use this. Possible backend is Quartz

I have more, but some are simply commands that are not so common. I'll add to this list if i can think of more.

Update

An effort is under way to currently solve the issue of not needing to create command wrapper classes around apps that are doing their own argument parsing anyway. An example of this is the JawkCommand that has arguments for acceptance and completion, but it simply passes the command line options verbatim to Jawk for it to process.

Footnotes for jRPM, JLzma, Loopy

Submitted by gchii on Fri, 05/08/2009 - 15:23.

jRPM provides an implementation of cpio and RPM. JLzma provides an implementation of LZMA compression. Loopy provides an implementation of ISO9660 based upon work done on JNode.

some links for your libraries

Submitted by Fabien D on Fri, 05/08/2009 - 19:04.

Loopy : interesting ... it might allow jnode to mount an iso file as device.
jRPM : I wonder if RPM is suitable for java applications too.
JLzma, there is also the project lzmajio that is built on top of it to add LzmaInputStream and LzmaOutputStream classes.

Fabien

my blog : en français, in english or both

Thanks for lzmajio

Submitted by gchii on Sat, 05/09/2009 - 05:27.

When searching, I did not find lzmajio so I built my own LzmaInputStream and LzmaOutputStream classes. I'd rather not do that. Thanks,

zcat, man

Submitted by gchii on Fri, 05/08/2009 - 15:19.

As I understand it, the zcat command is similar to cat. It displays a gzip-compressed file. In an object-oriented world, does zcat extend cat? I advise against this. Rather, cat and zcat should produce similar behavior by depending upon the same object-oriented libraries.

Of course, if we're going to implement zcat, we might need bcat to display bzip2-compressed files and lcat to display lzma-compressed files. But one command for each kind of compression seems to be a bad idea.

As I understand it, the man command displays a man page. A man page is a document in a well-defined directory, using a well-defined format. There is no zman to display gzip-compressed pages, just man. So cat should display compressed files, too.

You guessed it. My favorite cat command in Java displays compressed files.

Berzerklyism!

Submitted by Stephen Crawley on Sat, 05/09/2009 - 03:47.

Back when I was running a 4.1bsd machine on a VAX, old school UNIX folks use to complain bitterly about the way that Bill Joy et al at Berkeley had added "a bazillion" options to commands like "cat" and "ls" to do supposedly useful things. This sounds like your "favourite cat".

We need simple commands that do specific tasks well, not swiss army knife commands that try to do everything. To me, the even the idea of using "cat" as a display program is conceptually wrong. That is what "more" / "less" / "page" are designed for. The "cat" program is for file / stream concatenation. Period.

(OK, I've no problem if you want to "alias 'cat=more'" ... but that's a personal choice, not a decision that should be foisted on everyone.)

zcat/bzcat are both in jnode

Submitted by cluster on Fri, 05/08/2009 - 16:12.

zcat/bzcat are both in jnode already. They all base off of the same libraries as their sibling commands. gzip/gunzip/zcat are all wrapper commands that use a common class to do the work. Same as bzip2/bunzip2/bzcat. bzcat and zcat are not the same as cat, as they do not provide the flags that cat does. They provide a 'similar' feature, in that they can output data to stdout. But you do not need bzcat/zcat to do this, as bzip2 and gzip can do this. For example, all of the following provide the exact same feature...

bzip -dc [file1] [file2]
bunzip -c [file1] [file2]
bzcat [file1] [file2]

And you have touched on a subject that i haven't fully approached yet. This not only falls into cat, but like the URL/URI in commands, you also have all types of commands that can use compressed files. For example, grep, diff, less can all be found in bz/z form. The aliases need to be made available for compatibility reasons, but we really don't need to create commands for all of them. The approach i had in mind was to create a common Utility class that returned streams for files based on their file type. All the class needs to know is that it got back an InputStream, it doesn't care if that stream is a GZipInputStream that wraps a FileInputStream. I have a start on this via org.jnode.command.util.IOUtils in the cli subproject. This way i should be able to allow _every_ command that uses this interface to read any type of compressed file, and be none the wiser.

Opening files based on file type and signature

Submitted by gchii on Sat, 05/09/2009 - 05:25.

Every binary file format has (or should have) a signature. Let's say that I copy x.gz to x.bak. It is still a gzip-compressed file regardless of its name. Its signature stays the same. Opening in.bak based upon its signature provides the decompressed file.

So we have come full circle. A command depends upon some kind of a stream factory to open a stream. Deep inside a stream factory, a decision is made about how to connect a wide variety of stream filters to return the original data. (Given, we are assuming a long list of things.)

How are we going to make a stream factory available to all command classes? And if we make a stream factory available, should it be attached conveniently to FileArgument?

Who is going to build a stream factory for every kind of file? With tens of thousands of file formats, and more on the way, a stream factory should be open ended with a service provider interface. Does org.jnode.command.util.IOUtils provide a scalable architecture for hundreds of signatures? When opening an archive, such as (ar, cpio, deb, ear, ext3, jar, iso, ntfs, rpm, tar, war, vfat, zip), nothing prevents us from returning the corresponding archive input stream. An "archive" is any digital resource that has a table of contents and contents, including a file system.

We have identified the need for a very powerful stream factory. What will enable us to plug-in additional file formats? What will enable us to open a stream with respect to the current working directory of a command? And finally, should a very powerful stream factory be conviently attached to FileArgument?

Here is the gotcha. It becomes ambiguous. Abiguity is bad in an operating system. Let's say that I have a text file called a.txt. Then I compress it with gzip, giving b.txt.gz. The cat command is used to concatinate two files. Should it concatinate the compressed or uncompressed version of b.txt.gz?

cat a.txt b.txt.gz

The diff command is used to display the difference between these two files. Should it compare a.txt with the compressed or the decompressed version of b.txt.gz?

diff a.txt b.txt.gz

Applications using compressed files, etc

Submitted by Stephen Crawley on Sat, 05/09/2009 - 03:16.

I also think that creating new commands or adding options to existing commands to handle compressed files, archives, URL arguments and so on is the wrong approach.

The "classic UNIX way" to address this was to use pipelines; e.g.

   wget htttp://somewhere.com/foo.txt | less
   zcat foo.tar.zip | tar tvf -

... but that approach has its limits. The "modern Linux way" is to map things into the filesystem namespace; e.g. by "mounting" an ZIP file or a remote http service.

Some Linux commands (e.g. scp, rdist, etc) fake this by extending the UNIX pathname syntax, but I don't think that's a great idea ... from a design viewpoint. Though clearly, if we are going to provide Linix (POSIX?) compatible scp, rdist, etc for JNode we need to support the same command syntax.

I agree, and the reason i

Submitted by cluster on Sat, 05/09/2009 - 03:28.

I agree, and the reason i haven't gone this far yet is i want to draw a clear line, between what the command is capable of. All i was currently hoping to achieve, was to make things like bzgrep redundant. If you pass a bzip/gzip compressed file to grep, then it would have the IS wrapped with an appropriate decompression stream.

There would be a strict limit to this, and it would not extend to full blown archives. Only to compressed files. Bzip/gzip files are easy to identify by their header 'magic' bytes, and the command itself does not even need to know or care that it is compressed. I like adding features to commands when the command does not have to do anything to support it. Which is why i thought of this as different from the URL/File issue in commands.

Java-related commands

Submitted by gchii on Sat, 05/02/2009 - 01:55.

There are a few Java-related commands that are on my wish list.

appletviewer
jar
jarsigner
javac
javadoc
javah
javap
keytool
native2ascii
pack200
policytool
serialver
unpack200

Already in JNode

Submitted by lsantha on Sat, 05/02/2009 - 05:50.

Most of these commandsare already in JNode, I added them while integrating OpenJDK.
javah is missing though because it doesn't make sense under JNode with its original functionality.
I implemented a similar tool which creates JNode speciffic stubs in Java for classes with native methods. This tool extended and improved may become one day the JNode version of javah.

Command documentation, JNode wrappers

Submitted by Stephen Crawley on Sat, 05/02/2009 - 08:20.

It would be good if someone (not necessarily Levente) would create command pages for these commands.

There is also the issue that these commands use (AFAIK) the classic Java 'main' entry point. Levente has suggested an approach for "skinning" classic Java programs, but it could be some time before Cluster or I get around to implementing that. A quicker alternative for enabling JNode completion and help for these commands would be to create org.jnode.shell.Command wrappers for them. Clunky ... but quick.

Implementing command interface for an external command

Submitted by lsantha on Fri, 05/01/2009 - 08:47.

svn
Need to implement a command interface to SVN Kit.

There are already many commands in jnode with such requirement.
However instead of implementing the command interface by hand for each command a nicer approach would be to make it automatic.
I have described this idea in more detail in the past: http://www.jnode.org/node/2903 (see the original issue description, most comments are offtopic)
Actually I liked what you did in JAWK except that you still had to create a new JNode command to make it work. The idea is to make this process completely controlled by the plugin descriptor by using a new attribute for the alias tag or whatever but wihout the need for coding extra wrapper commands.
What do you think?

Especially in the case of

Submitted by cluster on Fri, 05/01/2009 - 22:46.

Especially in the case of svn kit, this is the right approach. I have read that issue before, and i agree. Creating wrapper command classes is more of a work-around than a proper solution. This is going to be true of alot of commands and even more so full java applications. We dont want to have to maintain 10s of thousands of command classes + plugin descriptors for any application that is run.

To point out what it was i did with Jawk. I created a syntax and a command class with arguments for that syntax. But within the execute(), i disregarded the arguments, and passed the arguments verbatim to Jawk, as it has its own parsing mechanisms.

The two big things that the Command brings are completion and help. Most commands that do they're own arg parsing are going to have a 'awk -h/-?/--help' type of help display. If we had some simple way of telling help that for 'help awk' it should run 'awk -h', then we don't need the ArgumentBundle from the Command. Completion is going to be a little trickier, as I would think we would want to avoid creating Argument objects. We already have the ability to put the argument flags into the syntax (we were wondering how this would be usefull to!), so we dont need the Argument for that either. Really it comes down to doComplete. Perhaps steve could shed some light on this better.

third party implementations

Submitted by Fabien D on Fri, 05/01/2009 - 08:16.

cluster: I have added links in your post for "patch/diff/diff3", "sha based checksum" and "vi" implementations.

Fabien

my blog : en français, in english or both

Some more candidates for the TODO list

Submitted by Stephen Crawley on Wed, 04/29/2009 - 01:16.

Can I propose the following as worthy of attention?

mount: There are currently separate incarnations of mount for different FS types. These should be refactored so that there is one command that knows how to handle all FS types.
format: Same as for 'mount'.
dhcp: IIRC, this currently makes an request to some service component which "succeeds" before the task has been done. A script that uses 'dhcp' has to follow it with a 'sleep 10', an has no way of knowing if the request succeeded or capturing the output.
ifconfig, etc: Judging from the sample shell ".ini" file that someone put into the shell tree, other net commands suffer from the same problem as "dhcp".

I think that fixing the above commands will entail delving into (and maybe changing) important JNode service APIs and implementations. Not a simple task.

The problem with the current 'ls' is that it is actually started life as a DOS 'dir' implementation. IMO, it would be better to recognize that 'ls' and 'dir' are really very different commands and implement them in distinct command classes.

If you need help i can take

Submitted by galatnm on Tue, 04/28/2009 - 16:56.

If you need help i can take a look on wget command to implement missing feature (i'm busy with HFS+ features but i can deal with the two ^^). I propose also to add ssh and scp commands to the TODO list. There are already librairies that implement these protocols in java thus we can implement a simple command interfaces to these libraries.

My 2 cents,

Fabien L.

For wget all i was hoping

Submitted by cluster on Tue, 04/28/2009 - 21:03.

For wget all i was hoping for in the near future was having the -i and -O options, allowing '-' to be used for the file name. Then at least it can be used in pipelines.

SSH and SCP definetly should be on the list. Will update. I've got some others i'm adding as i go.

Btw, how is work going on HFS? I haven't setup jnode with anything more than a ramfs as the last time i tried i ended up corrupting it, but i'd like to set on up again, as the ramfs isn't going to cut it for me much longer. If/when your ready to test, let me know and I can do some testing while a test commands and such.

Active forum topics

Recent blog posts