Bjorne shell tracking issue

After I set the bjorne interpreter the command history doesn't seem to be working as expected any more.

The history issue is now fixed. Thanks. I've also fixed some bugs in variable expansion.

An 'immediate' use of the Bjorne shell in JNode could be the execution of startup scripts distributed with many Java applications for supporting the standard way of starting up those applications as intended by their authors.
Such scripts usually contain actions for determining the location of the JVM, the location of the application, setting up the classpath and various JVM parameteres and system properties as well as passing on to the application the arguments provided to the script.
For this we need in Bjorne the specific constructs and builtins to work properly, we also need to change certain aspects of the JNode environemnt so that it can be recognazied as a valid environment by the script and we also need a reworked java command with a similar set of options to the standard one which in this case will start the application in a new isolate. Most relvant in this thread is the Bjorne part of the problem.
An example for this could be the SVNkit (http://svnkit.com/download.php) software which is a pure Java SVN client which includes a shell script (jsvn) to provide functionality similar to the standard command line SVN client. I think the scripts distributed with SVNKit canbe a good starting point for this activity but the same applies to the Jetty, Tomcat, Jython, JRuby etc. distributions too if you need more examples.

As a part of the effort to get Bjorne working, I'm building a test harness for "black-box" testing of command and scripts. The basic idea is to run a command/script with a given set of arguments and standard input content, checking the return code and the output written to it standard output and error streams. Later on, I will extend this to cover reading and writing of file system objects. The harness is being developed so that it will (as far as possible) work both within JNode and in the development environment.

BTW, I'm not currently soliciting input on the design and implementation of this harness. I've already figured out what it needs to do for my use-case, and that's all I care about at the moment.

Revision #5026 contains changes to JNode including 'core' infrastructure to support better handling of environment variables and (soon) system properties. I've tested the changes thoroughly, but if the changes cause breakages, someone please rollback the revision.

It would be interesting to have a special command for starting the Bjorne shell. This is a more natural startup method for the users under JNode than setting an obscure system property. It also makes the usage of Bjrone shell much easier outside JNode.

Further more it would be interesting to try to make the Bjrone shell very easy to use outside JNode just as a general purpose shell implementation on the Java platform. Then we could try to make it available as a separte download on the project website. I supose this might involve some refactoring and probably a separation of the features to JNode specific and general purpose ones, as well as the creation of a compatibility layer for general purpose usage outside JNode on the JDK.
Still the effor might make sense because then the Bjorne shell could be used in the wider Java community independently of JNode in a much large scale. Currently devolpers need shell features in complex applications and for this purpose they create custom shells which are quite limited in functionality most often. Such developers are usually familiar with bash and so they could use the Bjrone shell and reuse their bash scripting skills in the domain of their Java application.

In the same time the Bjorne shell could also give for the whole JNode project a better exposure in the community and attract more contributors to the project.

In the future the Bjorne shell could support the dynamic compilation (to bytecode) of bilt-in functions and scripts (a feature most probably not available in any popular linux shell) and provide outstanding script execution performance. It could also be connected to the generic scripting support of the Java 6 platform hence further improving its usability in embedded script execution scenarios.

Interesting ideas.

Re: "ease of use" in switching the console to use the Bjorne interpreter ... the simple solutions are 1) change the property name to something less obscure or 2) write a tiny command or script to do "set jnode.imterpreter bjorne". But actually, I would expect that most users would simply put "set jnode.interpreter bjorne" (or whatever) in the relevant "ini" file and forget it.

Another option would be "console ... -i bjorne". But that is doing something a bit different.

Re: getting Bjorne to run as a stand-alone Java application ... a good idea, but a lot of work. It would entail porting most of the JNode shell infrastructure (e.g. CommandShell, syntax mgr, alias mgr), and creating new invokers. The last is necessary because neither proclets or isolates can be implemented in a classic Java VM. You'd need to use Process to launch commands external to the current JVM.

Re: the last idea is ... IMO, this is not worth the effort. People have in the past written UNIX tools to compile shell scripts to (for example) C code. The conclusion is that you don't get much speed up by doing this for "typical" scripts. Most of the time running a typical SH script is consumed in launching commands, and in the commands themselves. And a large part of the execution time in the shell itself is string bashing (variable expansion, globbing, etc) and other complicated stuff that is not amenable to optimisation by compiling to byte-codes.

If people want a "fast" scripting language, they would do better using BeanShell, Perl, Python etc. These are more expressive (e.g. so fewer commands need to be called) and as a result more amenable to compilation to byte-codes. Another idea is to improve JNode's native code compiler which would make everything faster ... not just shell scripts.

Hi Steve,

I saw you "complaint" about not enough testers for bjorne. I had a bit time and played with bjorne and I like it very much, actually I like it that much that I'm putting an entry into my local jnode.ini. Promised!
As it wouldn't make sense to tell you what worked, I'm instead doing as requested and report you any bug or strange behaviour I found:

The 80-column linewrapping bug is pretty annoying btw, especially if you're in deeper subdirs.

If I type "cd <TAB><TAB>" I see nothing even if there are possible directories in the working dir. Btw, this works with the default interpreter, but I also see that this might be problematic as the path argument is optional. Anyway, FYI

I am used to alias "l" to "ls -alh" in Linux, so let's start with:

 alias l ls
 alias | grep -r "^l:"
 l

which results in an ShellException about no matching syntax.

Having a file "test" in the working directory from now on:

 ls > t<TAB>

results in a prompt with just "ls "

 ls | grep test > <TAB>

throws an IncompleteCommandException about missing file argument

 ls | grep test > t<TAB>

just does nothing, though I see that this is a hard one, but at least it should not differ in behaviour to the first one.

I'm not sure if it is described or discussed anywhere yet, but on my box I'm very used Ctrl+key commands. To name some I personally use: a (home), e (end), k (cut til eol), y (paste), r (reverse history search). All of them print strange characters in JNode. pageup/pagedown does not do anything, in my box it uses the string before the cursor position and reverse searches the history for a command starting like the string.

Another strange behaviour is on an empty prompt: If you press the key-down button you go into the history from the beginning. I'm not sure if it's a feature or a bug, but I was confused the first times it happend as I didn't expect it.

You state "Expression expansion '$((...))' is not implemented" but Back-tick expansion is implemented.
What I tried is the following:

 for i in "ls foo bar" ; do echo $i ; done
 for i in 'ls' ; do echo $i ; done
 for i in `ls` ; do echo $i ; done

the working dir contains several files, the above commands give me:

 ls foo bar
 ls
 `ls`

And while fiddling with the syntax of some shell builtins (e.g. the for loop) I discovered that anytime you have an error in your command it does not make it into the history. This makes it difficult to test the syntax as you have to retype the complete command each time you had an error

It took me a bit to recognize "page", but I'm really missing "less". Btw, I discovered you built in "/"-style search in page: Is there a way to stream until the next hit like "n" in "less" (I discovered "?" to repeat the search for a page)? And line-edit (where you enter the search-string in "page") does not work.

That's it for now, hope some of it is of help.

EDIT:
I forgot to mention that some commands throw SecurityExceptions. E.g. I can not javac, java and even startawt (though I wouldn't expect that to work due to still not working cirrus driver anyway).

Peter, thanks for the copious feedback . Some of your points are already listed in the "not working yet" section. For example, aliases are simply not implemented yet, and a lot of work is needed to make completion work properly.

You've found a bug in 'for' loops. The <words> list needs to be subjected to command substitution to handle the "backticks" ... and possibly more. (But the examples in single and double quotes are handled correctly ... 'bash' on my LINUX box gives the same output.) Update: fixed now.

I'll take a look at the Security exceptions. Bjorne could be causing java/javac/startawt to be executed differently, and that could be causing the exceptions. (IIRC, they are all commands whose primary entry point is a "main" method Update: yes ... it was the entrypoint issue ... and it was bjorne's fault. Fixed now.

History and line editing are handled mostly at the level of the CommandShell and the input drivers. Bugs and RFEs should be separate issues. Update: Created issues.

You are correct that erroneous commands should be added to the command history ... that may be bjorne's fault. Update: fixed now. This is now dealt with at the CommandShell level.

The 80 column bug(s) are really an input driver problem as well. I'll create a separate issue for it / them. Update: Created issue.

And finally, we need a separate issue for the bugs / enhancements in "page". I'm not particularly keen on implementing all of "less", but if there are particular features that people want, I'll give it a go some time. Update: Created issue.

I think this falls under file pattern expansion, but it wasnt working for me:

gzip data{1,2,3}

tells gzip to compress data{1,2,3} not data1 && data2 && data3.

Thanks. Strictly speaking '{...}' patterns are not a POSIX shell feature. But they are widely used, so I'll put them on the list to be implemented anyway.

The shell seems to be globbing text in quotes. ".*" gets expanded into . and .., which would be right, outside of quotes. I spent a few minutes and traced it down somewhere into the bjorne parser/tokenizer area. I think the inital BjorneTokens were coming out as ".*" but somewhere between there and MuParser it ends up as . and ..

Edit:
Just to be clear, this is any shell glob. And \ doesn't escape it. So there's no simple way to pass strings with glob characters to arguments.

I've partly fixed the problem in cluster's comment #12. The problem was caused by bjorne doing globbing after all quoting had been stripped. It now does globbing before quote stripping, but this has revealed other bugs in the PathnamePattern class which I'm working on now.

Update - those bugs are fixed now.