How to do interactive console input?

The patch I just submitted does not address the following important issues:

• How can we deliver properly encoded multibyte characters to an application's InputStream?
• How can we deliver VK codes that don't map to Unicode codepoints?

(In addition, the patch cannot handle the NUL control character due to the unfortunate use of '0x0000' to represent an unmapped keycode.)

The KeyboardInputStream (as modified by the patch) drops the top 8 bits of character values in KeyEvents to make them 'fit' into the InputStream byte stream abstraction. KeyEvents that the keyboard interpretter didn't map to char values are filtered to get rid of modifier codes (SHIFT, CTRL, ALT, etc), and then the remaining VK codes are injected into the stream as bytes. This is broken both in theory and in practice. For a start, Latin-1 characters > 127 are liable to collide with significant VK codes.

I can think of a number of better ways to do this. Here are some examples:

1. Insert chars into the stream as UTF-8 sequences, and use some pre-existing scheme for encoding non-character VK keycodes; e.g. VT220 keyboard escape sequences. (This doesn't generalize. No existing scheme can deal with all possible current and future VK codes.)
2. Insert chars as above and use a JNode specific scheme for embedding unmapped VK codes. For example; use ESC xxxx {x=0..F} to represent the VK code xxxx, and ESC ESC to represent a real escape character. Or use the 0xffff Unicode 'not-a-character' codepoint as the VK code introducer.
3. Use an alternative to the InputStream API that distinguishes between character (Unicode) and non-character (unmapped VK) data points. For example, replace the read methods with different named methods for character and non-character data. We would need to do this in a way that doesn't involve breaking the InputStream API and the mountains of code that depends on it.

Once we've properly encoded the stream from the keyboard, we also need to be able to consume the stream in an appropriate fashion The TerminalInputStream class implements a rough approximation to the behavior of a UNIX tty stream in "cooked" mode. (Ideally the app would see properly encoded Unicode chars; currently, the app sees other 'crap' ... see above.)

We also need an approximation to a tty in "raw" mode, where line editting is bypassed, and an application can see both character and non-character VK codes.

A final piece of the input-side equation is that we will eventually need to cope with legacy terminal and xterm input. One idea is to implement alternative "keyboard interpretters" that translate all kinds of legacy codes (e.g. VT220 codes) into characters and VKs.

(We also need to enhance the output side as well; e.g. by emulating (say) a VT220's functionality for editting the displayed text in the JNode console stack. But that's a separate topic.)

I'm aware of the problem of line editting, it has poped up at the testing scripting language interpreters too, recently, but it's older than that.

Anyway I think we should handle two distinct cases with the commands.
One is the normal java behaviour, where the command or application gets the input characters on enter and up to that point the line can be editted at will. This would be the case of a standard java application starting up from the shell.
The other case is the jnode specific application which gets direct access to the stream of events and can takes actions on arbitrary keys or characters. This is the case of the shell or the charwa integration or the console base editors etc.

We need some refactorings to support these and I need some time to handle it.

Also extending support for command line history would be nice. So that for insatnce I start up the rhino shell or bsh and I have real line editting but also history of the entered lines on the arrow keys.

This is a sketch of what's on my mind related to this.
The bits of code for doing this are mostly there in the shell and console layers we need to reorganize and genearlize it a though.

Regards, Levente