Move CatCommand to the shell tree

I don't see any problem to do this. Actually, we can keep in FS tree only the commands that are directly related to filesystems like format or (in the futur) checker commands.

Fabien L.

Starting to move over commands between plugins just for the sake of testing doesn't look like the solution to me. Instead I would try to make the test harness more flexible.

We need to run the tests under JNode first of all. How will new build targets solve that?
The tests should follow the modular structure of the code that they are written for. This modularity should apply to the execution of the tests too. So the test harness created for testing the Bjorne shell might need a revision for this.
An alternative solution is that the "testharness" will tests the most basic features of the shell like syntax and builtin functions and the commands will be tested separately.

Starting to move over commands between plugins just for the sake of testing doesn't look like the solution to me.

Agreed. That's why I said reason #2 was lame!

What is your response to my reason #1? And to Fabien's more general argument about most of the FS commands not belonging there?

We need to run the tests under JNode first of all. How will new build targets solve that?

And you think I hadn't realized that???

My respose to your first question is that 'cat' doesn't look very useful without files to me. (What else can it do beyod copying stdin to stdout?) Which are the other commands in fs that are useful without files? For the case you describe 'grep' is a good example but that is in the shell from the start.

You can create tests for shell commands outside the shell project based on the test harness, then the new test plugin will use the test harness plugin, bjorne plugin and whatever else it needs.

... cat' doesn't look very useful without files to me.

cat -u http://jnode.org | less

IIRC, it was you that argued vehemently that we retain the '-u' switch.

IMO, "is not useful without files" is a poor criterion for deciding what to put in fs.commands; consider the 'jar' and 'javac' commands. And "because it is there already" is equally bad. However "performs a disk / low-level fs maintenance task" (i.e. Fabien's criterion) makes a lot of sense.

You can create tests for shell commands outside the shell project based on the test harness, then the new test plugin will use the test harness plugin, bjorne plugin and whatever else it needs.

That approach doesn't work when running tests in the sandbox. Emu doesn't do JNode plugin loading ... it fakes it by extracting info from the XML descriptor files. I looked at implementing plugin loading in Emu some time back, but it was too difficult.

Running tests in the sandbox is the important use-case when developing the shell and commands. So far, I've found that nearly all the bugs show up when testing in the sandbox ... and the edit-compile-test cycle is MUCH faster.

"is not useful without files" is not the criteria for putting commands in the fs project. Just as jar and javac are not in the fs project though they are unusable without a file system. If something is (nearly) unusable without files it shouldn't go to the shell project (that is the point!).

My question was: which are the the commands in the FS project now that you think are in the wrong place?
(imho they are ok there, but you can take alook) The 'cat -u' points towards the net project except that the url can refer to something else than a file or a network resource identifier to (system:/,plugin:/ etc.?), this might justify it's place in the shell project one day maybe).

The more important topic here is to enable the test harness to test commands outside the shell.
For this under JNode we need the modular solution. Outside JNode a quick solution is to put all the plugins to the classpath that you need for development, as you said, until we find a better solution.

"is not useful without files" is not the criteria for putting commands in the fs project.

So what is the criterion?

If something is (nearly) unusable without files it shouldn't go to the shell project (that is the point!).

You mean like 'grep', 'bsh', 'ant', 'test', 'sort', 'javac'? The list goes on. Basically, that criterion: 1) doesn't match the current arrangement of classes, 2) doesn't have any practical or pedagogical justification, and 3) simply doesn't make sense ... because most of JNode is useless without a working FS.

My question was: which are the the commands in the FS project now that you think are in the wrong place?

All of them. I started out thinking that 'mount' and 'format' should stay in FS, but I now think that there are significant benefits in moving (at least) the command classes and associated plugin descriptor elements to Shell. (In the case of 'format', the classes that do the formatting should probably stay in FS. IIRC, the command is already split into generic and file system specific parts.)

The ONLY reason that the FS project depends on the Shell project is that FS project contains command classes (and their supporting classes). The same logic applies to any commands in the Net and Gui trees. If the command classes were moved to Shell, then you could (hypothetically) assemble a JNode in which there was no console and no commands, but a working file system. For example, a system that was administered through HTTP requests.

So the logic in moving commands in general and FS commands in particular to Shell is:

1. Commands are more part of the "command line environment" than the "file system", "networking system", etc. (You might want to argue, but there are 40+ years of OS pedagogy to support this view of the world.)
2. Command classes are easier to find if they are in one project.
3. Command aliases and syntaxes are easier to find if they are in one project.
4. Moving all command classes to Shell (should) allow us to eliminate the project dependencies of FS, and maybe Net, Gui and others on the Shell project.
5. Commands are "(nearly) unusable" without a shell. So following your line of argument (which I don't think bears close scrutiny), they would belong in the Shell project.

The more important topic here is to enable the test harness to test commands outside the shell.

That is an independent topic, and I don't intend to discuss it further in this issue. Just pretend that I never mentioned the test harness, and focus on the question of whether the command classes need reorganizing.

(Hint, look at my latest checkin.)

Currently it's perfectly possible to build a JNode system without file system support, network support, shell support or gui support.
Unfortunately you think that this is not possible because the fs project depends on the shell project.
But in spite of that this is still possible because what's relevant here is not the dependencies among JNode projects but the dependencies among JNode plugins. If you move all shell commands to the shell project then instead of all projects depending on the shell project (what's wrong in this anyway?) you will have the shell project depend on all the others (and what is so much superior in this one?). Whatever the depencency is among projects it has little importance compared to the dependencies among plugins.
I agree that there are various things which need to be cleaned up (like the ant command in the shell project) but reorganizing all commands the way you propose makes little sense to me and it seems to have little benefit if any at all for the runtime flexibility of JNode.

We better focus on things that are more important now.

I agree that there are various things which need to be cleaned up (like the ant command in the shell project) but reorganizing all commands the way you propose makes little sense to me

Please explain what about my proposal makes little sense to you. Or maybe explain WHY you think that it is a good idea to put generic user level commands (like 'dir', 'cp', 'cat') in the same logical category as file system drivers.

and it seems to have little benefit if any at all for the runtime flexibility of JNode.

It is about build time flexibility, not runtime flexibility.

Suppose that I want to work on a no-shell/no-commands. With the current project partitioning and build structure I have to do a bunch of dependency analysis, and some fairly major surgery on the build system if I want to stop the build from building the bits that I don't need. If the project partitioning was correct, I should be simply be able to comment out the 'leaf' packages that I don't need and the cut-down build would just work.

We better focus on things that are more important now.

You may think it is unimportant, but I'm REALLY fed up with having to wait 5 minutes each time I do a JNode full build. If the partitioning was right, I could leave out the bits I don't need ( e.g. probably > 75% of Core ) and my build times would be halved or better.

I think the partitioning problem is very important, which is why I'm prepared to spend hours debating it. If you really don't think it is important, why don't you just say:

    "OK Steve ... since this is not important enough to argue about, you may as 
     well just do it.  If it turns out to be a bad idea, we can always undo it.".

... or even simpler, don't bother responding and get on with doing whatever it is that is more important.

Remember, I started asking about moving just one command so that I could get on with building test cases. I think next time I will just do what I think is the right thing to do.

By the way, you have not commented on the other benefits of moving the commands; i.e. #3, #4 and #5. Should I assume that you accept these points as valid? ( Careful how you answer that )

In what cases do you need the make a full rebuild while working for instance on the Bjorne shell or its tests?

When regular builds fail due to mysterious build errors that only go away after I run './build.sh clean'. It happened to me at least half a dozen times last night while I was trying to debug the test harness running on JNode.

The reason most probably was that I switched from the GNU Classpath security stuff to OpenJDK.
When hundreds of dfiles go away and come during an SVN update a full rebuild is recommanded.
But I doubt you need it when work inside of any specific module. In fact for many changes you can avoid the JNode restart too using the class hotswapping.

Migrating towards a better project structure and build system is quite a dramatic change that needs a lot of thinking and obviously a separate thread.
And of course we should make sure in advance that the new one will be better then the current.

The reason most probably was that I switched from the GNU Classpath security stuff to OpenJDK.

That would explain the first time, but not the other 5 or so times I had to run 'build clean'. There is a bug in one of the custom ant tasks (a file handle leak previously reported) and probably another bug in the dependencies somewhere. Ant throws an exception because it cannot load/find a custom task class. I'll create a bug next time it happens.

But even when the incremental build works (i.e. w/o clean) it takes 40 secs. If the build could be cut down to 20 secs ... and the boot time reduced ... that would be a big win.

Where is "class hot-swapping" documented?

Migrating towards a better project structure and build system is quite a dramatic change that needs a lot of thinking and obviously a separate thread.

Yes. I raised this to demonstrate that partitioning of projects and build-time dependencies between projects are real issues with important consequences.

And of curse we should make sure in advance that the new one will be better then the current.

Yes ... to the extent that it is possible to know in advance. I actually favour a more incremental approach, improving things a little bit at a time and learning from our mistakes.

So back to the original topic ... shall I move those command classes. or (<sigh>) do you still want to keep debating it?

I'm on irc. Please join in.

FYI ... the "black box" test harness now can be run on JNode as well from Eclipse and (in theory) the Linux/Windows command line; see Running black-box command tests.

For the record, the outcome of the IRC discussion was that it is OK to move the "easy" FS commands to the Shell project. Commands like the "mount" and "format" commands are more difficult because of dependencies on other classes in the FS tree. Possible solutions were mentioned, but it was resolved do this in the context of a larger scale pre-planned reorganization.

Is that a fair summary?

Well, that wasn't my conclusion.
The best solution would be to tailor the testharness command so that it can run tests in any project or plugin. Wanting to put all shell commands and all shell command related tests in the same project or plugin doesn't match the modularity of the JNode project.
The modular solution is to develop tests in the FS project too and the plugin containing these tests will require the plugin which contains the test harness or whatever it needs for the shell project.
Let's the testharnes go after the code and not vice versa.
Having half of the FS commands in the shell project and the other half in the FS project will only increase confusion. Now all commands which conceptually are generic file utilities and those which are various file system utilities are in the FS project.

I give up. I'm moving CatCommand, and someone else (i.e. you Levente) can figure out the rest.

OK. I hope you don't need anything else. The rest of the fs commands doesn't have a single usecase without a file system. Therefor org/jnode/shell/command/ is the wrong place for them because the commands in this package are part of the minimal shell plugin list which doesn't have a file system.
The only usecase of cat in this context is: cat -u url where the url refers to an internal in-memory resource accessed using a JNode speciffic protocol.

Automatically closed -- issue fixed for two weeks with no activity.