Refactor commands to cli project
| Project: | JNode All |
| Component: | Code |
| Category: | task |
| Priority: | normal |
| Assigned: | cluster |
| Status: | closed |
Jump to:
Description
Throughout the seperate jnode projects there are dependencies on other projects. So long as a project is compiled before a project that has dependencies on it, everything will compile fine. But, if a project early in the build has dependencies in a project that builds after it, then it can't resolve those dependencies.
Of course this is probably obvious to many, the question still remains, how do we resolve this issue?
If your still wondering what im refering too, as an example, if fs/ has dependencies on shell/, which it does, shell/ has to be built before fs/ which it is. But if shell/ tries to access portions of fs/, then there is nothing to resolve the classes against, because they dont exist yet.
Maybe, this isn't really an issue and its by design that this is not to happen? If so, mark this as 'By Design', otherwise are there any possible solutions?
- Login to post comments
#1
It is a fact of life that dependencies between projects do ... and must exist. But the dependencies are not circular. Indeed, if you try to create circular dependencies between classes in different projects, you will find that JNode won't build using (at least) Eclipse.
It would be a good thing to document the project dependencies in the Document tree. But the dependencies are declared as part of the project metadata in (at least) Eclipse, so external documentation is not critical, I think.
One could debate whether the current project dependencies are logical, but this is actually consequence of the current partitioning of packages between projects. If you'd like to propose a more logical partitioning, I'd be interested in listening. But take note of the discussion on issue 2901.
#2
Regarding the issue you linked, i dont think commands belong in any project period. At best maybe the distr/ project should house the commands, which would resolve most any dependencies commands my have if they have dependencies on multiple subprojects.
distr/ would be a good home for commands, as its one of the last projects built, and IMO, is more appropriate. This also would not require the build have any major changes, except perhaps addition of cp-jnode be added to distr/build.xml instead of just cp
This is basically what i did with our new awk implementation.
Pros
----
- Commands can be certain that jnode's main projects have been built, and dependencies will resolve.
- It will reduce the size of subprojects (maybe not that much relatively speaking...)
Cons
----
- Man hours, obviously its going to take some time to move everything over and make sure no regressions form. I dont think this is going to be a huge con, and im willing to accept the task. As its mostly a straightforward `mv` this could probably be done inside of a day. Heck i probably could have done it in the time it took to write this
#3
Agreed. Commands don't belong in FS, GUI or Net. Those projects are for 2nd tier operating system level infrastructure; file system implementations, network protocol implementations, gui drivers and the like.
Arguably commands do not belong in the same project as the Shell, interpreters and isolate / proclet infrastructure. The connection between commands and the shell is via a couple of "narrow" APIs.
The intent of Distr is not clear, but I don't think that "core" JNode commands belong there either.
IMO, what we need is a separate 'project' tree for the JNode "native" commands that are currently found in Shell, FS, Net and GUI. This project should also include the "syntax" and "help" packages. If Shell and the commands + syntax are to be in separate projects, some work done on the command invokers (etc) to remove their dependency on 'syntax'. (That would be a good thing IMO. People should be free to use other command line parsers and have them hook into JNode completion.)
As I discussed in the earlier issue, some FS commands (and possibly some Net commands) are a bit problematic because they hook into other FS code in ad-hoc ways. (The commands for formatting and mounting file systems are a good example.) But I believe this can be addressed by creating new FS (or Core) services / APIs that hide all of the implementation details from commands.
If you want to pursue this, turn the issue into a "task" or "feature request" or something. It is no longer a "support request".
#4
So something analogous to the core-utils/fs-utils packages you'd get on linux then.
I'll start with what i'll classify as the core commands. This will include commands from most of the packages. In the fs project, commands that strictly have to do with filesystems will not be included. This includes the various formating and partitioning. I'll work up an exact list of where im going to put what. So save comments on what is being put where until i form that list.
As far as choosing the distr/ project. That was simply for the convenience, but i agree. A separate project should be created and used.
The layout i have in mind at the moment is a coreutils/ project and i plan to use the org.jnode.core namespace, as its not already taken, and seems rather fitting. A basic layout i have in mind goes something like...
org.jnode.core.command.*
Will be home to the various commands. All of which will lie in some subpackage or another.
org.jnode.core.util
To hold any utility classes that are needed. This may include classes that provide access to the tender bits of jnode so that commands do not have to interface with them directly.
org.jnode.core.syntax.?
About the syntax. Do we want to keep command args and Syntax/MuSyntax together? I know they are relatively tied together, at least the basic arg types. But there are arg types spread throughout the project, and if possible, would like to collect them into org.jnode.core.command.argument.
org.jnode.core.help
A straight refactor from org.jnode.shell.help
This will take some time. Especially with all the likely dependencies that are going to need resolving. I'm going to work on this through a git branch, which i'll post a link to once i have it up. Once this is complete, i'll push this upstream to svn all in one shot over multiple commits. I would only ask that no 'major' changes be made to any involved systems. I can rebase my work, so it doesn't matter about little updates and whatnot.
Also, im sure the IDE project files are going to need a rework because of this. I dont use an IDE. I'll give notice before pushing to svn so anyone interested can fix anything that needs fixing. Ideally if someone has git and could do this before i push upstream would be the ideal solution.
#5
Something else i thought of. What about the extension point for syntax? I imagine this would need to change to org.jnode.core.syntaxes to fit its new namespace. And while im on that, what about aliases?
#6
IMO, org.jnode.core.* is a bad choice of package name. The word "core" is not suggestive of commands at all, and besides "core" is already used for the core project ... and the commands (and the shell) are not "core" to JNode (It could easily be reconfigured with no commands or shell; e.g. to implement a stand-alone webserver or a 'kiosk' based on JNode.) Maybe org.jnode.command.* or org.jnode.cli.* would be a better name.
IMO, we also ought to pull all of the external apps into one place to make it easier to identify what is and what isn't JNode. Currently there are external apps in at least three projects that I know of ... four if you include the Sun apps in the "core/classlib" codebase. Putting them all into "distr" would be a good idea. But I think that the partition tool and other JNode specific apps ought to be in either the proposed JNode commands package. Or maybe another new package for GUI-based JNode tools.
#7
Using org.jnode.command namespace for commands. So far i have the following refactored.
org.jnode.fs.command -> org.jnode.command.file
org.jnode.fs.command.archive -> org.jnode.command.archive
All except two commands from org.jnode.fs.command have been moved. mount and eject commands will be factored into a different package. Likely org.jnode.command.fs which will also include the mke2fs and friends commands.
The tree for these updates can be tracked here
The commit and associated diff hunks are here
#8
I'm not sure if this activity is worth doing right right now.
I suspect there are more critical tasks than this one.
This comment from cluster: "i dont think commands belong in any project period" looks like the main motivator of the refactoring. In my opinion a little bit of analysis and logical reasoning wouldn't hurt either.
#9
Ya know, i had the thought that you might say something to that affect.
Care to explain 'why' cleaning up the clutter of the codebase is not worth doing?
And if you think there are more critical tasks, care to share?
This comment from cluster: "i dont think commands belong in any project period" looks like the main motivator of the refactoring. In my opinion a little bit of analysis and logical reasoning wouldn't hurt either.
I'm not sure exactly what you were trying to say or what your point was, but it comes across as just a bit insulting.
#10
All I want to say is that such a significant refactoring would need some prior thinking and a proof that it will result a clearly superior code structure comapred to the current code structure, as much so that it's worth doing it. But you appear to be working pretty hard on 'cleaning up the clutter' already.
#11
More important tasks are trying to fix bugs as many of them as possible. If you are running ot of bugs just take a look at the well over half a thousend failures here: http://testsuite.jnode.bucksch.org/2009-04-26-15-00/results/ .
Identify and fix usuability issues inside JNode.
Regarding the shell commands it would be interesting to have a priority list for what other unixsh commands do we want to have in JNode as a plan of how far do we go on this road.
There are many more of course but we are over three months asfter the last release.
#12
well over half a thousend failures here: http://testsuite.jnode.bucksch.org/2009-04-26-15-00/results/ .
It's the first I've seen of this page. Things like this need to be made more public. Either pull the issues out of that result page and into the issue tracker, or put a link to that bug page somewhere in the sidebars so it can be easily found.\
Regarding shell commands, i have a priority list, and im working on implementing it. This refactoring is something i believed was quite obviously needed. Saying that it should have had some 'analysis' and 'logical reasoning' put into it, Well there was, albeit most of it was implied, as i believed it to be quite obvious.
Anyway, enough bantering.
Is there anything about this you think is 'wrong'. To me it seemed quite logical in and of itself. The commands don't belong in most of the various subprojects becase those projects should be for things like drivers, protocols, and anything else that deals with the operating system resources. Commands are for human interface, and thus don't belong. Also there is the fact that commands are simply becoming far too numerous and spread out. Its getting difficult to keep track of and maintain all the commands. And its simply better for this to be done now, rather than after there are 500 commands to refactor.
At some point, the end game i forsee is pulling the commands/syntax/arguments out of jnode and into a seperatly maintained package. It would simply be included as a jar, or even better as a pre-built plugin into the jnode system. Development is still too volatile for something like this, but this refactoring makes it a very short step to pull this code right out of jnode when it is more stable.
#13
Updates
Commands from fs/net/shell have been mostly moved over. I'm not completely done. I still have to work out some issues like plugin includes and permissions. Everything is buildable, so I'm almost there. I'm imagining that this is going to cause some havok with the IDE files. I dont have an IDE, so...
Log
Tree-ish
Also, im likely not to stick with the coreutils/ name in case anyone wasn't liking that. I'm thinking to go with cli/ as this has basically become a large portion of the cli code. If the plan is later to move things like the syntax parser and arguments into this, then it would seem fitting.
Hopefully will have this ready to commit tomorrow.
#14
These results are easily accessible from any page of the jnode portal (http://testsuite.jnode.bucksch.org/2009-04-26-15-00/results/)
You go to 'Status' in the top menu and from there to 'Reports'. There you will find various reports and the testsuite run results are among them.
Did I mention already to include the Bjorne tests and other shell related tests in the testsuite?
Yet an other noteworthy task... Some protection against regressions would be helpful to you too.
#15
Regarding post #13.
I have scanned this thread again with the hope to learn where exectly are you moving over those fs, net and shell commands but I'm still not sure about it. The last thing I would like is to learn it from your commits. Such changes affect the code and team members significantly. This is why I do request a more serious planning, discussion and timing for such steps. What looks obvious to you it may not be so for others. I hope you also note that when you redesign a shell command you were working on that is up to you but when you are reorganizing a lot of code of many people without even telling it how that is a more serious thing where others might have an opinion.
I'm trying to save some time for you by discuraging you to do work which might not get through especially on knowing how painful even a simple refactoring can be when done by hand instead of a state of the art Java IDE.
#16
Levente: look at the links Cluster provided in comment #13. They are GIT logs that show what files Cluster is moving, and where they are moving to.
Cluster: it looks as if you are leaving the "syntax" and "help" packages in the "shell" tree for the moment. IMO, this is a wise move (!!)... whether or not we decide to leave them there in the long term.
I looked at what is involved in separating shell + invokers + interpreters from syntax + help and its a tricky job. You've probably noticed that I've already created a separate issue for it ... on the assumption that we do want to do it. We haven't come to a firm conclusion on that point, but I think that it is conceptually the right thing to do.
#17
I have scanned this thread again with the hope to learn where exectly are you moving over those fs, net and shell commands but I'm still not sure about it
Post #4 The layout i have in mind at the moment is a coreutils/ project which has a had a simple renaming to the cli/ project.
This is why I do request a more serious planning, discussion and timing for such steps
Step 1) Refactor core jnode commands to the cli/ project (almost done)
Step 2) Untangle the syntax/arg/help APIs from the shell (in progress)
Step 3) Refactor step 2 into cli/ project
Step 4) Stabilize cli/ project
Step 5) Move on to greener pastures
Sometime in the future) Remove cli/ project from jnode trunk into its own having it included like any other application.
I'm trying to save some time for you by discuraging you to do work which might not get through
I still can't believe how you don't see how obviously necessary this is. I seen that this had to be done weeks ago.
how painful even a simple refactoring can be when done by hand instead of a state of the art Java IDE.
Bash scripts > IDE
#18
OK, that was a serious oversight by me last night regarding the Git logs but the rest of my post still applies.
Now about moving all commands to one place...
According to my uderstanding in JNode commands were not intended to be kept in one dedicated project by design and for good reasons. Generally speaking dedicated shell commands might be created for any module and component, plugin, module or whatever entity you want. They belong to it and are related to it so they are prefered to be in the same project as the related component and not in a separate project. This is why there is complete support for commands in the plugin descriptors by default. If you want to put all commands in one project then why do you need support for them everywhere?
The current support for commands is quite flexible and it's worthy to make good use of that felxibility.
The coreutils project name looks a bit too generic and unexpressive to me, something like where you can throw many things. Do we really need it?
#19
This isn't going to remove any flexibility, and yes there still will be some commands that are spread throughout the projects. If you browse the tree-ish you will see that the filesystem formatting commands are untouched. AFAIK much of that code is still unstable and it will be left alone for now. Also there are driver commands that are not being touched. Commands that are in distr/ are untouched as many of them dont belong in this.
Regarding the coreutils name, i agree, it was too generic, as it will eventually include args/syntax/help systems once they have been dealt with. And, in my last post, chose to rename it cli/, as this will be the base code for the command line interface, separating it from the shell code.
#20
This is the last im going to say about any of this, but regarding some of your comments...
Where is the lengthy discussion about refactoring classlib? I dont see any issues such as this one regarding it, with updates on its progress for the public to consume. It seems as if the classlib reorginization has caused more headaches then it helps. The whole point of this issue is to make the refactoring public so that people can see what is happening and comment on it. If there is something that doesn't jive then anyone and everyone is free to comment as to why its a good/bad idea. I don't remember seeing such a discussion over your classlib refactoring. Something like that is what i would consider a major restructuring, while this is quite minor in comparison. And yet you seem to think that this requires serious discussion but yours doesn't? If your going to make rules for this project, then the rules apply to you to. If you dont follow them, why should anyone else?
Again, im done discussing this. Please post some positive constructive criticism, or i will simply refrain from discussing my work on this forum any more. I'm still angry at you over the way you simply dropped into this issue and blantly called it 'unimportant'. As far as im concerned, its my time, i will choose what is important. And if what i'm doing is unimportant to you, too bad.
#21
You know cluster, the classlib integration (among others) was my zip command ...
Plese post here anyone who has contributed to the classlib also tell it how much...
#22
Now your hijacking my issue.
Do NOT use this issue thread for any more of this senseless bickering.
#23
it looks as if you are leaving the "syntax" and "help" packages in the "shell" tree for the moment.
Exactly. As your untangling the syntax/arg/help code, i'll refactor it into the cli project.
Also, as i've kind of hinted at, can we separate arguments from syntax? From what i've seen, it seems like the Syntax/MuSyntax is in ways dependant on the arguments, but not vice versa. What i have in mind is to separate the arguments into an org.jnode.command.argument package. The syntax is still going to have dependencies on the arguments, which is expected and fine. But if there are other syntaxes created, at least they too will have the Arguments as a separate API that they can interface with.
Just to be clear. I see commands and arguments as more or less inseparable. At least i don't see any good reason for them to be abstracted from each other. So i'm not implying this anywhere. But i think as it stands, i should be able to refactor the arguments into this project as well. Though that will be for after the commands have been dealt with.
#24
Ok, the final commit has been pushed to the git repo. If anyone notices anything let me know and i'll fix it up. If there are no objections, i'd like to get this out of the way and pushed upstream, so i can do something 'more important'.
To clarify.
This refactoring is moving a large number of commands from the various projects into a 'cli' project. The purpose of this project will eventually be to house the various systems that attribute to JNode's command line interface. This will likely include commands(in this issue), command arguments, syntax parser, and the help system. The namespace being used is org.jnode.command, and includes the following packages.
Anything that deals with archiving or compression goes here.
This is for most of the commands that deal with files. Wether it be organizing of directory structures searching or modifying of files.
This is one of the larger packages, as it contains the majority of the commands that deal with displaying and setting of jnode system resources.
Commands that deal with network interfacing go here.
This one is pretty empty at the moment, but will be the home for development tools and commands. The ant command is located in the ant subpackage of this package.
This is mostly a collection of commands that didnt belong elsewhere. They are 'common' command line utilities.
This is not for commands, but rather for utility classes that aid commands.
That will be it for now, as there is more work to be done before more refactoring to this project can happen.
#25
#26
Submitted. I'll leave this active for a couple of days to deal with any related issues.
This is going to require the IDE related files be updated.
If you get any ClassNotFoundExceptions, let me know, thats an unresolved import in the descriptor. I'm pretty sure i got them all sorted out.
Same goes for permission exceptions.
#27
I've added / updated the necessary Eclipse project and build path metadata. Eclipse builds are now running clean (no errors) for me.
Some empty directories have been checked in as trunk/coreutils/src/commands/org/jnode/command/util. I assume that this an accident ...
#28
Good to hear.
I noticed that svn dir in the build log on the site. I'm not sure why that got left. I think it has something to do with the difference between git and svn in that git does not track empty directories.
#29
org.jnode.command.devel would would a better name than org.jnode.command.dev for development releated commands.
basename and dirname conceptually belong to org.jnode.command.file.
#30
I've removed the empty dir (trunk\coreutils) from svn
#31
org.jnode.command.devel would would a better name than org.jnode.command.dev
Ok, will change
basename and dirname conceptually belong to org.jnode.command.file.
Those and a i think there are some others that could use a proper home besides 'common', which is a kind of catch-all package.
#32
IMO, cluster is correct about basename and dirname. They are actually operations on strings ... which may or may not be names of actual files / directories.
#33
#34
Manually closed.