plugin/ driver/service managers integration
Submitted by Valentin Chira on Mon, 04/19/2004 - 12:18.
I was thinking now about a discussion that I had with ewout some time ago about the way an feature OS should be like and started to look around the code. We agreed that there should be no applications or drivers or plugins but only services and service providers. My question is now how is this true for our current implementation? Is plugin manager/ drivermanager dufferent modules? Can we maybe merge this modules into 1 module called service manager? what about the plugin loaders? Maybe we should call them service loaders and should work with service providers which can be links like they are now for plugins. I have a dream..I dream of a world where the companies don't buy monolithic systems anymore, but services...a world where the code is truly reused and there is only 1 OS: JNode all the pother companys write services...a world where the OS loads its modules from different service providers..a world where the OS knows how to solve yourt problems and doesn't ask for user imput for every action...a world where we are free to have the OS that we need and don't install al the junk just to open once a word document...I have a dream..
- Login to post comments
Are you dreaming of Jini?
Hi Valentin,
your thoughts sound very familiar to what the Jini community (http://jini.org) wants to accomplish (I'm a Jinian, too
).
They also have a project called 'surrogate' (at http://surrogate.jini.org) that allows devices to advertise as Jini service even if they aren't able to run a JVM. AFAIK it also allows default drivers (in the form of Jini services) to be loaded when the device doesn't supply it's own. There are plans to supports various interconnects (i.e. IP networks, Bluetooth, USB ...).
Maybe it can be integrated into JNode.
Sebastian
Want some suggestion from you
Hi, Monochromata,
I'm new to JNode, and would like to have some suggestion from you on how to master JNode. How did you study JNode besides reading the simple handbook (which even does not present an architecture pic)? Code reading? then where should I start? and what's the premise? Could you give a roadmap of the src of JNode? I think every JNode newbie needs that
Thx
Birkey Ma
----------------------------------------------
Beijing, China
----------------------------------------------
Things should be made as simple as possible -
but no simpler. (Einstein)
How to start
Well, that depends on what you want to do with Jnode. Of course you will have to read much code.
You will first have to learn about the build process since this is the way to put your code into Jnode currently. This will also cover choosing a way to boot Jnode. You can use an Emulator like VMWare or run it on a real PC. If you choose the latter option you can boot from CD-ROM, from the network or (using a CompactFlash-to-IDE adapter) from a CompactFlash card.
The following steps depend heavily on what you are trying to achieve. E.g. when trying to run Jini on Jnode you will first have to run a simple test program to figure out if Sockets, ServerSockets and MulticastSockets are working ...
Sebastian
Is it also related to 'network devices' ?
Is it also related to 'network devices' (printer directly connected to a network, ...) ?
I have seen their site and it looks interesting but it seems that's
under a sun licence. If so, we can't use their code/implementation (I think).
I have tested their cpd that detect copy and pasted code with the JNode sources and found a lot of copy and pasted code. That's can help us to put this code in a more appropriate place.
Network Devices
Yes you can also implement network devices using Jini. But the cool thing about Jini is, that everything becomes a service. Every service implements a Java interface that specifies how it can be used. It's really flexible. Most people think that Jini is for devices (which is what Sun told everyone when they introduced it in 1999) but you can also implement 'software services' using Jini. I.e. in the scenario outlined by Valentin you could have a 'software service' that is accessible via the network (acting as some kind of service provider) and is able to display Word documents. On the other hand you can also have a printer that advertises itself on the network as a Jini printer service. (Well, no such printer exists yet
).
The licensing stuff is a bit complicated. There is currently an ongoing debate about it in the Jini community. But Jnode should be able to use the Sun implementation. Indeed this would mean that all Jnode user would have to accept the Sun Community Source License (SCSL) license for the Jini stuff. The SCSL is intended to ensure compatability among different implementations of the Jini standards. Those parts of Jnode that would only use the Jini implementation might still be able to be under a license of your choice (LGPL is used by Jnode, I think). The license point might need further investigation, but it won't prevent using Jini in Jnode.
I just had a look at the cpd project. Seems to be nice, but as far as I can see they don't use Jini features (at least the project page states the project is currently non-distributed).
Sebastian
Components
Hi,
For several months now I have been about the buisness of writing a powerful, flexible component architecture for jnode. Some of you may remember my discussing it in other places. After I finished a preliminary version of that system I began to run tests and found that the constraints of the system were to difficult to program around. Nothing of any great usefullness could be made.
So I re-examined my foundational theories and tried agian. This new system is much better that the first. Now I monitor this discussion and I wonder if my system could not be better.
Jini is not the answer to our problem. And I believe that sortly, I will be able to show that conclusivly. Jnode's needs are specifice and unlike any befor it. We will not find a solution in any single, pre-existing software.
This is what I suggest, we should compile a list of the requrirements of the jnode component architecture. Comments should be concise and definiit. I will add my comments when I am through here, you may use them as a model.
When we have finished the list, we will all be better able to descuss what systems we need, and from where we will get them. We will se where jnodes current system is lacking, where jini falls short, and what any composit system (like mine) must do.
Once again, please keep comments short and clear.
Thanks.
Integration
I would prefer any approach that allows Jini and other services to be integrated seamlessly (must be able to discover them). I don't know the requirements for local components, but I would like to see easy integration for ones from the network.
Please feel free to outline why you think Jnode needs a radically new component model and what it will look like.
Regards.
Sebastian
Neo-Component Systems
Jini is a destributed component architecture. Jnode is a local os. This is the fundamental diference between the two. This differnce shows itself primarily in Jini's use of RMI.
All Jini services are required to be rmi enabled (extend UnicastRemoteObject etc.) This structure is not neccessary for jnode components; RemoteExceptions do not occure during local method calls.
Neither would it be wise to proxy jnode components to RMI (Jini) components; the reason RemoteException must be declared is so that the client of a remote service will be alerted to network failure.
Mostly for these reasons, and for a few smaller ones, it would be unwise to force direct intigration of Jini into the jnode components system, this does not mean that Jini cannot be used, nor does not mean that jnode's component system must be isolated to local components (keep reading). It takes aproximatly 10-20 lines for a Jini lookup, depending on neatness, and nearly all of that may be abstracted into a utility class (which I recomend).
I hope that my reasoning on this is understood.
Now, having been invited, I would like to present my reasoning about what jnode's component system should do.
Jnode's components are unlike any other (except languages like perl) in that they include the idea of libraries. Libraries may be thought of as components that dont do anything by themselves. They are only a resourse to other components. Libraries add greatly to the ideal of code reuse. If a group or person will produce one small set of classes that fullfil a specific purpose and does it well, then those classes may be used by any interested party. This is code reuse. A good example would be a set of Collections.
The smaller the components, the more code reuse, the more flexible (and logical) the os will be.
Elsewhere in this discussion I stated a list of key features that must be present in jnode. I hope and will assume that no one disagrees with these points. I will re-state some of them here for discussion.
Must manage multiple versions of a component.
Versions are inevitable, it cannot be stopped. Jnode must have a plan to handle them.
Must manage dependencies on specific versions of components.
Must manage class loading, especially of specific versions.
This is key. A component must be able to specify what version of another component it is dependent on, and the classes of the proper versions must be loaded and used.
This idea and that of Libraries, are the foundations of my theory and my code.
My theory is this; that jnode plugins are assemblies of java classes (jars) that come with internal documentation. No restriction is placed on the struture of the plugin (unlike a Jini service). The documentation includes info about the plugin (name, provider, version), the type of plugin, and the dependencies of the plugin. This is of course the same as jnode's current system. However, the dependecy information is used directly by classloading systems. For this reason dependencies are stated by class name and version. (It will be possible to import whole packages.) As I say, version dependencies are used during the class loading proccess. If the proper version of a class is not found a ClassNotFoundException will be thrown regardless of if there is another version on the system.
Must manage install, removal, and update of compoents.
These three proccesses involve two ideas, first the transfer of the actual plugin jar data, and second the manipulation of external configuration states. Uninstalling a plugin requires the removal of the plugin jar and the removal of any data that plugin was using.
The opposit is true of installs and durring updates, that data must be transfered or translated.
This data manipulation cannot be done by the component system directly. They must be handled by the plugin itself. In my system I have designated a not-library a service. Unlike a library a service does do something usefull, and that useful thing may be persistent. Thus in the documentation for a plugin of type service there is an optional data-agent attribute. This should be the relative jar-path to a subclass of DataAgent. DataAgent defines methods install() uninstall() and update(DataAgent old); These methods are called by the component system at appropriate times. (This is not yet fully implemented, I am working on it as time allows.)
Must provide shutdown signal to components with threads.
One other attribute of services is that they may be running threads. These threads must be given warning of a shutdown so they can stop.
For this reason services may supply the optional attribute shutdown-manager with the relative path to a subclass of ShutdownManager. ShutdownManager defines shutdown(); This method should end all running threads for that plugin. (This is not yet implemented, I am working on it as time allows.)
Must provide a means to expand the component idea to net services.
This is the interesting part, and ultimatly the reason for this descussion. In order to adapt in the age of the network, jnode's component system must provide support for net services. I define net services as any service that is not hosted on the local machine.
I was puzzeled with this problem when first I was designing my new model. Then this series of thought occured to me.
The proccess of loading the classes of a pluggin produces exact information about what version of what class is needed. That exact information is compared to the contents of other plugins. If a match is found the class is returned. Thus requrements are specified presicely and plugins are searched precicely. Localy.
What if it wasen't local?
The current version of my component system does this: Upon recieving a request for a class and version pair, it searches the local system. If no match is found, it then sends a udp datagram containing the class and version on a multicast chanel. This datagram is recived by other component managers on the network. These systems use the information to search their local systems if a match is found, the entire containing jar is sent to the originator of the request. The originater installs the plugin jar and uses its contents to fill the original request.
In this way dependencies are handled automatically; the user will never have to think about what a plugin needs, because the plugin gets what it needs. Also the system will only install components it is trying to use, not "all the junk" or bonus software.
New downloads and net services are both accomplished by the use of a special URI. For example:
'jnodepl:ClassName!vX[#arguments]' where ClassName is the name of a class and vX is a version number for that class. This information is used to lookup a plugin in the component system. This results in the download and install of the pluggin jar that contains that class. Next 'arguements' may be passed to the main method of ClassName. The main method should do whatever is appropriate, start the program, thank the user for downloading, or connect to a net service using the argument values.
This URI would need to be specially handled by a jnode browser.
I hope that I have been clear. If not please ask questions.
I joined this descussion both to give and recive help, so let me know what you think.
Thanks.
Class loading caching
One thing i've been thinking is that we should get away from the conceptual model of a computer we install things to. What I mean by this is that environments are stored and loaded automatically. As a user I can log into any jnode box that has the same user server and my system will follow me around, with all installed apps and dependencies. Ok if I use a different computer it will be a lot slower at first. In this scenario the hard disk just becomes a conceptual cache. For some types of applications this might not be possible, ie a game that required the graphics card to support some advanced feature, but in general this approach should be possible.
Anyway sorry i'm digressing, the UDP idea is great. How many times have I done a network install of Linux knowing full well a local Linux machine has all the binaries I want, yes I know I could set the local box up as an install server, but wouldn't it be great if this behaviour was automatic. Security could be built in with digital signatures. So I like the UDP idea except what if the box that has the class file we want is on a different subnet? Perhaps it would be better to delegate the searching to a classfilefinder plugin, that could search for servers etc. The plugin could keep stats on network latency and bandwidth between nodes etc. We should also build in the capability for parrallel downloads, so we can get the same file from x different places and glue the results together.
about the downloads ...
The downloads could also be made on a pear to pear basis : each node may act as a server and give a list of other nodes he know to its clients.
About the parallel downloads, since a long time I dream of a library that can do it in JAVA. I think the NetComponent library (we already use it in JNode) is a good basis for the protocol implementation part.
For these kind of downloads, we also need to do a 'checksum' on the downloaded files. However, JNode must treat these as dangerous code (virus, corrupted bytecode ...) ... maybe with isolation and a special SecurityManager (to avoid access to system parts).
Why checksum downloads
Why not use digital signatures? With a digital signature we can be guaranteed that the file hasn't been tampered with no matter what the source. Its far better than checksumming. The originator of the classfile provides the digital signature (which is tiny). All other downloads of the classfile are checked against the digital signature of the originator. The sweet thing about this system is even if a JNode system on a network is compromised and classfiles are tempered with so when requested they will compromise the requesting systems. How can the hacker temper with the file in such a way that the digital signature will still match, they can't. Also When each classfile has a digital signature we can boot a box from say a CD-ROM which run through JNode and checks each classfiles digital signature against the digital signatures stored on a server somewhere. They way we can find quickly if a box has been tampered with.
Yes, no need for checksum
Yes, there is no need for checksum for these downloads.
When I talked about this, I thought about my own experience with downloading linux distro : ftp servers give generally a checksum file with the ISO of the distro.
So, it's better for JNode to use digital signatures.
Peer download
I was thinking of using JXTA for the peer work.
It has systems for using http proxy to get arround nats and fire walls. My early research says its possible.
What do you think?
Re: Neo-Component Systems
Just to update you knowledge of Jini:
1) No service is to required to extend UnicastRemoteObject or to otherwise perform remote computation. It is perfectly legal to create local components / services that are Serializable and perform all operations locally. I.e. there is no restriction placed on the structure of a plugin.
The reason for most methods of Jini services to throw RemoteExceptions is to be able to implement the service either as a local or remote one. This allows seamless integration of local and remote services into one component management framework.
2) It is possible to discover a Jini services with just 6 lines of code:
try {
ServiceDiscoveryManager sdm = new ServiceDiscoveryManager(null, null);
ServiceItem item = sdm.lookup(new ServiceTemplate(null, new Class[] { ServiceType.class }, new Entry[0]), null, 3*60*1000L);
sdm.terminate();
} catch(Exception e) {
}
I still think that Jini would do well acting as the foundation of the component model to allow remote components to be integrated seamlessly. Nonetheless many of the problems you pointed out (e.g. versioning issues) haven't been solved by Jini yet, too. But I'm sure they can be solved with reasonable effort by using the extension mechanisms provided by Jini.
I'm currently working on some Jini projects that deal with creating a Jini client-side component model. This model will also have to deal with the issues you pointed out regarding threads started by a component etc.
You approach to classloading is very interesting too. I'm currently prototyping a system that also loads classes from the net. In this system, no jar files need to be created because the class dependencies are processed (using the BCEL library) when a class is requested from the classfile server. Additionally I thought of using a delegation hierarchy similar to that of the DNS to find classes, since classes are already placed in packages that resemble domain names.
Why do you think components should be libraries like in Perl? Why shouldn't there be components that do perform a real task i.e. a Mail application component?
Regards,
Sebastian
Jini
I have worked with Jini; I have seen it in action. It accomplishes its goal excellently. My point in all of this is that this goal does not align with ours. Jini is used to make a flexible, adaptable, selfhealing network. We are trying to make a system that builds up the os.
When Jini downloads (by serialization) a live service its classes are loaded by RMI procedure. When the service is out of scope it will be reaped by the garbage collector. The classes for that service will be reaped then or latter when the system shuts down. The classes are not permanent, thus they must be downloaded again the next time they are needed.
This was the driving force behind my remote pluggin download system. Once the service is used, it becomes part of the system. Pluggin jars are sent in whole so signatures remain intact and will be used to dictate the privlages of the code. Because the service has become part of the system, it may be reference (looked up) just like every other part; by class name. Lookup is accomplished through class loaders.
This idea is not found in Jini, because it is not a part of Jini's goal. What Jini does is connect destributed components, in a flexible way. Jini is an excellent choice for connecting to a remote printer or maybe a network filesystem. Jini should be used by components to communicate with services that will remain remote. You have shown that it takes only six lines to find a Jini service. These six lines should be used to connect to remote systems, not to run the local component archetecture.
Alex
Re: Jini
If you don't want the classes of the service to be downloaded every time you can use a caching classloader that writes the downloaded classes to disk.
For services that don't require a backend it will also be possible to cache the service too so it is available from the cache during network outages.
Just to get a clearer understanding of what you what to achieve: What sorts components are you thinking of? Do you mean e.g. the browser application or the X server? Do you also think of drivers and OS-level things to be components? I think Jini is well suited even for local components because even those are always downloaded from the network.
I think it would be cool to download a Jini service acting as a video driver. You plug in your new video card, Jnode asks the Lookup service for a matching driver and gets it. Of course the video driver service wouldn't be a remote service. The service and its classes could be cached persistently (maybe we can even have a CacheDescriptor saying that the driver won't be updated within the next week / month so you don't have to look it up again.)
Sebastian
Re: Jini
I view every thing that is not a core part of the os, as a component; libraries, services (apps), and drivers. Everything.
We are talking about the same thing. The only difference in our oppinions is that I think we should make a new system to fit our goals, and you think we should do a large re-write of an existing system so that it will fit our goals.
It is unwise to use a rake as a shovel. I believe strongly that we should use concepts from Jini and other systems to make an architecture that fits Jnode perfectly.
What do you say?
rake and shovel
I think it might take rather much time to create a solution that perfectly fits Jnodes needs. Therefore we could use what Jini has already acheived (after 5 years of development!) and customize it to Jnode. Maybe we should figure out how many pronks the Jini-rake
has?
I have to admit that your and mine position are contrary but both perfectly legal. I think a competitive approach will help to find a good solution. So how should we proceed? Maybe we can do some more detailed sketching / design to see what needs to be done. I think I will use the weekend to create a draft of adapting J***
to use for a component model for Jnode.
The reason why I adhere to Jini is because I think it's dynamic nature can solve many problems generically that would otherwise needed to solved again and again.
But thanks for your opposition. I think it's useful to think about whether Jini needs to be everywhere, even for me
Have a nice we! I look forward to continue the discussion but I will have to think it over again, first.
Regards,
Sebastian
Competition
This is a prudent descision. Let us go our separate ways for now, and report here with developments.
I apreciate your good humor in this matter.
Thanks.
Alex.
i agree
I agree with you that JINI component model is what we preaty much need. Maybe you can write a draft about this..and how we cou integrate such a component model in JNode. What do you say?
I will write a draft ...
... but please give me some time. I will create a document describing what I think is currently possible with Jini. This will add to the discussion and, as I wrote in my earlier post will lead to some open questions jalynch did already point out when writing about component requirements in general.
Jnode's component architecture
Jnode's component architecture:
Must manage multiple versions of a component.
Must manage install, removal, and update of compoents.
Must manage dependencies.
Must manage dependencies on specific versions of components.
Must manage class loading, especially of specific versions.
Must provide means of purging a component from the system (for devel)
Must provide shutdown signal to components with threads.
Must manage the storage of installed components in a secure location.
Must provide a means to expand the component idea to net services.
Local Components and services
Hi,
I agree completly with you.
About the Jnode's component architecture:
this is the same as what I'd like to have also in the JNode System (Service) Installer. But also adding some Security Features.
About JINI:
I know JINI very little, but I don't think it's the right solution (it could introduce a level of complexity that it's not wanted for some local Services like that of an OS).
But we could make an adapter to manage some other Services (non-critical in the OS), all JINI-related.
For example, if we would implement a distributed Domain (Users, etc. account for a LAN, using LDAP) here having JINI for these Services could be useful, but later, in a second stage.
For Generic Local Services, I've seen the Jakarta-Commons Daemon ... could be a good starting point for our Services.
Bye,
Sandro