Plugin engine in use?

Hi all,

Just joined the group. A buddy of mine and I have developed a plugin engine similar to the eclipse engine. It's about 40K in size with an ebmeded xml pull parser, it is based on extension points and extensions. We deploy plugins as open directories OR .par archive files. .par files are jar files with a plugin.xml at the root, a /classes dir for plugin classes, and a /lib dir for any embeded jar/zip libraries the plugin depends on or makes available to dependent plugins. At runtime, our code pulls .class files out of the .par /classes and out of embeded /lib jar/zip files and creates classes on the fly, as opposed to requiring to unzip.

The reason for this post, my co-developer, Evert, told me you guys are also using an engine similar to the eclipse engine. I want to get the scoop on how you guys are actually handling plugins, how you add new plugins, etc? I also wanted to offer any help. I have become very good with classloaders. In our engine, we load classes like so:

* if (classname == java. or javax., use System.class.getClassLoader() (static ref in the engine) to use the boot loader to load classes.
* if (classname == gpe. or kxml. or xmlpull., use the parent loader to load the plugin engine specific classes.
* If we get this far, try loading from plugin classpath, including the cache of the super URLClassLoader (first), then along the super URLs added as the plugins classpath including the /classes directory and any jar/zip files in the /lib directory.
* If we get this far with no class, we next check any dependent loaders. At load time of plugins, we resolve all dependencies by adding each plugins classloader to a list of dependent loaders for a given plugin's dependency list. This allows faster runtime lookups without requiring to go to the engine at runtime to find and verify a dependency and get its loader then.
* Lastly, if still no class is found, we call the parent loader, which can be set to a "shared" classloader for all plugins, or the loader that loaded the engine.

I have not looked at any source yet for jnode, I plan to do so. The purpose of our engine is to allow any java application to have what I believe is the best plugin engine available, in that following the lead of the eclipse engine using extension points and extensions, it is super easy to add plugins to specific points, many if need be. It is a superior design over anything else I have seen thus far.

We do support dynamic xml for extensions, although we dont validate xml chunks against a dtd or schema like eclipse does. We opt for small size, 40K without debug info, about 33K without debug info and obfuscated with yguard (for size only). While one goal is to make it ready for J2ME, J2ME doesn't support the notion of classloaders, so I dont know if we'll venture down that path anytime soon. I am hoping J2ME will eventually go away as PDAs now support 1GB of memory or more with plugin-in (no pun intended) memory cards.

Anywho, I am interested in your approach to using dynamically reloaded plugins. Our engine is "almost" there with reloading and unloading of plugins. More so, we are building a full UI framework around the plugin engine, similar to the eclipse RCP. We'll have help, preferences, user access levels, file i/o dialogs, components like wizards, status bar, outlook style bar, etc.

I invite anyone to check out the project at www.sourceforge.net/projects/genpluginengine. We don't have a good catchy name for it yet, but currently we use my old Theseus name that I had for a MVC web framework I wrote. We are adding some more features as well, such as automatic dependency resolution, automatic event/listener resolution, etc.

If anyone can fill me in on the details of how the plugin engine workes within JNode, point me to links that explain it, or want to shoot an email offline to discuss it I would really appreciate it.

Thank you.

I got few infos, but think you can help us.

Hi,
I don't know much about our plugin engine, but I think we have similarities with eclipse plugin engine (extension points and extensions).

I think you can help us with your knowledge of classloader.
Do you handle plugin version and isolation ?

For isolation, the main problem is the static members that must be duplicated in different classloaders. Have you tried doing such a thing ? Maybe it's not possible with only different classloaders.

However, your help is welcome.

Fabien

isolation

Here you can find some information about isolation:
http://research.sun.com/projects/barcelona/

They create a read only Class that contains no static code, what class is shared by all applications, static fields and static methods are in separeted classes were every application has his own copy of.
So if you have a class Hello.class, Hello$sfields.class contains the static fields and Hello$aMehods.class the static methods.

I am working on the same project as buckman1 (I am the co-developer he is talking about who wrote the par code).

I am interrested in JNode, maybe I can help?

Isolation?

Hi there,

I am working on version stuff now. Basically, we have a 3 point version. x.y.z. If a plugin depends on, or uses another plugin, their versions must match the rules. The problem is, if you build your 1.0 plugin against a 3.3 plugin, you need a way to specify the minimum plugin yours depends on. I am working out those details still. It somewhat works now.

I am not sure I understand isolation though? I can tell you this, just today my codeveloper and I have been talking about automatic dependencies. The thing is, anytime a plugin with an EP (ExtensionPoint) goes through its list of Extensions resolved to it, each of those extensions implement an interface provided by the EP plugin, so each E plugin's classloader must have a ref or a way of asking the EP Plugin loader to find the interface the EP defines (provides) that each E implements. After many an hour watching the output of loadClass(), resolveClass(), etc we noticed something we should have known, but anyway when the E class is found, it implements the interface in the EP, before the E class can be resolved the JVM now needs to find the EP interface, so another loadClass call is made on the E's classloader. That is why each E's loader must somehow gain access to the EP's loader to find the interface.

Anyway, long story short..er short for me, we came up with a method called getExtensionClass(), and it automatically added the EP's loader to the E's loader as a "delegate"... a special purpose one-time use loader. At runtime, the E would use this delegate loader to find the interface the EP provides. Case solved..now the E and EP work great..

but there is a problem. What happens if your EP wants to provide a way for each E to control it? Imagine that an EP plugin provides a menu bar API, in other words, the EP isn't really requiring each E to implement an interface the EP provides. Instead, the EP wants each E to get a reference to an implementation of an interface BOTH of which are in the EP plugin. That is, the EP provides an interface AND an implementation of this interface. It passes its implementation as a "callback" to each E, so that each E can make calls BACK to the EP implementation of the interface it provides. So, each E can control the EP as needed. Perhaps the E adds a listener somewhere else, and when that listener is triggered, it uses the ref to the EP implementation to do something. This is like a service, basicall. As an example, an email service plugin provides an interface AND an implementation. Other plugins don't "extend" it per se, they want to USE it. They want to control WHEN they use the email service plugin, NOT wait for the email service plugin to request each E to use it. It's sort of the reverse of how eclipse works its plugins.

One problem, and I think you may be insinuating this, is how does one plugin USE another plugins classes, make instances of it, etc? In Eclipse, I have noticed the way to do this is provide a static instance method getInstance(), and then plugins that your plugin with static in it, can grab a static instance ref directly. There is no way to prevent this even with my engine. However, I find this troublesome to allow simply because the dynamic ability to load/unload/reload plugins at runtime could fail if you have a plugin that gets a static ref and never gives it up. Without using AOP or some sort of interceptor on every method of every class in every plugin, there is no way to know when one plugin makes a static ref to another and ties it up...such that that plugin could never unload/reload and thus halts the ability of the engine to properly reload a plugin at runtime.

Even with all this said, there is no real way to prevent this. Reflection can also be used to allow any plugin to use any other plugins code.

My solution which worked in the past, then I changed my engine to work more like Eclipse, but am changing it back, is every time any plugin E resolves to anothers EP, the EP plugin classloader is added as a ref to the E's list of dependent loaders, and vice versa. The reason is, this allows at runtime any EP class to use an E class, and any E class to use any EP class. It provides the ability for callbacks to work, thus avoiding the need to require static refs to plugin classes. It is still a developer learning issue. Any plugin developer can "break" this rule if they wish, but hopefully with good documentation, developers will find it useful.

What it will also allow is for my engine, at runtime, to automatically connect dependencies through various plugin engine methods. For example, a plugin can be compiled against another plugin but not declare a dependency to it in the plugin.xml manifest. At runtime it will ask the engine for the plugin, and if found, use it. The engine, if found, will interconnect the classloaders. Anytime either is unloaded the other gets a dependency unloaded event. If done right, most plugins that extend EPs via E's will not bother with dependency events, but EP's might. If an EP's E is unloaded, an extension removed event is sent, so that the EP can do any cleanup of resources so that the E plugin can be unloaded if that's the course it is going.

Phew! I don't even know if I touched on what you were asking..?? If not, please fill me in a bit more, I'll see what I can figure out.

Ya'll are more than welcome to disect my engine, see if any of it can be retrofitted into your core.

Plugins

Hi,

For some time now I have been working on a plugin manager to replace the current implementation. This was my purpose from the begining, so I acconted for ideas like versions, isolation and reloading. Also included are the neccessary ideas of data management during install, unininstal, and update. Also included is a shutdown signal (for plugins with threads).

Our systems are not very differnent, thought yours may be further allong. I use an xml descriptor to specify dependencies. I have used class names (soon package names also) to specify dependencies. In this way a plugin can clame dependence directly on the classes that it needs, not on some alternate name.

Plugins are loaded in a contained classloader environment. The classloader of any class in a plugin has the PluginManager (formerly ServiceManager) as its parent loader. Thus when a plugin class resolves a reference to another class the loading procedure passes through the PluginManager, which performes the neccessary comparisions.

This is the primary reason that dependencies are stated in terems of class names, so that explicit and implicit dependencies can be compared.

Version compatibility can be expressed in definite (1.4.2_6) or abstract terms (2X or 1.4+ or 2-). All explicit dependencies (in the xml descriptor) must define the version of the class that is wanted. Implicit dependencies are asumed to be Version.ALL, which will return any version of the required class.

Every plugin is maintianed in its own class loader thus static field isolation is accomplished automatically.

Because dependencies can be assesed dynamically, and by class name, any plugin may reference any class of any other plugin in normal java code. For example:

public class MyClass //in plugin A
{
public MyClass()
{
SomePluginLibrary.staticField++;
//reference some class in another plugin.
}
}

In this case the neccessary plugin will be loaded from storage and placed in a new classloader. That classloader will maintain isolation for the static fields contained in that plugin.

This is all working quite nicely.
Reloading plugins is more relative in the PluginManager. When a plugin is uninstalled, any other plugin that has a reference to it is not asked or forced to release it. The plugin (the first one) is simply removed from the controle area so that no other parties can gain references to it. When the referncing parties finaly do release the plugin, it and its classloader will fall out of scope, and will have been completely removed.

The PluginManager also has systems to loacate plugins that are not currently installed on the local system. You can see more at http://jnode.sourceforge.net/portal/node/view/240.

As for disecting your engine, I am confident that we all would prefer that you join us in deciding what the perfect plugin manager would be and help us implement it.

Thanks,
Alex

re: Plugins..specific question..

Hey Alex,

Reread your post. You mention that a plugin developer uses other plugin classes as if they were not plugins. If one plugin depends on another, that is fine. But how do you handle a plugin that provides an extension point and goes through all the extensions resolved to it? If each extension provides a class that implements an interface the EP provides, then the EP code has to grab the class from the Extension somehow, instantiate it (ideally the Extension classloader instantiates the class it provides that implements the EP's interface), and so on.

Sooo.. in this particular case, I take it you do NOT just program in normal coding mode..you have to use dynamic features of the engine in some manner to get an extension class instance for the EP code to use it? Or does your engine not even support this type of coding?

Thanks.

Control flow

Hi,

This is one of the major diferences between our systems. The PluginManager makes no attempt to controle program flow. No atempt is made to activate the plugin, nor are there any structural requriments for plugins to comply with.

All plugins are peers; there is no such destinction between E and EP. All classes may be extended, implemented or utilized across plugins, without using any code to lookup or link peer plugins. All of this is handled by the system.

The fact that there is no activation proccess requires that an external party use reflection start a plugin, but once execution begins inside the plugin (which is inside the PluginManager), code procedes as normal.

This is the proccess to instanciate a an object from a plugin:

PluginManager sm = null;
try
{
sm = PluginManager.getInstance();
Class tmc = sm.loadClass("org.jnode.services.aplugin.APluginClass");
tmc.newInstance();
}catch(Exception e)
{
e.printStackTrace();
}

Once newInstance() is called, execution switches to the constructor, and one there other plugins are loaded and linked simply by resolving references to them in normal java code (things link new PluginClass()).

I hope I answered your question. If not, or if you have more just ask.
Thanks,
Alex

re: Plugins

Hi alex,

Nice stuff. I don't know that I am further along. Each plugin in my engine has its own classloader as well. Here is how to use my engine:

List locations = new ArrayList();
locations.add(new URL("file","","somepath/"));

try
{
PluginEngine.start(locations, false, null, 0);
}
catch (PluginEngineException pee)
{
}

That starts the engine. Any directory found that has a plugin.xml in it below any directory specified as a URL, as well as any .par files found in the directories specified are loaded, parsed and resolved against. Each plugin descriptor looks like so:

[plugin loadOnStartup="true"]
[name]name[/name]
[version]1.0.0[/version]
[uid]someUniqueIdentifier[/uid]
[class]net.sourceforge.package.LifeCycleClass[/class]

[dependencies]
[dependency uid="anotherPluginUID" version="1.1.0"/]
[dependency uid="andAnotherPluginUID" version="2.2.3"/]
[/dependencies]

[extension-points]
[point uid="someEPPoint" interface="package.InterfaceClass"/]
[point uid="anotherEPPoint"/]
[/extension-points]

[extensions]
[point uid="someExtPoint"]
[!-- dynamic xml --]
[someNodeName someAttr="someValue"/]
[anotherNode]TextBetweenNode[/anotherNode]
[/point]

[point uid="anEPWithInterfaceDefined" class="package.Class"/]
[/extensions]

[/plugin]

In the above snippet, you see a "similar" yet different layout to
what eclipse plugins would have. Rather than have
and below the main node, I figured it
would look a little better to put them in their own parent node
sections, to help identify extension points and extensions from
one another. I am adding a listener/event feature as well, just
not sure how I want it to work yet.

Basically, as I said before an EP can "provide" an interface. E's
like in Eclipse implement this interface. The difference is, in
my engine, if the EP definition above has an interface="", the
engine ENFORCES that all E's must implement the interface before
the class can be instantiated and assigned as the extension executable
class.

Below is a simple plugin:

public class MyPlugin extends AbstractPlugin
{
public void activate()
{
ExtensionPoint ep = PluginEngine.getExtensionPoint("somePoint");

if (null != ep)
{
PluginEngine.walkExtensions(ep, new ExtensionCallBack()
{
public void handleExtension(ExtensionPoint extp, Extension ext)
{
// do somethign with the Extension resolved to this
// EP.
}
});
}
}

public void deactivate()
{
}

public void destroy()
{
}
}

The activate() is called by the engine via an event fired to a plugin
when it is first activated. Like eclipse, activation occurs on a plugin
the very first time a class within its classpath is accessed BEFORE it
is returned. Hence, lazy initialization of plugins. In the above
means activate the plugin as soon as
it is loaded and resovled. If set to false, its PluginModel is created
including the PluginClassLoader instance, name, version, any EPs are
added to the engine, and any E's are added to an unresolved repository.

Each and every time a plugin is loaded, after it's plugin.xml is parsed,
it goes through a resolution process. This is to allow plugin A depend
on B, but A is loaded first. When B is finally loaded, it resolves
immediately to A's dependency on it. Resolutions occur for dependencies
and for extensions to extension points.

All of this works pretty well right now. At runtime, if the engine
resolves an extension to an extension point and the plugin owner of
the extension point is already activated, the engine sends it an
EXTENSION_ADDED event so that it can respond to a runtime loaded
extension. HOWEVER, this is only done AFTER the initial startup()
method is done. That is to say, when you first start the engine,
the start() call may find 100 plugins to load. Naturally by the time
all 100 are loaded, everything will be resolved and there is no need
to send any events to inform others of extensions/dependencies. After
the initial start() sequence, if the loadPlugin() is called, it then
sends the event, as well as DEPENDENCY_ADDED if that is the case.

Anyway, I don't know what else I could contribute to what you have. It
sounds like its working well. I think in my engine we'll send out
events if dependencies are unloaded to hopefully allow GC of the
plugins faster. We also strive for most everything being done at
load/start time of a plugin, to avoid performance hits at runtime
looking for plugins. I thought of just keeping a big repository
of plugins, and at runtime when a class is looked for it just
looks in ALL plugins until it finds it. I didn't like the idea of how
long it may take to look through all the classes in every plugin
including embeded libraries at runtime.

One thing my co-developer did, he wrote the code for the .par stuff,
is to keep an index of package names and the plugin as well as the
embeded library they are in. This allows lookups to be much faster
at runtime by directly finding the plugin and embeded library (if that
is the case) and get the .class out of the URL location.

So how can I help? I don't know if I can help much, don't have much
time. I wish I could get paid to do this stuff!! That would be great!

P.S. How do you use > and < symbols? I tkae it this must be HTML formatted in this forum? Ok, after previwing this, I see I have to use the ampersand g t ; stuff. Is there any formatting commands that you can wrap code, xml, etc in?

Thanks.