Prevayler, Prevalence layer for Java - integration in JNode

Hi,
I think this new OpenSource Project is very interesting:
it's a free-software Prevalence layer for Java.

Excerpt from their Intro Page:
"Ridiculously simple, Prevalence is by far the fastest and most transparent persistence, fault-tolerance and load-balancing architecture for Plain Old Java Objects (POJOs). "

I'm using it for some Web Applications, ind it's very interesting, but the price to pay is to have all data in RAM ...

But, the question related to JNode is this:
I've sen that they are creating ad Adapter (MBean) to use this also inside te J2EE Server JBoss as a Service. In this manner, more Web Applicatione, EJB, etc can refer to this as a System per data retrieval and storage, without DB and without SQL.

Anyone is interested to integrate this also as a System Persistence Service, that JNode could give to other Applications ?

And, in JNode someone has already implemented a JNDI provider to share some objects between applications ? Like Tomcat or JBoss, or other J2EE Application Server.

You can find it at: http://www.prevayler.org
They have also a Demo of the classic PetStore J2EE called Presto, that you can try and download ...

Bye,
Sandro

Prevalence and the JNFS

I created a standin oodb for use with ServiceManager testing. There are a lot of problems with it but it suited my purposes.

I have visited the prevayler site, where they laid out most of their design stratagies. After some thought I concluded that these stratagies could be added to an oodb (like mine) to for a hybrid.

If once an object is retreaved from persistent storage, it was cached with a SoftReference, then it would be quickly reachable for any subsequent updates. Also by using SoftReference's we will be able to optimize the relationship between ram an hdd (not ram and vertual memory).

When a vm doesn't have enough memory to allocate a new object it is supposed to remove SoftReference's untill it has enough space. When the SoftReference's are removed they will be sent to the associated ReferenceQueue, from which we will be able to retrieve the object and store it persistently again.

This coupled with reliable loging techniques (as described at prevayler) would make the oodb faster and also provide for systems that have something less than 2gig of ram.

Actually I have already designed and tested a reliable logger that manages store\update style persistence, so that would be easy.

I am very interested in seeing an oodb as the jNode file system, and I would like to see more discussion on the subject.

JNodeFS

Thanks to everyone for the sugestions, they were very helpful, though perhaps not in they way that they were intended.

I consider component architectures to be my forte, and a first found it hard going to design a oodbms. My first atempt was among the ugliest things I have ever writen. It had a complicated locking system, very ineficient persistence mechinism, and clumsy symantics. Over all it was uneligent and dificult, not something you would expect from java or a first class os.

After reading the prevalence comment above I commented that SoftReference caching would make an oodb more efficient. After that more comments came in with suggestions. As I was reading the documentation for these systems it came to me that an oodbms at its core is not that very diferent from a component architecture.

With this enlightened point of view, I began to redesign the JNodeFileSystem (wich I renamed JNodeFS). This is what i did:

1) I added SoftReference caching as was sugested. This makes every aspect of the system enormously faster. (The JNodeFileSystem was deserializing the object from harddrive at every call. Ah!)

2) I added a store/update style persistence mechinism. The stored object state must be kept curent with the live object state, in case of a system crash. Instead of serializing the entire object to drive at every update, only store the changed part. A store saves the entire object. A update saves only the modified part. Many updates may follow a store, but a store replaces the old store, and all of the updates. This proccess keeps the persistent state identical with the live state, while minimizing repetative data storage.

//This is the cool part
3)I combined dynamic subclassing as in EJB CMP, and a proxy system like RMI, to produce a dynamic class proxy. The system works like this, a client adds a pre-existing object to the JNodeFS. The jnfs caches the object and examins the class of the object and writes a subclass implementation that overwrites all of the public methods of the original object. This class is compiled and instanciated with a key to the original cached object. Finally this new class proxy object is returned to the client.

The overloaded methods of the class proxy object cominicate to the jnfs which method was called and with what arguments. The jnfs uses the key from the class proxy to look up the original object, and then uses the supplied data to invoke the method dynamically and then returns the result.

All arguments are wrapped and unwrapped properly and exceptions are handeld accuratly.

Because the proxy object extends the supplied object's class, it can be handled just like the supplied object. For example:

Hashtable t = (Hashtable) jnfs.putObject(new Hashtable()); //get proxy
t.put("two","one"); // Use normal api
t.put("date",new Date());
System.out.println( t.get("date"));
t.remove("date");
t.add("Cool","Huh?");

This looks just like a normal Hashtable interaction, but it is all done through proxy. The persistent state of the object is updated every time a method is called on the proxy object. If the entire state of the object had to be writen, then that could get expensive, but only the method name, args, and signiture are saved in an update.

There are some syntactic requirements for an object to be elegible for JnodeFS persistence. Most of them are trivial.

Object can not:
Be final or abstract.
Object must:
Be instance of Serializable, or Externalizable, and provide an empty (no-arg) constructor.

These are the enforced requirements. For the system to work properly the object must be deterministic, and all of the methods that need to be exposed in the proxy must be public.

If it is not deterministic (if it does not do the same thing, provided it has the same args) then the live and persistent states will lose synchronization. If proxy methods are not public, they will not be read by reflection, will not be put in the proxy class, and will not relay invocation properly.

The object cache is built on a stack. When ever an object is looked up (this happens at every proxy invocation) the retreved object is moved from its place to the top of the stack. The stack is seached from top down, so objects that are used often are found quickly. The cache will not excede 90% of the vm memory, and since it is based on SoftReferences, if the vm suddenly needs a lot of space it can just clear cache entries.

Once an object has been added to the jnfs it can be retrieved agian using queries. A Query is based intirely on reflection, and comparative equality (.equals()). For example to look up the Hashtable we just made you could do this:

Query q = new Query(Hashtable.class); //Class of object to search
Method m = Hashtable.class.getMethod("get",new Class[]{Object.class});
q.addConstraint(new QueryMethod(m, new Object[]{"two"},true,"one");

// this calls .get("two").equals("one");
jnfs.executeQuery(q);

executeQuery() returns a collection of keys to object that matched the constraints.

The jnfs cross references objects by interface and super class recursively, so we could just as easily searched for Map.class, or Dictionary.class.

In sumary the JnodeFS alows for the transparent persistence of objects in an extreamly efficient manor.

The JnodeFS is a major improvement over my first design.

What do you think?

Alex

JNodeFS and Daemon problems

Hi Alex,
I think you are doing an impressive job: good luck.

Only one question:
what will happen to the JNode System if the File System OODBMS hang ? All the rest of the System will be able to keep stable ? And can I configure the FileSystem Daemon to use x MB of Ram, like the SGA in Oracle ? In this manner we could tune that Service depending to the Pc and the use we want (for ex. if I use the System as a FileServer I could give to your Daemon more memory ...).

Excuse me but I haven't understand:
this will be the only FileSystem supported in JNode, or is an Extended FileSystem ?

Have you seen that in Microsoft LongHorn (the Next Release of Windows XP) they will implement a similar FileSystem ... I think that's a good idea. Good work.

Bye,
Sandro

Deadlock

Thanks, Im glad you like it.

>what will happen to the JNode System if the File System OODBMS hang ? All the rest of the System will be able to keep stable ?

Good question. Any lowlevel system must be very stable in order to support its clients. I have not yet done any extensive multi proccess testing, however I can tell you what I expect. The JnodeFS does not use any interal threads. All functions are carried out in the callers thread. This was done parcially to prevent complex deadlocking, but also because the caller expects only to be interacting with a normal java object, so having asynchonous events take place seamed impoper. So not having any internal thread to complicate things, if an exception arises it will be propigated back to the calling client in the form of an UncaughtThrowableException. That clients thread may or may not die, but the rest of the system should continue working.

Right now the jnfs does not use any syntactic synchronization. Because of the synchronization (redundency) between the live objects and the stored objects multithreaded object loading should be handled cleanly. If an object is loaded twice close together, one will always be higher on the cache stack. This top one will be the one in use and the other will float to the bottom and be removed by the space clearing algorithm. Neat and tidy.

>And can I configure the FileSystem Daemon to use x MB of Ram, like the SGA in Oracle ? In this manner we could tune that Service depending to the Pc and the use we want (for ex. if I use the System as a FileServer I could give to your Daemon more memory ...).

This is not the case right now. However it should be. I will add the proper methods as soon as I can.

>Excuse me but I haven't understand:
this will be the only FileSystem supported in JNode, or is an Extended FileSystem ?

The JnodeFS is more of a persistence mechinism than a file system. This was my motive for shortening FileSystem to FS. The jnfs can be placed over any supported file system, because it only uses io. The jnfs places all of its data in a tree starting with a dir call jnfs in the root directory. Access to that dir will need to be protected so that outside forces cannot alter persistent data. In my idealistic world programs would use the jnfs for persistence, and only users would access the underlying file system (for things like developing java code).

>Have you seen that in Microsoft LongHorn (the Next Release of Windows XP) they will implement a similar FileSystem

I have heard of it, though I though they were using a relational system.

Thanks for the thumbs up.

Alex

I will work on a JNodeFileSystem

Alex,

If I understood your JnodeFS is not a FileSystem but an oodb. What do you think of renaming it to something not suggesting it's a FileSystem ? What do you think of JNodeObjectSystem ?
I ask you this question because I want to avoid a possible confusion with JNodeFileSystem that implements FileSystem.

I think we can have easily a working FileSystem by creating a new one. So, I will start working on the specification and put it in the documentation.

Fabien

Sure

That is fine. I will rename it next chance I get.
Thanks for the heads up.

wehn can you check in an initial version?

or is it allready in? what about implementing some basic commands using it so we have some examples..I can't wait to see it heapend. I also think that this JNFS should be more of a persistance mechanism or a kind of virtual filesystem and it should work with any FS driver written. Maybe for performance reasons we could implement a native driver or so..but this later.

Other ideas based on an Object filesystem

I think it's an interesting idea because everything in JAVA is an Object (but special case for primitive types). So, why not file themself !

This give me other ideas :
- the virtual memory (ie memory on disk also called swap file) can contain unused Objects (at a given time). Perhaps, we could think this memory as a folder and all files in it as "swapped Objects".

- We could extend the previous idea to a "suspend to disk" functionality

- The cache disk could be a Collection of Objects

- A ".jpg" file (for example) could be thought as JPGFile that extends (or implements) ImageFile. And the JNode GUI can operate on Objects and their Classes instead of Files.

Objectivity

I am an idealist. I would like nothing better that to see .txt be Strings and .jpg be Images. This idea would enable the complete removal of the hierarchal file system. However that kind of revelution would come with a price. The system would be so diferent from anything that we have every used, it would feel dificult at first even if it was truely easier. Another dificulty lies in the fact that some things have hierarchal ideas build into them, most importantly java packages. To enable the construction of packages in a developement environment would require something like a DevelPackage object that could contain other packages and classes.

A file system has the primary key of location. If you know nothing about a file but its location you can still find it. The object system would need a primary key. Humans can not be expected to remember all of the important attributes of every object they want to find. We need primay keys. I have theorized a linear time system that would be build upon the jnfs above. In this system time index would serve as the primary key.

Imagine yourself at a desktop. On the left of the screen is a line from top to bottom. On this line there are images representing object types. You see a String in the middle, a Document toward the bottom and a URL at the top. All of the objects are flowing gently down from the top (which represents the current time) to the bottom (which repressents some time in the past). You can scroll down this time stream to see what is further in the past. You notice a JarObject which you downloaded yesterday. You don't need it any more so you delete it from the stream. Next you see a ClassObject which you have been working on. You click on it an the desktop opens the apropriat viewer. You edit the class, and save it. Because the object was modified it is moved from its old location to the top of the stream (which is now).

I believe that this sort of display combined with attribute based queries (searches ?) would enable a user to adapt do a non hierarchal object system.

What do you think?

Alex

Lifestreams

The idea of 'time streams' is a known research subject at Yale started many years ago: http://www.cs.yale.edu/homes/freeman/lifestreams.html

Since many type of information is inherently hierarchical it might be a good idea to support that. The next step might be to eficiently work with networks of objects. The time based linear organisation might be an alternative way for acessing you objects but by no means the only way.

You can also have a look at the following: www.lazysoft.com . (They had an applet version of the Sentences product but I cannot find it any more. You can stilll download the personal edition for free if you want to have a look).
This is a database product based on a somewhat new approach.

Whats wrong with giving the user the option of a hierarchy ?

This comment is interesting. However, I would not like to be using it. I am used to hierarchical systems, and this is evident with the way I do things, including java sources (as you commented on).

All that would be needed is a Directory object. This will be a collection object which can hold other Directory objects, and other File (or whatever the super class of ImageFile and StringFile, or just plain Object if that is the case).

By doing this, the user now has the option of listing the files by date, by type, by attribute, by interface, by superclass, by <insert other cool mechanism here> or by hierarchy.

This can be done quite easily in a RDBMS so I can only assume this will be easy to implement in an OODBMS.

Matt.

Directory

Good observation.
I used this aproach on my first pass at the JNodeFS, but the shortcomings of the first design made it impossible. Perhaps I will give it another go under the new version.
I have concluded that hierarchies themselfs are not unobject oriented, so I would be just as pleased to see a file system of objects as an object system.

Interactive Objects

I wrote a short paper on interactive objects, that is, decorating objects with Controller behaviors using Reflection and the Objects' public interfaces.

http://ucsub.colorado.edu/~reids/papers/Interactive_Objects-Sam_Reid-Fal...

My thoughts on objects in user desktops:
1. Objects need a long term persistence, like xml or ascii, not a brittle object serialization.
2. Interaction with Objects or files needs to be easily manipulable. Whether clients should ever be exposed to bare objects is not clear.

No need to remove hierarchical aspect

The organisation of your objects is clearly oriented on productivity (it's a kind of cache with recently objects on the top). The number of use of an Object could also be considered.

I didn't think about removing the hierarchical of the FS because if we have a lot of objects (files) it will be difficult to find one I used several months ago. If folders are logically organised (for the user), it will be easier.

JISP

IfF you need a simple indexed object store, you might want to have a look at JISP: http://www.coyotegulch.com/jisp/ to see if it helps.

OODBMS and Prevalence

Hi,
if you need some Help about Prevayler (but I don't know it very much), contact me. At the moment, Prevayler hasn't some features:
having a mix of Prevayler and an OODBMS could be a good thing.

Have you also tried to use Ozone-Db, a OODBMS OpenSource, from www.ozone-db.org ?

Only one question:
what will happen to the JNode System if the File System OODBMS hang ? All the rest of the System will be able to keep stable ? And can I configure the FileSystem Daemon to use x MB of Ram, like the SGA in Oracle ? In this manner we could tune that Service depending to the Pc and the use we want (for ex. if I use the System as a FileServer I could give to your Daemon more memory ...).

Excuse me but I haven't understand:
this will be the only FileSystem supported in JNode, or is an Extended FileSystem ?

Have you seen that in Microsoft LongHorn (the Next Release of Windows XP) they will implement a similar FileSystem ... I think that's a good idea. Good work.

Bye,
Sandro

not enough memory for object

I am working on joafip to be able to manage so huge collection that can not keep in few memory I have, this whithout using a database and less intrusion in data model code.
http://joafip.sourceforge.net/

It would be interesting to

It would be interesting to see if your project works on JNode!