Formatting an ext2 filesystem (i.e. mke2fs) is possible now. It would be great if someone could do some testing before the new release (sunday). I've just finished it so I didn't do very extensive testing yet, but it looks like it is working.
To create an ext2 filesystem on a partition:
- the partition type has to be set to ext2 (0x83). You have to do it with fdisk under Linux as jnode's fdisk does not yet support setting partition codes.
- stop the device: "device stop [PARTITION_DEVICE]"
- "format -t ext2 [PARTITION_DEVICE] [blocksize]" creates the filesystem. Blocksize is optional and can be 1, 2, 4 (in KBytes), the default is 4KB.
- don't forget to stop the device before shutting down JNode.
I had to make some corrections in Device and in the VFS layer to get things working properly:
- In org.jnode.driver.Device.start(), fireStartedEvent() was called _before_ started was set to true. As a result of this, if a partition device was stopped and then restarted, the filesystem was not mounted as FileSystemMounter was triggered before the started attribute of the device was set to true, so I've switched the order of "started=true" and fireStartedEvent().
- An other bug prevented mounting a filesystem more than once: when a device was stopped, the filesystem was unregistered but the FSEntryCache still held references to FSEntry-s from the old filesystem instance. As a result of this, even though a new filesystem instance was created, the system used the old, already closed fs from the cached entries. So I've added a call to flush the entries from the FSEntryCache which belong to a filesystem when it is unregistered.
I am continuously working on it but I have run into some problems that delay my work a bit. Nevertheless I'll try to commit it today or tomorrow as it is.
BTW I have just updated from CVS and now I get a lot of AccessControlExceptions so first I'll have to take care of that (I see that there is some documentation about that).
I have committed the new version of the ext2fs driver. It adds several bugfixes for reading an ext2 partition and adds preliminary write support, with the following limitations:
- format() is not yet ready
- removing a file is not yet ready
- some filesystem options are not supported, namely
--- sparse_super option is not supported
--- dir_index option is not supported
--- filetype option is always assumed
So the recommended way of creating an ext2fs partition for use with JNode is (under Linux): mke2fs -O ^dir_index -O filetype -O ^sparse_super DEVICE_NAME
IF YOU WANT TO USE THIS NOW, PLEASE READ THE FOLLOWINGS
This is an intermediate release for those who can't live without a writable ext2fs filesystem now (or would like to help testing) but i strongly recommend to NOT USE THIS ON A FILESYSTEM WHERE YOU HAVE VALUEABLE DATA!!! I have tested this with ext2 filesystems created on loopback devices and not on my real partitions with real data.
If anyone would like to help testing, I recommend to run "e2fsck -vf DEVICE" after using a partition to check for any possible inconsistencies.
It is very important to shut down the device that contains the ext2fs filesystem after writing anything to it to have all FS metadata properly flushed to disk (just like umount in Linux, although it is not unmounted here). To do that, simply say device stop YOURDEVICE
on whatever device you placed your ext2 partition (same as the JNode mount point) before shutting down JNode.
There are some ugly hacks that I've had to make for the time being:
- the IDE driver hangs somtimes, mostly during write operations. I see that the IDEBus has an executeAndWait method with a timeout, but using that does not help: when it hangs, it does not time out either. It looks like there is some kind of a deadlock, although I have not yet looked at it deeper. To overcome this, I have modified the disk read and write operations in the ext2 driver to wait for a timeout and retry if it hangs. This is certainly only a temporary solution until the problem is resolved where it should be (probably in the org.jnode.driver.ide package).
- a strange bug: with certain methods, it looks like the return value gets screwed up between the return statement and the caller (e.g. a method returns 33 and the caller sees something like 67675646567575478785). It only happens in certain methods, but with those, every time. I made a workaround by returning a Long instead of a long where it happens, but this looks kinda like a bug in the VM and is very weird. If you want to test this bug, open a file for appending and you will get this at FileOutputStream in the FileOutputStream(File, boolean) constructor. When fh.getLength() is called, garbage is received although if you println the return value in getLength(), it correct (at least this is what used to happen for me).
- I have made a LinkedHashMap based LRU cache for the blocks and inodes, and it is working fine in test programs but is doing strange things when it is used within Ext2FileSystem (values().iterator() only returns a single element, removeEldestEntry() is never called, etc.). Until this problem is resolved I use a simple Hashtable for the inode cache. It is obviously wrong because it only grows as the entries are never cleaned up, but should do it until the issue with INodeCache and BlockCache is solved. So for now if you use a lot of things on your filesystem, remember all of it will end up in your memory
Expect the above things to be corrected in the next couple of days with a new release. I will add the special features of ext2 (symlinks, indexed directories, etc.) when the basic stuff is working stable.
By the way, the FS access is very slow, partially due to the following factors:
- The FS does not know when a file is closed. As a result of this, the written data is flushed after every write access, together with the necessary FS metadata update. I strongly recommend to add close() to the org.jnode.fs.FSFile interface. If there is no objection, I will add it, but someone is needed to implement it for the FAT fs.
- a lot of debugging is turned on for now until the driver gets stable
- every write is synchronous (SYNC) [neccessary until the cache thing is resolved. After that I plan to change block writes to ASYNC and add a separate flusher thread that writes the dirty blocks to disk].
Andras, since FSFile is intended to be used by a number of concurrent users & processes, it will not be easy to add a close method to it, without breaking this front end behaviour.
However we can add a flush method. This method can then be called in VMFileHandle.close.
Finally JNode has a filesystem implementation that gives decent read access to a phisical partition.
I didn't try the write operations yet. However I copied the Kopi compiler (kjc) to an ext2 partition and started it up from that partition under JNode. Due to an error where the classloader couldn't load some resource files I didn't get to the point where I could actually compile somthing. But that error has nothing to do with the fs implementation.
Soon we can write, compile and run Java programs in JNode. Your work is a big step towards that.
Please try to make this async write stuff as generic as possible. I hope it can be used in all filesystems (so maybe even without strict knowledge of a filesystem)
format
Formatting an ext2 filesystem (i.e. mke2fs) is possible now. It would be great if someone could do some testing before the new release (sunday). I've just finished it so I didn't do very extensive testing yet, but it looks like it is working.
To create an ext2 filesystem on a partition:
- the partition type has to be set to ext2 (0x83). You have to do it with fdisk under Linux as jnode's fdisk does not yet support setting partition codes.
- stop the device: "device stop [PARTITION_DEVICE]"
- "format -t ext2 [PARTITION_DEVICE] [blocksize]" creates the filesystem. Blocksize is optional and can be 1, 2, 4 (in KBytes), the default is 4KB.
- don't forget to stop the device before shutting down JNode.
I had to make some corrections in Device and in the VFS layer to get things working properly:
- In org.jnode.driver.Device.start(), fireStartedEvent() was called _before_ started was set to true. As a result of this, if a partition device was stopped and then restarted, the filesystem was not mounted as FileSystemMounter was triggered before the started attribute of the device was set to true, so I've switched the order of "started=true" and fireStartedEvent().
- An other bug prevented mounting a filesystem more than once: when a device was stopped, the filesystem was unregistered but the FSEntryCache still held references to FSEntry-s from the old filesystem instance. As a result of this, even though a new filesystem instance was created, the system used the old, already closed fs from the cached entries. So I've added a call to flush the entries from the FSEntryCache which belong to a filesystem when it is unregistered.
today or tomorrow
I am continuously working on it but I have run into some problems that delay my work a bit. Nevertheless I'll try to commit it today or tomorrow as it is.
BTW I have just updated from CVS and now I get a lot of AccessControlExceptions so first I'll have to take care of that (I see that there is some documentation about that).
Andras
r/w committed
I have committed the new version of the ext2fs driver. It adds several bugfixes for reading an ext2 partition and adds preliminary write support, with the following limitations:
- format() is not yet ready
- removing a file is not yet ready
- some filesystem options are not supported, namely
--- sparse_super option is not supported
--- dir_index option is not supported
--- filetype option is always assumed
So the recommended way of creating an ext2fs partition for use with JNode is (under Linux):
mke2fs -O ^dir_index -O filetype -O ^sparse_super DEVICE_NAME
IF YOU WANT TO USE THIS NOW, PLEASE READ THE FOLLOWINGS
This is an intermediate release for those who can't live without a writable ext2fs filesystem now (or would like to help testing) but i strongly recommend to NOT USE THIS ON A FILESYSTEM WHERE YOU HAVE VALUEABLE DATA!!! I have tested this with ext2 filesystems created on loopback devices and not on my real partitions with real data.
If anyone would like to help testing, I recommend to run "e2fsck -vf DEVICE" after using a partition to check for any possible inconsistencies.
It is very important to shut down the device that contains the ext2fs filesystem after writing anything to it to have all FS metadata properly flushed to disk (just like umount in Linux, although it is not unmounted here). To do that, simply say
device stop YOURDEVICE
on whatever device you placed your ext2 partition (same as the JNode mount point) before shutting down JNode.
There are some ugly hacks that I've had to make for the time being:
- the IDE driver hangs somtimes, mostly during write operations. I see that the IDEBus has an executeAndWait method with a timeout, but using that does not help: when it hangs, it does not time out either. It looks like there is some kind of a deadlock, although I have not yet looked at it deeper. To overcome this, I have modified the disk read and write operations in the ext2 driver to wait for a timeout and retry if it hangs. This is certainly only a temporary solution until the problem is resolved where it should be (probably in the org.jnode.driver.ide package).
- a strange bug: with certain methods, it looks like the return value gets screwed up between the return statement and the caller (e.g. a method returns 33 and the caller sees something like 67675646567575478785). It only happens in certain methods, but with those, every time. I made a workaround by returning a Long instead of a long where it happens, but this looks kinda like a bug in the VM and is very weird. If you want to test this bug, open a file for appending and you will get this at FileOutputStream in the FileOutputStream(File, boolean) constructor. When fh.getLength() is called, garbage is received although if you println the return value in getLength(), it correct (at least this is what used to happen for me).
- I have made a LinkedHashMap based LRU cache for the blocks and inodes, and it is working fine in test programs but is doing strange things when it is used within Ext2FileSystem (values().iterator() only returns a single element, removeEldestEntry() is never called, etc.). Until this problem is resolved I use a simple Hashtable for the inode cache. It is obviously wrong because it only grows as the entries are never cleaned up, but should do it until the issue with INodeCache and BlockCache is solved. So for now if you use a lot of things on your filesystem, remember all of it will end up in your memory
Expect the above things to be corrected in the next couple of days with a new release. I will add the special features of ext2 (symlinks, indexed directories, etc.) when the basic stuff is working stable.
By the way, the FS access is very slow, partially due to the following factors:
- The FS does not know when a file is closed. As a result of this, the written data is flushed after every write access, together with the necessary FS metadata update. I strongly recommend to add close() to the org.jnode.fs.FSFile interface. If there is no objection, I will add it, but someone is needed to implement it for the FAT fs.
- a lot of debugging is turned on for now until the driver gets stable
- every write is synchronous (SYNC) [neccessary until the cache thing is resolved. After that I plan to change block writes to ASYNC and add a separate flusher thread that writes the dirty blocks to disk].
greets,
Andras
FSFile.close or not
Andras, since FSFile is intended to be used by a number of concurrent users & processes, it will not be easy to add a close method to it, without breaking this front end behaviour.
However we can add a flush method. This method can then be called in VMFileHandle.close.
Does that help?
Ewout
flush()
I see.
However we can add a flush method. This method can then be called in VMFileHandle.close.
Yes, that would be fine.
Andras
In CVS
flush has been added.
Nice work!
Great work, András!
Finally JNode has a filesystem implementation that gives decent read access to a phisical partition.
I didn't try the write operations yet. However I copied the Kopi compiler (kjc) to an ext2 partition and started it up from that partition under JNode. Due to an error where the classloader couldn't load some resource files I didn't get to the point where I could actually compile somthing. But that error has nothing to do with the fs implementation.
Soon we can write, compile and run Java programs in JNode. Your work is a big step towards that.
Levente
Generic async writes
Andras,
Please try to make this async write stuff as generic as possible. I hope it can be used in all filesystems (so maybe even without strict knowledge of a filesystem)
Ewout
test commands
I have committed a plugin with some commands that may help testing:
writeTest filename [some_text]
writes the text into the file.
copyTest fileFrom fileTo
copies a file.
fillTest fileName noOfMegabytes
creates a file that is the specified number of megabytes large.
Andras