ext2 bugfixes against 0.1.10

Submitted by zoku777 on Sat, 03/12/2005 - 18:38.

I have a significant number of patches against the org.jnode.fs.ext2 package, specifically related to using Ext2FileSystem.create() to correctly "format" a Device with an ext2 filesystem. With these patches, it is possible to format a device up to 2GB in size (I'm still having trouble at larger sizes, possibly related to the 4GB int limit) and write files into it with Ext2File.write(). The filesystem will pass Linux e2fsck with no errors.

What is my best strategy for getting this code committed? I'm a newbie to this whole sourceforge dev thing...

zoku

‹ Accessing directly harddrive from JAVA/Eclipse ? Format a HD ›

welcome !

Submitted by Fabien D on Sat, 03/12/2005 - 21:49.

welcome !

blind wrote most of the ext2 stuff. Do you know if he wasn't working on the same bugs or features ?

Have you run junit FS tests (that I wrote) ?

Fabien D

Have not run tests yet

Submitted by zoku777 on Sat, 03/12/2005 - 22:35.

I have not run the JUnit tests yet. I will try to do that before posting the remaining patches that I have.

I did not see any mailing list or CVS activity related to the problems that I was running into. The patches that I have relate to several problems that I faced:

  • Bugfixes: True bugfixes for the ext2 code, mostly encountered when using Ext2FileSystem.create() and Ext2File.write()
  • GCJ pedantics: I'm mostly working under GCJ which catches some syntax problems that the Sun javac ignores
  • Logging performance: I've added log.isDebugEnabled() calls to most of the log.debug() calls; this sped up performance of formatting around 40% when using a fast Device, like the ByteArrayDevice
  • Class access: I want to be able to subclass some of the ext2 classes to implement customized versions of them and some tweaks were necessary to the API to be able to do this

Obviously, the API changes will need some discussion but that's why I signed up Smiling

More fixes

Submitted by zoku777 on Mon, 03/21/2005 - 02:53.

I have another change that I'm organizing patches for. In many places, the block number on the filesystem (usually in a variable named blockNr or something similar) is stored using an "integer". However, according to the ext2 spec, the limit for the number of blocks per filesystem is unsigned 32-bit. Therefore, we need to ensure that these block number values are always stored as "long" types instead of "integer" (since everything is signed in Java and we'll overflow the field if we have more than ~2 billion blocks). I think we'll run into problems because of this on larger (>8GB) filesystems.

I should have some small patches to partially address this during this week.

int / long issue

Submitted by blind on Mon, 03/21/2005 - 16:15.

This is quite a problem indeed that I have faced at several places in the code but the problem is that usually everything boils down to array indexing and you can not index an array with a long, only with an int.
However even with this limitation you can use filesystems of at least 16TB if I remember correctly.

One idea that has just occured to me is to use two-dimensional arrays where the first index can be in the range of [0,1] to compensate for the lost bit.
What is your solution?

Andras

BlockCache thoughts

Submitted by zoku777 on Fri, 03/25/2005 - 02:02.

Oops, kilo * giga = tera (not giga)... Sticking out tongue I must have been sleep-deprived when I sent that last message.

My main concern with the int/long stuff is the keys used for the BlockCache. Since this is a regular java.util.Collection, we might as well use Long keys instead of Integer keys.

Another thought; it's a little strange to split the block caching up like it is now. The Ext2FileSystem class has to manage the interactions between writing to the BlockDeviceAPI and maintaining the BlockCache. It would be easier to simply wrap the BlockDeviceAPI with a caching layer that can be written to using block numbers instead of raw offsets. This would take a lot of the low-level block offset calculations out of Ext2FileSystem and make it somewhat easier to unit test.

I agree with you

Submitted by Fabien D on Fri, 03/25/2005 - 18:52.

I agree with you about caching layer and access with block number.

For cache part, I have started several weeks ago but I stopped because FS part and especially BlockDeviceAPI need heavy refactoring (+ use of ByteBuffer). But, before, all of this we need good working FS impl and BlockDeviceAPI impl tests. These two are too closely linked.

For FS impl tests, it's ok. but for BlockDeviceAPI impl tests, it's progressing (too) slowly.

Fabien

Fork?

Submitted by zoku777 on Tue, 03/29/2005 - 01:18.

I'm considering forking the org.jnode.fs.ext2 package. I think that the cleanup that I could get done now would save enough refactoring time later that it would be worth it. The ByteBuffer transition shouldn't really be a big deal but with the BlockDeviceAPI access split up like it is in the current package, it will be more difficult than necessary.

I've spent a lot of time with the code and I can't really block for this long while the rest of the unit testing/refactoring/ByteBuffer work is tied up. I'd like to start getting my jungle of patches committed back but it looks like that won't be an option until that other work is finished. I'll be glad to provide updates on my progress to anyone who is interested in the effort.

Is better to commit

Submitted by lsantha on Tue, 03/29/2005 - 14:09.

From your posts it seems to me that you have done a lot of useful improvements on the ext2 fs support. I think it would be better to commit your work so that other team memebers and users can potentially try it out and use it if they want to do so. I personally would like to see a better ext2 support in JNode the sooner the better (Did you look into deleting files on ext2?). On the other hand the ByteBuffer refactorings look like a long and tedious, time consuming process. It might not be a good idea to delay other things because of it which do not strictly depend on it.

I am not sure to understand

Submitted by Fabien D on Tue, 03/29/2005 - 13:07.

About forking ext2 package, do you mean it will be easier to refactor on BlockDeviceAPI with modification you plan to do ? Is it only for ext2 ? If it make it easier (even if only for ext2), I can wait that you do it. During this, I will continue the testing to prepare the way for refactoring of ByteBuffer.

I understand that you don't want to wait too much time to commit your changes. Have you send the patches to Ewout ? Or do you have the right to commit ?

In my current devs for testing, I am not on the FileSystem implementations but on the underlying layer : driver, device ... that use the BlockDeviceAPI. So, If you only work on ext2, I don't think it will interfere with my work ... until the refactoring of BlockDeviceAPI.

Fabien

Explanation

Submitted by zoku777 on Tue, 04/05/2005 - 03:15.

The changes that I'd like to make would involve a lot of reorganization of the code. Not API changes; just some reorganization with the following goals:

  • Consolidate all block layer access to simplify the code. This will have the side effect of making it easier to move to the ByteBuffer BlockDeviceAPI later.
  • Make the filesystem 64-bit clean.
  • Improve performance if possible (block allocations, etc.).

I think at this point, it will be easier for me to work and easier to integrate the code if I work on a branched copy of the ext2 code. It will be difficult for me to make patches (and for you guys to audit the changes in the patches) because of the number of files I need to touch. After I can get things cleaned up, I'll send the entire package back in and we can all decide which parts of the code fit in.

I have not had a lot of time to work on this lately, but I will send in any updates when I have them.

Tarball submitted

Submitted by zoku777 on Sun, 07/17/2005 - 18:53.

I have submitted a tarball of my changes to Ewout. They were made against the 0.1.10 code so they're a bit out of date but judging by the CVS mailing list, not much has changed in the ext2 code except for the move to typed collections with the Java 5 generics syntax.

There are several features that are incomplete in my code; if you have questions about it, please contact me on this thread.

Do you mean BlockDeviceAPI ?

Submitted by Fabien D on Sun, 03/13/2005 - 10:00.

When you say API, do you mean BlockDeviceAPI (FSBlockDeviceAPI) ?

If so, let me inform you that BlockDeviceAPI is currently under heavy change : migrating from byte[] to ByteBuffer in the JNode-FS branch named "BlockDeviceAPI_with_ByteBuffer".

But I am running into big problems with JNode-FS testability and regression tests. So, I switched back to the CVS head and I am writing more tests that must all succeed before continuing in the CVS branch.

The main problem in the tests and that become more and more recurrent is : running tests outside of jnode to speedup the dev process.

Fabien

Ran tests; mailed first patch

Submitted by zoku777 on Tue, 03/15/2005 - 03:57.

I've gotten the fs unit tests to run now and I have eclipse installed so it should be easier for me to integrate my work with the project.

I have mailed a first patch (made against HEAD) to epr; I still need to comb through my sandbox and see what else I have that needs to be extracted so I can provide more patches.

My work should not interfere with any of the ByteBuffer work. The API changes that I was referring to were mostly just class access (public vs. protected vs. private) issues within the ext2 classes. Pretty simple stuff.

Hello zoku, You can made p

Submitted by galatnm on Sat, 03/12/2005 - 20:54.

Hello zoku,

You can made patches with eclipse and send it to ewout and/or the creator of Ext2 classes. They can integrate and test patches before add them to cvs.

Fabien L.

jnode-devel?

Submitted by zoku777 on Sat, 03/12/2005 - 22:14.

I'll just go ahead and post the patches to the jnode-devel list. That way, anyone who is interested should be able to see them.

welcome !

Submitted by Fabien D on Sat, 03/12/2005 - 19:10.

welcome !
You should send the patches to epr (our project team leader).
After, he may decide to give you CVS right for commit.

blind wrote most of the ext2 stuff. Do you know if he wasn't working on the same bugs or features ?

Have you run junit FS tests (that I wrote) ?

Fabien