Committing changes to an ext3_inode structure

Committing changes to an ext3_inode structure

[Felipe Franciosi]

Dear all,

I am experimenting on a 2.6.20.3-ubuntu1 kernel to prototype and  
evaluate different methods for migrating datablocks for a given inode  
on the fly.

I have created an ioctl call that implements some verifications and  
invokes a function I wrote in balloc.c for such migration.

Upon receiving a struct inode*, I retrieve the corresponding iloc  
using the ext3_get_inode_iloc() function and then the struct  
ext3_inode* using the ext3_raw_inode() on the iloc. I have confirmed  
the pointer is correct by printing several ext3_inode relevant fields.

After making my modifications to the i_block array, I am finding it  
impossible to commit these changes to the inode itself.

I have tried different things such as marking the iloc.bh buffer_head  
as dirty and submitting it for writing, marking the page as dirty,  
etc, but I think I'm missing something more fundamental. Most of my  
attempts were planned according to what I've seen inside similar code  
of buffer_head manipulation.

Can anyone shed some light on what I'm missing?

Regards,
Felipe

Re: Committing changes to an ext3_inode structure

[Andreas Dilger]

On 2010-10-19, at 09:23, Felipe Franciosi wrote:
> I am experimenting on a 2.6.20.3-ubuntu1 kernel to prototype and evaluate different methods for migrating datablocks for a given inode on the fly.
> 
> I have created an ioctl call that implements some verifications and invokes a function I wrote in balloc.c for such migration.

Are you aware that there already is the ability to do on-the-fly migration of files in ext4?  This was developed quite some time ago and is in ext4 in newer kernels (not sure of the exact version, maybe 2.6.32 or so).

I would strongly recommend that you investigate that code, since it could definitely use some good testing/inspection/enhancement (as needed).  It is in fs/ext4/migrate.c, though I'm not sure where the user-space tools are located (maybe e2fsprogs?).

Cheers, Andreas

Re: Committing changes to an ext3_inode structure

[Andreas Dilger]

On 2010-10-19, at 09:23, Felipe Franciosi wrote:
> I am experimenting on a 2.6.20.3-ubuntu1 kernel to prototype and evaluate different methods for migrating datablocks for a given inode on the fly.
> 
> I have created an ioctl call that implements some verifications and invokes a function I wrote in balloc.c for such migration.

Are you aware that there already is the ability to do on-the-fly migration of files in ext4?  This was developed quite some time ago and is in ext4 in newer kernels (not sure of the exact version, maybe 2.6.32 or so).

I would strongly recommend that you investigate that code, since it could definitely use some good testing/inspection/enhancement (as needed).  It is in fs/ext4/migrate.c, though I'm not sure where the user-space tools are located (maybe e2fsprogs?).

Cheers, Andreas

Re: Committing changes to an ext3_inode structure

[Greg Freemyer]

On Tue, Oct 19, 2010 at 5:56 PM, Andreas Dilger <adilger@dilger.ca> wrote:
> On 2010-10-19, at 09:23, Felipe Franciosi wrote:
>> I am experimenting on a 2.6.20.3-ubuntu1 kernel to prototype and evaluate different methods for migrating datablocks for a given inode on the fly.
>>
>> I have created an ioctl call that implements some verifications and invokes a function I wrote in balloc.c for such migration.
>
> Are you aware that there already is the ability to do on-the-fly migration of files in ext4?  This was developed quite some time ago and is in ext4 in newer kernels (not sure of the exact version, maybe 2.6.32 or so).
>
> I would strongly recommend that you investigate that code, since it could definitely use some good testing/inspection/enhancement (as needed).  It is in fs/ext4/migrate.c, though I'm not sure where the user-space tools are located (maybe e2fsprogs?).
>
> Cheers, Andreas

I'm not familiar with that code, I'm curious what it does?

But there is also the EXT4_IOC_MOVE_EXT ioctl which is called by
e4defrag primarily.

The code is in fs/ext4/move_extent.c

The concept is you allocate new blocks in a donor inode.  Once you
have that done, then EXT4_IOC_MOVE_EXT will move the data from the
original data blocks to the new ones and replace the old original
blocks with the new donor blocks.

All done on the fly.

Greg
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Committing changes to an ext3_inode structure

[Felipe Franciosi]

Dear Greg and Andreas,

Thanks for the replies.

I've read about migration in ext4, but never really touched the code  
because as I understand there is a lot of conceptual differences in  
the inode structure, block allocation and other metadata in general.  
I'm stuck to this (2.6.20) kernel, though (as there are other people  
working on the machine for quite a while and they are not keen to  
upgrade it), and the rest of my work was already done in ext3 anyway.

I will look into the code you pointed out and see if I can get any  
clues out of it. Otherwise I shall bother you again. =)

My problem is pretty much what I said: I am probably doing something  
wrong, but I have no idea how to commit the changes to an struct  
ext3_inode back to disk. Maybe I should actually be modifying  
something else, like a struct ext3_inode_info.

As I said, I also tried to modify the datablock of the inode table  
directly on the disk (and committing the changes, syncing, remounting,  
etc), but maybe because I am marking the inode dirty, the cache gets  
written back to disk and overwriting my changes after I do the  
alteration.

Cheers,
Felipe

On 19 Oct 2010, at 23:55, Greg Freemyer wrote:

> On Tue, Oct 19, 2010 at 5:56 PM, Andreas Dilger <adilger@dilger.ca>  
> wrote:
>> On 2010-10-19, at 09:23, Felipe Franciosi wrote:
>>> I am experimenting on a 2.6.20.3-ubuntu1 kernel to prototype and  
>>> evaluate different methods for migrating datablocks for a given  
>>> inode on the fly.
>>>
>>> I have created an ioctl call that implements some verifications  
>>> and invokes a function I wrote in balloc.c for such migration.
>>
>> Are you aware that there already is the ability to do on-the-fly  
>> migration of files in ext4?  This was developed quite some time ago  
>> and is in ext4 in newer kernels (not sure of the exact version,  
>> maybe 2.6.32 or so).
>>
>> I would strongly recommend that you investigate that code, since it  
>> could definitely use some good testing/inspection/enhancement (as  
>> needed).  It is in fs/ext4/migrate.c, though I'm not sure where the  
>> user-space tools are located (maybe e2fsprogs?).
>>
>> Cheers, Andreas
>
> I'm not familiar with that code, I'm curious what it does?
>
> But there is also the EXT4_IOC_MOVE_EXT ioctl which is called by
> e4defrag primarily.
>
> The code is in fs/ext4/move_extent.c
>
> The concept is you allocate new blocks in a donor inode.  Once you
> have that done, then EXT4_IOC_MOVE_EXT will move the data from the
> original data blocks to the new ones and replace the old original
> blocks with the new donor blocks.
>
> All done on the fly.
>
> Greg

Re: Committing changes to an ext3_inode structure

[Amir Goldstein]

On Tue, Oct 19, 2010 at 5:23 PM, Felipe Franciosi <felipe@paradoxo.org> wrote:
> Dear all,
>
> I am experimenting on a 2.6.20.3-ubuntu1 kernel to prototype and evaluate
> different methods for migrating datablocks for a given inode on the fly.
>
> I have created an ioctl call that implements some verifications and invokes
> a function I wrote in balloc.c for such migration.
>
> Upon receiving a struct inode*, I retrieve the corresponding iloc using the
> ext3_get_inode_iloc() function and then the struct ext3_inode* using the
> ext3_raw_inode() on the iloc. I have confirmed the pointer is correct by
> printing several ext3_inode relevant fields.
>

you've complicated things too much.
all you have to do is modify EXT3_I(inode)->i_data and then call
ext3_mark_inode_dirty().
the rest of the iloc and raw inode stuff is taken care of by ext3_put_inode().

> After making my modifications to the i_block array, I am finding it
> impossible to commit these changes to the inode itself.
>

that is because your changes are being overriden by the unmodified ei->i_data.

> I have tried different things such as marking the iloc.bh buffer_head as
> dirty and submitting it for writing, marking the page as dirty, etc, but I
> think I'm missing something more fundamental. Most of my attempts were
> planned according to what I've seen inside similar code of buffer_head
> manipulation.
>

maybe you should have looked at code that modifies inodes instead...

> Can anyone shed some light on what I'm missing?
>

hope this helps,
Amir.

Re: Committing changes to an ext3_inode structure

[Felipe Franciosi]

Hi Amir,

I finally got it working this morning, but thanks for your reply.  
Indeed all I had to do was to modify i_data... I got confused because  
of this:

On the kernel I have (haven't checked other versions), I see:

include/linux/ext3_fs_i.h:
struct ext3_inode_info {
    __le32 i_data[15];   /* unconverted */
    ...
}

include/linux/ext3_fs.h:
struct ext3_inode {
    ...
    __le32 i_block[EXT3_N_BLOCKS};   /* Pointers to blocks */
    ...
}

I never associated the two as being the same, even though I obviously  
knew EXT3_N_BLOCKS is 15, and the comments on ext3_fs_i.h are not very  
useful, are they? :-/
When I glanced at them, I just assumed it was some raw unconverted  
stuff that had nothing to do with what I wanted.

And yes, as you pointed out and as I said late last night, my fears  
were confirmed: my changes were being overwritten by the  
ext3_inode_info later on. :-(

But I'm glad it's (almost) all working now. I had tried to look for  
the code that manipulate inodes (and not buffer_heads), but I was  
overlooking anything related to an ext3_inode_info struct (I had in my  
head that things were related to the ext3_inode struct). After  
checking out the code pointed out to me here in the list (fs/ext4/ 
migrate.c) I finally got it working. Thanks again to Greg and Andreas  
as well.

Regards,
Felipe

On 23 Oct 2010, at 17:48, Amir Goldstein wrote:

> On Tue, Oct 19, 2010 at 5:23 PM, Felipe Franciosi  
> <felipe@paradoxo.org> wrote:
>> Dear all,
>>
>> I am experimenting on a 2.6.20.3-ubuntu1 kernel to prototype and  
>> evaluate
>> different methods for migrating datablocks for a given inode on the  
>> fly.
>>
>> I have created an ioctl call that implements some verifications and  
>> invokes
>> a function I wrote in balloc.c for such migration.
>>
>> Upon receiving a struct inode*, I retrieve the corresponding iloc  
>> using the
>> ext3_get_inode_iloc() function and then the struct ext3_inode*  
>> using the
>> ext3_raw_inode() on the iloc. I have confirmed the pointer is  
>> correct by
>> printing several ext3_inode relevant fields.
>>
>
> you've complicated things too much.
> all you have to do is modify EXT3_I(inode)->i_data and then call
> ext3_mark_inode_dirty().
> the rest of the iloc and raw inode stuff is taken care of by  
> ext3_put_inode().
>
>> After making my modifications to the i_block array, I am finding it
>> impossible to commit these changes to the inode itself.
>>
>
> that is because your changes are being overriden by the unmodified  
> ei->i_data.
>
>> I have tried different things such as marking the iloc.bh  
>> buffer_head as
>> dirty and submitting it for writing, marking the page as dirty,  
>> etc, but I
>> think I'm missing something more fundamental. Most of my attempts  
>> were
>> planned according to what I've seen inside similar code of  
>> buffer_head
>> manipulation.
>>
>
> maybe you should have looked at code that modifies inodes instead...
>
>> Can anyone shed some light on what I'm missing?
>>
>
> hope this helps,
> Amir.