e4defrag and immutable files

e4defrag and immutable files

[H. Peter Anvin]

Hi all,

I have looked a bit at e4defrag, and it does not appear to prevent the
relocation of files marked immutable.  This is a problem for boot
loaders, which may need to have a block pointer to a specific file in
order to bootstrap themselves.

Would it be possible to either:

a) prevent e4defrag and/or EXT4_IOC_MOVE_EXT from relocating an
immutable file, or
b) add a new attribute with the above property (in case there is
legitimate need to move around immutable files)?

	-hpa

Re: e4defrag and immutable files

[Greg Freemyer]

On Fri, May 28, 2010 at 4:14 PM, H. Peter Anvin <hpa@zytor.com> wrote:
>
> Hi all,
>
> I have looked a bit at e4defrag, and it does not appear to prevent the
> relocation of files marked immutable.  This is a problem for boot
> loaders, which may need to have a block pointer to a specific file in
> order to bootstrap themselves.
>
> Would it be possible to either:
>
> a) prevent e4defrag and/or EXT4_IOC_MOVE_EXT from relocating an
> immutable file, or
> b) add a new attribute with the above property (in case there is
> legitimate need to move around immutable files)?
>
>        -hpa


Other projects / tools than e4defrag use EXT4_IOC_MOVE_EXT, so it
would be best to do this in the kernel in my opinion.

Greg
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: e4defrag and immutable files

[H. Peter Anvin]

On 05/28/2010 02:21 PM, Greg Freemyer wrote:
>
> Other projects / tools than e4defrag use EXT4_IOC_MOVE_EXT, so it
> would be best to do this in the kernel in my opinion.
>

I fully agree.  If nothing else, it will provide protection against 
broken userspace programs.

	-hpa

Re: e4defrag and immutable files

[Eric Sandeen]

H. Peter Anvin wrote:
> Hi all,
> 
> I have looked a bit at e4defrag, and it does not appear to prevent the
> relocation of files marked immutable.  This is a problem for boot
> loaders, which may need to have a block pointer to a specific file in
> order to bootstrap themselves.
> 
> Would it be possible to either:
> 
> a) prevent e4defrag and/or EXT4_IOC_MOVE_EXT from relocating an
> immutable file, or
> b) add a new attribute with the above property (in case there is
> legitimate need to move around immutable files)?

seems like the immutable flag isn't necessarily the right thing to
key on; at least it's an addition to what the manpage says immutable
means...

>        A file with the ‘i’ attribute cannot be modified: it cannot
>        be  deleted or renamed, no link can be created to this file
>        and no data can be written to the file.  Only the superuser
>        or  a process possessing the CAP_LINUX_IMMUTABLE capability
>        can set or clear this attribute.

Changing the physical mapping doesn't do any of those things.

I think a new flag, no-move or something, may be better, if it's
needed.  But how many files would ever get this; maybe defragging
boot files falls into the category of "don't do that?" :)

-Eric

> 	-hpa

--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: e4defrag and immutable files

[Sunil Mushran]

On 05/28/2010 01:14 PM, H. Peter Anvin wrote:
> Hi all,
>
> I have looked a bit at e4defrag, and it does not appear to prevent the
> relocation of files marked immutable.  This is a problem for boot
> loaders, which may need to have a block pointer to a specific file in
> order to bootstrap themselves.
>
> Would it be possible to either:
>
> a) prevent e4defrag and/or EXT4_IOC_MOVE_EXT from relocating an
> immutable file, or
> b) add a new attribute with the above property (in case there is
> legitimate need to move around immutable files)?
>    

We (ocfs2) are looking to add a new attribute to denote files that
have a fixed allocation on disk. But at the same time, allow writes
that do not change the allocation on disk. No truncating, extending,
filling holes, etc. We were thinking of calling it "Static" files.

No meddling with the allocation will allow ocfs2 to not have any cluster
locks associated with that inode. And no cluster locks means no need
to pause ios during node recovery. This could be of interest to vm users
who are willing have fully allocated images for no stoppage during cluster
reconfiguration.

Thanks
Sunil

Re: e4defrag and immutable files

[tytso]

On Tue, Jun 01, 2010 at 12:32:29PM -0700, Sunil Mushran wrote:
> 
> We (ocfs2) are looking to add a new attribute to denote files that
> have a fixed allocation on disk. But at the same time, allow writes
> that do not change the allocation on disk. No truncating, extending,
> filling holes, etc. We were thinking of calling it "Static" files.

That's an interesting set of semantics, and it might make sense to
conflate that with a local disk "don't move or defrag" the file
option.  I'm not crazy with the name "static", since it could mean a
number of other things in other contexts, but I admit I can't think of
a better name.

					- Ted

Re: e4defrag and immutable files

[Greg Freemyer]

On Tue, Jun 1, 2010 at 3:49 PM,  <tytso@mit.edu> wrote:
> On Tue, Jun 01, 2010 at 12:32:29PM -0700, Sunil Mushran wrote:
>>
>> We (ocfs2) are looking to add a new attribute to denote files that
>> have a fixed allocation on disk. But at the same time, allow writes
>> that do not change the allocation on disk. No truncating, extending,
>> filling holes, etc. We were thinking of calling it "Static" files.
>
> That's an interesting set of semantics, and it might make sense to
> conflate that with a local disk "don't move or defrag" the file
> option.  I'm not crazy with the name "static", since it could mean a
> number of other things in other contexts, but I admit I can't think of
> a better name.

FROZEN?

PRE-ALLOCATED?

FALLOCATE_ONLY?

FROZEN_ALLOCATION?

STATIC_ALLOCATION?

I like the last 2 choices myself.

Greg
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: e4defrag and immutable files

[Eric Sandeen]

tytso@mit.edu wrote:
> On Tue, Jun 01, 2010 at 12:32:29PM -0700, Sunil Mushran wrote:
>> We (ocfs2) are looking to add a new attribute to denote files that
>> have a fixed allocation on disk. But at the same time, allow writes
>> that do not change the allocation on disk. No truncating, extending,
>> filling holes, etc. We were thinking of calling it "Static" files.
> 
> That's an interesting set of semantics, and it might make sense to
> conflate that with a local disk "don't move or defrag" the file
> option.  I'm not crazy with the name "static", since it could mean a
> number of other things in other contexts, but I admit I can't think of
> a better name.
> 
> 					- Ted

fixed-mapping files maybe?  

-Eric

Re: e4defrag and immutable files

[Sunil Mushran]

On 06/01/2010 02:00 PM, Eric Sandeen wrote:
> tytso@mit.edu wrote:
>    
>> On Tue, Jun 01, 2010 at 12:32:29PM -0700, Sunil Mushran wrote:
>>      
>>> We (ocfs2) are looking to add a new attribute to denote files that
>>> have a fixed allocation on disk. But at the same time, allow writes
>>> that do not change the allocation on disk. No truncating, extending,
>>> filling holes, etc. We were thinking of calling it "Static" files.
>>>        
>> That's an interesting set of semantics, and it might make sense to
>> conflate that with a local disk "don't move or defrag" the file
>> option.  I'm not crazy with the name "static", since it could mean a
>> number of other things in other contexts, but I admit I can't think of
>> a better name.
>>
>> 					- Ted
>>      
> fixed-mapping files maybe?
>    

We had thought of the term static because we wanted the inode to
be frozen. Entire metadata including the mapping. Our aim is to
do away with cluster locks for such inodes. For ios, we were planning
to limit it to only odirect and not change the mtime. That also means
no links, touch. Yes, "static" does not fit it like a glove but that's the
best we could come up with.

Sunil

Re: e4defrag and immutable files

[tytso]

On Tue, Jun 01, 2010 at 02:28:08PM -0700, Sunil Mushran wrote:
> 
> We had thought of the term static because we wanted the inode to
> be frozen. Entire metadata including the mapping. Our aim is to
> do away with cluster locks for such inodes. For ios, we were planning
> to limit it to only odirect and not change the mtime. That also means
> no links, touch. Yes, "static" does not fit it like a glove but that's the
> best we could come up with.

Hmm, maybe "fixed_metadata"?  The one thing that worries a bit is not
touching mtime is definitely icky.  I suppose it would improve
performance even on local-disk file systems if we don't update mtime,
ctime, or atime.  So maybe that would be useful for folks who are
trying to extract the last bit of performance out of their enterprise
database.

One thing I do like about fixed_metadata is that as far as chattr and
lsattr is concerned 's' and 'S' are already taken.  But 'f' and 'F'
aren't allocated yet.  :-)

					- Ted

Re: e4defrag and immutable files

[Sunil Mushran]

On 06/01/2010 03:26 PM, tytso@mit.edu wrote:
> On Tue, Jun 01, 2010 at 02:28:08PM -0700, Sunil Mushran wrote:
>    
>> We had thought of the term static because we wanted the inode to
>> be frozen. Entire metadata including the mapping. Our aim is to
>> do away with cluster locks for such inodes. For ios, we were planning
>> to limit it to only odirect and not change the mtime. That also means
>> no links, touch. Yes, "static" does not fit it like a glove but that's the
>> best we could come up with.
>>      
> Hmm, maybe "fixed_metadata"?  The one thing that worries a bit is not
> touching mtime is definitely icky.  I suppose it would improve
> performance even on local-disk file systems if we don't update mtime,
> ctime, or atime.  So maybe that would be useful for folks who are
> trying to extract the last bit of performance out of their enterprise
> database.
>
> One thing I do like about fixed_metadata is that as far as chattr and
> lsattr is concerned 's' and 'S' are already taken.  But 'f' and 'F'
> aren't allocated yet.  :-)
>
>    

Fixed Metadata sounds good to me.

The mtime restriction is icky. But for us it is a requirement
considering we cannot safely update it without cluster locks.
I am ok if we can make that file system specific.

Re: e4defrag and immutable files

[H. Peter Anvin]

On 06/01/2010 12:49 PM, tytso@mit.edu wrote:
> On Tue, Jun 01, 2010 at 12:32:29PM -0700, Sunil Mushran wrote:
>>
>> We (ocfs2) are looking to add a new attribute to denote files that
>> have a fixed allocation on disk. But at the same time, allow writes
>> that do not change the allocation on disk. No truncating, extending,
>> filling holes, etc. We were thinking of calling it "Static" files.
>
> That's an interesting set of semantics, and it might make sense to
> conflate that with a local disk "don't move or defrag" the file
> option.  I'm not crazy with the name "static", since it could mean a
> number of other things in other contexts, but I admit I can't think of
> a better name.
>

For what it's worth, this sort of seems to be what one would expect if a 
file is *both* "fixed" and "immutable".

	-hpa

Re: e4defrag and immutable files

[Sunil Mushran]

On 06/02/2010 12:28 AM, H. Peter Anvin wrote:
> On 06/01/2010 12:49 PM, tytso@mit.edu wrote:
>> On Tue, Jun 01, 2010 at 12:32:29PM -0700, Sunil Mushran wrote:
>>>
>>> We (ocfs2) are looking to add a new attribute to denote files that
>>> have a fixed allocation on disk. But at the same time, allow writes
>>> that do not change the allocation on disk. No truncating, extending,
>>> filling holes, etc. We were thinking of calling it "Static" files.
>>
>> That's an interesting set of semantics, and it might make sense to
>> conflate that with a local disk "don't move or defrag" the file
>> option.  I'm not crazy with the name "static", since it could mean a
>> number of other things in other contexts, but I admit I can't think of
>> a better name.
>>
>
> For what it's worth, this sort of seems to be what one would expect if 
> a file is *both* "fixed" and "immutable".

"Immutable" means the contents do not change. But the file mappings
could change.

"Fixed mapping" means the mappings do not change but contents
could (as long as the ondisk mappings don't).

"Fixed metadata" means the entire inode (mappings included) cannot
change but the contents could (as long as the ondisk mappings don't).
(This does have the side effect of allowing writes without touching the
mtime. Like XFS' invisible i/o.)

What the boot loader needs is "Fixed mapping". I am sure there could be
other use cases for it too.

My suggestion would be to implement "Fixed mapping" as is and not have
it tied to "Fixed metadata". While we would like to use a chattr flag 
for this
feature, it is not a requirement.

Sunil

Re: e4defrag and immutable files

[H. Peter Anvin]

On 06/02/2010 11:02 AM, Sunil Mushran wrote:
> On 06/02/2010 12:28 AM, H. Peter Anvin wrote:
>> On 06/01/2010 12:49 PM, tytso@mit.edu wrote:
>>> On Tue, Jun 01, 2010 at 12:32:29PM -0700, Sunil Mushran wrote:
>>>>
>>>> We (ocfs2) are looking to add a new attribute to denote files that
>>>> have a fixed allocation on disk. But at the same time, allow writes
>>>> that do not change the allocation on disk. No truncating, extending,
>>>> filling holes, etc. We were thinking of calling it "Static" files.
>>>
>>> That's an interesting set of semantics, and it might make sense to
>>> conflate that with a local disk "don't move or defrag" the file
>>> option.  I'm not crazy with the name "static", since it could mean a
>>> number of other things in other contexts, but I admit I can't think of
>>> a better name.
>>>
>>
>> For what it's worth, this sort of seems to be what one would expect if 
>> a file is *both* "fixed" and "immutable".
> 
> "Immutable" means the contents do not change. But the file mappings
> could change.
> 
> "Fixed mapping" means the mappings do not change but contents
> could (as long as the ondisk mappings don't).
> 
> "Fixed metadata" means the entire inode (mappings included) cannot
> change but the contents could (as long as the ondisk mappings don't).
> (This does have the side effect of allowing writes without touching the
> mtime. Like XFS' invisible i/o.)
> 

Actually, if you're going to have three flags you might as well make
them orthogonal.  That is, separate "fixed contents", "fixed mappings",
"fixed metadata" -- and don't consider the mapping as metadata for this
purpose.

	-hpa

Re: e4defrag and immutable files

[Joel Becker]

On Tue, Jun 08, 2010 at 03:12:04PM -0700, H. Peter Anvin wrote:
> On 06/02/2010 11:02 AM, Sunil Mushran wrote:
> > "Immutable" means the contents do not change. But the file mappings
> > could change.
> > 
> > "Fixed mapping" means the mappings do not change but contents
> > could (as long as the ondisk mappings don't).
> > 
> > "Fixed metadata" means the entire inode (mappings included) cannot
> > change but the contents could (as long as the ondisk mappings don't).
> > (This does have the side effect of allowing writes without touching the
> > mtime. Like XFS' invisible i/o.)
> > 
> 
> Actually, if you're going to have three flags you might as well make
> them orthogonal.  That is, separate "fixed contents", "fixed mappings",
> "fixed metadata" -- and don't consider the mapping as metadata for this
> purpose.

	I think Sunil was defining terms, not suggesting three flags ;-)
That said, it does allow all the possible characteristics in the
discussion.  The only ugly think I can see is that we'll have three new
flags, plus the old immutable flag that means the same as setting all
three new flags.

Joel

-- 

"But all my words come back to me
 In shades of mediocrity.
 Like emptiness in harmony
 I need someone to comfort me."

Joel Becker
Principal Software Developer
Oracle
E-mail: joel.becker@oracle.com
Phone: (650) 506-8127

Re: e4defrag and immutable files

[tytso]

On Tue, Jun 08, 2010 at 06:10:28PM -0700, Joel Becker wrote:
> On Tue, Jun 08, 2010 at 03:12:04PM -0700, H. Peter Anvin wrote:
> > On 06/02/2010 11:02 AM, Sunil Mushran wrote:
> > > "Immutable" means the contents do not change. But the file mappings
> > > could change.
> > > 
> > > "Fixed mapping" means the mappings do not change but contents
> > > could (as long as the ondisk mappings don't).
> > > 
> > > "Fixed metadata" means the entire inode (mappings included) cannot
> > > change but the contents could (as long as the ondisk mappings don't).
> > > (This does have the side effect of allowing writes without touching the
> > > mtime. Like XFS' invisible i/o.)
> > 
> > Actually, if you're going to have three flags you might as well make
> > them orthogonal.  That is, separate "fixed contents", "fixed mappings",
> > "fixed metadata" -- and don't consider the mapping as metadata for this
> > purpose.
> 
> 	I think Sunil was defining terms, not suggesting three flags ;-)
> That said, it does allow all the possible characteristics in the
> discussion.  The only ugly think I can see is that we'll have three new
> flags, plus the old immutable flag that means the same as setting all
> three new flags.

Umm, what I think is being proposed --- and certainly what I think
makes sense --- is that immutable just means the data contents (just
as it has always meant), and we add two new flags, one which means
"fixed mappings", or that the location on disk must not change, and
"fixed metadata" which means all of the file metadata (not including
the mappings) must not change.

					- Ted