All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH] ext4: allow defrag (EXT4_IOC_MOVE_EXT) in 32bit compat mode
@ 2010-03-07 20:32 Christian Borntraeger
  2010-03-07 23:27 ` defrag deployment status (was Re: [PATCH] ext4: allow defrag (EXT4_IOC_MOVE_EXT) in 32bit compat mode) Jeff Garzik
                   ` (4 more replies)
  0 siblings, 5 replies; 20+ messages in thread
From: Christian Borntraeger @ 2010-03-07 20:32 UTC (permalink / raw)
  To: linux-ext4; +Cc: linux-kernel, Akira Fujita

I have an x86_64 kernel with i386 userspace. e4defrag fails on the
EXT4_IOC_MOVE_EXT ioctl because it is not wired up for the compat
case. It seems that struct move_extent is compat save, only types
with fixed widths are used:
{
        __u32 reserved;         /* should be zero */
        __u32 donor_fd;         /* donor file descriptor */
        __u64 orig_start;       /* logical start offset in block for orig */
        __u64 donor_start;      /* logical start offset in block for donor */
        __u64 len;              /* block length to be moved */
        __u64 moved_len;        /* moved block length */
};

Lets just wire up EXT4_IOC_MOVE_EXT for the compat case.

Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
CCed: Akira Fujita <a-fujita@rs.jp.nec.com> 

---
 fs/ext4/ioctl.c |    2 ++
 1 file changed, 2 insertions(+)

--- a/fs/ext4/ioctl.c
+++ b/fs/ext4/ioctl.c
@@ -375,6 +375,8 @@ long ext4_compat_ioctl(struct file *file
 		break;
 	case EXT4_IOC_GROUP_ADD:
 		break;
+	case EXT4_IOC_MOVE_EXT:
+		break;
 	default:
 		return -ENOIOCTLCMD;
 	}

^ permalink raw reply	[flat|nested] 20+ messages in thread

* defrag deployment status (was Re: [PATCH] ext4: allow defrag (EXT4_IOC_MOVE_EXT) in 32bit compat mode)
  2010-03-07 20:32 [PATCH] ext4: allow defrag (EXT4_IOC_MOVE_EXT) in 32bit compat mode Christian Borntraeger
@ 2010-03-07 23:27 ` Jeff Garzik
  2010-03-08  5:43   ` Eric Sandeen
  2010-03-08  7:53     ` Christian Borntraeger
  2010-03-08  5:47 ` [PATCH] ext4: allow defrag (EXT4_IOC_MOVE_EXT) in 32bit compat mode Eric Sandeen
                   ` (3 subsequent siblings)
  4 siblings, 2 replies; 20+ messages in thread
From: Jeff Garzik @ 2010-03-07 23:27 UTC (permalink / raw)
  To: linux-ext4; +Cc: Christian Borntraeger, linux-kernel, Akira Fujita

On 03/07/2010 03:32 PM, Christian Borntraeger wrote:
> I have an x86_64 kernel with i386 userspace. e4defrag fails on the
> EXT4_IOC_MOVE_EXT ioctl because it is not wired up for the compat
> case. It seems that struct move_extent is compat save, only types
> with fixed widths are used:
> {
>          __u32 reserved;         /* should be zero */
>          __u32 donor_fd;         /* donor file descriptor */
>          __u64 orig_start;       /* logical start offset in block for orig */
>          __u64 donor_start;      /* logical start offset in block for donor */
>          __u64 len;              /* block length to be moved */
>          __u64 moved_len;        /* moved block length */
> };
>
> Lets just wire up EXT4_IOC_MOVE_EXT for the compat case.
>
> Signed-off-by: Christian Borntraeger<borntraeger@de.ibm.com>
> CCed: Akira Fujita<a-fujita@rs.jp.nec.com>

I'm curious, what is the overall deployment status of ext4 defragging? 
I actually worked on this problem years ago[1], and am hopeful that I 
will see defragging in a Linux distribution sometime in my lifetime!  ;-)

Looking at Fedora rawhide, I do not see anything resembling e4defrag in 
any of the RPM packages like e2fsprogs.

Thanks to everyone working on this,

	Jeff




[1] http://linux.yyz.us/misc/ext2meta.c


^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: defrag deployment status (was Re: [PATCH] ext4: allow defrag (EXT4_IOC_MOVE_EXT) in 32bit compat mode)
  2010-03-07 23:27 ` defrag deployment status (was Re: [PATCH] ext4: allow defrag (EXT4_IOC_MOVE_EXT) in 32bit compat mode) Jeff Garzik
@ 2010-03-08  5:43   ` Eric Sandeen
  2010-03-08  7:53     ` Christian Borntraeger
  1 sibling, 0 replies; 20+ messages in thread
From: Eric Sandeen @ 2010-03-08  5:43 UTC (permalink / raw)
  To: Jeff Garzik; +Cc: linux-ext4, Christian Borntraeger, linux-kernel, Akira Fujita

Jeff Garzik wrote:
> On 03/07/2010 03:32 PM, Christian Borntraeger wrote:
>> I have an x86_64 kernel with i386 userspace. e4defrag fails on the
>> EXT4_IOC_MOVE_EXT ioctl because it is not wired up for the compat
>> case. It seems that struct move_extent is compat save, only types
>> with fixed widths are used:
>> {
>>          __u32 reserved;         /* should be zero */
>>          __u32 donor_fd;         /* donor file descriptor */
>>          __u64 orig_start;       /* logical start offset in block for
>> orig */
>>          __u64 donor_start;      /* logical start offset in block for
>> donor */
>>          __u64 len;              /* block length to be moved */
>>          __u64 moved_len;        /* moved block length */
>> };
>>
>> Lets just wire up EXT4_IOC_MOVE_EXT for the compat case.
>>
>> Signed-off-by: Christian Borntraeger<borntraeger@de.ibm.com>
>> CCed: Akira Fujita<a-fujita@rs.jp.nec.com>
> 
> I'm curious, what is the overall deployment status of ext4 defragging? I
> actually worked on this problem years ago[1], and am hopeful that I will
> see defragging in a Linux distribution sometime in my lifetime!  ;-)

on ext4 you mean, I guess - you could use XFS if defragging is a high priority,
see xfs_fsr(8)

> Looking at Fedora rawhide, I do not see anything resembling e4defrag in
> any of the RPM packages like e2fsprogs.

I had it for a while, but with the various problems and general uneasiness
with the code so far, I took it back out lest people lose data to it.

-Eric

> Thanks to everyone working on this,
> 
>     Jeff
> 
> 
> 
> 
> [1] http://linux.yyz.us/misc/ext2meta.c


^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: [PATCH] ext4: allow defrag (EXT4_IOC_MOVE_EXT) in 32bit compat mode
  2010-03-07 20:32 [PATCH] ext4: allow defrag (EXT4_IOC_MOVE_EXT) in 32bit compat mode Christian Borntraeger
  2010-03-07 23:27 ` defrag deployment status (was Re: [PATCH] ext4: allow defrag (EXT4_IOC_MOVE_EXT) in 32bit compat mode) Jeff Garzik
@ 2010-03-08  5:47 ` Eric Sandeen
  2010-03-08  8:42 ` Akira Fujita
                   ` (2 subsequent siblings)
  4 siblings, 0 replies; 20+ messages in thread
From: Eric Sandeen @ 2010-03-08  5:47 UTC (permalink / raw)
  To: Christian Borntraeger; +Cc: linux-ext4, linux-kernel, Akira Fujita

Christian Borntraeger wrote:
> I have an x86_64 kernel with i386 userspace. e4defrag fails on the
> EXT4_IOC_MOVE_EXT ioctl because it is not wired up for the compat
> case. It seems that struct move_extent is compat save, only types
> with fixed widths are used:

and they're well-aligned.

> {
>         __u32 reserved;         /* should be zero */
>         __u32 donor_fd;         /* donor file descriptor */
>         __u64 orig_start;       /* logical start offset in block for orig */
>         __u64 donor_start;      /* logical start offset in block for donor */
>         __u64 len;              /* block length to be moved */
>         __u64 moved_len;        /* moved block length */
> };
> 
> Lets just wire up EXT4_IOC_MOVE_EXT for the compat case.
> 
> Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
> CCed: Akira Fujita <a-fujita@rs.jp.nec.com> 

Looks fine to me.

Reviewed-by: Eric Sandeen <sandeen@redhat.com>

> ---
>  fs/ext4/ioctl.c |    2 ++
>  1 file changed, 2 insertions(+)
> 
> --- a/fs/ext4/ioctl.c
> +++ b/fs/ext4/ioctl.c
> @@ -375,6 +375,8 @@ long ext4_compat_ioctl(struct file *file
>  		break;
>  	case EXT4_IOC_GROUP_ADD:
>  		break;
> +	case EXT4_IOC_MOVE_EXT:
> +		break;
>  	default:
>  		return -ENOIOCTLCMD;
>  	}
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: defrag deployment status (was Re: [PATCH] ext4: allow defrag (EXT4_IOC_MOVE_EXT) in 32bit compat mode)
  2010-03-07 23:27 ` defrag deployment status (was Re: [PATCH] ext4: allow defrag (EXT4_IOC_MOVE_EXT) in 32bit compat mode) Jeff Garzik
@ 2010-03-08  7:53     ` Christian Borntraeger
  2010-03-08  7:53     ` Christian Borntraeger
  1 sibling, 0 replies; 20+ messages in thread
From: Christian Borntraeger @ 2010-03-08  7:53 UTC (permalink / raw)
  To: Jeff Garzik; +Cc: linux-ext4, linux-kernel, Akira Fujita

Am Montag 08 März 2010 00:27:58 schrieb Jeff Garzik:
> I'm curious, what is the overall deployment status of ext4 defragging? 
> I actually worked on this problem years ago[1], and am hopeful that I 
> will see defragging in a Linux distribution sometime in my lifetime!  ;-)

I had to make the following patch. (There was the same patch already on the
e2fsprogs mailing list, but I cannot find that at the moment. Would be good
if this patch can make it into the next version).

--- a/misc/e4defrag.c
+++ b/misc/e4defrag.c
@@ -1605,7 +1605,7 @@ static int file_defrag(const char *file, const struct stat64 *buf,
                return 0;
        }

-       fd = open64(file, O_RDONLY);
+       fd = open64(file, O_RDWR);
        if (fd < 0) {
                if (mode_flag & DETAIL) {
                        PRINT_FILE_NAME(file);


I did use it on two machines and it does not seem to break anything. 
Some bigger things are missing in the e4defrag tool:
- make the indirect to extend change for old files - e.g. converted ext3
  file systems. Adding the MIGRATE ioctl does not look very hard, but who knows?
- defragging directories (and also remove deleted entries)
- free space optimization is only done as a side effect
- overall layout considerations (e.g. putting files close to its directory or
  use the atime to move often used files to the beginning of a disk etc.)

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: defrag deployment status (was Re: [PATCH] ext4: allow defrag (EXT4_IOC_MOVE_EXT) in 32bit compat mode)
@ 2010-03-08  7:53     ` Christian Borntraeger
  0 siblings, 0 replies; 20+ messages in thread
From: Christian Borntraeger @ 2010-03-08  7:53 UTC (permalink / raw)
  To: Jeff Garzik; +Cc: linux-ext4, linux-kernel, Akira Fujita

Am Montag 08 März 2010 00:27:58 schrieb Jeff Garzik:
> I'm curious, what is the overall deployment status of ext4 defragging? 
> I actually worked on this problem years ago[1], and am hopeful that I 
> will see defragging in a Linux distribution sometime in my lifetime!  ;-)

I had to make the following patch. (There was the same patch already on the
e2fsprogs mailing list, but I cannot find that at the moment. Would be good
if this patch can make it into the next version).

--- a/misc/e4defrag.c
+++ b/misc/e4defrag.c
@@ -1605,7 +1605,7 @@ static int file_defrag(const char *file, const struct stat64 *buf,
                return 0;
        }

-       fd = open64(file, O_RDONLY);
+       fd = open64(file, O_RDWR);
        if (fd < 0) {
                if (mode_flag & DETAIL) {
                        PRINT_FILE_NAME(file);


I did use it on two machines and it does not seem to break anything. 
Some bigger things are missing in the e4defrag tool:
- make the indirect to extend change for old files - e.g. converted ext3
  file systems. Adding the MIGRATE ioctl does not look very hard, but who knows?
- defragging directories (and also remove deleted entries)
- free space optimization is only done as a side effect
- overall layout considerations (e.g. putting files close to its directory or
  use the atime to move often used files to the beginning of a disk etc.)
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: [PATCH] ext4: allow defrag (EXT4_IOC_MOVE_EXT) in 32bit compat mode
  2010-03-07 20:32 [PATCH] ext4: allow defrag (EXT4_IOC_MOVE_EXT) in 32bit compat mode Christian Borntraeger
  2010-03-07 23:27 ` defrag deployment status (was Re: [PATCH] ext4: allow defrag (EXT4_IOC_MOVE_EXT) in 32bit compat mode) Jeff Garzik
  2010-03-08  5:47 ` [PATCH] ext4: allow defrag (EXT4_IOC_MOVE_EXT) in 32bit compat mode Eric Sandeen
@ 2010-03-08  8:42 ` Akira Fujita
  2010-03-12  7:01 ` [PATCH resend] " Christian Borntraeger
  2010-04-02 21:48 ` [PATCH] " tytso
  4 siblings, 0 replies; 20+ messages in thread
From: Akira Fujita @ 2010-03-08  8:42 UTC (permalink / raw)
  To: Christian Borntraeger; +Cc: linux-ext4, linux-kernel


Hi Christian,

(2010/03/08 5:32), Christian Borntraeger wrote:
> I have an x86_64 kernel with i386 userspace. e4defrag fails on the
> EXT4_IOC_MOVE_EXT ioctl because it is not wired up for the compat
> case. It seems that struct move_extent is compat save, only types
> with fixed widths are used:
> {
>          __u32 reserved;         /* should be zero */
>          __u32 donor_fd;         /* donor file descriptor */
>          __u64 orig_start;       /* logical start offset in block for orig */
>          __u64 donor_start;      /* logical start offset in block for donor */
>          __u64 len;              /* block length to be moved */
>          __u64 moved_len;        /* moved block length */
> };
>
> Lets just wire up EXT4_IOC_MOVE_EXT for the compat case.
>
> Signed-off-by: Christian Borntraeger<borntraeger@de.ibm.com>
> CCed: Akira Fujita<a-fujita@rs.jp.nec.com>
>
> ---
>   fs/ext4/ioctl.c |    2 ++
>   1 file changed, 2 insertions(+)
>
> --- a/fs/ext4/ioctl.c
> +++ b/fs/ext4/ioctl.c
> @@ -375,6 +375,8 @@ long ext4_compat_ioctl(struct file *file
>   		break;
>   	case EXT4_IOC_GROUP_ADD:
>   		break;
> +	case EXT4_IOC_MOVE_EXT:
> +		break;
>   	default:
>   		return -ENOIOCTLCMD;
>   	}

Looks good for me.
Thank you for your work.

Regards,
Akira Fujita



^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: defrag deployment status (was Re: [PATCH] ext4: allow defrag (EXT4_IOC_MOVE_EXT) in 32bit compat mode)
  2010-03-08  7:53     ` Christian Borntraeger
  (?)
@ 2010-03-08 15:33     ` David Newall
  2010-03-08 16:00         ` Christian Borntraeger
  2010-03-08 16:22       ` jim owens
  -1 siblings, 2 replies; 20+ messages in thread
From: David Newall @ 2010-03-08 15:33 UTC (permalink / raw)
  To: Christian Borntraeger; +Cc: Jeff Garzik, linux-ext4, linux-kernel, Akira Fujita

Christian Borntraeger wrote:
> Some bigger things are missing in the e4defrag tool:
> ...
> - overall layout considerations (e.g. putting files close to its directory or
>   use the atime to move often used files to the beginning of a disk etc.)

Shouldn't oft-used files be placed closer to the middle?  If you place 
them at the beginning of the file, it's only possible for the head-stack 
to be close to the file from the inner direction.  Place them in the 
middle and it's possible for the head-stack to be close from the outer 
direction, too, which sounds like a doubling of probability.  It seems 
that it's the least frequently used files that should be placed at one 
end of the disk or the other.

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: defrag deployment status (was Re: [PATCH] ext4: allow defrag (EXT4_IOC_MOVE_EXT) in 32bit compat mode)
  2010-03-08 15:33     ` David Newall
@ 2010-03-08 16:00         ` Christian Borntraeger
  2010-03-08 16:22       ` jim owens
  1 sibling, 0 replies; 20+ messages in thread
From: Christian Borntraeger @ 2010-03-08 16:00 UTC (permalink / raw)
  To: David Newall; +Cc: Jeff Garzik, linux-ext4, linux-kernel, Akira Fujita

Am Montag 08 März 2010 16:33:46 schrieb David Newall:
> Christian Borntraeger wrote:
> > Some bigger things are missing in the e4defrag tool:
> > ...
> > - overall layout considerations (e.g. putting files close to its directory or
> >   use the atime to move often used files to the beginning of a disk etc.)
> 
> Shouldn't oft-used files be placed closer to the middle?  If you place 
> them at the beginning of the file, it's only possible for the head-stack 
> to be close to the file from the inner direction.  Place them in the 
> middle and it's possible for the head-stack to be close from the outer 
> direction, too, which sounds like a doubling of probability.  It seems 
> that it's the least frequently used files that should be placed at one 
> end of the disk or the other.
> 

Maybe. This was just an example of things that you can come
up with and which are not yet possible up in e4defrag. 

Christian

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: defrag deployment status (was Re: [PATCH] ext4: allow defrag (EXT4_IOC_MOVE_EXT) in 32bit compat mode)
@ 2010-03-08 16:00         ` Christian Borntraeger
  0 siblings, 0 replies; 20+ messages in thread
From: Christian Borntraeger @ 2010-03-08 16:00 UTC (permalink / raw)
  To: David Newall; +Cc: Jeff Garzik, linux-ext4, linux-kernel, Akira Fujita

Am Montag 08 März 2010 16:33:46 schrieb David Newall:
> Christian Borntraeger wrote:
> > Some bigger things are missing in the e4defrag tool:
> > ...
> > - overall layout considerations (e.g. putting files close to its directory or
> >   use the atime to move often used files to the beginning of a disk etc.)
> 
> Shouldn't oft-used files be placed closer to the middle?  If you place 
> them at the beginning of the file, it's only possible for the head-stack 
> to be close to the file from the inner direction.  Place them in the 
> middle and it's possible for the head-stack to be close from the outer 
> direction, too, which sounds like a doubling of probability.  It seems 
> that it's the least frequently used files that should be placed at one 
> end of the disk or the other.
> 

Maybe. This was just an example of things that you can come
up with and which are not yet possible up in e4defrag. 

Christian
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: defrag deployment status (was Re: [PATCH] ext4: allow defrag (EXT4_IOC_MOVE_EXT) in 32bit compat mode)
  2010-03-08 15:33     ` David Newall
  2010-03-08 16:00         ` Christian Borntraeger
@ 2010-03-08 16:22       ` jim owens
  2010-03-08 16:31           ` Greg Freemyer
                           ` (2 more replies)
  1 sibling, 3 replies; 20+ messages in thread
From: jim owens @ 2010-03-08 16:22 UTC (permalink / raw)
  To: David Newall
  Cc: Christian Borntraeger, Jeff Garzik, linux-ext4, linux-kernel,
	Akira Fujita

David Newall wrote:
> Christian Borntraeger wrote:
>> Some bigger things are missing in the e4defrag tool:
>> ...
>> - overall layout considerations (e.g. putting files close to its
>> directory or
>>   use the atime to move often used files to the beginning of a disk etc.)
> 
> Shouldn't oft-used files be placed closer to the middle?  If you place
> them at the beginning of the file, it's only possible for the head-stack
> to be close to the file from the inner direction.  Place them in the
> middle and it's possible for the head-stack to be close from the outer
> direction, too, which sounds like a doubling of probability.  It seems
> that it's the least frequently used files that should be placed at one
> end of the disk or the other.

No.  Your logic would be correct if rotating disks had
similar speed at all locations.  Current disks are much
faster at the 0 end than at the middle or highest address.

It is not unusual to see 2x difference in transfer speed
so you always want the important stuff as low as possible.

jim

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: defrag deployment status (was Re: [PATCH] ext4: allow defrag  (EXT4_IOC_MOVE_EXT) in 32bit compat mode)
  2010-03-08 16:22       ` jim owens
@ 2010-03-08 16:31           ` Greg Freemyer
  2010-03-08 19:38         ` Valdis.Kletnieks
  2010-03-09 16:19         ` David Newall
  2 siblings, 0 replies; 20+ messages in thread
From: Greg Freemyer @ 2010-03-08 16:31 UTC (permalink / raw)
  To: jim owens
  Cc: David Newall, Christian Borntraeger, Jeff Garzik, linux-ext4,
	linux-kernel, Akira Fujita

On Mon, Mar 8, 2010 at 11:22 AM, jim owens <owens6336@gmail.com> wrote:
> David Newall wrote:
>> Christian Borntraeger wrote:
>>> Some bigger things are missing in the e4defrag tool:
>>> ...
>>> - overall layout considerations (e.g. putting files close to its
>>> directory or
>>>   use the atime to move often used files to the beginning of a disk etc.)
>>
>> Shouldn't oft-used files be placed closer to the middle?  If you place
>> them at the beginning of the file, it's only possible for the head-stack
>> to be close to the file from the inner direction.  Place them in the
>> middle and it's possible for the head-stack to be close from the outer
>> direction, too, which sounds like a doubling of probability.  It seems
>> that it's the least frequently used files that should be placed at one
>> end of the disk or the other.
>
> No.  Your logic would be correct if rotating disks had
> similar speed at all locations.  Current disks are much
> faster at the 0 end than at the middle or highest address.
>
> It is not unusual to see 2x difference in transfer speed
> so you always want the important stuff as low as possible.
>
> jim

Jim, I should know this, but is sector 0 on the outside edge, or the inner edge?

I assume outer so that the linear speed of the platter under the head
is faster and thus more data per second is passing under the head.

Greg

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: defrag deployment status (was Re: [PATCH] ext4: allow defrag (EXT4_IOC_MOVE_EXT) in 32bit compat mode)
@ 2010-03-08 16:31           ` Greg Freemyer
  0 siblings, 0 replies; 20+ messages in thread
From: Greg Freemyer @ 2010-03-08 16:31 UTC (permalink / raw)
  To: jim owens
  Cc: David Newall, Christian Borntraeger, Jeff Garzik, linux-ext4,
	linux-kernel, Akira Fujita

On Mon, Mar 8, 2010 at 11:22 AM, jim owens <owens6336@gmail.com> wrote:
> David Newall wrote:
>> Christian Borntraeger wrote:
>>> Some bigger things are missing in the e4defrag tool:
>>> ...
>>> - overall layout considerations (e.g. putting files close to its
>>> directory or
>>>   use the atime to move often used files to the beginning of a disk etc.)
>>
>> Shouldn't oft-used files be placed closer to the middle?  If you place
>> them at the beginning of the file, it's only possible for the head-stack
>> to be close to the file from the inner direction.  Place them in the
>> middle and it's possible for the head-stack to be close from the outer
>> direction, too, which sounds like a doubling of probability.  It seems
>> that it's the least frequently used files that should be placed at one
>> end of the disk or the other.
>
> No.  Your logic would be correct if rotating disks had
> similar speed at all locations.  Current disks are much
> faster at the 0 end than at the middle or highest address.
>
> It is not unusual to see 2x difference in transfer speed
> so you always want the important stuff as low as possible.
>
> jim

Jim, I should know this, but is sector 0 on the outside edge, or the inner edge?

I assume outer so that the linear speed of the platter under the head
is faster and thus more data per second is passing under the head.

Greg
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: defrag deployment status (was Re: [PATCH] ext4: allow defrag (EXT4_IOC_MOVE_EXT) in 32bit compat mode)
  2010-03-08 16:31           ` Greg Freemyer
  (?)
@ 2010-03-08 17:11           ` jim owens
  2010-03-09 13:23             ` jim owens
  -1 siblings, 1 reply; 20+ messages in thread
From: jim owens @ 2010-03-08 17:11 UTC (permalink / raw)
  To: Greg Freemyer
  Cc: David Newall, Christian Borntraeger, Jeff Garzik, linux-ext4,
	linux-kernel, Akira Fujita

Greg Freemyer wrote:
> 
> Jim, I should know this, but is sector 0 on the outside edge, or the inner edge?
> 
> I assume outer so that the linear speed of the platter under the head
> is faster and thus more data per second is passing under the head.

Correct.  AFAIK everyone starts 0 at the outer edge for that reason,
it makes for better benchmarks ;)

When I only worried about a few OEM drives, I used to read the zone
geometry from the drive to see where each speed transition was as the
density decreased.  But that is just not worth the effort in linux
filesystems IMO, it is enough to pack low.

jim

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: defrag deployment status (was Re: [PATCH] ext4: allow defrag (EXT4_IOC_MOVE_EXT) in 32bit compat mode)
  2010-03-08 16:22       ` jim owens
  2010-03-08 16:31           ` Greg Freemyer
@ 2010-03-08 19:38         ` Valdis.Kletnieks
  2010-03-08 20:48           ` jim owens
  2010-03-09 16:19         ` David Newall
  2 siblings, 1 reply; 20+ messages in thread
From: Valdis.Kletnieks @ 2010-03-08 19:38 UTC (permalink / raw)
  To: jim owens
  Cc: David Newall, Christian Borntraeger, Jeff Garzik, linux-ext4,
	linux-kernel, Akira Fujita

[-- Attachment #1: Type: text/plain, Size: 625 bytes --]

On Mon, 08 Mar 2010 11:22:15 EST, jim owens said:

> No.  Your logic would be correct if rotating disks had
> similar speed at all locations.  Current disks are much
> faster at the 0 end than at the middle or highest address.
> 
> It is not unusual to see 2x difference in transfer speed
> so you always want the important stuff as low as possible.

On the flip side, seek time is so much larger than the time spent
actually reading that minimizing the seeks will improve total throughput
more.  Sure, maybe you spend 0.05ms reading instead of 0.1ms - but if
the seek took 0.75ms rather than 0.5ms you're still in the hole.

[-- Attachment #2: Type: application/pgp-signature, Size: 227 bytes --]

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: defrag deployment status (was Re: [PATCH] ext4: allow defrag (EXT4_IOC_MOVE_EXT) in 32bit compat mode)
  2010-03-08 19:38         ` Valdis.Kletnieks
@ 2010-03-08 20:48           ` jim owens
  0 siblings, 0 replies; 20+ messages in thread
From: jim owens @ 2010-03-08 20:48 UTC (permalink / raw)
  To: Valdis.Kletnieks
  Cc: David Newall, Christian Borntraeger, Jeff Garzik, linux-ext4,
	linux-kernel, Akira Fujita

Valdis.Kletnieks@vt.edu wrote:
> On Mon, 08 Mar 2010 11:22:15 EST, jim owens said:
> 
>> No.  Your logic would be correct if rotating disks had
>> similar speed at all locations.  Current disks are much
>> faster at the 0 end than at the middle or highest address.
>>
>> It is not unusual to see 2x difference in transfer speed
>> so you always want the important stuff as low as possible.
> 
> On the flip side, seek time is so much larger than the time spent
> actually reading that minimizing the seeks will improve total throughput
> more.  Sure, maybe you spend 0.05ms reading instead of 0.1ms - but if
> the seek took 0.75ms rather than 0.5ms you're still in the hole.

Agree that seek proximity is important, but seeking at the low end
for N blocks is also faster than seeking the same N blocks at the
slow high end of the disk. 

So my point is you want to pack the data together when you can
at the low address end of the disk to maximize performance.
And we should include some free space in the fast zone because
much of our performance involves short-lived writes.

If you are filling and accessing all of the disk, then maybe
"pack from the middle" will average out better, but we hope
most filesystems have a small hot subset.  If you are often
accessing outside the hot zone a lot then you probably will
get end-to-end seek penalties anyway if you are defragging
to an "old, free, hot, free, old" layout.

jim 

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: defrag deployment status (was Re: [PATCH] ext4: allow defrag (EXT4_IOC_MOVE_EXT) in 32bit compat mode)
  2010-03-08 17:11           ` jim owens
@ 2010-03-09 13:23             ` jim owens
  0 siblings, 0 replies; 20+ messages in thread
From: jim owens @ 2010-03-09 13:23 UTC (permalink / raw)
  To: Greg Freemyer
  Cc: David Newall, Christian Borntraeger, Jeff Garzik, linux-ext4,
	linux-kernel, Akira Fujita

After thinking about it overnight, I realized I think in terms
of 1 drive is 1 filesystem.  That is a fatal trap for defragment.
 
> When I only worried about a few OEM drives, I used to read the zone
> geometry from the drive to see where each speed transition was as the
> density decreased.  But that is just not worth the effort in linux
> filesystems IMO, it is enough to pack low.

So I retract that we don't care about zone geometry, we need to
care deeply, but not in the sense of how moving short distances
on a drive affects the performance.  What we need to ensure is
that the placer algorithm does not span across partitions as in:

["/" 100GB created] [300GB other] [100G LVM added to "/"]

so the filesystem thinks it is 200GB contiguous and the
defragmenter thinks address 90GB is closer to address 110 GB
than 90GB is to 50GB.

jim

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: defrag deployment status (was Re: [PATCH] ext4: allow defrag (EXT4_IOC_MOVE_EXT) in 32bit compat mode)
  2010-03-08 16:22       ` jim owens
  2010-03-08 16:31           ` Greg Freemyer
  2010-03-08 19:38         ` Valdis.Kletnieks
@ 2010-03-09 16:19         ` David Newall
  2 siblings, 0 replies; 20+ messages in thread
From: David Newall @ 2010-03-09 16:19 UTC (permalink / raw)
  To: jim owens
  Cc: Christian Borntraeger, Jeff Garzik, linux-ext4, linux-kernel,
	Akira Fujita

jim owens wrote:
> No.  Your logic would be correct if rotating disks had
> similar speed at all locations.  Current disks are much
> faster at the 0 end than at the middle or highest address.
>   

I think  my logic is still correct, although I wished I had said "closer 
to the middle."  In fact, simplistic ideas for placement of files are 
unlikely to produce fabulous results (and that includes placing commonly 
used files towards the middle of the disk, say at the inside edge of the 
outermost zone.)  The effort that BSD went to in FFS, placing 
directories with files and meta-data in cylinder groups, illustrates 
that disk performance is a sophisticated problem.

Why don't we use BSD FFS/FFS2?

^ permalink raw reply	[flat|nested] 20+ messages in thread

* [PATCH resend] ext4: allow defrag (EXT4_IOC_MOVE_EXT) in 32bit compat mode
  2010-03-07 20:32 [PATCH] ext4: allow defrag (EXT4_IOC_MOVE_EXT) in 32bit compat mode Christian Borntraeger
                   ` (2 preceding siblings ...)
  2010-03-08  8:42 ` Akira Fujita
@ 2010-03-12  7:01 ` Christian Borntraeger
  2010-04-02 21:48 ` [PATCH] " tytso
  4 siblings, 0 replies; 20+ messages in thread
From: Christian Borntraeger @ 2010-03-12  7:01 UTC (permalink / raw)
  To: tytso; +Cc: linux-ext4, linux-kernel, Akira Fujita, Eric Sandeen

Ted, can you consider this patch for ext4?

From: Christian Borntraeger <borntraeger@de.ibm.com>

I have an x86_64 kernel with i386 userspace. e4defrag fails on the
EXT4_IOC_MOVE_EXT ioctl because it is not wired up for the compat
case. It seems that 
struct move_extent is compat save, only types with fixed widths are used.
{
        __u32 reserved;         /* should be zero */
        __u32 donor_fd;         /* donor file descriptor */
        __u64 orig_start;       /* logical start offset in block for orig */
        __u64 donor_start;      /* logical start offset in block for donor */
        __u64 len;              /* block length to be moved */
        __u64 moved_len;        /* moved block length */
};


Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
Reviewed-by: Eric Sandeen <sandeen@redhat.com>
Acked-by: Akira Fujita <a-fujita@rs.jp.nec.com> 

---
 fs/ext4/ioctl.c |    2 ++
 1 file changed, 2 insertions(+)

--- a/fs/ext4/ioctl.c
+++ b/fs/ext4/ioctl.c
@@ -375,6 +375,8 @@ long ext4_compat_ioctl(struct file *file
 		break;
 	case EXT4_IOC_GROUP_ADD:
 		break;
+	case EXT4_IOC_MOVE_EXT:
+		break;
 	default:
 		return -ENOIOCTLCMD;
 	}


^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: [PATCH] ext4: allow defrag (EXT4_IOC_MOVE_EXT) in 32bit compat mode
  2010-03-07 20:32 [PATCH] ext4: allow defrag (EXT4_IOC_MOVE_EXT) in 32bit compat mode Christian Borntraeger
                   ` (3 preceding siblings ...)
  2010-03-12  7:01 ` [PATCH resend] " Christian Borntraeger
@ 2010-04-02 21:48 ` tytso
  4 siblings, 0 replies; 20+ messages in thread
From: tytso @ 2010-04-02 21:48 UTC (permalink / raw)
  To: Christian Borntraeger; +Cc: linux-ext4, linux-kernel, Akira Fujita

On Sun, Mar 07, 2010 at 09:32:10PM +0100, Christian Borntraeger wrote:
> I have an x86_64 kernel with i386 userspace. e4defrag fails on the
> EXT4_IOC_MOVE_EXT ioctl because it is not wired up for the compat
> case. It seems that struct move_extent is compat save, only types
> with fixed widths are used:
> {
>         __u32 reserved;         /* should be zero */
>         __u32 donor_fd;         /* donor file descriptor */
>         __u64 orig_start;       /* logical start offset in block for orig */
>         __u64 donor_start;      /* logical start offset in block for donor */
>         __u64 len;              /* block length to be moved */
>         __u64 moved_len;        /* moved block length */
> };
> 
> Lets just wire up EXT4_IOC_MOVE_EXT for the compat case.

Thanks, added to the ext4 patch queue.

							- Ted

^ permalink raw reply	[flat|nested] 20+ messages in thread

end of thread, other threads:[~2010-04-02 21:48 UTC | newest]

Thread overview: 20+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2010-03-07 20:32 [PATCH] ext4: allow defrag (EXT4_IOC_MOVE_EXT) in 32bit compat mode Christian Borntraeger
2010-03-07 23:27 ` defrag deployment status (was Re: [PATCH] ext4: allow defrag (EXT4_IOC_MOVE_EXT) in 32bit compat mode) Jeff Garzik
2010-03-08  5:43   ` Eric Sandeen
2010-03-08  7:53   ` Christian Borntraeger
2010-03-08  7:53     ` Christian Borntraeger
2010-03-08 15:33     ` David Newall
2010-03-08 16:00       ` Christian Borntraeger
2010-03-08 16:00         ` Christian Borntraeger
2010-03-08 16:22       ` jim owens
2010-03-08 16:31         ` Greg Freemyer
2010-03-08 16:31           ` Greg Freemyer
2010-03-08 17:11           ` jim owens
2010-03-09 13:23             ` jim owens
2010-03-08 19:38         ` Valdis.Kletnieks
2010-03-08 20:48           ` jim owens
2010-03-09 16:19         ` David Newall
2010-03-08  5:47 ` [PATCH] ext4: allow defrag (EXT4_IOC_MOVE_EXT) in 32bit compat mode Eric Sandeen
2010-03-08  8:42 ` Akira Fujita
2010-03-12  7:01 ` [PATCH resend] " Christian Borntraeger
2010-04-02 21:48 ` [PATCH] " tytso

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.