e2defrag 0.81 released

e2defrag 0.81 released

[Phillip Susi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

It's back!

Back in the days when dinosaurs walked the earth ( the '90s ), there was
a defragger for ext2 written by Stephen Tweedie with contributions from
others including Ted Ts'o.  After many years of abandonment and bit rot,
I have decided to take over maintainership of the package.  I now feel
that it is in good enough of a state for a wider audience ( including
working with extents and other ext4 features ), so I am announcing it
here and asking for testing.  The project page is
http://launchpad.net/e2defrag.

The program opens an unmounted block device and parses the filesystem
itself, assigns new locations for all blocks, packing them to the left
within their native block group if possible, then moves all the blocks
around quickly and efficiently.  The process it fast, but unsafe: should
anything go wrong or the process be interrupted, your fs will be toast.
 Therefore:

DO NOT USE ON A FS YOU CARE ABOUT AND/OR HAVE NOT BACKED UP FIRST

Should a bug cause it to trash your fs, a raw e2image ( e2image -r
/dev/sda1 - | bzip2 -c > sda1.e2i.bz2 ) made before the defrag, and
obviously saved on another fs, would be most helpful in debugging.

One of the more interesting features is the ability to pass a list of
inodes to be given priority over others.  This can be used to pack a set
of files together at the start of the disk to allow for faster booting.
 I cobbled together a simple python script, dump2inodes, that can obtain
the list of files that ureadahead ( Ubuntu ) loads during boot and
generates the inode priority listing you can pass to e2defrag, and this
gives some nice boot time improvements.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJP9243AAoJEJrBOlT6nu75GPAIAJuZ7IiBH1YinJXFTil+R6z9
EtLI4lZkHeWUGbVr/CnhV9l1iYIVo/aqPCBo4yH0efFGPSxF9z/KN8zIEvqs3Q30
nDe1dNACyvkeIfzyP7wkFxuqK7hX7wkF+cIYT24MftTGTfaXssnoJ5bGd9TyJ0+s
8fNM/Erq4saRBnq4aty/Q3FsAF9sc4ke5042FHNSM+xCFfHjZcsciTdPZP5svAcH
8crQFmR9ieDIbn1M54mWWp3hYTd368ScJSoiM+K+d8ofbrXRSapJ9bSvuhLjvoxE
Tx7QXFdHoJb+Id4yf6xI/OZjg9egPW6otkPWwUJfAIE2Wh2AAfnAsGNg4QA86Ws=
=PK87
-----END PGP SIGNATURE-----

Re: e2defrag 0.81 released

[Andreas Dilger]

[-- Attachment #1: Type: text/plain, Size: 3155 bytes --]

On 2012-07-06, at 5:01 PM, Phillip Susi wrote:
> Back in the days when dinosaurs walked the earth ( the '90s ), there was
> a defragger for ext2 written by Stephen Tweedie with contributions from
> others including Ted Ts'o.  After many years of abandonment and bit rot,
> I have decided to take over maintainership of the package.  I now feel
> that it is in good enough of a state for a wider audience ( including
> working with extents and other ext4 features ), so I am announcing it
> here and asking for testing.  The project page is
> http://launchpad.net/e2defrag.
> 
> The program opens an unmounted block device and parses the filesystem
> itself, assigns new locations for all blocks, packing them to the left
> within their native block group if possible, then moves all the blocks
> around quickly and efficiently.  The process it fast, but unsafe: should
> anything go wrong or the process be interrupted, your fs will be toast.

I hate to rain on your parade, but, are you aware of e4defrag?

It is already in e2fsprogs, and can be used on a mounted ext4 filesystem...
It also needs some lovin' to make it really robust, but would definitely
be a better starting point than the ancient e2defrag code...

That's why it is always a good idea to post to the mailing list _before_
you start on a project, to see what else is going on...  Definitely there
is a need for such a tool, but I hate to see effort being spent in two
different directions to make two so-so tools, when it could be going in
the same direction to make one excellent tool.

Cheers, Andreas

> Therefore:
> DO NOT USE ON A FS YOU CARE ABOUT AND/OR HAVE NOT BACKED UP FIRST
> 
> Should a bug cause it to trash your fs, a raw e2image ( e2image -r
> /dev/sda1 - | bzip2 -c > sda1.e2i.bz2 ) made before the defrag, and
> obviously saved on another fs, would be most helpful in debugging.
> 
> One of the more interesting features is the ability to pass a list of
> inodes to be given priority over others.  This can be used to pack a set
> of files together at the start of the disk to allow for faster booting.
> I cobbled together a simple python script, dump2inodes, that can obtain
> the list of files that ureadahead ( Ubuntu ) loads during boot and
> generates the inode priority listing you can pass to e2defrag, and this
> gives some nice boot time improvements.
> 
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.11 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
> 
> iQEcBAEBAgAGBQJP9243AAoJEJrBOlT6nu75GPAIAJuZ7IiBH1YinJXFTil+R6z9
> EtLI4lZkHeWUGbVr/CnhV9l1iYIVo/aqPCBo4yH0efFGPSxF9z/KN8zIEvqs3Q30
> nDe1dNACyvkeIfzyP7wkFxuqK7hX7wkF+cIYT24MftTGTfaXssnoJ5bGd9TyJ0+s
> 8fNM/Erq4saRBnq4aty/Q3FsAF9sc4ke5042FHNSM+xCFfHjZcsciTdPZP5svAcH
> 8crQFmR9ieDIbn1M54mWWp3hYTd368ScJSoiM+K+d8ofbrXRSapJ9bSvuhLjvoxE
> Tx7QXFdHoJb+Id4yf6xI/OZjg9egPW6otkPWwUJfAIE2Wh2AAfnAsGNg4QA86Ws=
> =PK87
> -----END PGP SIGNATURE-----
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


Cheers, Andreas






[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 235 bytes --]

Re: e2defrag 0.81 released

[Phillip Susi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 07/07/2012 02:02 AM, Andreas Dilger wrote:
> I hate to rain on your parade, but, are you aware of e4defrag?

Of course.  I started working on e2fsprogs two years ago though, when e4defrag was still on the drawing board, and just haven't gotten around to polishing and announcing it until now.

My understanding of e4defrag is that it works essentially like tools such as shake: find a fragmented file and copy it, hoping the copy will be less fragmented.  Thus, it does nothing for free space fragmentation, file locality, etc.  Plus, it doesn't have cool ansi graphics ;)

> It is already in e2fsprogs, and can be used on a mounted ext4 filesystem...
> It also needs some lovin' to make it really robust, but would definitely
> be a better starting point than the ancient e2defrag code...
> 
> That's why it is always a good idea to post to the mailing list _before_
> you start on a project, to see what else is going on...  Definitely there
> is a need for such a tool, but I hate to see effort being spent in two
> different directions to make two so-so tools, when it could be going in
> the same direction to make one excellent tool.

I'm hoping that eventually e4defrag will be able to do everything e2defrag can, and not too much slower, but for the moment, e2defrag already works, and I haven't seen any indication that e4defrag is planned to do free space defragging and file packing.  Perhaps having a little friendly competition will change that.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJP+E1MAAoJEJrBOlT6nu75bkYH/2rXRYqMYAA+D/EUwxfvq18p
ltrb3nOWXA7ZA6Gy1Lg+zcoG203F44XOKg8aNADS/1sYKkyiF5NhFcGRgXrp/gM/
0586WqpkAc0rhuk0vGcDinr8KOdaeTH3Ye5zN/6jju/bEls3Q5wpSCU71LfYYj62
Q+YOJK+TRq6r5NudIe6rIbNArVPMedPDw17L6yP4opi2iPrylW/WndStFFzjDAgK
Fa0/YRLl+SLwfKvCSorBtuSpuOToC1bfdq2ZmqIX7VCe7WThTVAzE9avDoaDhzIj
LK3ErsyztJZWPcfr34A1BEYZn2ZH5+Hsf+/Pmh4M8grMRgGfUhy1UU1/t7Zhyq0=
=Bxln
-----END PGP SIGNATURE-----

Re: e2defrag 0.81 released

[Greg Freemyer]

On Sat, Jul 7, 2012 at 10:53 AM, Phillip Susi <psusi@ubuntu.com> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 07/07/2012 02:02 AM, Andreas Dilger wrote:
>> I hate to rain on your parade, but, are you aware of e4defrag?
>
> Of course.  I started working on e2fsprogs two years ago though, when e4defrag was still on the drawing board, and just haven't gotten around to polishing and announcing it until now.

e4defrag has been around more than 2 years.  It's been at least 2
years since I did a code review of how it works.

> My understanding of e4defrag is that it works essentially like tools such as shake: find a fragmented file and copy it, hoping the copy will be less fragmented.  Thus, it does nothing for free space fragmentation, file locality, etc.  Plus, it doesn't have cool ansi graphics ;)

More or less true except iirc the process is more like:

find a fragmented file.
fallocate a new donor file with uninitialized data blocks..  (ie. data
blocks get assigned, but not written to.)
if new donor file is less fragmented than orig file:
   copy data from orig_blocks to donor_blocks
   swap donor blocks with original blocks at the inode level.
end if
delete donor

At a minimum I think e4defrag should work better with sparse files and
defrag each block group separately.

Also it used to hold the file lock for the entire time the copy data /
swap donor blocks operation took place.  For a large file that could
be a very long time.  I don't know if that was ever made less
intrusive or not.  ie. defragging a 100 GB file used to require
putting a i/o lock in place for the entire time it took to copy that
100GB of blocks to the donor blocks.

Of course, even that is better than having to unmount the filesystem
like you new tool apparently requires.

>> It is already in e2fsprogs, and can be used on a mounted ext4 filesystem...
>> It also needs some lovin' to make it really robust, but would definitely
>> be a better starting point than the ancient e2defrag code...
>>
>> That's why it is always a good idea to post to the mailing list _before_
>> you start on a project, to see what else is going on...  Definitely there
>> is a need for such a tool, but I hate to see effort being spent in two
>> different directions to make two so-so tools, when it could be going in
>> the same direction to make one excellent tool.
>
> I'm hoping that eventually e4defrag will be able to do everything e2defrag can, and not too much slower, but for the moment, e2defrag already works, and I haven't seen any indication that e4defrag is planned to do free space defragging and file packing.  Perhaps having a little friendly competition will change that.

I think you're right e4defrag userspace code ignores consolidating
freespace to make bigger extents possible and I don't think userspace
has any file packing specific knowledge.

Does fallocate itself assist with either?

ie. since e4defrag uses fallocate to allocate the donor file blocks,
logically it would make sense to add the intelligence you are looking
for into fallocate.  And hopefully it is already there.

Greg

Re: e2defrag 0.81 released

[Phillip Susi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 7/9/2012 12:16 PM, Greg Freemyer wrote:
> Of course, even that is better than having to unmount the
> filesystem like you new tool apparently requires.

Old tool... very old tool ;)

I remember using the thing with one of the first releases of slackware
when I first started playing with Linux, and iirc, DOS hadn't yet
included its own defragger.

> I think you're right e4defrag userspace code ignores consolidating 
> freespace to make bigger extents possible and I don't think
> userspace has any file packing specific knowledge.
> 
> Does fallocate itself assist with either?
> 
> ie. since e4defrag uses fallocate to allocate the donor file
> blocks, logically it would make sense to add the intelligence you
> are looking for into fallocate.  And hopefully it is already
> there.

You need a method of requesting specific blocks instead of just asking
for the right number.  Then you need to analyze all of the files on
the disk to figure out where their blocks are, and decide where you
want them to be instead.  Then you could use the donor inode method to
move things around.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJP+yfPAAoJEJrBOlT6nu75Y8AH/09qDUt0jdhPjNyXMUUFRVaV
U7Ih5j6r0srRHH9VmS7ppq6ingqpbrFEExbalg8ynTRX1nYDGL0nLiBU79rfK+KK
XOTK0ZYm+CUMElLdO5bYnaHti9vsSHL5xRyGmI5OR0CY7LwfnyJeDw6wS7+WsgyQ
o6ziTPcBEfaJzn9BBS+EP/1WI06lQ/cJbWUqUAJhefu2my3IXJUrjUYjnH46HKGM
2ptPbIJM2fT4QubBNnuPpdTNI5yc56+auzoB3hrK2Nf/Yh17w5R0MomHk/VitzPt
AD7zKLgpk6R9G/IbnajsMVzbLsjBbn8Tcyrhher2uyw3xE0XtJgj2fmPq32ZrL0=
=hMeu
-----END PGP SIGNATURE-----