State of dump utility

State of dump utility

[Lukas Czerner]

Hello,

lately we've had couple of bugs against dump utility and a after a quick
look at the code I realized that it is very much outdated at least on
the extN side of things and would need some work and attention to make it
work reliably with modern ext4 features.

However the code has been neglected for a while and talking to the
maintainer he is pretty much done with it. At this point I am ready to
pull the plug on dump/restore in Fedora, but before I do I was wondering
whether there is any interest in moving dump/restore, or part of it, into
e2fsprogs ?

I have not looked at the code close enought to say whether it's worth it
or whether it would be better to write something from scratch. There is
also a question about what to do with the tape code - that's not
something I have any interest in digging into.

In my eyes dump had a good run and I would be happy just dumping it, but
it is worth asking here on the list. Is there anyone interested in
maintaining dump/restore, or is there interest in or objections agains
merging it into e2fsprogs ?

Thanks!
-Lukas

Re: State of dump utility

[Theodore Y. Ts'o]

On Tue, Sep 29, 2020 at 04:37:13PM +0200, Lukas Czerner wrote:
> 
> lately we've had couple of bugs against dump utility and a after a quick
> look at the code I realized that it is very much outdated at least on
> the extN side of things and would need some work and attention to make it
> work reliably with modern ext4 features.
> 
> However the code has been neglected for a while and talking to the
> maintainer he is pretty much done with it. At this point I am ready to
> pull the plug on dump/restore in Fedora, but before I do I was wondering
> whether there is any interest in moving dump/restore, or part of it, into
> e2fsprogs ?
> 
> I have not looked at the code close enought to say whether it's worth it
> or whether it would be better to write something from scratch. There is
> also a question about what to do with the tape code - that's not
> something I have any interest in digging into.
> 
> In my eyes dump had a good run and I would be happy just dumping it, but
> it is worth asking here on the list. Is there anyone interested in
> maintaining dump/restore, or is there interest in or objections agains
> merging it into e2fsprogs ?

One of the interesting questions is how reliable the dump utility
really is; that's because it works by reading the metadata directly
--- while the file system is mounted.  So it's quite possible for the
metadata to be changing out from under the dump/restore process.
Especially with metadata checksums, I suspect dump/restore is going
much more unreliable in terms of the libext2fs returning checksum
failures.

In the future, if we ever try to bypass the use of the buffer cache,
and instead have jbd2 write out directly to the bio layer so we cant
get better write error codes.  There was a discussion about this
recently, and there are two problems.  First, we need to worry about
programs like tune2fs and e2label that need to be able to read and
modify the superblock while the file system is modified.  We'd want to
add ioctl's to set and get the superblock, and update e2fsprogs to try
to use those system calls first.  And then.... there is dump/restore.i

I could imagine adding ioctl's which allow safe read-only access to
all metadata blocks, and not just for the superblock.  The question,
though is... is it worth it, especially if it's only to make
dump/restore work?

On the other hand, if we want to try to implement some kind of on-line
fsck work, then -perhaps safe metadata reading would be part of that
interface.  So I'd never say never, but I do wonder if it's time to
pull the plug on dump/restore --- especially if we want to allow it to
support not just inline files/directories, but also things like
extended attributes and ACL's.

						- Ted

Re: State of dump utility

[Lukas Czerner]

On Tue, Sep 29, 2020 at 10:06:46PM -0400, Theodore Y. Ts'o wrote:
> On Tue, Sep 29, 2020 at 04:37:13PM +0200, Lukas Czerner wrote:
> > 
> > lately we've had couple of bugs against dump utility and a after a quick
> > look at the code I realized that it is very much outdated at least on
> > the extN side of things and would need some work and attention to make it
> > work reliably with modern ext4 features.
> > 
> > However the code has been neglected for a while and talking to the
> > maintainer he is pretty much done with it. At this point I am ready to
> > pull the plug on dump/restore in Fedora, but before I do I was wondering
> > whether there is any interest in moving dump/restore, or part of it, into
> > e2fsprogs ?
> > 
> > I have not looked at the code close enought to say whether it's worth it
> > or whether it would be better to write something from scratch. There is
> > also a question about what to do with the tape code - that's not
> > something I have any interest in digging into.
> > 
> > In my eyes dump had a good run and I would be happy just dumping it, but
> > it is worth asking here on the list. Is there anyone interested in
> > maintaining dump/restore, or is there interest in or objections agains
> > merging it into e2fsprogs ?
> 
> One of the interesting questions is how reliable the dump utility
> really is; that's because it works by reading the metadata directly
> --- while the file system is mounted.  So it's quite possible for the
> metadata to be changing out from under the dump/restore process.
> Especially with metadata checksums, I suspect dump/restore is going
> much more unreliable in terms of the libext2fs returning checksum
> failures.

Hi Ted,

this is a very good point. I have not even thought about checksums, but
that is just one example where it is likely to fail miserably. Granted
that's a relatively new feature, as well as inline data however it can't
even handle uninitialized extents correctly and hardly anyone is
noticing.

> 
> In the future, if we ever try to bypass the use of the buffer cache,
> and instead have jbd2 write out directly to the bio layer so we cant
> get better write error codes.  There was a discussion about this
> recently, and there are two problems.  First, we need to worry about
> programs like tune2fs and e2label that need to be able to read and
> modify the superblock while the file system is modified.  We'd want to
> add ioctl's to set and get the superblock, and update e2fsprogs to try
> to use those system calls first.  And then.... there is dump/restore.i
> 
> I could imagine adding ioctl's which allow safe read-only access to
> all metadata blocks, and not just for the superblock.  The question,
> though is... is it worth it, especially if it's only to make
> dump/restore work?

I have a feeling that the answer is no, it's not worth it for
dump/restore alone.

> 
> On the other hand, if we want to try to implement some kind of on-line
> fsck work, then -perhaps safe metadata reading would be part of that
> interface.  So I'd never say never, but I do wonder if it's time to
> pull the plug on dump/restore --- especially if we want to allow it to
> support not just inline files/directories, but also things like
> extended attributes and ACL's.
> 
> 						- Ted
> 

Thanks Ted, you made some good points and while there are some good
ideas for the future, there is no place for dump there. I think we're in
agreement to pull the plug on dump.

Thanks!
-Lukas

Re: State of dump utility

[Christoph Hellwig]

On Tue, Sep 29, 2020 at 10:06:46PM -0400, Theodore Y. Ts'o wrote:
> One of the interesting questions is how reliable the dump utility
> really is; that's because it works by reading the metadata directly
> --- while the file system is mounted.  So it's quite possible for the
> metadata to be changing out from under the dump/restore process.
> Especially with metadata checksums, I suspect dump/restore is going
> much more unreliable in terms of the libext2fs returning checksum
> failures.

Before using a tool like dumpe2fs you'll need to do a fsfreeze,
and then everything should be fine.  Best would be to patch the
tool to issue the freeze/unfreeze ioctls itself.

Re: State of dump utility

[Theodore Y. Ts'o]

On Fri, Oct 02, 2020 at 02:01:58PM +0200, Lukas Czerner wrote:
> 
> Thanks Ted, you made some good points and while there are some good
> ideas for the future, there is no place for dump there. I think we're in
> agreement to pull the plug on dump.

I think the only strong argument for dump/restore is that there might
be some crazy^H^H^H^H^H^H dedicated hobbiests over at The Unix
Historical Society who might find a 9-track tape with some 2.10 BSD
backup on it, and dump/restore is format compatible with it.  So
_restore_ might be interesting for some of those folks.  Then again,
they'd probably prefer to actually restore it on their 2.10 BSD system
running on a PDP-11 emulator.... so we're probably good.  :-)

	     	    		    	  - Ted