Backref walking utilities

Backref walking utilities

[liubo]

Hi,

As one of my plans, I'm going to take this project over unless someone has been working on it.

>From wiki, quote:
    Backref walking utilities

    Given a block number on a disk, the Btrfs metadata can find all the files and directories
    that use or care about that block.  Some utilities to walk these back refs and print the
    results would help debug corruptions.

    Given an inode, the Btrfs metadata can find all the directories that point to the inode.
    We should have utils to walk these back refs as well. 
end quote.

And I have some thoughts to share with you:

    - Clearly, this is going to be another command.  Just like the command "btrfs-debug-tree",
      btrfs-walk-backref also needs to be able to track btrfs's metadata in
          a) the offline situation (at a umount state), or
          b) the corrupted situation.

    - For block number, the main goal is to find relative extent backrefs.  When it comes to
      those shared blocks, maybe things will be more complex.

    - For inode, the main goal is to find relative inode refs.  And we should be cautious about
      a) an inode with hard links, b) snapshot.

Did I miss or misunderstand something?  Any comments are welcomed. :)

thanks,
liubo

Re: Backref walking utilities

[Arne Jansen]

Hi liubo,

On 23.05.2011 11:53, liubo wrote:
> As one of my plans, I'm going to take this project over unless someone has been working on it.

Jan Schmidt has a patch for scrub nearly ready, that does some
ref-walking to report affected files to the user. While this is
kernel code and you're planning to add user-space code, it might
still be possible to share some of it. Maybe the efforts can be
coordinated.

Thanks
Arne

> 
>>From wiki, quote:
>     Backref walking utilities
> 
>     Given a block number on a disk, the Btrfs metadata can find all the files and directories
>     that use or care about that block.  Some utilities to walk these back refs and print the
>     results would help debug corruptions.
> 
>     Given an inode, the Btrfs metadata can find all the directories that point to the inode.
>     We should have utils to walk these back refs as well. 
> end quote.
> 
> And I have some thoughts to share with you:
> 
>     - Clearly, this is going to be another command.  Just like the command "btrfs-debug-tree",
>       btrfs-walk-backref also needs to be able to track btrfs's metadata in
>           a) the offline situation (at a umount state), or
>           b) the corrupted situation.
> 
>     - For block number, the main goal is to find relative extent backrefs.  When it comes to
>       those shared blocks, maybe things will be more complex.
> 
>     - For inode, the main goal is to find relative inode refs.  And we should be cautious about
>       a) an inode with hard links, b) snapshot.
> 
> Did I miss or misunderstand something?  Any comments are welcomed. :)
> 
> thanks,
> liubo
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Backref walking utilities

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 2407 bytes --]

On Mon, May 23, 2011 at 12:02:21PM +0200, Arne Jansen wrote:
> Hi liubo,
> 
> On 23.05.2011 11:53, liubo wrote:
> > As one of my plans, I'm going to take this project over unless someone has been working on it.
> 
> Jan Schmidt has a patch for scrub nearly ready, that does some
> ref-walking to report affected files to the user. While this is
> kernel code and you're planning to add user-space code, it might
> still be possible to share some of it. Maybe the efforts can be
> coordinated.

   Another use for walking backrefs for a block (or an extent) is to
work out the differential size of a snapshot -- i.e. how much space
will be freed up if the snapshot is deleted. (You need to look at
every extent of every file in the snapshot, and work out whether those
extents are used anywhere outside the subvolume).

   Hugo.

> Thanks
> Arne
> 
> > 
> >>From wiki, quote:
> >     Backref walking utilities
> > 
> >     Given a block number on a disk, the Btrfs metadata can find all the files and directories
> >     that use or care about that block.  Some utilities to walk these back refs and print the
> >     results would help debug corruptions.
> > 
> >     Given an inode, the Btrfs metadata can find all the directories that point to the inode.
> >     We should have utils to walk these back refs as well. 
> > end quote.
> > 
> > And I have some thoughts to share with you:
> > 
> >     - Clearly, this is going to be another command.  Just like the command "btrfs-debug-tree",
> >       btrfs-walk-backref also needs to be able to track btrfs's metadata in
> >           a) the offline situation (at a umount state), or
> >           b) the corrupted situation.
> > 
> >     - For block number, the main goal is to find relative extent backrefs.  When it comes to
> >       those shared blocks, maybe things will be more complex.
> > 
> >     - For inode, the main goal is to find relative inode refs.  And we should be cautious about
> >       a) an inode with hard links, b) snapshot.
> > 
> > Did I miss or misunderstand something?  Any comments are welcomed. :)
> > 
> > thanks,
> > liubo

-- 
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 515C238D from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
   --- We are all lying in the gutter,  but some of us are looking ---   
                              at the stars.                              

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 190 bytes --]

Re: Backref walking utilities

[Jan Schmidt]

On 05/23/2011 12:02 PM, Arne Jansen wrote:
> Hi liubo,
> 
> On 23.05.2011 11:53, liubo wrote:
>> As one of my plans, I'm going to take this project over unless someone has been working on it.
> 
> Jan Schmidt has a patch for scrub nearly ready, that does some
> ref-walking to report affected files to the user. While this is
> kernel code and you're planning to add user-space code, it might
> still be possible to share some of it. Maybe the efforts can be
> coordinated.

The patches are ready and should be flexible enough to use for your
purpose. However I use them in context of the scrub code, thus I'm
planning to send them out as soon as the current version of scrub is
included in Chris' master.

If anybody wants to test the patches before that (apply well against
Arnes scrub branch), drop me an email.

-Jan

Re: Backref walking utilities

[liubo]

On 05/25/2011 11:08 PM, Jan Schmidt wrote:
> On 05/23/2011 12:02 PM, Arne Jansen wrote:
>> Hi liubo,
>>
>> On 23.05.2011 11:53, liubo wrote:
>>> As one of my plans, I'm going to take this project over unless someone has been working on it.
>> Jan Schmidt has a patch for scrub nearly ready, that does some
>> ref-walking to report affected files to the user. While this is
>> kernel code and you're planning to add user-space code, it might
>> still be possible to share some of it. Maybe the efforts can be
>> coordinated.
> 
> The patches are ready and should be flexible enough to use for your
> purpose. However I use them in context of the scrub code, thus I'm
> planning to send them out as soon as the current version of scrub is
> included in Chris' master.
> 
> If anybody wants to test the patches before that (apply well against
> Arnes scrub branch), drop me an email.
> 

I'd like to have a look ahead.  Would you please give the address of these patches?

thanks,
liubo

> -Jan
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>