find subvolume directories

find subvolume directories

[Ulli Horlacher]

I need to find (all) subvolume directories.
I know, btrfs subvolumes root directories have inode #256, but a
"find / -inum 256" is horrible slow!

Next idea: "btrfs subvolume list /" is really fast, but its output is not
always direct usable to find the subvolume directory.

Example output on a SUSE system:

root@trulla:~# btrfs subvolume list /
ID 257 gen 2280099 top level 5 path @
ID 258 gen 2280769 top level 257 path @/home
ID 259 gen 2280947 top level 257 path @/opt
ID 260 gen 2280098 top level 257 path @/srv
ID 261 gen 2280954 top level 257 path @/tmp
ID 262 gen 2280187 top level 257 path @/usr/local
ID 263 gen 2280099 top level 257 path @/var/crash
ID 264 gen 2280949 top level 257 path @/var/log
ID 265 gen 2280099 top level 257 path @/var/opt
ID 266 gen 2280954 top level 257 path @/var/spool
ID 267 gen 2280947 top level 257 path @/var/tmp
ID 270 gen 2280222 top level 257 path @/.snapshots
ID 453 gen 2280954 top level 270 path @/.snapshots/128/snapshot
ID 1235 gen 2280099 top level 257 path @/var/lib/machines
ID 12392 gen 2123118 top level 270 path @/.snapshots/1065/snapshot
ID 12393 gen 2123120 top level 270 path @/.snapshots/1066/snapshot
ID 13273 gen 2176640 top level 270 path @/.snapshots/1089/snapshot
ID 13274 gen 2176651 top level 270 path @/.snapshots/1090/snapshot
ID 13553 gen 2203681 top level 270 path @/.snapshots/1103/snapshot

There is no /@/ directory in the default filesystem because of:

root@trulla:/# stat /@/.snapshots/128/snapshot
stat: cannot stat '/@/.snapshots/128/snapshot': No such file or directory

root@trulla:~# btrfs subvolume get-default /
ID 453 gen 2280954 top level 270 path @/.snapshots/128/snapshot

root@trulla:/# mount | grep " / "
/dev/sda2 on / type btrfs (rw,relatime,space_cache,subvolid=453,subvol=/@/.snapshots/128/snapshot)

On this particular system I could remove "@" from the subvolume path to
get the subvolume directory:

root@trulla:/# stat /.snapshots/128/snapshot
  File: '/.snapshots/128/snapshot'
  Size: 198             Blocks: 0          IO Block: 4096   directory
Device: 27h/39d Inode: 256         Links: 1
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2019-07-13 01:03:08.543830085 +0200
Modify: 2019-07-13 01:03:37.336445461 +0200
Change: 2019-07-13 01:03:37.336445461 +0200
 Birth: -

But what if a btrfs filesystem does not have a toplevel /@/ directory, but
anything else, like /this/is/my/top/directory ?

Will be the first output line of "btrfs subvolume list /"
always look like

ID 257 gen 2280099 top level 5 path this/is/my/top/directory 

?



-- 
Ullrich Horlacher              Server und Virtualisierung
Rechenzentrum TIK         
Universitaet Stuttgart         E-Mail: horlacher@tik.uni-stuttgart.de
Allmandring 30a                Tel:    ++49-711-68565868
70569 Stuttgart (Germany)      WWW:    http://www.tik.uni-stuttgart.de/
REF:<20190712231705.GA16856@tik.uni-stuttgart.de>

Re: find subvolume directories

[Andrei Borzenkov]

13.07.2019 2:17, Ulli Horlacher пишет:
> I need to find (all) subvolume directories.
> I know, btrfs subvolumes root directories have inode #256, but a
> "find / -inum 256" is horrible slow!
> 
> Next idea: "btrfs subvolume list /" is really fast, but its output is not
> always direct usable to find the subvolume directory.
> 
> Example output on a SUSE system:
> 
> root@trulla:~# btrfs subvolume list /
> ID 257 gen 2280099 top level 5 path @
> ID 258 gen 2280769 top level 257 path @/home
> ID 259 gen 2280947 top level 257 path @/opt
> ID 260 gen 2280098 top level 257 path @/srv
> ID 261 gen 2280954 top level 257 path @/tmp
> ID 262 gen 2280187 top level 257 path @/usr/local
> ID 263 gen 2280099 top level 257 path @/var/crash
> ID 264 gen 2280949 top level 257 path @/var/log
> ID 265 gen 2280099 top level 257 path @/var/opt
> ID 266 gen 2280954 top level 257 path @/var/spool
> ID 267 gen 2280947 top level 257 path @/var/tmp
> ID 270 gen 2280222 top level 257 path @/.snapshots
> ID 453 gen 2280954 top level 270 path @/.snapshots/128/snapshot
> ID 1235 gen 2280099 top level 257 path @/var/lib/machines
> ID 12392 gen 2123118 top level 270 path @/.snapshots/1065/snapshot
> ID 12393 gen 2123120 top level 270 path @/.snapshots/1066/snapshot
> ID 13273 gen 2176640 top level 270 path @/.snapshots/1089/snapshot
> ID 13274 gen 2176651 top level 270 path @/.snapshots/1090/snapshot
> ID 13553 gen 2203681 top level 270 path @/.snapshots/1103/snapshot
> 
> There is no /@/ directory in the default filesystem because of:
> 
> root@trulla:/# stat /@/.snapshots/128/snapshot
> stat: cannot stat '/@/.snapshots/128/snapshot': No such file or directory
> 
> root@trulla:~# btrfs subvolume get-default /
> ID 453 gen 2280954 top level 270 path @/.snapshots/128/snapshot
> 
> root@trulla:/# mount | grep " / "
> /dev/sda2 on / type btrfs (rw,relatime,space_cache,subvolid=453,subvol=/@/.snapshots/128/snapshot)
> 
> On this particular system I could remove "@" from the subvolume path to
> get the subvolume directory:
> 

That is just coincidence because @/.snapshot subvolume is mounted on
/.snapshot. It could also be mounted under /var/lib/snapper (insert your
path here).

> root@trulla:/# stat /.snapshots/128/snapshot
>   File: '/.snapshots/128/snapshot'
>   Size: 198             Blocks: 0          IO Block: 4096   directory
> Device: 27h/39d Inode: 256         Links: 1
> Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
> Access: 2019-07-13 01:03:08.543830085 +0200
> Modify: 2019-07-13 01:03:37.336445461 +0200
> Change: 2019-07-13 01:03:37.336445461 +0200
>  Birth: -
> 
> But what if a btrfs filesystem does not have a toplevel /@/ directory, but
> anything else, like /this/is/my/top/directory ?
> 

btrfs does not have "top level directory" beyond single /. It is
entirely up to the user who creates it how subvolumes are named and
structured. You can well have /foo, /bar, /baz mounted as /, /var and /home.

> Will be the first output line of "btrfs subvolume list /"
> always look like
> 
> ID 257 gen 2280099 top level 5 path this/is/my/top/directory 
> 
> ?
> 

If /foo and /foo/bar are subvolumes, then /foo/bar will have higher ID
than /foo (because it must have been created after /foo) so /foo will be
listed before /foo/bar in default sort order. Whether it will have ID
257 is entirely up to whoever created this filesytsem and its history.

Re: find subvolume directories

[Ulli Horlacher]

On Sat 2019-07-13 (06:59), Andrei Borzenkov wrote:
> 13.07.2019 2:17, Ulli Horlacher ?8H5B:
> 
> > I need to find (all) subvolume directories.

> That is just coincidence because @/.snapshot subvolume is mounted on
> /.snapshot. It could also be mounted under /var/lib/snapper (insert your
> path here).

Yes, this is the problem for me!


> > But what if a btrfs filesystem does not have a toplevel /@/ directory, but
> > anything else, like /this/is/my/top/directory ?
> > 
> 
> btrfs does not have "top level directory" beyond single /.

I used the wrong naming.
I have meant "top level directory beyond /"


> It is entirely up to the user who creates it how subvolumes are named and
> structured. You can well have /foo, /bar, /baz mounted as /, /var and
> /home.

And how can I find them in my mounted filesystem?
THIS is my problem.
As I wrote: "find / -inum 256" is too slow.


-- 
Ullrich Horlacher              Server und Virtualisierung
Rechenzentrum TIK         
Universitaet Stuttgart         E-Mail: horlacher@tik.uni-stuttgart.de
Allmandring 30a                Tel:    ++49-711-68565868
70569 Stuttgart (Germany)      WWW:    http://www.tik.uni-stuttgart.de/
REF:<75e5bd20-fafa-07fd-afd9-159e9aacb7db@gmail.com>

Re: find subvolume directories

[Andrei Borzenkov]

13.07.2019 11:27, Ulli Horlacher пишет:
> On Sat 2019-07-13 (06:59), Andrei Borzenkov wrote:
>> 13.07.2019 2:17, Ulli Horlacher ?8H5B:
>>
>>> I need to find (all) subvolume directories.
> 
>> That is just coincidence because @/.snapshot subvolume is mounted on
>> /.snapshot. It could also be mounted under /var/lib/snapper (insert your
>> path here).
> 
> Yes, this is the problem for me!
> 
> 
>>> But what if a btrfs filesystem does not have a toplevel /@/ directory, but
>>> anything else, like /this/is/my/top/directory ?
>>>
>>
>> btrfs does not have "top level directory" beyond single /.
> 
> I used the wrong naming.
> I have meant "top level directory beyond /"
> 
> 
>> It is entirely up to the user who creates it how subvolumes are named and
>> structured. You can well have /foo, /bar, /baz mounted as /, /var and
>> /home.
> 
> And how can I find them in my mounted filesystem?
> THIS is my problem.

I am not sure what problem you are trying to solve, but you can use list
of current mounts to build path to each subvolume (as long as it is
either below one of mounted subvolumes or explicitly mounted).

> As I wrote: "find / -inum 256" is too slow.
> 
> 

Re: find subvolume directories

[Ulli Horlacher]

On Sat 2019-07-13 (14:10), Andrei Borzenkov wrote:

> >> It is entirely up to the user who creates it how subvolumes are named and
> >> structured. You can well have /foo, /bar, /baz mounted as /, /var and
> >> /home.
> > 
> > And how can I find them in my mounted filesystem?
> > THIS is my problem.
> 
> I am not sure what problem you are trying to solve

I want a list of all subvolumes directories (which I can access with UNIX
tools like cd and ls or btrfs subvolume ...).


> but you can use list of current mounts to build path to each subvolume
> (as long as it is either below one of mounted subvolumes or explicitly
> mounted).

Is there an easy/standard way to do this?


-- 
Ullrich Horlacher              Server und Virtualisierung
Rechenzentrum TIK         
Universitaet Stuttgart         E-Mail: horlacher@tik.uni-stuttgart.de
Allmandring 30a                Tel:    ++49-711-68565868
70569 Stuttgart (Germany)      WWW:    http://www.tik.uni-stuttgart.de/
REF:<62366a29-a8ea-a889-f857-0305eba99051@gmail.com>

Re: find subvolume directories

[Andrei Borzenkov]

13.07.2019 14:28, Ulli Horlacher пишет:
> On Sat 2019-07-13 (14:10), Andrei Borzenkov wrote:
> 
>>>> It is entirely up to the user who creates it how subvolumes are named and
>>>> structured. You can well have /foo, /bar, /baz mounted as /, /var and
>>>> /home.
>>>
>>> And how can I find them in my mounted filesystem?
>>> THIS is my problem.
>>
>> I am not sure what problem you are trying to solve
> 
> I want a list of all subvolumes directories (which I can access with UNIX
> tools like cd and ls or btrfs subvolume ...).
> 
> 
>> but you can use list of current mounts to build path to each subvolume
>> (as long as it is either below one of mounted subvolumes or explicitly
>> mounted).
> 
> Is there an easy/standard way to do this?
> 
> 

Not that I'm aware of.

Re: find subvolume directories

[Piotr Szymaniak]

On Sat, Jul 13, 2019 at 01:28:32PM +0200, Ulli Horlacher wrote:
> On Sat 2019-07-13 (14:10), Andrei Borzenkov wrote:
> 
> > >> It is entirely up to the user who creates it how subvolumes are named and
> > >> structured. You can well have /foo, /bar, /baz mounted as /, /var and
> > >> /home.
> > > 
> > > And how can I find them in my mounted filesystem?
> > > THIS is my problem.
> > 
> > I am not sure what problem you are trying to solve
> 
> I want a list of all subvolumes directories (which I can access with UNIX
> tools like cd and ls or btrfs subvolume ...).

Hi,

what about btrfs sub list [options]? (see man btrfs-subvolume)

You can make ie.:
root@ed:~# btrfs sub list -a / | head -10
ID 259 gen 142795 top level 5 path <FS_TREE>/@rut
ID 267 gen 1599 top level 259 path @rut/BUP/190417-1748_Image_SYSVOL
ID 268 gen 2516 top level 259 path @rut/BUP/190417-1750_Image_C
ID 326 gen 1599 top level 259 path @rut/BUP/190418-1009
ID 359 gen 1599 top level 259 path @rut/BUP/190418-1751
ID 361 gen 1599 top level 259 path @rut/BUP/190419-0812
ID 364 gen 1599 top level 259 path @rut/BUP/190423-1025
ID 369 gen 2086 top level 259 path @rut/BUP/190423-2232

But, I'm a bit like Andrei, and not sure what are you looking for. You
already asked about "mounted" and then about "list of all subvols"...
So you want to find mounted subvolumes or all subvolumes or all mounted
subvolumes or ...?


Best regards,
Piotr Szymaniak.
-- 
Najgoretsze miejsce w piekle szykowane jest nie tym, ktorzy zabijaja,
ale tym, ktorzy sie bezczynnie temu przygladaja.
  -- Dante Alighieri

Re: find subvolume directories

[Ulli Horlacher]

On Mon 2019-07-15 (15:22), Piotr Szymaniak wrote:

> > I want a list of all subvolumes directories (which I can access with UNIX
> > tools like cd and ls or btrfs subvolume ...).
> 
> what about btrfs sub list [options]? (see man btrfs-subvolume)
> 
> You can make ie.:
> root@ed:~# btrfs sub list -a / | head -10
> ID 259 gen 142795 top level 5 path <FS_TREE>/@rut
> ID 267 gen 1599 top level 259 path @rut/BUP/190417-1748_Image_SYSVOL

There is no directory "<FS_TREE>/@rut" or
"@rut/BUP/190417-1748_Image_SYSVOL" which I cann access directly with
standard UNIX commands.


> But, I'm a bit like Andrei, and not sure what are you looking for. You
> already asked about "mounted" and then about "list of all subvols"...
> So you want to find mounted subvolumes or all subvolumes or all mounted
> subvolumes or ...?

I need a list of all subvolumes DIRECTORIES, to be accessible with
standard UNIX commands like cd and ls or btrfs subvolume show



-- 
Ullrich Horlacher              Server und Virtualisierung
Rechenzentrum TIK         
Universitaet Stuttgart         E-Mail: horlacher@tik.uni-stuttgart.de
Allmandring 30a                Tel:    ++49-711-68565868
70569 Stuttgart (Germany)      WWW:    http://www.tik.uni-stuttgart.de/
REF:<20190715132228.GF4212@pontus.sran>

Re: [RFC] a standard user-friendly way to find a snapshot in nested subvolumes [was: find subvolume directories]

[Nicholas D Steeves]

[-- Attachment #1: Type: text/plain, Size: 3160 bytes --]

Hi Ulli,

On Tue, Jul 16, 2019 at 12:40:51AM +0200, Ulli Horlacher wrote:
> On Mon 2019-07-15 (15:22), Piotr Szymaniak wrote:
> 
> > > I want a list of all subvolumes directories (which I can access with UNIX
> > > tools like cd and ls or btrfs subvolume ...).
> > 
> > what about btrfs sub list [options]? (see man btrfs-subvolume)
> > 
> > You can make ie.:
> > root@ed:~# btrfs sub list -a / | head -10
> > ID 259 gen 142795 top level 5 path <FS_TREE>/@rut
> > ID 267 gen 1599 top level 259 path @rut/BUP/190417-1748_Image_SYSVOL
> 
> There is no directory "<FS_TREE>/@rut" or
> "@rut/BUP/190417-1748_Image_SYSVOL" which I cann access directly with
> standard UNIX commands.
> 
> 
> > But, I'm a bit like Andrei, and not sure what are you looking for. You
> > already asked about "mounted" and then about "list of all subvols"...
> > So you want to find mounted subvolumes or all subvolumes or all mounted
> > subvolumes or ...?
> 
> I need a list of all subvolumes DIRECTORIES, to be accessible with
> standard UNIX commands like cd and ls or btrfs subvolume show
> 

"a list of all subvolumes DIRECTORIES" doesn't make sense...  It
sounds like you want to list all available subvolumes (presumably
snapshots, given that the path has BUP in it), to find a specific one
you want, and then access an older copy of one of your files.

Something like the following method might do the trick:

First, mount /dev/sdX to /btrfs-admin without using a subvol option.
This will wonly work if you haven't changed the default subvol.

  sudo btrfs sub list -at /btrfs-admin/ | sed 's:<FS_TREE>:btrfs-admin:

It should then be obvious how to get out of @rut ;-)

If for some reason it's not obvious what to do next, here's a simple
method that should show the full path for (presumably) snapshots such as
"@rut/BUP/190417-1748_Image_SYSVOL"

  sudo btrfs sub list -a /btrfs-admin/ \
    | sed 's:<FS_TREE>:/btrfs-admin:' \
    | grep -v /btrfs-admin \
    | awk '{print "/btrfs-admin/"$9}'

If by "volume DIRECTORIES" you actually mean independent "btrfs
volumes" then this function will output a list of devices you can
mount the top-level of:

  https://github.com/kdave/btrfsmaintenance/blob/f7a8ed25805ed321a1dfc20b65cd35728c3723c4/btrfsmaintenance-functions#L11-L34

On the topic of the absence of a standard way to get a top-down
tree-like view of all subvolumes, without duplicates...it could be
nice to have /btrfs-admin or /.btrfs-admin mountpoint officially
recognised as the cross-distribution standard method.  In cases where
there are multiple volumes, this would be
/btrfs-admin/{volume-name0,volume-name1,etc}.  Then, if btrfs sub list
was run with a hypothetical "--tree" option it could provide
user-friendly output like findmnt.

Alternatively, are there any reasons why we don't have some VFS magic
providing such a view in /proc or /sys?

Whatever the implementation, I think that this class of wishlist bug
would be solved with something like "sudo btrfs fi show" with no other
arguments would output a comprehensive btrfs view of all detected
volumes.


Cheers,
Nicholas

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [RFC] a standard user-friendly way to find a snapshot in nested subvolumes [was: find subvolume directories]

[Ulli Horlacher]

On Mon 2019-07-15 (19:58), Nicholas D Steeves wrote:

> > I need a list of all subvolumes DIRECTORIES, to be accessible with
> > standard UNIX commands like cd and ls or btrfs subvolume show
> > 
> 
> "a list of all subvolumes DIRECTORIES" doesn't make sense...  It
> sounds like you want to list all available subvolumes (presumably
> snapshots, given that the path has BUP in it), to find a specific one
> you want

Yes, but not limited to snapshots. I want a list of ALL subvolumes
directories. 


> and then access an older copy of one of your files.

Sometimes yes, sometimes not.
Sometimes I want make a new snapshot. But then I have to know what
directories are "snapshootable", aka btrfs subvolumes.
"btrfs subvolume snapshot" needs as first parameter the directory name of
the subvolume, so I have to know it!


> Something like the following method might do the trick:
> 
> First, mount /dev/sdX to /btrfs-admin without using a subvol option.
> This will wonly work if you haven't changed the default subvol.

Only root can do mounting.
I need a solution for every user!


>   sudo btrfs sub list -at /btrfs-admin/ | sed 's:<FS_TREE>:btrfs-admin:

Then I still do not know where it is in my standard UNIX filesystems.

I am looking for a faster (native btrfs command) version of:

root@trulla:~# find / -type d -inum 256
/
/home
/home/tux/blubb
/opt
/opt/.snapshot/2019-07-11_0000.daily
/opt/.snapshot/2019-07-12_0000.daily
/opt/.snapshot/2019-07-13_0000.daily
/opt/.snapshot/2019-07-15_0000.daily
/opt/.snapshot/2019-07-16_0000.daily
/opt/.snapshot/2019-07-16_0100.hourly
/opt/.snapshot/2019-07-16_0200.hourly
/srv
/tmp
/tmp/test
/usr/local
/var/crash
/var/log
/var/opt
/var/spool
/var/tmp
/var/lib/machines
/sys/fs/cgroup/systemd/system.slice/wickedd-auto4.service
/mnt/tmp
/mnt/tmp/ss
/.snapshots
find: File system loop detected; '/.snapshots/128/snapshot' is part of the same file system loop as '/'.
/.snapshots/1065/snapshot
/.snapshots/1066/snapshot
/.snapshots/1089/snapshot
/.snapshots/1090/snapshot
/.snapshots/1103/snapshot
/.snapshots/1104/snapshot
(...)

This is VERY slow, because it scans through all of my files, even on nfs!

My next idea was:

root@fex:~# df --output=fstype,target | awk '/^btrfs/{print "find "$2" -type d -inum 256 -xdev"}' | sh -x
+ find /backup -type d -inum 256 -xdev
/backup
/backup/fex
/backup/diaspora
+ find /local -type d -inum 256 -xdev
/local
/local/home
/local/spool/fex

But this does not descend subvolumes, because "find" thinks it is on a
different filesystem (-xdev)

For example, it does not find:

root@fex:~# ll /local/home/.snapshot/
drwxr-xr-x  root     root     - 2017-08-31 08:28:11  /local/home/.snapshot/2019-07-13_0000.daily
drwxr-xr-x  root     root     - 2017-08-31 08:28:11  /local/home/.snapshot/2019-07-14_0000.weekly
drwxr-xr-x  root     root     - 2017-08-31 08:28:11  /local/home/.snapshot/2019-07-15_0000.daily
drwxr-xr-x  root     root     - 2017-08-31 08:28:11  /local/home/.snapshot/2019-07-15_2300.hourly
drwxr-xr-x  root     root     - 2017-08-31 08:28:11  /local/home/.snapshot/2019-07-16_0000.daily
drwxr-xr-x  root     root     - 2017-08-31 08:28:11  /local/home/.snapshot/2019-07-16_0100.hourly
drwxr-xr-x  root     root     - 2017-08-31 08:28:11  /local/home/.snapshot/2019-07-16_0200.hourly

root@fex:~# btrfs subvolume show /local/home/.snapshot/2019-07-13_0000.daily
home/.snapshot/2019-07-13_0000.daily
        Name:                   2019-07-13_0000.daily
        UUID:                   084b1f18-b700-5845-a32d-f151db6a9f57
        Parent UUID:            ba4d388f-44bf-7b46-b2b8-00e2a9a87181
        Received UUID:          -
        Creation time:          2019-07-13 00:00:01 +0200
        Subvolume ID:           17957
        Generation:             1645903
        Gen at creation:        1645903
        Parent ID:              350
        Top level ID:           350
        Flags:                  readonly
        Snapshot(s):



-- 
Ullrich Horlacher              Server und Virtualisierung
Rechenzentrum TIK         
Universitaet Stuttgart         E-Mail: horlacher@tik.uni-stuttgart.de
Allmandring 30a                Tel:    ++49-711-68565868
70569 Stuttgart (Germany)      WWW:    http://www.tik.uni-stuttgart.de/
REF:<20190715235821.rh7elbip3dgzkq7y@DigitalMercury.dynalias.net>

Re: find subvolume directories

[Ulli Horlacher]

On Sat 2019-07-13 (14:10), Andrei Borzenkov wrote:

> I am not sure what problem you are trying to solve, but you can use list
> of current mounts to build path to each subvolume (as long as it is
> either below one of mounted subvolumes or explicitly mounted).

I have tried it this way (many hours of programming), but it is a horrible
complex task, because subvolumes can be mounted several times at different
places (mount points).

Why is such a common command "list all subvolumes directories" not possible
with the standard btrfs tools?
Am I really the only one who wants to know it?


-- 
Ullrich Horlacher              Server und Virtualisierung
Rechenzentrum TIK         
Universitaet Stuttgart         E-Mail: horlacher@tik.uni-stuttgart.de
Allmandring 30a                Tel:    ++49-711-68565868
70569 Stuttgart (Germany)      WWW:    http://www.tik.uni-stuttgart.de/
REF:<62366a29-a8ea-a889-f857-0305eba99051@gmail.com>

Re: find subvolume directories

[Ulli Horlacher]

On Sat 2019-07-13 (01:17), Ulli Horlacher wrote:
> I need to find (all) subvolume directories.
> I know, btrfs subvolumes root directories have inode #256, but a
> "find / -inum 256" is horrible slow!

At least, I want to exclude non-btrfs filesystems to speed up the find
run, but find cannot detect snapshots (as part of btrfs):

root@trulla:~# find /opt -inum 256 -printf "%p %F\n"
/opt btrfs
/opt/.snapshot/2019-07-10_0000.daily unknown
/opt/.snapshot/2019-07-11_0000.daily unknown
/opt/.snapshot/2019-07-12_0000.daily unknown
/opt/.snapshot/2019-07-13_0000.daily unknown
/opt/.snapshot/2019-07-15_0000.daily unknown
/opt/.snapshot/2019-07-15_1200.hourly unknown
/opt/.snapshot/2019-07-15_1300.hourly unknown

root@trulla:~# find /opt -inum 256 \( -fstype btrfs -o -fstype unknown \) -printf "%p %F\n"
/opt btrfs
root@trulla:~# 

root@trulla:~# btrfs subvolume list -o /opt
ID 14930 gen 2276304 top level 259 path @/opt/.snapshot/2019-07-10_0000.daily
ID 14957 gen 2277856 top level 259 path @/opt/.snapshot/2019-07-11_0000.daily
ID 14981 gen 2279354 top level 259 path @/opt/.snapshot/2019-07-12_0000.daily
ID 15005 gen 2280891 top level 259 path @/opt/.snapshot/2019-07-13_0000.daily
ID 15052 gen 2283953 top level 259 path @/opt/.snapshot/2019-07-15_0000.daily
ID 15064 gen 2284721 top level 259 path @/opt/.snapshot/2019-07-15_1200.hourly
ID 15065 gen 2284788 top level 259 path @/opt/.snapshot/2019-07-15_1300.hourly

root@trulla:~# stat -f -c %T /opt/.snapshot/2019-07-10_0000.daily
btrfs


-- 
Ullrich Horlacher              Server und Virtualisierung
Rechenzentrum TIK         
Universitaet Stuttgart         E-Mail: horlacher@tik.uni-stuttgart.de
Allmandring 30a                Tel:    ++49-711-68565868
70569 Stuttgart (Germany)      WWW:    http://www.tik.uni-stuttgart.de/
REF:<20190712231705.GA16856@tik.uni-stuttgart.de>

Re: find subvolume directories

[Ulli Horlacher]

On Sat 2019-07-13 (01:17), Ulli Horlacher wrote:
> I need to find (all) subvolume directories.
> I know, btrfs subvolumes root directories have inode #256, but a
> "find / -inum 256" is horrible slow!

Because it scans also non-btrfs filesystems. This is especially a bad idea
for nfs.

I have now implemented the search for inode 256 in Perl using the algorithm: 

- get a list of all mounted filesystems (via df)
- for each btrfs filesystem do a recursive search for directories with
  inode 256
- stop recursion if the directory is on another (mounted) filesystem


This speeds up the search:

root@trulla:/opt/btrfs-tools/bin# sysinfo 
System:        Linux trulla 4.4.180-94.97-default x86_64
Distribution:  SUSE Linux Enterprise Server 12 SP3
Hardware:      VMware, Inc. VMware Virtual Platform None (VMware)
CPU:           Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz, 1 x 2394 MHz
RAM:           1024 MB

root@trulla:/opt/btrfs-tools/bin# dropfscache;time btrfs_list -r
ACCESS-MODE SUBVOLUME <- SNAPSHOT-PARENT
rw / <- -
rw /.snapshots
ro /.snapshots/1065/snapshot <- /
ro /.snapshots/1066/snapshot <- /
ro /.snapshots/1089/snapshot <- /
ro /.snapshots/1090/snapshot <- /
ro /.snapshots/1103/snapshot <- /
ro /.snapshots/1104/snapshot <- /
ro /.snapshots/1107/snapshot <- /
ro /.snapshots/1108/snapshot <- /
ro /.snapshots/1127/snapshot <- /
ro /.snapshots/1128/snapshot <- /
ro /.snapshots/1131/snapshot <- /
ro /.snapshots/1132/snapshot <- /
ro /.snapshots/1133/snapshot <- /
ro /.snapshots/1134/snapshot <- /
ro /.snapshots/1135/snapshot <- /
ro /.snapshots/1136/snapshot <- /
ro /.snapshots/1137/snapshot <- /
ro /.snapshots/1138/snapshot <- /
ro /.snapshots/1139/snapshot <- /
ro /.snapshots/1140/snapshot <- /
rw /.snapshots/128/snapshot <- -
rw /home
rw /home/tux/blubb
rw /mnt/tmp
ro /mnt/tmp/ss
rw /opt
ro /opt/.snapshot/2019-07-11_0000.daily <- /opt
ro /opt/.snapshot/2019-07-12_0000.daily <- /opt
ro /opt/.snapshot/2019-07-13_0000.daily <- /opt
ro /opt/.snapshot/2019-07-15_0000.daily <- /opt
ro /opt/.snapshot/2019-07-16_0000.daily <- /opt
ro /opt/.snapshot/2019-07-16_1100.hourly <- /opt
ro /opt/.snapshot/2019-07-16_1200.hourly <- /opt
rw /srv
rw /tmp
rw /tmp/blubb
ro /tmp/blubb/.snapshot/2017-09-20_1801.test <- /tmp/blubb
ro /tmp/blubb/.snapshot/2017-12-02_0940.test <- /tmp/blubb
ro /tmp/blubb/.snapshot/2018-01-11_1415.test <- /tmp/blubb
ro /tmp/test
rw /usr/local
rw /var/crash
rw /var/lib/machines
rw /var/log
rw /var/opt
rw /var/spool
rw /var/tmp

real    0m4.301s
user    0m1.352s
sys     0m1.438s


(dropfscache drops the filesystem cache to have a reproducable test)

-- 
Ullrich Horlacher              Server und Virtualisierung
Rechenzentrum TIK         
Universitaet Stuttgart         E-Mail: horlacher@tik.uni-stuttgart.de
Allmandring 30a                Tel:    ++49-711-68565868
70569 Stuttgart (Germany)      WWW:    http://www.tik.uni-stuttgart.de/
REF:<20190712231705.GA16856@tik.uni-stuttgart.de>

Re: find subvolume directories

[Axel Burri]

On 13/07/2019 01.17, Ulli Horlacher wrote:
> I need to find (all) subvolume directories.
> I know, btrfs subvolumes root directories have inode #256, but a
> "find / -inum 256" is horrible slow!

Having all required frameworks for this in btrbk, implementing a "list
all subvolumes below <path>" command was quite easy to implement:

https://github.com/digint/btrbk/commit/e12d980502

 - get mounted filesystems from /proc/self/mountinfo
 - fetch subvolumes using "btrfs subvolume list" (fast, needs root)
 - filter and print subvolumes below mount point

Note that this approach needs root, as "btrfs subvolume list" requires
"cap_sys_admin" and "cap_dac_read_search".


Try it:

Download btrbk from "action-ls" feature branch (no dependencies needed):

# cd /tmp
# wget https://raw.githubusercontent.com/digint/btrbk/action-ls/btrbk
# chmod +x /tmp/btrbk


List subvolumes below /home:

# ./btrbk ls /home
# ./btrbk ls /home -t


Comprehensive list of all accessible subvolumes:

# ./btrbk ls / --format=long


Show commands run by btrbk:

# ./btrbk ls / -l debug


If you need to run this as a regular user (and if you are brave), you
can install setcap enabled btrfs binaries from:
https://github.com/digint/btrfs-progs-btrbk

Hope this helps,

- Axel

Re: find subvolume directories

[Andrei Borzenkov]

18.07.2019 15:00, Axel Burri пишет:
> On 13/07/2019 01.17, Ulli Horlacher wrote:
>> I need to find (all) subvolume directories.
>> I know, btrfs subvolumes root directories have inode #256, but a
>> "find / -inum 256" is horrible slow!
> 
> Having all required frameworks for this in btrbk, implementing a "list
> all subvolumes below <path>" command was quite easy to implement:
> 
> https://github.com/digint/btrbk/commit/e12d980502
> 
>  - get mounted filesystems from /proc/self/mountinfo
>  - fetch subvolumes using "btrfs subvolume list" (fast, needs root)
>  - filter and print subvolumes below mount point
> 
> Note that this approach needs root, as "btrfs subvolume list" requires
> "cap_sys_admin" and "cap_dac_read_search".
> 
> 
> Try it:
> 
> Download btrbk from "action-ls" feature branch (no dependencies needed):
> 
> # cd /tmp
> # wget https://raw.githubusercontent.com/digint/btrbk/action-ls/btrbk
> # chmod +x /tmp/btrbk
> 
> 
> List subvolumes below /home:
> 
> # ./btrbk ls /home
> # ./btrbk ls /home -t
> 
> 
> Comprehensive list of all accessible subvolumes:
> 
> # ./btrbk ls / --format=long
> 

Seems to work, also across non-btrfs mount point. One thing missing is
actual subvolume path (not current mount point).

> 
> Show commands run by btrbk:
> 
> # ./btrbk ls / -l debug
> 
> 
> If you need to run this as a regular user (and if you are brave), you
> can install setcap enabled btrfs binaries from:
> https://github.com/digint/btrfs-progs-btrbk
> 
> Hope this helps,
> 
> - Axel
> 

Re: find subvolume directories

[Axel Burri]

On 18/07/2019 19.48, Andrei Borzenkov wrote:
> 18.07.2019 15:00, Axel Burri пишет:
>> On 13/07/2019 01.17, Ulli Horlacher wrote:
>>> I need to find (all) subvolume directories.
>>> I know, btrfs subvolumes root directories have inode #256, but a
>>> "find / -inum 256" is horrible slow!
>>
>> Having all required frameworks for this in btrbk, implementing a "list
>> all subvolumes below <path>" command was quite easy to implement:
>>
>> https://github.com/digint/btrbk/commit/e12d980502
>>
>>  - get mounted filesystems from /proc/self/mountinfo
>>  - fetch subvolumes using "btrfs subvolume list" (fast, needs root)
>>  - filter and print subvolumes below mount point
>>
>> Note that this approach needs root, as "btrfs subvolume list" requires
>> "cap_sys_admin" and "cap_dac_read_search".
>>
>>
>> Try it:
>>
>> Download btrbk from "action-ls" feature branch (no dependencies needed):
>>
>> # cd /tmp
>> # wget https://raw.githubusercontent.com/digint/btrbk/action-ls/btrbk
>> # chmod +x /tmp/btrbk
>>
>>
>> List subvolumes below /home:
>>
>> # ./btrbk ls /home
>> # ./btrbk ls /home -t
>>
>>
>> Comprehensive list of all accessible subvolumes:
>>
>> # ./btrbk ls / --format=long
>>
> 
> Seems to work, also across non-btrfs mount point. One thing missing is
> actual subvolume path (not current mount point).

Displaying the subvolume path might be useful as well, but I think this
should better be hidden from the user, except he is investigating btrfs
internas.
Note that the subvolume path (from "btrfs subvolume list") is ambiguous:
You can not tell what parts are "path-to-subvolume", and which are
"subvolume-name" (especially for nested subvolumes within directories).


Nevertheless, I updated "btrbk/action-ls" to print subvolume_path as
well as subvolume_rel_path (relative to top_level subvolume) for
format=rel [1]:

# ./btrbk ls / --format=rel

 [1] https://github.com/digint/btrbk/commit/c0f2896ae1

> 
>>
>> Show commands run by btrbk:
>>
>> # ./btrbk ls / -l debug
>>
>>
>> If you need to run this as a regular user (and if you are brave), you
>> can install setcap enabled btrfs binaries from:
>> https://github.com/digint/btrfs-progs-btrbk
>>
>> Hope this helps,
>>
>> - Axel
>>
> 

Re: find subvolume directories

[Ulli Horlacher]

On Thu 2019-07-18 (14:00), Axel Burri wrote:

> > I need to find (all) subvolume directories.
> > I know, btrfs subvolumes root directories have inode #256, but a
> > "find / -inum 256" is horrible slow!
> 
> https://github.com/digint/btrbk/commit/e12d980502
> 
>  - get mounted filesystems from /proc/self/mountinfo
>  - fetch subvolumes using "btrfs subvolume list" (fast, needs root)
>  - filter and print subvolumes below mount point

Meanwhile I have come to the same solution :-)
Only I use /proc/mounts instead of /proc/self/mountinfo

framstag@fex:/tmp: btrfs_list
ACCESS-MODE SUBVOLUME
rw /local/home/framstag/blubb
ro /local/home/framstag/blubb/.snapshot/2017-09-15_1811.test
ro /local/home/framstag/blubb/.snapshot/2017-11-30_1007.test
ro /local/home/framstag/blubb/.snapshot/2017-12-02_1026.test
ro /local/home/framstag/blubb/.snapshot/2019-07-18_0139.test
rw /local/tmp/test

My first (wrong) approach was to use 
btrfs subvolume list TOPLEVEL_SUBVOLUME_MOUNTPOINT 
only, but not all btrfs mountpoints.


> Note that this approach needs root, as "btrfs subvolume list" requires
> "cap_sys_admin" and "cap_dac_read_search".

My btrfs_list uses sudo and lists only subvolumes belonging to the user.


> # cd /tmp
> # wget https://raw.githubusercontent.com/digint/btrbk/action-ls/btrbk
> # chmod +x /tmp/btrbk
> 
> 
> List subvolumes below /home:
> 
> # ./btrbk ls /home

root@fex:~# /tmp/btrbk ls /local/
ERROR: Configuration file not found: /etc/btrbk.conf, /etc/btrbk/btrbk.conf
ERROR: Failed to parse configuration file

root@fex:~# touch /etc/btrbk.conf

root@fex:~# /tmp/btrbk ls /local/
/local
/local/home
/local/home/.snapshot/2019-07-14_0000.weekly
/local/home/.snapshot/2019-07-18_0000.daily
/local/home/.snapshot/2019-07-19_0000.daily
/local/home/.snapshot/2019-07-20_0000.daily
/local/home/.snapshot/2019-07-20_0900.hourly
/local/home/.snapshot/2019-07-20_1000.hourly
/local/home/.snapshot/2019-07-20_1100.hourly
/local/home/framstag/blubb
/local/home/framstag/blubb/.snapshot/2017-09-15_1811.test
/local/home/framstag/blubb/.snapshot/2017-11-30_1007.test
/local/home/framstag/blubb/.snapshot/2017-12-02_1026.test
/local/home/framstag/blubb/.snapshot/2019-07-18_0139.test
/local/home/smc/test
/local/spool/fex
/local/tmp/test

-- 
Ullrich Horlacher              Server und Virtualisierung
Rechenzentrum TIK         
Universitaet Stuttgart         E-Mail: horlacher@tik.uni-stuttgart.de
Allmandring 30a                Tel:    ++49-711-68565868
70569 Stuttgart (Germany)      WWW:    http://www.tik.uni-stuttgart.de/
REF:<04d6d375-ee33-8d77-d139-a81302efd7f8@tty0.ch>