subvolumes as partitions and mount options

subvolumes as partitions and mount options

[Matt Zagrabelny]

Greetings,

I have a root partition btrfs file system.

I need to have /tmp, /var, /var/tmp, /var/log, and other directories
under separate partitions so that certain mount options can be set for
those partitions/directories.

I'm testing out a subvolume mount with the subvolume /subv_content
mounted at /subv_mnt.

For instance, the noexec mount option can be circumvented:
root@ziti:/# findmnt --kernel /subv_mnt
TARGET    SOURCE                                FSTYPE OPTIONS
/subv_mnt /dev/nvme0n1p2[/@rootfs/subv_content] btrfs
rw,nosuid,nodev,noexec,relatime,ssd,space_cache=v2,subvolid=257,subvol=/@rootfs/subv_content

root@ziti:/# echo '#!/usr/bin/bash' > /subv_mnt/foo ; echo 'echo foo'
>> /subv_mnt/foo ; chmod 0755 /subv_mnt/foo
root@ziti:/# /subv_mnt/foo
bash: /subv_mnt/foo: Permission denied
root@ziti:/# /subv_content/foo
foo
root@ziti:/#

Am I missing some mechanism to restrict subvolume with mount options
that cannot be worked around by accessing the files in the subvolume
as opposed to the mount point?

Thanks for any help!

-m

Re: subvolumes as partitions and mount options

[Andrei Borzenkov]

On 27.03.2023 21:48, Matt Zagrabelny wrote:
> Greetings,
> 
> I have a root partition btrfs file system.
> 
> I need to have /tmp, /var, /var/tmp, /var/log, and other directories
> under separate partitions so that certain mount options can be set for
> those partitions/directories.
> 
> I'm testing out a subvolume mount with the subvolume /subv_content
> mounted at /subv_mnt.
> 
> For instance, the noexec mount option can be circumvented:

"exec/noexec" option applies to mount instance, it is not persistent 
property of underlying filesystem. It is not specific to btrfs at all.

bor@bor-Latitude-E5450:/tmp/tst$ ./bin/foo.sh
Hello, world!
bor@bor-Latitude-E5450:/tmp/tst$ mkdir exec noexec
bor@bor-Latitude-E5450:/tmp/tst$ sudo mount -o bind,exec bin exec
bor@bor-Latitude-E5450:/tmp/tst$ sudo mount -o bind,noexec bin noexec
bor@bor-Latitude-E5450:/tmp/tst$ ./exec/foo.sh
Hello, world!
bash: ./noexec/foo.sh: Permission denied
bor@bor-Latitude-E5450:/tmp/tst$




> root@ziti:/# findmnt --kernel /subv_mnt
> TARGET    SOURCE                                FSTYPE OPTIONS
> /subv_mnt /dev/nvme0n1p2[/@rootfs/subv_content] btrfs
> rw,nosuid,nodev,noexec,relatime,ssd,space_cache=v2,subvolid=257,subvol=/@rootfs/subv_content
> 
> root@ziti:/# echo '#!/usr/bin/bash' > /subv_mnt/foo ; echo 'echo foo'
>>> /subv_mnt/foo ; chmod 0755 /subv_mnt/foo
> root@ziti:/# /subv_mnt/foo
> bash: /subv_mnt/foo: Permission denied
> root@ziti:/# /subv_content/foo
> foo
> root@ziti:/#
> 
> Am I missing some mechanism to restrict subvolume with mount options
> that cannot be worked around by accessing the files in the subvolume
> as opposed to the mount point?
> 
> Thanks for any help!
> 
> -m

Re: subvolumes as partitions and mount options

[Matt Zagrabelny]

On Mon, Mar 27, 2023 at 2:25 PM Andrei Borzenkov <arvidjaar@gmail.com> wrote:
>
> On 27.03.2023 21:48, Matt Zagrabelny wrote:
> > Greetings,
> >
> > I have a root partition btrfs file system.
> >
> > I need to have /tmp, /var, /var/tmp, /var/log, and other directories
> > under separate partitions so that certain mount options can be set for
> > those partitions/directories.
> >
> > I'm testing out a subvolume mount with the subvolume /subv_content
> > mounted at /subv_mnt.
> >
> > For instance, the noexec mount option can be circumvented:
>
> "exec/noexec" option applies to mount instance, it is not persistent
> property of underlying filesystem. It is not specific to btrfs at all.

Agreed. My email was more about subvolumes and the mount point has the
"noexec", but the actual subvolume doesn't - so there exists a path on
disk where folks can exec the same file by circumventing the mount
option by directly invoking the full path under the subvolume.

>
> bor@bor-Latitude-E5450:/tmp/tst$ ./bin/foo.sh
> Hello, world!
> bor@bor-Latitude-E5450:/tmp/tst$ mkdir exec noexec
> bor@bor-Latitude-E5450:/tmp/tst$ sudo mount -o bind,exec bin exec
> bor@bor-Latitude-E5450:/tmp/tst$ sudo mount -o bind,noexec bin noexec
> bor@bor-Latitude-E5450:/tmp/tst$ ./exec/foo.sh
> Hello, world!
> bash: ./noexec/foo.sh: Permission denied
> bor@bor-Latitude-E5450:/tmp/tst$

Agreed completely.

If an attacker can gain access to a system, I'd like /tmp to be
mounted "noexec".

The attacker can execute the foo.sh via /tmp/tst/bin/foo.sh even
though the bind mount (/tmp/tst/noexec) restricts the executing of
programs.

That seems to be the position I am in right now with subvolumes as
opposed to an actual partition.

If I create a separate partition for /tmp and mount it noexec, there
is no backdoor bind mount where the attacker can execute programs
from.

It seems mounting subvolumes works similarly to bind mounts - is there
a way to mimic /tmp being on a separate partition and mounted with
noexec using subvolumes?

Thanks for the help!

-m

Re: subvolumes as partitions and mount options

[Graham Cobb]

On 27/03/2023 20:50, Matt Zagrabelny wrote:
> On Mon, Mar 27, 2023 at 2:25 PM Andrei Borzenkov <arvidjaar@gmail.com> wrote:
>>
>> On 27.03.2023 21:48, Matt Zagrabelny wrote:
>>> Greetings,
>>>
>>> I have a root partition btrfs file system.
>>>
>>> I need to have /tmp, /var, /var/tmp, /var/log, and other directories
>>> under separate partitions so that certain mount options can be set for
>>> those partitions/directories.
>>>
>>> I'm testing out a subvolume mount with the subvolume /subv_content
>>> mounted at /subv_mnt.
>>>
>>> For instance, the noexec mount option can be circumvented:
>>
>> "exec/noexec" option applies to mount instance, it is not persistent
>> property of underlying filesystem. It is not specific to btrfs at all.
> 
> Agreed. My email was more about subvolumes and the mount point has the
> "noexec", but the actual subvolume doesn't - so there exists a path on
> disk where folks can exec the same file by circumventing the mount
> option by directly invoking the full path under the subvolume.

So, create the subvolume inside a non-world-readable directory? In fact,
I always create all the subvolumes inside top level (subvolid=5)
subvolume but that subvolume is not normally mounted. /, /tmp, /var, etc
are all subvolumes and subvolid=5 is not mounted at all (or can be
mounted with a mount point somewhere not world accessible).

Don't make the mistake of thinking that subvolumes have to be visible
anywhere in the filesystem except the place you mount them.

Graham

Re: subvolumes as partitions and mount options

[Matthew Warren]

If you want something like this, you will want to have those
subvolumes outside of the root subvolume. For instance, My BTRFS
subvolumes look like this
/ root subvol - The subvolume which is created on mkfs
/@arch - The subvolume I have mounted as /
/@home - The subvolume I have mounted as /home

If you do something like that, then you prevent access by having it
hidden in the root subvolume.

Matthew Warren

On Mon, Mar 27, 2023 at 3:57 PM Matt Zagrabelny <mzagrabe@d.umn.edu> wrote:
>
> On Mon, Mar 27, 2023 at 2:25 PM Andrei Borzenkov <arvidjaar@gmail.com> wrote:
> >
> > On 27.03.2023 21:48, Matt Zagrabelny wrote:
> > > Greetings,
> > >
> > > I have a root partition btrfs file system.
> > >
> > > I need to have /tmp, /var, /var/tmp, /var/log, and other directories
> > > under separate partitions so that certain mount options can be set for
> > > those partitions/directories.
> > >
> > > I'm testing out a subvolume mount with the subvolume /subv_content
> > > mounted at /subv_mnt.
> > >
> > > For instance, the noexec mount option can be circumvented:
> >
> > "exec/noexec" option applies to mount instance, it is not persistent
> > property of underlying filesystem. It is not specific to btrfs at all.
>
> Agreed. My email was more about subvolumes and the mount point has the
> "noexec", but the actual subvolume doesn't - so there exists a path on
> disk where folks can exec the same file by circumventing the mount
> option by directly invoking the full path under the subvolume.
>
> >
> > bor@bor-Latitude-E5450:/tmp/tst$ ./bin/foo.sh
> > Hello, world!
> > bor@bor-Latitude-E5450:/tmp/tst$ mkdir exec noexec
> > bor@bor-Latitude-E5450:/tmp/tst$ sudo mount -o bind,exec bin exec
> > bor@bor-Latitude-E5450:/tmp/tst$ sudo mount -o bind,noexec bin noexec
> > bor@bor-Latitude-E5450:/tmp/tst$ ./exec/foo.sh
> > Hello, world!
> > bash: ./noexec/foo.sh: Permission denied
> > bor@bor-Latitude-E5450:/tmp/tst$
>
> Agreed completely.
>
> If an attacker can gain access to a system, I'd like /tmp to be
> mounted "noexec".
>
> The attacker can execute the foo.sh via /tmp/tst/bin/foo.sh even
> though the bind mount (/tmp/tst/noexec) restricts the executing of
> programs.
>
> That seems to be the position I am in right now with subvolumes as
> opposed to an actual partition.
>
> If I create a separate partition for /tmp and mount it noexec, there
> is no backdoor bind mount where the attacker can execute programs
> from.
>
> It seems mounting subvolumes works similarly to bind mounts - is there
> a way to mimic /tmp being on a separate partition and mounted with
> noexec using subvolumes?
>
> Thanks for the help!
>
> -m

Re: subvolumes as partitions and mount options

[Matt Zagrabelny]

Hi Matthew,

On Mon, Mar 27, 2023 at 3:32 PM Matthew Warren
<matthewwarren101010@gmail.com> wrote:
>
> If you want something like this, you will want to have those
> subvolumes outside of the root subvolume. For instance, My BTRFS
> subvolumes look like this
> / root subvol - The subvolume which is created on mkfs
> /@arch - The subvolume I have mounted as /
> /@home - The subvolume I have mounted as /home
>
> If you do something like that, then you prevent access by having it
> hidden in the root subvolume.

Do you know if I can retrofit my current btrfs install to implement
the structure you've suggested?

To my knowledge I've got my root filesystem mounted on the "parent" subvolume:

root@ziti:~# btrfs subvolume list / -a
ID 256 gen 606645 top level 5 path <FS_TREE>/@rootfs
ID 257 gen 606389 top level 256 path @rootfs/subv_content

root@ziti:~# mount | grep btrfs
/dev/nvme0n1p2 on / type btrfs
(rw,relatime,ssd,space_cache=v2,subvolid=256,subvol=/@rootfs)
/dev/nvme0n1p2 on /subv_mnt type btrfs
(rw,nosuid,nodev,noexec,relatime,ssd,space_cache=v2,subvolid=257,subvol=/@rootfs/subv_content)

The subv_content subvolume is just for testing and can be deleted.

Thanks for any pointers!

-m

Re: subvolumes as partitions and mount options

[Matthew Warren]

It looks like you already have it mostly set up correctly. You will
want to mount your filesystem somewhere without specifying a
subvolume. Then you can put all the subvolumes you want "hidden" in
there. This should be as simple as unomunting /subv_mnt, moving
subv_content to the btrfs root subvolume, and then re-mounting it with
the new position. This is what it looks like for me when I run the
subvolume list command.

sudo btrfs sub list / -a
ID 258 gen 680918 top level 5 path <FS_TREE>/@arch
ID 259 gen 680918 top level 5 path <FS_TREE>/@home
ID 260 gen 680915 top level 5 path <FS_TREE>/@snapshots
ID 726 gen 658581 top level 260 path <FS_TREE>/@snapshots/ROOT.20230320T0100
ID 727 gen 658582 top level 260 path <FS_TREE>/@snapshots/home.20230320T0100
... trimmed...
ID 740 gen 678482 top level 260 path <FS_TREE>/@snapshots/ROOT.20230327T0100
ID 741 gen 678483 top level 260 path <FS_TREE>/@snapshots/home.20230327T0100

And this is what the root of my btrfs file system looks like with ls

ls
'@arch'/  '@home'/  '@snapshots'/

Matthew Warren

On Mon, Mar 27, 2023 at 5:06 PM Matt Zagrabelny <mzagrabe@d.umn.edu> wrote:
>
> Hi Matthew,
>
> On Mon, Mar 27, 2023 at 3:32 PM Matthew Warren
> <matthewwarren101010@gmail.com> wrote:
> >
> > If you want something like this, you will want to have those
> > subvolumes outside of the root subvolume. For instance, My BTRFS
> > subvolumes look like this
> > / root subvol - The subvolume which is created on mkfs
> > /@arch - The subvolume I have mounted as /
> > /@home - The subvolume I have mounted as /home
> >
> > If you do something like that, then you prevent access by having it
> > hidden in the root subvolume.
>
> Do you know if I can retrofit my current btrfs install to implement
> the structure you've suggested?
>
> To my knowledge I've got my root filesystem mounted on the "parent" subvolume:
>
> root@ziti:~# btrfs subvolume list / -a
> ID 256 gen 606645 top level 5 path <FS_TREE>/@rootfs
> ID 257 gen 606389 top level 256 path @rootfs/subv_content
>
> root@ziti:~# mount | grep btrfs
> /dev/nvme0n1p2 on / type btrfs
> (rw,relatime,ssd,space_cache=v2,subvolid=256,subvol=/@rootfs)
> /dev/nvme0n1p2 on /subv_mnt type btrfs
> (rw,nosuid,nodev,noexec,relatime,ssd,space_cache=v2,subvolid=257,subvol=/@rootfs/subv_content)
>
> The subv_content subvolume is just for testing and can be deleted.
>
> Thanks for any pointers!
>
> -m

Re: subvolumes as partitions and mount options

[Matt Zagrabelny]

On Mon, Mar 27, 2023 at 8:42 PM Matthew Warren
<matthewwarren101010@gmail.com> wrote:
>
> It looks like you already have it mostly set up correctly. You will
> want to mount your filesystem somewhere without specifying a
> subvolume.

Sure. Things are starting to make a little more sense.

 Then you can put all the subvolumes you want "hidden" in
> there. This should be as simple as unomunting /subv_mnt, moving
> subv_content to the btrfs root subvolume,

Right. My rootfs is mounted with a subvolume option, so I still can't
get at the "root" subvolume:

# mount | grep btrfs
/dev/nvme0n1p2 on / type btrfs
(rw,relatime,ssd,space_cache=v2,subvolid=256,subvol=/@rootfs)

Thusly, I would need to unmount my root partition (presumably through
a live-cd or equivalent) and then mount:

mount /dev/nvme0n1p2 /mnt

and create my subvolume:

btrfs subvolume create /mnt/@foo

then boot back into my system with the regular root fs mount entry in
/etc/fstab and then I can mount the subvolume as desired:

mount /dev/nvme0n1p2 /path/to/foo -o subvol=@foo

It looks like I can mount the root:

sudo -i
mkdir /btrfs-fixer
mount /dev/nvme0n1p2 /btrfs-fixer
btrfs subvolume create /btrfs-fixer/@foo
umount /dev/nvme0n1p2
rm -rf /btrfs-fixer
mount /dev/nvme0n1p2 /path/to/foo -o subvol=@foo

Not a bad work-around.

Thanks for all the help!

-m

Re: subvolumes as partitions and mount options

[Andrei Borzenkov]

On 28.03.2023 22:45, Matt Zagrabelny wrote:
> On Mon, Mar 27, 2023 at 8:42 PM Matthew Warren
> <matthewwarren101010@gmail.com> wrote:
>>
>> It looks like you already have it mostly set up correctly. You will
>> want to mount your filesystem somewhere without specifying a
>> subvolume.
> 
> Sure. Things are starting to make a little more sense.
> 
>   Then you can put all the subvolumes you want "hidden" in
>> there. This should be as simple as unomunting /subv_mnt, moving
>> subv_content to the btrfs root subvolume,
> 
> Right. My rootfs is mounted with a subvolume option, so I still can't
> get at the "root" subvolume:
> 
> # mount | grep btrfs
> /dev/nvme0n1p2 on / type btrfs
> (rw,relatime,ssd,space_cache=v2,subvolid=256,subvol=/@rootfs)
> 
> Thusly, I would need to unmount my root partition (presumably through

There is no need for this, multiple subvolumes can be mounted concurrently.

mount -o subvol=/ ...

> a live-cd or equivalent) and then mount:
> 
> mount /dev/nvme0n1p2 /mnt
> 

This depends on what default subvolume is and will not necessarily mount 
the top level of btrfs.

> and create my subvolume:
> 
> btrfs subvolume create /mnt/@foo
> 

"@" is just a convention, there is no magic in it, it is not *needed*.

> then boot back into my system with the regular root fs mount entry in
> /etc/fstab and then I can mount the subvolume as desired:
> 
> mount /dev/nvme0n1p2 /path/to/foo -o subvol=@foo
> 
> It looks like I can mount the root:
> 
> sudo -i
> mkdir /btrfs-fixer
> mount /dev/nvme0n1p2 /btrfs-fixer
> btrfs subvolume create /btrfs-fixer/@foo
> umount /dev/nvme0n1p2
> rm -rf /btrfs-fixer
> mount /dev/nvme0n1p2 /path/to/foo -o subvol=@foo
> 
> Not a bad work-around.
> 
> Thanks for all the help!
> 
> -m