ZFS on Debian GNU/kFreeBSD

ZFS on Debian GNU/kFreeBSD

[Tuco]

Hello

I've been doing some experiments with ZFS on Debian GNU/kFreeBSD. My
goal is to setup ZFS as root filesystem. I setup the ZFS tools in
Debian with some tricks (my blog has detail about them:
http://tucobsd.blogspot.com/2010/07/how-to-enable-zfs-on-debian-gnukfreebsd.html),
but when trying to install GRUB in a ZFS filesystem I run into
trouble:

/usr/sbin/grub-probe: error: cannot find a device for /boot/grub (is
/dev mounted?).

I checked the command run by grub-install; in a normal (UFS) system
this command prints /dev/ad0s2.

Is this a known problem? Should I investigate? Can you give some guideline?

Re: ZFS on Debian GNU/kFreeBSD

[Vladimir 'φ-coder/phcoder' Serbinenko]

[-- Attachment #1: Type: text/plain, Size: 1038 bytes --]

On 07/21/2010 01:22 PM, Tuco wrote:
> Hello
>
> I've been doing some experiments with ZFS on Debian GNU/kFreeBSD. My
> goal is to setup ZFS as root filesystem. I setup the ZFS tools in
> Debian with some tricks (my blog has detail about them:
> http://tucobsd.blogspot.com/2010/07/how-to-enable-zfs-on-debian-gnukfreebsd.html),
> but when trying to install GRUB in a ZFS filesystem I run into
> trouble:
>
> /usr/sbin/grub-probe: error: cannot find a device for /boot/grub (is
> /dev mounted?).
>
> I checked the command run by grub-install; in a normal (UFS) system
> this command prints /dev/ad0s2.
>
> Is this a known problem? Should I investigate? Can you give some guideline?
>
>   
Yes it is. ZFS is in separate repo and isn't added to grub_probe source
so it doesn't work in grub-probe.
> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> http://lists.gnu.org/mailman/listinfo/grub-devel
>
>   


-- 
Regards
Vladimir 'φ-coder/phcoder' Serbinenko



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 294 bytes --]

Re: ZFS on Debian GNU/kFreeBSD

[Seth Goldberg]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1314 bytes --]



Quoting Vladimir 'φ-coder/phcoder' Serbinenko, who wrote the following on...:

> On 07/21/2010 01:22 PM, Tuco wrote:
>> Hello
>>
>> I've been doing some experiments with ZFS on Debian GNU/kFreeBSD. My
>> goal is to setup ZFS as root filesystem. I setup the ZFS tools in
>> Debian with some tricks (my blog has detail about them:
>> http://tucobsd.blogspot.com/2010/07/how-to-enable-zfs-on-debian-gnukfreebsd.html),
>> but when trying to install GRUB in a ZFS filesystem I run into
>> trouble:
>>
>> /usr/sbin/grub-probe: error: cannot find a device for /boot/grub (is
>> /dev mounted?).
>>
>> I checked the command run by grub-install; in a normal (UFS) system
>> this command prints /dev/ad0s2.
>>
>> Is this a known problem? Should I investigate? Can you give some guideline?
>>
>>
> Yes it is. ZFS is in separate repo and isn't added to grub_probe source
> so it doesn't work in grub-probe.

  The main problem is that the st_dev that you'll get from a call to stat() of 
a file on a ZFS filesystem won't match the major/minor of any device in /dev 
(at least on Solaris).  You have to look deeper to figure out the pool name, 
then query the pool to figure out its devices.  And if you're REALLY good, 
you'd invoke grub-setup for each member of a root pool mirror :).

  --S

Re: ZFS on Debian GNU/kFreeBSD

[Robert Millan]

Hi,

This has been on my radar for a while now, I'll have a look at it
(today I start my vacation and have some spare time (but don't hold
your breath...)).

2010/7/21, Seth Goldberg <seth.goldberg@oracle.com>:
> The main problem is that the st_dev that you'll get from a call to stat() of
> a file on a ZFS filesystem won't match the major/minor of any device in /dev
> (at least on Solaris).  You have to look deeper to figure out the pool name,
> then query the pool to figure out its devices.

I guess the API exported by libzfs can serve for this?  But first, I
have some concern about zfs.mod, I'll write a detailed mail about that
later.

> And if you're REALLY good,
> you'd invoke grub-setup for each member of a root pool mirror :).

Uhm this sounds wrong, grub-setup isn't meant to handle filesystems
(it does something like you describe for RAID1, but only because it
can rely on the storage backend being 1:1).

Re: ZFS on Debian GNU/kFreeBSD

[Seth Goldberg]

Hi,

Quoting Robert Millan, who wrote the following on Thu, 22 Jul 2010:

> Hi,
>
> This has been on my radar for a while now, I'll have a look at it
> (today I start my vacation and have some spare time (but don't hold
> your breath...)).

   Nice to see you around again.  I didn't know you were doing anything with 
the zfs module :).

>
> 2010/7/21, Seth Goldberg <seth.goldberg@oracle.com>:
>> The main problem is that the st_dev that you'll get from a call to stat() of
>> a file on a ZFS filesystem won't match the major/minor of any device in /dev
>> (at least on Solaris).  You have to look deeper to figure out the pool name,
>> then query the pool to figure out its devices.
>
> I guess the API exported by libzfs can serve for this?  But first, I
> have some concern about zfs.mod, I'll write a detailed mail about that
> later.

   That's how I'm implementing it, though libzfs is technically an uncommitted 
interface.

>
>> And if you're REALLY good,
>> you'd invoke grub-setup for each member of a root pool mirror :).
>
> Uhm this sounds wrong, grub-setup isn't meant to handle filesystems
> (it does something like you describe for RAID1, but only because it
> can rely on the storage backend being 1:1).

  zfs is rather unique in this respect -- it's sorta like a filesystem but, 
really, it's pool-based, and the right thing to do is to update the boot 
blocks on all mirrors of the root pool to ensure they stay in synch.  When you 
invoke grub-setup (at least on Solaris), you'll pass it the boot slice (when 
used with a Solaris partition (the sunpc partmap).  When using it with GPT, 
you'll pass it the GPT ZFS partition (into which the core image is embedded on 
BIOS systems -- UEFI is not an issue, since the ESP is used).

  Yes, it's true that the disks used for the mirror can be different, and that 
would require grub-setup to write the boot and core image with different 
embedded information, hence the extra points for that ;).  To be truly 
fault-tolerant, if we're booting from a zfs pool, there must be an embedded 
config file that searches for a (non-faulted) root device from the pool's 
mirrors as well.  Separate invocations for each device in the root pool mirror 
are fine, though.

  BTW, I've run into a similar issue (comparing major/minors) in updating 
grub-probe to deal with zfs -- grub_make_system_path_relative_to_its_root() 
compares st_dev with the st_dev's of each path component.  This won't work on 
ZFS either because GRUB files are stored in the top-level filesystem and 
because the root filesystem (for the purposes of GRUB2's $root) is a different 
filesystem than the top-level filesystem.  GRUB2 was never designed to deal 
with this kind of disparity (even at the RAID level).

  --S

Re: ZFS on Debian GNU/kFreeBSD

[Robert Millan]

2010/7/22 Seth Goldberg <seth.goldberg@oracle.com>
>> I guess the API exported by libzfs can serve for this?  But first, I
>> have some concern about zfs.mod, I'll write a detailed mail about that
>> later.
>
>  That's how I'm implementing it, though libzfs is technically an uncommitted interface.

What are you currently implementing? I just want to make sure I don't
duplicate existing work.

>> Uhm this sounds wrong, grub-setup isn't meant to handle filesystems
>> (it does something like you describe for RAID1, but only because it
>> can rely on the storage backend being 1:1).
>
>  [...] To be truly fault-tolerant, if we're booting from a zfs pool, there must be an embedded config file that searches for a (non-faulted) root device from the pool's mirrors as well.  Separate invocations for each device in the root pool mirror are fine, though.

Ah, I understand you now. You want to take advantage of ZFS' fault-tollerancy
for boot blocks. I'm not sure this would make sense, because we have to give
up at some point (e.g. boot.img can't benefit from this), so we're just moving
the line. In any case, I just want the basic functionality to work and it's not
currently in my plan to implement this (sorry ;-)).

>  BTW, I've run into a similar issue (comparing major/minors) in updating grub-probe to deal with zfs -- grub_make_system_path_relative_to_its_root() compares st_dev with the st_dev's of each path component.  This won't work on ZFS either because GRUB files are stored in the top-level filesystem and because the root filesystem (for the purposes of GRUB2's $root) is a different filesystem than the top-level filesystem.  GRUB2 was never designed to deal with this kind of disparity (even at the RAID level).

Can you map st_dev to zpool name (and later to zpool devices) using libzfs?

Re: ZFS on Debian GNU/kFreeBSD

[Robert Millan]

2010/7/21 Tuco <tuco.xyz@gmail.com>:
> Is this a known problem? Should I investigate? Can you give some guideline?

If you want to help, you could have a look at porting libzfs over to
GNU/kFreeBSD.
It's likely that GRUB will need it, and in either case the Debian
folks definitely will.

Re: ZFS on Debian GNU/kFreeBSD

[Robert Millan]

The problem I see with current zfs.mod is that it integrates with GRUB using
the "filesystem model", i.e. each pool has one
backend (the storage device) and one frontend (the filesystem).

But ZFS is different in both ends: each pool can use N storage devices
and provide M filesystems. In the current representation this yields
two problems
for GRUB:

 - zpools where more than one device is essential (i.e. not just
mirror like RAID1)
   aren't accessible.

 - To allow more more than one filesystem in a zpool, filesystems are
selected in
   a special manner, e.g.:
   (hd0)/@/
   (hd0)/foo@/
   This collides with an assumption consistently made by GRUB utilities
   and scripts: that you can construct a GRUB path by combining the result
   of make_path_relative_to_its_root() (known at install time) with the device
   that contains this filesystem (usually known at run time).

My impression is representing ZFS as a filesystem is doable, but not
the best fit.
ZFS is much more like what GRUB calls an "abstraction" (RAID or LVM). In fact
if you take LVM interface:

grub> ls
(hd0) (hd1)
grub> insmod lvm
grub> ls
(hd0) (hd1) (foo) (bar)

the only difference for ZFS is that (foo) and (bar) are provided directly as
filesystems (like pxe.mod does) instead of block devices for GRUB to probe.

So my intention is to readjust zfs.mod to follow this interface. Aside
from enabling
multi-device pools, this will also make it much easier to support ZFS
in the install
utilities and scripts later.

Comments / objections / suggestions?

Re: ZFS on Debian GNU/kFreeBSD

[Seth Goldberg]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 3155 bytes --]



Quoting Robert Millan, who wrote the following on Fri, 23 Jul 2010:

> 2010/7/22 Seth Goldberg <seth.goldberg@oracle.com>
>>> I guess the API exported by libzfs can serve for this?  But first, I
>>> have some concern about zfs.mod, I'll write a detailed mail about that
>>> later.
>>
>>  That's how I'm implementing it, though libzfs is technically an uncommitted interface.
>
> What are you currently implementing? I just want to make sure I don't
> duplicate existing work.

  Sure -- The code I currently have in getroot.c and util.c is ugly (it's 
filesystem-aware), because of the disparity in the design as you mentioned. 
The code currently stat()s the given file, gets the st_dev and uses that to 
correlate the filesystem with the major/minor available via getextmntent(). 
It then looks at mnt_special (which is the zpool/zfs name) and looks up the 
pool via libzfs's zpool_get_physpath() (which only returns one device, 
incidentally, which is fine for now for testing, but is not good enough for a 
fault-tolerant configuration).  I also made modifications to 
grub_make_system_path_relative_to_its_root() to adapt the pathname passed in 
to Vladimir's current zfs design (/<fsname>@/<path>).  I now have working 
grub-probe and grub-setup (For Solaris's uses, I'm modifying grub-setup to use 
the boot slice in a Solaris partition instead of the zfs filesystem slice 
(because that's where our tools look for the boot loader on disk). Other OSes 
that use zfs can simply embed core into the zfs boot block area (which is over 
3M in size).

I like the idea of having the zfs stuff conform to the abstraction (though I 
think it's unique in its ability to use different-size disks, arbitrary 
partitions, etc, as part of its pool, so assuming that's workable in the GRUB2 
design, I'd much rather see (<poolname>)/filesystem[@snapshot]/<path> at the 
top-level than have to deal with the disk partitions individually).  Since 
it's not an abstraction today, the code to deal with it has been a bit ugly -- 
I'm all for clean design here.


>
>>> Uhm this sounds wrong, grub-setup isn't meant to handle filesystems
>>> (it does something like you describe for RAID1, but only because it
>>> can rely on the storage backend being 1:1).
>>
>>  [...] To be truly fault-tolerant, if we're booting from a zfs pool, there must be an embedded config file that searches for a (non-faulted) root device from the pool's mirrors as well.  Separate invocations for each device in the root pool mirror are fine, though.
>
> Ah, I understand you now. You want to take advantage of ZFS' fault-tollerancy
> for boot blocks. I'm not sure this would make sense, because we have to give
> up at some point (e.g. boot.img can't benefit from this), so we're just moving
> the line. In any case, I just want the basic functionality to work and it's not

  Actually, that's not true.  Even boot.img can benefit if the system firmware 
cannot read from the disk (then fails over to the next bootable device in the 
device list -- and that's exactly what we do today on BIOS systems, and it 
works well).

  --S

Re: ZFS on Debian GNU/kFreeBSD

[Seth Goldberg]

Quoting Robert Millan, who wrote the following on Fri, 23 Jul 2010:

> The problem I see with current zfs.mod is that it integrates with GRUB using
> the "filesystem model", i.e. each pool has one
> backend (the storage device) and one frontend (the filesystem).
>
> But ZFS is different in both ends: each pool can use N storage devices
> and provide M filesystems. In the current representation this yields
> two problems
> for GRUB:
>
> - zpools where more than one device is essential (i.e. not just
> mirror like RAID1)
>   aren't accessible.

   Very true, and if non-mirrored root pools are supported in the future, that 
will be a problem.

>
> - To allow more more than one filesystem in a zpool, filesystems are
> selected in
>   a special manner, e.g.:
>   (hd0)/@/
>   (hd0)/foo@/
>   This collides with an assumption consistently made by GRUB utilities
>   and scripts: that you can construct a GRUB path by combining the result
>   of make_path_relative_to_its_root() (known at install time) with the device
>   that contains this filesystem (usually known at run time).

   Well, that still does work -- you just need to modify 
make_path_relative_to_its_root ;).


> My impression is representing ZFS as a filesystem is doable, but not
> the best fit.
> ZFS is much more like what GRUB calls an "abstraction" (RAID or LVM). In fact
> if you take LVM interface:
>
> grub> ls
> (hd0) (hd1)
> grub> insmod lvm
> grub> ls
> (hd0) (hd1) (foo) (bar)
>
> the only difference for ZFS is that (foo) and (bar) are provided directly as
> filesystems (like pxe.mod does) instead of block devices for GRUB to probe.

   How do you deal with multiple pools each with the same filesystem names?  We 
do need a namespace -- top-level should be (<poolname>) (though keep in mind 
it's possible for different pools with the same pool name to exist, so we'd 
need a uniquifier here also).

> So my intention is to readjust zfs.mod to follow this interface. Aside from 
> enabling multi-device pools, this will also make it much easier to support 
> ZFS in the install utilities and scripts later.
>
> Comments / objections / suggestions?

   Sounds like an interesting experiment, at least :).

  --S

Re: ZFS on Debian GNU/kFreeBSD

[Vladimir 'φ-coder/phcoder' Serbinenko]

[-- Attachment #1: Type: text/plain, Size: 2950 bytes --]

On 07/23/2010 02:53 PM, Robert Millan wrote:
> The problem I see with current zfs.mod is that it integrates with GRUB using
> the "filesystem model", i.e. each pool has one
> backend (the storage device) and one frontend (the filesystem).
>
> But ZFS is different in both ends: each pool can use N storage devices
> and provide M filesystems. In the current representation this yields
> two problems
> for GRUB:
>
>  - zpools where more than one device is essential (i.e. not just
> mirror like RAID1)
>    aren't accessible.
>
>   
They are simply not implemented yet. When it will be you'll have to just
access any underlying device and other devices will be autoscanned.
>  - To allow more more than one filesystem in a zpool, filesystems are
> selected in
>    a special manner, e.g.:
>    (hd0)/@/
>    (hd0)/foo@/
>    This collides with an assumption consistently made by GRUB utilities
>    and scripts: that you can construct a GRUB path by combining the result
>    of make_path_relative_to_its_root() (known at install time) with the device
>    that contains this filesystem (usually known at run time).
>
>   
This syntax is the one that reflects the real structure of ZFS. OS often
needs some kind of tricks which warp the reality and show it in a way
which is the best in achieving many OS goals. GRUB goals are much
simpler and such warping would only complicate the design. One example
of this is using (<DISK>)/<path> instead of mount points. It results in
more difficulty in userland tools but there we can manage complexity
easier since we have all the usual tools.
> My impression is representing ZFS as a filesystem is doable, but not
> the best fit.
> ZFS is much more like what GRUB calls an "abstraction" (RAID or LVM). In fact
> if you take LVM interface:
>
> grub> ls
> (hd0) (hd1)
> grub> insmod lvm
> grub> ls
> (hd0) (hd1) (foo) (bar)
>
> the only difference for ZFS is that (foo) and (bar) are provided directly as
> filesystems (like pxe.mod does) instead of block devices for GRUB to probe.
>
>   
It was the way I originally did it but it turned out that ZFS is goot at
representing tree-like structure of subvolumes, it may have thousands of
subvolumes and snapshots. Trying to "ls" it would be useless. Handling
tree-like structure in devices is possible but probably not necessary,
just for ZFS.
> So my intention is to readjust zfs.mod to follow this interface. Aside
> from enabling
> multi-device pools, this will also make it much easier to support ZFS
> in the install
> utilities and scripts later.
>
> Comments / objections / suggestions?
>
>   
I don't think that the design needs changes but I'm open to discussion.
> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> http://lists.gnu.org/mailman/listinfo/grub-devel
>
>   


-- 
Regards
Vladimir 'φ-coder/phcoder' Serbinenko



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 294 bytes --]

Re: ZFS on Debian GNU/kFreeBSD

[Robert Millan]

2010/7/28, Vladimir 'φ-coder/phcoder' Serbinenko <phcoder@gmail.com>:
> I don't think that the design needs changes but I'm open to discussion.

My proposal seemed like an improvement in design, which I wanted to do before
dealing with grub-install/grub-mkconfig. But obviously I've missed some details,
and things aren't as nice as they appear at first.

So I'll move on to next step now. Thanks for the explanation :-)