[dm-crypt] btrfs and "requested offset is beyond real size of device"

[dm-crypt] btrfs and "requested offset is beyond real size of device"

[Jay Sullivan]

[-- Attachment #1: Type: text/plain, Size: 1477 bytes --]

Hi,

I don't know if this belongs on the btrfs, GRUB, or LUKS mailing list, but
it is a cryptsetup error, so I figure maybe someone here can help me
understand what this error means.

*What I'm doing*

I'm trying to set up a system with a btrfs /boot partition, and a few
encrypted btrfs-over-LUKS partitions. Nothing special, except that I'm
using btrfs for all of my partitions.

*The Problem*

I did get the setup "working", I'm able to successfully boot into it.
Here's the weird part: it only successfully boots about half of the time.

The other half of the time,  I see this error just after I successfully
enter my password:

    requested offset is beyond real size of device

When this happens, it continues to fail when it prompts me again for my
password, until I reboot.  It seems to fall into this issue about 50% of
the boots.

The culprit seems to be when the /boot partition is btrfs. If I use ext4 as
/boot, everything works fine and dandy.

This should be reproducible on any machine, I've tried several times on
different machines to rule out hardware. I can give complete instructions
to reproduce if someone wants it, but before I go through the effort, has
anyone heard of this behavior before? Does anyone have any insight on why
this may be happening, especially why it only happens sometimes.

I'm using Ubuntu 14.10's LiveCD to install the operating system, where
cryptsetup's version says 1.6.1, and btrfs's version is 3.14.1.

Thanks,
Jay Sullivan

[-- Attachment #2: Type: text/html, Size: 1765 bytes --]

Re: [dm-crypt] btrfs and "requested offset is beyond real size of device"

[Arno Wagner]

Just one data-point: LUKS does not store device size. It gets 
queried on mapping. What it stores is the data-offset, i.e.
where the data-area starts after they keyslot-area.

Can you reproduce this with strace and a password that is not 
secret? ("strace cryptsetup ...") That should tell us in which 
call the error happens.

Arno 

On Sun, Feb 15, 2015 at 23:43:22 CET, Jay Sullivan wrote:
> Hi,
> 
> I don't know if this belongs on the btrfs, GRUB, or LUKS mailing list, but
> it is a cryptsetup error, so I figure maybe someone here can help me
> understand what this error means.
> 
> *What I'm doing*
> 
> I'm trying to set up a system with a btrfs /boot partition, and a few
> encrypted btrfs-over-LUKS partitions. Nothing special, except that I'm
> using btrfs for all of my partitions.
> 
> *The Problem*
> 
> I did get the setup "working", I'm able to successfully boot into it.
> Here's the weird part: it only successfully boots about half of the time.
> 
> The other half of the time,  I see this error just after I successfully
> enter my password:
> 
>     requested offset is beyond real size of device
> 
> When this happens, it continues to fail when it prompts me again for my
> password, until I reboot.  It seems to fall into this issue about 50% of
> the boots.
> 
> The culprit seems to be when the /boot partition is btrfs. If I use ext4 as
> /boot, everything works fine and dandy.
> 
> This should be reproducible on any machine, I've tried several times on
> different machines to rule out hardware. I can give complete instructions
> to reproduce if someone wants it, but before I go through the effort, has
> anyone heard of this behavior before? Does anyone have any insight on why
> this may be happening, especially why it only happens sometimes.
> 
> I'm using Ubuntu 14.10's LiveCD to install the operating system, where
> cryptsetup's version says 1.6.1, and btrfs's version is 3.14.1.
> 
> Thanks,
> Jay Sullivan

> _______________________________________________
> dm-crypt mailing list
> dm-crypt@saout.de
> http://www.saout.de/mailman/listinfo/dm-crypt


-- 
Arno Wagner,     Dr. sc. techn., Dipl. Inform.,    Email: arno@wagner.name
GnuPG: ID: CB5D9718  FP: 12D6 C03B 1B30 33BB 13CF  B774 E35C 5FA1 CB5D 9718
----
A good decision is based on knowledge and not on numbers. -- Plato

If it's in the news, don't worry about it.  The very definition of 
"news" is "something that hardly ever happens." -- Bruce Schneier

Re: [dm-crypt] btrfs and "requested offset is beyond real size of device"

[Milan Broz]

On 02/15/2015 11:43 PM, Jay Sullivan wrote:
> Hi,
> 
> I don't know if this belongs on the btrfs, GRUB, or LUKS mailing
> list, but it is a cryptsetup error, so I figure maybe someone here
> can help me understand what this error means.
> 
> *What I'm doing*
> 
> I'm trying to set up a system with a btrfs /boot partition, and a few
> encrypted btrfs-over-LUKS partitions. Nothing special, except that
> I'm using btrfs for all of my partitions.
> 
> *The Problem*
> 
> I did get the setup "working", I'm able to successfully boot into it.
> Here's the weird part: it only successfully boots about half of the
> time.
> 
> The other half of the time,  I see this error just after I
> successfully enter my password:
> 
> requested offset is beyond real size of device

Hi,

this means that reporeted device size is for some reason smaller than 
LUKS data starting offset (payload) in header.

Unfortunately exactly this code was changed, so it would be nice to reproduce
it with new cryptsetup (1.6.6 ideally).

> When this happens, it continues to fail when it prompts me again for
> my password, until I reboot.  It seems to fall into this issue about
> 50% of the boots.

ok, this is strange. I would say some udev script interfering here
locking the device or so...

Does your system use cryptsetup directly or systemd-cryptsetup to activate partition?

> The culprit seems to be when the /boot partition is btrfs. If I use
> ext4 as /boot, everything works fine and dandy.
> 
> This should be reproducible on any machine, I've tried several times
> on different machines to rule out hardware. I can give complete
> instructions to reproduce if someone wants it, but before I go
> through the effort, has anyone heard of this behavior before? Does
> anyone have any insight on why this may be happening, especially why
> it only happens sometimes.
> 
> I'm using Ubuntu 14.10's LiveCD to install the operating system,
> where cryptsetup's version says 1.6.1, and btrfs's version is
> 3.14.1.

Please can you describe exactly how to reproduce it?
How the partition table looks like (MBR or GPT?) and how is the devices mapped?
(Paste lsblk output from correctly booted system).

What is in /etc/cryptab and fstab and on kernel boot line?
(Do you always use underlying partition for dmcrypt with btrfs on top of it?)

Maybe it would be better to create new issue here just to track debug data
https://code.google.com/p/cryptsetup/issues/list

Thanks,
Milan

Re: [dm-crypt] btrfs and "requested offset is beyond real size of device"

[Jay Sullivan]

[-- Attachment #1: Type: text/plain, Size: 4265 bytes --]

So,

I must have been doing some step in the wrong order before, because, after
cleaning up my notes for others to read, it now seems to work fine.  I'm
still sort of confused why I was running into that error consistently, but
clearly I'm doing something differently this time.  Attached steps for
creating full disk encryption with btrfs with Ubuntu 14.10, which worked
successfully for me.

----
---- Start of steps to reproduce
----

I'm using Ubuntu Desktop 14.10 LiveCD, with only one hard drive in my
computer: a 32GB flash drive.  (Note: I tried this on a 500GB hard drive as
well, same results.)  I'm doing all of this offline (as in not connected to
Internet.)

I use GParted to partition the flash drive as follows (GPT):

    /dev/sda1     1MiB      bios_grub     ( < I make sure to set the
bios_grub flag here )
    /dev/sda2   256MiB    btrfs         ( /boot )
    /dev/sda3   6.84GiB   crypt-luks    ( encrypted swap )
    /dev/sda4   22.82GiB  crypt-luks    ( encrypted / )

I then format/open the LUKS partitons:

    cryptsetup luksFormat /dev/sda3
    cryptsetup luksFormat /dev/sda4
    cryptsetup luksOpen /dev/sda3 neoswap
    cryptsetup luksOpen /dev/sda4 neoroot

I then format all my boot, swap, and root partitions:

    mkfs.btrfs -f -L neoboot /dev/sda2
    mkswap -L neoswap /dev/mapper/neoswap
    mkfs.btrfs -f -L neoroot /dev/mapper/neoroot

I then run the graphical "Install Ubuntu 14.10" installer from the
desktop.  When it asks how I want to install, I choose "Something else",
and tell it to use the following partitioning scheme, instructing it NOT to
format any partitions (since we already did):

    /dev/sda2                btrfs    /boot
    /dev/mapper/neoswap      swap
    /dev/mapper/neoroot      btrfs    /

(Note: It will insist on formatting the swap partiton, that's fine.  It
will insist on installing a bootloader to /dev/sda, that's fine, although
we will override this step in a second.)  I continue with the installation,
waiting until the installation is finished.

At this point, it's possible for me to list all of the partition GUIDs,
which I'll need for crypttab and fstab:

    /dev/sda1     1MiB      bios_grub
    /dev/sda2   256MiB    btrfs         ( /boot )
UUID=23fa61d2-0b74-465a-be83-e026ebb373bc
    /dev/sda3   6.84GiB   crypt-luks
UUID=5d60aeca-6e5d-4a0c-9d28-b31c79213589
        /dev/mapper/neoswap             ( swap  )
UUID=ee13478f-b310-4c3c-b14a-579dea9ee25c
    /dev/sda4   22.82GiB  crypt-luks
UUID=badb304c-38de-4d04-9456-e9759c3150ab
        /dev/mapper/neoroot             ( /     )
UUID=aa67f7ba-08cc-4760-aa30-fa5bc10b88cc

Once installation is done, and before I restart, I open a root shell and
chroot into the newly-installed setup:

    # mount /dev/mapper/neoroot /mnt -o subvol=@
    # mount /dev/mapper/neoroot /mnt/home -o subvol=@home
    # mount /dev/sda2 /mnt/boot
    # mount --bind /dev /mnt/dev
    # mount --bind /proc /mnt/proc
    # mount --bind /sys /mnt/sys
    # mount --bind /etc/resolv.conf /mnt/etc/resolv.conf
    # chroot /mnt

Now that I'm in, I update /etc/crypttab as follows:

    neoswap UUID=5d60aeca-6e5d-4a0c-9d28-b31c79213589 none luks
    neoroot UUID=badb304c-38de-4d04-9456-e9759c3150ab none luks

I then update /etc/fstab:

    UUID=23fa61d2-0b74-465a-be83-e026ebb373bc /boot           btrfs
defaults,noatime              0       2
    UUID=ee13478f-b310-4c3c-b14a-579dea9ee25c none            swap
sw                            0       0
    UUID=aa67f7ba-08cc-4760-aa30-fa5bc10b88cc /               btrfs
defaults,noatime,subvol=@     0       1
    UUID=aa67f7ba-08cc-4760-aa30-fa5bc10b88cc /home           btrfs
defaults,noatime,subvol=@home 0       2

Almost done.  Still chroot'ed, I run:

    update-initramfs -k all -c
    update-grub
    grub-install /dev/sda

I restart my computer, boot into GRUB, and boot with the following:

    insmod gzio
    insmod part_gpt
    insmod btrfs
    search --no-floppy --fs-uuid --set=root
23fa61d2-0b74-465a-be83-e026ebb373bc
    linux /vmlinuz-3.16.0-23-generic
root=UUID=aa67f7ba-08cc-4760-aa30-fa5bc10b88cc rootflags=noatime,subvol=@
    initrd /initrd.img-3.16.0-23-generic

----
---- End of steps to reproduce
----

I just did this and it booted successfully 15 times in a row.

Thanks,
Jay

[-- Attachment #2: Type: text/html, Size: 5175 bytes --]

Re: [dm-crypt] btrfs and "requested offset is beyond real size of device"

[Arno Wagner]

On Mon, Feb 16, 2015 at 22:28:54 CET, Jay Sullivan wrote:
> So,
> 
> I must have been doing some step in the wrong order before, because, after
> cleaning up my notes for others to read, it now seems to work fine.  I'm
> still sort of confused why I was running into that error consistently, but
> clearly I'm doing something differently this time.  Attached steps for
> creating full disk encryption with btrfs with Ubuntu 14.10, which worked
> successfully for me.

Interesting. Thanks for reporting the success and the details!

Arno
 
> ----
> ---- Start of steps to reproduce
> ----
> 
> I'm using Ubuntu Desktop 14.10 LiveCD, with only one hard drive in my
> computer: a 32GB flash drive.  (Note: I tried this on a 500GB hard drive as
> well, same results.)  I'm doing all of this offline (as in not connected to
> Internet.)
> 
> I use GParted to partition the flash drive as follows (GPT):
> 
>     /dev/sda1     1MiB      bios_grub     ( < I make sure to set the
> bios_grub flag here )
>     /dev/sda2   256MiB    btrfs         ( /boot )
>     /dev/sda3   6.84GiB   crypt-luks    ( encrypted swap )
>     /dev/sda4   22.82GiB  crypt-luks    ( encrypted / )
> 
> I then format/open the LUKS partitons:
> 
>     cryptsetup luksFormat /dev/sda3
>     cryptsetup luksFormat /dev/sda4
>     cryptsetup luksOpen /dev/sda3 neoswap
>     cryptsetup luksOpen /dev/sda4 neoroot
> 
> I then format all my boot, swap, and root partitions:
> 
>     mkfs.btrfs -f -L neoboot /dev/sda2
>     mkswap -L neoswap /dev/mapper/neoswap
>     mkfs.btrfs -f -L neoroot /dev/mapper/neoroot
> 
> I then run the graphical "Install Ubuntu 14.10" installer from the
> desktop.  When it asks how I want to install, I choose "Something else",
> and tell it to use the following partitioning scheme, instructing it NOT to
> format any partitions (since we already did):
> 
>     /dev/sda2                btrfs    /boot
>     /dev/mapper/neoswap      swap
>     /dev/mapper/neoroot      btrfs    /
> 
> (Note: It will insist on formatting the swap partiton, that's fine.  It
> will insist on installing a bootloader to /dev/sda, that's fine, although
> we will override this step in a second.)  I continue with the installation,
> waiting until the installation is finished.
> 
> At this point, it's possible for me to list all of the partition GUIDs,
> which I'll need for crypttab and fstab:
> 
>     /dev/sda1     1MiB      bios_grub
>     /dev/sda2   256MiB    btrfs         ( /boot )
> UUID=23fa61d2-0b74-465a-be83-e026ebb373bc
>     /dev/sda3   6.84GiB   crypt-luks
> UUID=5d60aeca-6e5d-4a0c-9d28-b31c79213589
>         /dev/mapper/neoswap             ( swap  )
> UUID=ee13478f-b310-4c3c-b14a-579dea9ee25c
>     /dev/sda4   22.82GiB  crypt-luks
> UUID=badb304c-38de-4d04-9456-e9759c3150ab
>         /dev/mapper/neoroot             ( /     )
> UUID=aa67f7ba-08cc-4760-aa30-fa5bc10b88cc
> 
> Once installation is done, and before I restart, I open a root shell and
> chroot into the newly-installed setup:
> 
>     # mount /dev/mapper/neoroot /mnt -o subvol=@
>     # mount /dev/mapper/neoroot /mnt/home -o subvol=@home
>     # mount /dev/sda2 /mnt/boot
>     # mount --bind /dev /mnt/dev
>     # mount --bind /proc /mnt/proc
>     # mount --bind /sys /mnt/sys
>     # mount --bind /etc/resolv.conf /mnt/etc/resolv.conf
>     # chroot /mnt
> 
> Now that I'm in, I update /etc/crypttab as follows:
> 
>     neoswap UUID=5d60aeca-6e5d-4a0c-9d28-b31c79213589 none luks
>     neoroot UUID=badb304c-38de-4d04-9456-e9759c3150ab none luks
> 
> I then update /etc/fstab:
> 
>     UUID=23fa61d2-0b74-465a-be83-e026ebb373bc /boot           btrfs
> defaults,noatime              0       2
>     UUID=ee13478f-b310-4c3c-b14a-579dea9ee25c none            swap
> sw                            0       0
>     UUID=aa67f7ba-08cc-4760-aa30-fa5bc10b88cc /               btrfs
> defaults,noatime,subvol=@     0       1
>     UUID=aa67f7ba-08cc-4760-aa30-fa5bc10b88cc /home           btrfs
> defaults,noatime,subvol=@home 0       2
> 
> Almost done.  Still chroot'ed, I run:
> 
>     update-initramfs -k all -c
>     update-grub
>     grub-install /dev/sda
> 
> I restart my computer, boot into GRUB, and boot with the following:
> 
>     insmod gzio
>     insmod part_gpt
>     insmod btrfs
>     search --no-floppy --fs-uuid --set=root
> 23fa61d2-0b74-465a-be83-e026ebb373bc
>     linux /vmlinuz-3.16.0-23-generic
> root=UUID=aa67f7ba-08cc-4760-aa30-fa5bc10b88cc rootflags=noatime,subvol=@
>     initrd /initrd.img-3.16.0-23-generic
> 
> ----
> ---- End of steps to reproduce
> ----
> 
> I just did this and it booted successfully 15 times in a row.
> 
> Thanks,
> Jay

> _______________________________________________
> dm-crypt mailing list
> dm-crypt@saout.de
> http://www.saout.de/mailman/listinfo/dm-crypt


-- 
Arno Wagner,     Dr. sc. techn., Dipl. Inform.,    Email: arno@wagner.name
GnuPG: ID: CB5D9718  FP: 12D6 C03B 1B30 33BB 13CF  B774 E35C 5FA1 CB5D 9718
----
A good decision is based on knowledge and not on numbers. -- Plato

If it's in the news, don't worry about it.  The very definition of 
"news" is "something that hardly ever happens." -- Bruce Schneier