bcache and hibernation

bcache and hibernation

[Mathijs Kwik]

Hi all,

Today, I lost most my data (don't worry, got backups) after the cache
got corrupted somehow. I suspected a recent suspend-to-disk to be the
cause. I checked how my distribution (NixOS) handles suspend/resume and
I have some concerns about how bcache fits into this.

Normally, the kernel and initrd get loaded. The initrd loads required
kernel modules, waits for udev to settle, activates luks&lvm, then
finally asks the kernel to resume from the resume device.

The kernel documentation on suspend is VERY clear you should NOT touch
anything on disk between suspend and resume. So activating luks and LVM
is probably risky already, but it apppears both luks and LVM do not make
any on-disk changes when activated and any in-memory state (within the
resumed image) is still valid. The benefit of activating luks and LVM
before resume seems to be that it allows resuming from encrypted/lvm
volumes. 

Now, with bcache added, things probably get a bit hairy. NixOS supports
bcache inside the initrd and uses udev rules to activate/attach. I
suspect this is probably unsafe. Probably bcache starts to see if any
dirty pages exist, to write them to the backing store. Even without
writeback caching, the activation of lvm will read some sectors, which
might trigger the cache to update. Then after resuming the image, the
in-memory state is corrupted and further damage occurs. 

- Does this sound plausible? 
- Is there any way to tell bcache to make absolutely no changes to
  either the backing device or the cache?
  Basically like a readaround+writearound which can be triggered on
  hibernate and switched off on resume.

Thanks,
Mathijs

Re: bcache and hibernation

[Mathijs Kwik]

As a followup question:

Is there a safe way to use hiberation in the presence of bcache?

Assuming a (initrd) boot procedure should at least wait for the
hard-disk to settle, this will mean some udev rules will fire...
With the bcache udev rules in place, this means activating/attaching a
cache+backing device. Am I assuming correctly that this might
immediately lead to on-disk changes (writeback dirty pages for example)?

So what's the way to go?
Make sure the bcache module doesn't get loaded when resuming from
hibernation? Tell udev not to activate the device automatically until we
check for a resume-image?





Mathijs Kwik <mathijs@bluescreen303.nl> writes:

> Hi all,
>
> Today, I lost most my data (don't worry, got backups) after the cache
> got corrupted somehow. I suspected a recent suspend-to-disk to be the
> cause. I checked how my distribution (NixOS) handles suspend/resume and
> I have some concerns about how bcache fits into this.
>
> Normally, the kernel and initrd get loaded. The initrd loads required
> kernel modules, waits for udev to settle, activates luks&lvm, then
> finally asks the kernel to resume from the resume device.
>
> The kernel documentation on suspend is VERY clear you should NOT touch
> anything on disk between suspend and resume. So activating luks and LVM
> is probably risky already, but it apppears both luks and LVM do not make
> any on-disk changes when activated and any in-memory state (within the
> resumed image) is still valid. The benefit of activating luks and LVM
> before resume seems to be that it allows resuming from encrypted/lvm
> volumes. 
>
> Now, with bcache added, things probably get a bit hairy. NixOS supports
> bcache inside the initrd and uses udev rules to activate/attach. I
> suspect this is probably unsafe. Probably bcache starts to see if any
> dirty pages exist, to write them to the backing store. Even without
> writeback caching, the activation of lvm will read some sectors, which
> might trigger the cache to update. Then after resuming the image, the
> in-memory state is corrupted and further damage occurs. 
>
> - Does this sound plausible? 
> - Is there any way to tell bcache to make absolutely no changes to
>   either the backing device or the cache?
>   Basically like a readaround+writearound which can be triggered on
>   hibernate and switched off on resume.
>
> Thanks,
> Mathijs

Re: bcache and hibernation

[Mathijs Kwik]

Josep Lladonosa <jlladono@gmail.com> writes:

> On 13 November 2014 16:52, Mathijs Kwik <mathijs@bluescreen303.nl> wrote:
>
>> As a followup question:
>>
>> Is there a safe way to use hiberation in the presence of bcache?
>>
>
> Hello,
>
> I am using (just now) a laptop with root filesystem on a bcache partition,
> a separate boot partition (non bcache, small) and a swap partition used for
> hibernation and it works perfectly.

I'm glad to hear that. So how does your system boot/resume?
Does the initrd postpone registering/activating bcache until after the
resume-check? Or are my assumptions just wrong?


>
> Josep
>
>
>>
>> Assuming a (initrd) boot procedure should at least wait for the
>> hard-disk to settle, this will mean some udev rules will fire...
>> With the bcache udev rules in place, this means activating/attaching a
>> cache+backing device. Am I assuming correctly that this might
>> immediately lead to on-disk changes (writeback dirty pages for example)?
>>
>> So what's the way to go?
>> Make sure the bcache module doesn't get loaded when resuming from
>> hibernation? Tell udev not to activate the device automatically until we
>> check for a resume-image?
>>
>>
>>
>>
>>
>> Mathijs Kwik <mathijs@bluescreen303.nl> writes:
>>
>> > Hi all,
>> >
>> > Today, I lost most my data (don't worry, got backups) after the cache
>> > got corrupted somehow. I suspected a recent suspend-to-disk to be the
>> > cause. I checked how my distribution (NixOS) handles suspend/resume and
>> > I have some concerns about how bcache fits into this.
>> >
>> > Normally, the kernel and initrd get loaded. The initrd loads required
>> > kernel modules, waits for udev to settle, activates luks&lvm, then
>> > finally asks the kernel to resume from the resume device.
>> >
>> > The kernel documentation on suspend is VERY clear you should NOT touch
>> > anything on disk between suspend and resume. So activating luks and LVM
>> > is probably risky already, but it apppears both luks and LVM do not make
>> > any on-disk changes when activated and any in-memory state (within the
>> > resumed image) is still valid. The benefit of activating luks and LVM
>> > before resume seems to be that it allows resuming from encrypted/lvm
>> > volumes.
>> >
>> > Now, with bcache added, things probably get a bit hairy. NixOS supports
>> > bcache inside the initrd and uses udev rules to activate/attach. I
>> > suspect this is probably unsafe. Probably bcache starts to see if any
>> > dirty pages exist, to write them to the backing store. Even without
>> > writeback caching, the activation of lvm will read some sectors, which
>> > might trigger the cache to update. Then after resuming the image, the
>> > in-memory state is corrupted and further damage occurs.
>> >
>> > - Does this sound plausible?
>> > - Is there any way to tell bcache to make absolutely no changes to
>> >   either the backing device or the cache?
>> >   Basically like a readaround+writearound which can be triggered on
>> >   hibernate and switched off on resume.
>> >
>> > Thanks,
>> > Mathijs
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>

Re: bcache and hibernation

[Mathijs Kwik]

Josep Lladonosa <jlladono@gmail.com> writes:

> On 13 Nov 2014 17:52, "Mathijs Kwik" <mathijs@bluescreen303.nl> wrote:
>>
>> Josep Lladonosa <jlladono@gmail.com> writes:
>>
>> > On 13 November 2014 16:52, Mathijs Kwik <mathijs@bluescreen303.nl>
> wrote:
>> >
>> >> As a followup question:
>> >>
>> >> Is there a safe way to use hiberation in the presence of bcache?
>> >>
>> >
>> > Hello,
>> >
>> > I am using (just now) a laptop with root filesystem on a bcache
> partition,
>> > a separate boot partition (non bcache, small) and a swap partition used
> for
>> > hibernation and it works perfectly.
>>
>> I'm glad to hear that. So how does your system boot/resume?
>> Does the initrd postpone registering/activating bcache until after the
>> resume-check? Or are my assumptions just wrong?
>
> I do not know. Worked without any special setup.

Well that's nice, but I would like to know for sure this is guaranteed
to be safe :)

>
> System boots as usual, but during kernel boot it finds a hibernation in
> swap area and just restores it.

I understand how suspend works. My question is regarding the steps
_before_ finding the hibernation image. My guess is there are situations
where detecting/activating a bcache device before restoring the
hibernation image can lead to disaster.

So either this is not the case (I would like an explanation why), or I
need a way to make sure udev does not register bcache before resuming.




>
>>
>>
>> >
>> > Josep
>> >
>> >
>> >>
>> >> Assuming a (initrd) boot procedure should at least wait for the
>> >> hard-disk to settle, this will mean some udev rules will fire...
>> >> With the bcache udev rules in place, this means activating/attaching a
>> >> cache+backing device. Am I assuming correctly that this might
>> >> immediately lead to on-disk changes (writeback dirty pages for
> example)?
>> >>
>> >> So what's the way to go?
>> >> Make sure the bcache module doesn't get loaded when resuming from
>> >> hibernation? Tell udev not to activate the device automatically until
> we
>> >> check for a resume-image?
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> Mathijs Kwik <mathijs@bluescreen303.nl> writes:
>> >>
>> >> > Hi all,
>> >> >
>> >> > Today, I lost most my data (don't worry, got backups) after the cache
>> >> > got corrupted somehow. I suspected a recent suspend-to-disk to be the
>> >> > cause. I checked how my distribution (NixOS) handles suspend/resume
> and
>> >> > I have some concerns about how bcache fits into this.
>> >> >
>> >> > Normally, the kernel and initrd get loaded. The initrd loads required
>> >> > kernel modules, waits for udev to settle, activates luks&lvm, then
>> >> > finally asks the kernel to resume from the resume device.
>> >> >
>> >> > The kernel documentation on suspend is VERY clear you should NOT
> touch
>> >> > anything on disk between suspend and resume. So activating luks and
> LVM
>> >> > is probably risky already, but it apppears both luks and LVM do not
> make
>> >> > any on-disk changes when activated and any in-memory state (within
> the
>> >> > resumed image) is still valid. The benefit of activating luks and LVM
>> >> > before resume seems to be that it allows resuming from encrypted/lvm
>> >> > volumes.
>> >> >
>> >> > Now, with bcache added, things probably get a bit hairy. NixOS
> supports
>> >> > bcache inside the initrd and uses udev rules to activate/attach. I
>> >> > suspect this is probably unsafe. Probably bcache starts to see if any
>> >> > dirty pages exist, to write them to the backing store. Even without
>> >> > writeback caching, the activation of lvm will read some sectors,
> which
>> >> > might trigger the cache to update. Then after resuming the image, the
>> >> > in-memory state is corrupted and further damage occurs.
>> >> >
>> >> > - Does this sound plausible?
>> >> > - Is there any way to tell bcache to make absolutely no changes to
>> >> >   either the backing device or the cache?
>> >> >   Basically like a readaround+writearound which can be triggered on
>> >> >   hibernate and switched off on resume.
>> >> >
>> >> > Thanks,
>> >> > Mathijs
>> >> --
>> >> To unsubscribe from this list: send the line "unsubscribe
> linux-bcache" in
>> >> the body of a message to majordomo@vger.kernel.org
>> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> >>

Re: bcache and hibernation

[Kent Overstreet]

On Thu, Nov 13, 2014 at 02:52:02PM +0100, Mathijs Kwik wrote:
> Hi all,
> 
> Today, I lost most my data (don't worry, got backups) after the cache
> got corrupted somehow. I suspected a recent suspend-to-disk to be the
> cause. I checked how my distribution (NixOS) handles suspend/resume and
> I have some concerns about how bcache fits into this.

Augh :(

> Normally, the kernel and initrd get loaded. The initrd loads required
> kernel modules, waits for udev to settle, activates luks&lvm, then
> finally asks the kernel to resume from the resume device.
> 
> The kernel documentation on suspend is VERY clear you should NOT touch
> anything on disk between suspend and resume. So activating luks and LVM
> is probably risky already, but it apppears both luks and LVM do not make
> any on-disk changes when activated and any in-memory state (within the
> resumed image) is still valid. The benefit of activating luks and LVM
> before resume seems to be that it allows resuming from encrypted/lvm
> volumes. 

Yeah, this is handled for in kernel stuff with the freezing mechanism, which
bcache uses.

> Now, with bcache added, things probably get a bit hairy. NixOS supports
> bcache inside the initrd and uses udev rules to activate/attach. I
> suspect this is probably unsafe. Probably bcache starts to see if any
> dirty pages exist, to write them to the backing store. Even without
> writeback caching, the activation of lvm will read some sectors, which
> might trigger the cache to update. Then after resuming the image, the
> in-memory state is corrupted and further damage occurs. 
> 
> - Does this sound plausible? 
> - Is there any way to tell bcache to make absolutely no changes to
>   either the backing device or the cache?
>   Basically like a readaround+writearound which can be triggered on
>   hibernate and switched off on resume.

So, userspace shouldn't have to do anything to tell bcache about hibernation.

The dev branch is getting a true read only mode (still in progress), but this
isn't relevant to hibernation.

bcache kernel threads (allocator thread, gc thread) should be correct w.r.t.
hibernation, but - maybe the workqueue usage isn't.

I'm probably not going to be able to get to this in the next couple days, but
this is a pretty serious issue. Can you ping me again every couple days until I
get a fix out for this, and myabe file a bug somewhere? (i think
bugzilla.kernel.org has been used for bcache bugs before...)

Re: bcache and hibernation

[Mathijs Kwik]

Kent Overstreet <kmo@daterainc.com> writes:

> On Thu, Nov 13, 2014 at 02:52:02PM +0100, Mathijs Kwik wrote:
>> The kernel documentation on suspend is VERY clear you should NOT touch
>> anything on disk between suspend and resume. So activating luks and LVM
>> is probably risky already, but it apppears both luks and LVM do not make
>> any on-disk changes when activated and any in-memory state (within the
>> resumed image) is still valid. The benefit of activating luks and LVM
>> before resume seems to be that it allows resuming from encrypted/lvm
>> volumes. 
>
> Yeah, this is handled for in kernel stuff with the freezing mechanism, which
> bcache uses.
>
>> 
>
> So, userspace shouldn't have to do anything to tell bcache about
> hibernation.

I understand bcache knows when the system hibernates so it can do some
bookkeeping/flushing. What I don't get, is how this will protect the
system in the short phase before resume, when my initrd activates bcache
and lvm, _before_ it checks for a resume image.

I guess bcache will just start running as usual. Maybe flushing some
dirty buckets, cache some new stuff (when lvm searches for volumes and
udev reads labels & more).

Then finally the resume mechanism loads the old hibernated system, while
the cache has changed in the mean time! Won't this cause issues?


>
> The dev branch is getting a true read only mode (still in progress), but this
> isn't relevant to hibernation.
>
> bcache kernel threads (allocator thread, gc thread) should be correct w.r.t.
> hibernation, but - maybe the workqueue usage isn't.
>
> I'm probably not going to be able to get to this in the next couple days, but
> this is a pretty serious issue. Can you ping me again every couple days until I
> get a fix out for this, and myabe file a bug somewhere? (i think
> bugzilla.kernel.org has been used for bcache bugs before...)

I will file a report once I'm sure what the exact cause for my data-loss
was. If I understand you correctly, all should be safe and well, no
matter what (initramfs) userspace does in between and the only thing
that might not be safe is the workqueue?

Re: bcache and hibernation

[Kent Overstreet]

On Sun, Nov 30, 2014 at 07:25:03PM +0100, Mathijs Kwik wrote:
> Kent Overstreet <kmo@daterainc.com> writes:
> 
> > On Thu, Nov 13, 2014 at 02:52:02PM +0100, Mathijs Kwik wrote:
> >> The kernel documentation on suspend is VERY clear you should NOT touch
> >> anything on disk between suspend and resume. So activating luks and LVM
> >> is probably risky already, but it apppears both luks and LVM do not make
> >> any on-disk changes when activated and any in-memory state (within the
> >> resumed image) is still valid. The benefit of activating luks and LVM
> >> before resume seems to be that it allows resuming from encrypted/lvm
> >> volumes. 
> >
> > Yeah, this is handled for in kernel stuff with the freezing mechanism, which
> > bcache uses.
> >
> >> 
> >
> > So, userspace shouldn't have to do anything to tell bcache about
> > hibernation.
> 
> I understand bcache knows when the system hibernates so it can do some
> bookkeeping/flushing. What I don't get, is how this will protect the
> system in the short phase before resume, when my initrd activates bcache
> and lvm, _before_ it checks for a resume image.

Wait what? Ohh... you must be talking about hibernate, not suspend...

> I guess bcache will just start running as usual. Maybe flushing some
> dirty buckets, cache some new stuff (when lvm searches for volumes and
> udev reads labels & more).
> 
> Then finally the resume mechanism loads the old hibernated system, while
> the cache has changed in the mean time! Won't this cause issues?

I have no idea how hibernate works, but yeah, quite possibly...

> > The dev branch is getting a true read only mode (still in progress), but this
> > isn't relevant to hibernation.
> >
> > bcache kernel threads (allocator thread, gc thread) should be correct w.r.t.
> > hibernation, but - maybe the workqueue usage isn't.
> >
> > I'm probably not going to be able to get to this in the next couple days, but
> > this is a pretty serious issue. Can you ping me again every couple days until I
> > get a fix out for this, and myabe file a bug somewhere? (i think
> > bugzilla.kernel.org has been used for bcache bugs before...)
> 
> I will file a report once I'm sure what the exact cause for my data-loss
> was. If I understand you correctly, all should be safe and well, no
> matter what (initramfs) userspace does in between and the only thing
> that might not be safe is the workqueue?

I was thinking suspend to ram, not hibernate. gonna have to look into how
hibernate works now.

Re: bcache and hibernation

[Kent Overstreet]

On Sun, Nov 30, 2014 at 07:25:03PM +0100, Mathijs Kwik wrote:
> Kent Overstreet <kmo@daterainc.com> writes:
> 
> > On Thu, Nov 13, 2014 at 02:52:02PM +0100, Mathijs Kwik wrote:
> >> The kernel documentation on suspend is VERY clear you should NOT touch
> >> anything on disk between suspend and resume. So activating luks and LVM
> >> is probably risky already, but it apppears both luks and LVM do not make
> >> any on-disk changes when activated and any in-memory state (within the
> >> resumed image) is still valid. The benefit of activating luks and LVM
> >> before resume seems to be that it allows resuming from encrypted/lvm
> >> volumes. 
> >
> > Yeah, this is handled for in kernel stuff with the freezing mechanism, which
> > bcache uses.
> >
> >> 
> >
> > So, userspace shouldn't have to do anything to tell bcache about
> > hibernation.
> 
> I understand bcache knows when the system hibernates so it can do some
> bookkeeping/flushing. What I don't get, is how this will protect the
> system in the short phase before resume, when my initrd activates bcache
> and lvm, _before_ it checks for a resume image.
> 
> I guess bcache will just start running as usual. Maybe flushing some
> dirty buckets, cache some new stuff (when lvm searches for volumes and
> udev reads labels & more).
> 
> Then finally the resume mechanism loads the old hibernated system, while
> the cache has changed in the mean time! Won't this cause issues?

BTW - it sounds like you're ahead of me on how this is put together - could you
point me at the userspace side of hibernate that you're using (those initramfs
scripts, and in particular whatever device mapper does)? that'll help a lot.

Re: bcache and hibernation

[Mathijs Kwik]

Kent Overstreet <kmo@daterainc.com> writes:
>
> BTW - it sounds like you're ahead of me on how this is put together - could you
> point me at the userspace side of hibernate that you're using (those initramfs
> scripts, and in particular whatever device mapper does)? that'll help a lot.

Please see
https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/system/boot/stage-1-init.sh 
lines 128 onward. I'm just using the in-kernel suspend/hibernate
functionality (swsusp) but the same probably applies to tuxonice and
other solutions as well.

As far as I understand the events leading up to hibernation are very
similar to suspend. The kernel will notify processes and kernel threads
they will be frozen. Then, when everything has prepared for suspension,
instead of just putting the system to suspend, the kernel will write the
full contents of the system RAM to a swap device. 

I'm pretty sure that's all still OK and things are in a consistent
state. However, when resuming the system, some basic initialization is
performed in initrd. At least SCSI/SATA controller modules and other
stuff needed to find the hibernated RAM image are needed of course, but
most distributions will include more stuff and use udev to find hardware
and load the appropriate modules.

This is where things might get nasty though. On nixos, we normally
initialize bcache using udev rules. Modern versions of udev will do a
quick scan of block devices when found to find their labels/types. So
while waiting for stuff like disks/usb/whatever to appear, my bcache
partitions get found and activated. I'm pretty sure bcache will take
over from here and do some bookkeeping / flush dirty buckets, whatever.
Even without writeback, things might change on-disk: udev and tools like
vgscan (lvm) and "btrfs scan" might probe some magic sectors inside the
newly-activated bcache device. If those aren't in the cache, they will
be put there, once again changing the on-disk state.

Then finally (line 190) the kernel gets instructed to check if the swap
device contains a hibernated RAM image and restore that. For everything
running "inside the RAM image", it's just like waking up from a normal
suspend.

From this explanation, it should be clear that it is vital that no
on-disk state is changed in the brief period that the initrd is setting
up the system, or bcache's in-memory state (inside the resumed RAM
image) will be corrupted, probably leading to disasters. Either that, or
bcache should assume nothing on resume and make sure to reassemble its
entire in-memory state from disk.

The temporary solution I found was to not include the udev rules in the
initrd so it will not get found and activated before resume. Then for
normal booting (I have my root on bcache) I manually load and activate
bcache _after_ seeing there is no resume image. However, this solution
is ugly, because I need to repeat all other initialization steps
(lvm/btrfs) afterwards. 

Hope this helps,
Mathijs

Re: bcache and hibernation

[Kai Krakow]

Mathijs Kwik <mathijs@bluescreen303.nl> schrieb:

>> System boots as usual, but during kernel boot it finds a hibernation in
>> swap area and just restores it.
> 
> I understand how suspend works. My question is regarding the steps
> _before_ finding the hibernation image. My guess is there are situations
> where detecting/activating a bcache device before restoring the
> hibernation image can lead to disaster.
> 
> So either this is not the case (I would like an explanation why), or I
> need a way to make sure udev does not register bcache before resuming.

Are you sure that user space runs before the resume image is loaded? I know 
that initrd could do that but as far as I see, if you supply the resume 
option to the kernel cmdline, the kernel will take care of resuming before 
initrd userspace does it - which should pretty well circumvent your problem.

BTW: I'm using the resume= option and had no problem with hibernation. It 
seems to work fine.

-- 
Replies to list only preferred.