Unclean shutdowns cause google-chrome profile to be corrupted in various ways

Unclean shutdowns cause google-chrome profile to be corrupted in various ways

[Marc MERLIN]

Someone just told me yesterday they had the same problem, so I filed a
bug:
https://bugzilla.kernel.org/show_bug.cgi?id=83041

Fairly often (over 20 times for me so far with various kernel versions),
when I reboot after a crash, my google-chrome profile is damaged in one
of 2 ways:
1) open tabs don't reopen
2) google-chrome says that my profile is corrupted.

In both cases rsyncing ~/.config/google-chrome from the last hourly
snapshot has fixed the problem every time.

Given that, I would say that google-chrome does not have a bug of half
unclean states since the state present in a snapshot has always worked
for me, but that's anecdotal, not proof.

But if my kernel hangs due to a bug that isn't btrfs' fault and I need
to power off and back on, after reboot my google-chrome profile is
almost always broken in some way.
This with a samsung evo 840 SSD which I believe does reasonable enough
things on power shutdowns, although I can't prove that.

File formats are some kind of raw data and sqlite 3.x
~/.config/google-chrome-beta/Profile 1:
Last Session:                     data
Last Tabs:                        data
Login Data:                       SQLite 3.x database

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: Unclean shutdowns cause google-chrome profile to be corrupted in various ways

[Eric Sandeen]

On 8/22/14, 10:50 AM, Marc MERLIN wrote:

> But if my kernel hangs due to a bug that isn't btrfs' fault and I need
> to power off and back on, after reboot my google-chrome profile is
> almost always broken in some way.

Given my experiences with userspace in general, I'd lay money on
google-chrome simply not DTRT WRT data integrity syscalls (or
maybe it's the sqlite database handling underneath...)

Does it behave any differently on other filesystems?

-Eric

Re: Unclean shutdowns cause google-chrome profile to be corrupted in various ways

[Duncan]

Marc MERLIN posted on Fri, 22 Aug 2014 08:50:40 -0700 as excerpted:

> Fairly often (over 20 times for me so far with various kernel versions),
> when I reboot after a crash, my google-chrome profile is damaged in one
> of 2 ways:
> 1) open tabs don't reopen 2) google-chrome says that my profile is
> corrupted.
> 
> In both cases rsyncing ~/.config/google-chrome from the last hourly
> snapshot has fixed the problem every time.

I've had a similar issue with firefox, tho I've narrowed it down to a 
single file, IIRC preferences.js, which firefox keeps plenty of backups 
of (I see seven, going back quite some way), and recovering it from the 
last firefox backup or from my own backups has worked the three-ish times 
it happened here.  But it was a bit disconcerting to see firefox pop up 
its new profile wizard the first time it happened, and it took me awhile 
to figure out which file I needed to restore to get my skins and 
extensions back to normal.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Unclean shutdowns cause google-chrome profile to be corrupted in various ways

[Marc MERLIN]

On Fri, Aug 22, 2014 at 12:32:27PM -0500, Eric Sandeen wrote:
> On 8/22/14, 10:50 AM, Marc MERLIN wrote:
> 
> > But if my kernel hangs due to a bug that isn't btrfs' fault and I need
> > to power off and back on, after reboot my google-chrome profile is
> > almost always broken in some way.
> 
> Given my experiences with userspace in general, I'd lay money on
> google-chrome simply not DTRT WRT data integrity syscalls (or
> maybe it's the sqlite database handling underneath...)
> 
> Does it behave any differently on other filesystems?

Well
1) I used to work on chromeos until some months back, and haven't heard
on this problem with chrome/linux
2) never had a chrome corruption problem on a non btrfs filesystem
3) as mentioned, my btrfs snapshots are somehow always consistent for
chrome, which if chrome were not flushing its buffers right, wouldn't
happen.

Incidently, I just had to rebuild my mysqldb on my laptop after it got
corrupted in a way that was non recoverable.
Mysql surely knows how to flush and have consistent state points.

In my experience, linux crashes cause btrfs to somehow end up with non
consistent states on disk. Whether it's a bug in the btrfs code, the
linux block subsystem or sata driver, or firmware bugs on all 5 SSDs
I've had so far, I can't say, but there is a problem somewhere for sure.

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: Unclean shutdowns cause google-chrome profile to be corrupted in various ways

[Marc MERLIN]

On Fri, Aug 22, 2014 at 06:17:38PM +0000, Duncan wrote:
> Marc MERLIN posted on Fri, 22 Aug 2014 08:50:40 -0700 as excerpted:
> 
> > Fairly often (over 20 times for me so far with various kernel versions),
> > when I reboot after a crash, my google-chrome profile is damaged in one
> > of 2 ways:
> > 1) open tabs don't reopen 2) google-chrome says that my profile is
> > corrupted.
> > 
> > In both cases rsyncing ~/.config/google-chrome from the last hourly
> > snapshot has fixed the problem every time.
> 
> I've had a similar issue with firefox, tho I've narrowed it down to a 

Ok, so so far we've had:
1) gogole-chrome
2) firefox
3) mysql

and 3 different people reporting this at least.

Google-chrome is complicated because it has many state files and I
haven't narrowed down which one got corrupted.
In your firefox example, did you find what corruption you got in that
file, or was it just truncated?

For mysql, I got:
InnoDB: Page directory corruption: infimum not pointed to
140708 11:53:58  InnoDB: Page dump in ascii and hex (16384 bytes):
 len 16384; hex 00000000(16KB of 0's).

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: Unclean shutdowns cause google-chrome profile to be corrupted in various ways

[Duncan]

Marc MERLIN posted on Fri, 22 Aug 2014 11:49:19 -0700 as excerpted:

> On Fri, Aug 22, 2014 at 06:17:38PM +0000, Duncan wrote:
>> Marc MERLIN posted on Fri, 22 Aug 2014 08:50:40 -0700 as excerpted:
>> 
>> > Fairly often (over 20 times for me so far with various kernel
>> > versions), when I reboot after a crash, my google-chrome profile
>> > is damaged in one of 2 ways:
>> > 1) open tabs don't reopen
>> > 2) google-chrome says that my profile is corrupted.
>> > 
>> > In both cases rsyncing ~/.config/google-chrome from the last hourly
>> > snapshot has fixed the problem every time.
>> 
>> I've had a similar issue with firefox, tho I've narrowed it down to a
> 
> Ok, so so far we've had:
> 1) gogole-chrome 2) firefox 3) mysql
> 
> and 3 different people reporting this at least.
> 
> Google-chrome is complicated because it has many state files and I
> haven't narrowed down which one got corrupted.
> In your firefox example, did you find what corruption you got in that
> file, or was it just truncated?

Unfortunately I can't say for sure.  I only notice it when I launch 
firefox the next time and it comes up skinless, with intro dialogs from a 
some of the extensions (which are still loaded from other files, just not 
configured).  By that time it has rewritten the file in question, *MUCH* 
smaller than my fully configured version, apparently rewriting the firefox 
defaults and possibly some extension defaults, but without any of the 
custom configuration.

I don't know if the file disappears entirely (missing directory entry), 
or whether it gets truncated to zero-size, or whether it's there but 
corrupted, but whatever the case, firefox obviously decides it can' work 
with it and restores the settings in that file to default (not restoring 
from the last backup it made tho I can do that manually, but to defaults 
for that file).

And firefox has already overwritten it with the defaults version by the 
time I notice it after launching firefox and seeing the defaults.

BTW, I just looked at my profile and it's for sure prefs.js, with 
firefox's automated backups being prefs-1.js, prefs-2.js ... prefs-7.js, 
the latter being the latest.  An educated guess is that firefox makes a 
new numbered backup every time it upgrades, which would mean those are 
the backups going back seven versions before current (v31).  Or maybe 
only when it changes something big and thus those seven backups go back 
to v4 or whatever.

> For mysql, I got:
> InnoDB: Page directory corruption: 
> infimum not pointed to 140708 11:53:58
> InnoDB: Page dump in ascii and hex (16384 bytes):
> len 16384; hex 00000000(16KB of 0's).

Is that on ssd or spinning rust, and if ssd, do you run with trim/discard 
and/or have you filled the device yet if not (since mkfs.btrfs trims the 
device as part of the process)?  I'm wondering if that's 4 4 KiB btrfs 
data blocks of trimmed and unwritten SSD?

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Unclean shutdowns cause google-chrome profile to be corrupted in various ways

[Marc MERLIN]

On Sat, Aug 23, 2014 at 02:52:16AM +0000, Duncan wrote:
> > For mysql, I got:
> > InnoDB: Page directory corruption: 
> > infimum not pointed to 140708 11:53:58
> > InnoDB: Page dump in ascii and hex (16384 bytes):
> > len 16384; hex 00000000(16KB of 0's).
> 
> Is that on ssd or spinning rust, and if ssd, do you run with trim/discard 
> and/or have you filled the device yet if not (since mkfs.btrfs trims the 
> device as part of the process)?  I'm wondering if that's 4 4 KiB btrfs 
> data blocks of trimmed and unwritten SSD?

It's on SSD, I do have trim/discard, I never filled the device.

But I could totally remove trim and see what happens. I'll do that.

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: Unclean shutdowns cause google-chrome profile to be corrupted in various ways

[Naohiro Aota]

On Sat, Aug 23, 2014 at 12:10 PM, Marc MERLIN <marc@merlins.org> wrote:
> On Sat, Aug 23, 2014 at 02:52:16AM +0000, Duncan wrote:
>> > For mysql, I got:
>> > InnoDB: Page directory corruption:
>> > infimum not pointed to 140708 11:53:58
>> > InnoDB: Page dump in ascii and hex (16384 bytes):
>> > len 16384; hex 00000000(16KB of 0's).
>>
>> Is that on ssd or spinning rust, and if ssd, do you run with trim/discard
>> and/or have you filled the device yet if not (since mkfs.btrfs trims the
>> device as part of the process)?  I'm wondering if that's 4 4 KiB btrfs
>> data blocks of trimmed and unwritten SSD?
>
> It's on SSD, I do have trim/discard, I never filled the device.
>
> But I could totally remove trim and see what happens. I'll do that.

I'm experiencing the same google-chrome profile corruption on my HDD too.
It almost always happen to me when the power got lost or kernel get panic.

Naohiro

Re: Unclean shutdowns cause google-chrome profile to be corrupted in various ways

[Duncan]

Marc MERLIN posted on Fri, 22 Aug 2014 20:10:55 -0700 as excerpted:

> On Sat, Aug 23, 2014 at 02:52:16AM +0000, Duncan wrote:
>> > For mysql, I got:
>> > InnoDB: Page directory corruption:
>> > infimum not pointed to 140708 11:53:58 InnoDB: Page dump in ascii and
>> > hex (16384 bytes):
>> > len 16384; hex 00000000(16KB of 0's).
>> 
>> Is that on ssd or spinning rust, and if ssd, do you run with
>> trim/discard and/or have you filled the device yet if not (since
>> mkfs.btrfs trims the device as part of the process)?  I'm wondering if
>> that's 4 4 KiB btrfs data blocks of trimmed and unwritten SSD?
> 
> It's on SSD, I do have trim/discard, I never filled the device.
> 
> But I could totally remove trim and see what happens. I'll do that.

You'd probably have to mostly fill up the device with garbage and then 
delete it (with discard off) before it's return anything but already 
trimmed/zeroed blocks, since if I'm correct it's pre-allocating a bunch 
but then not actually writing it before the crash.  It'd be pre-
allocating from what it thought was free space, so as long as most of 
that free space hasn't been written at all since it was trimmed, you'd 
still likely get zeros even after turning trim off.  Only after you have 
written something to that space and then deleted it, would the chance of 
coming up "dirty" increase dramatically.

That is of course assuming the pre-allocation doesn't pre-zero as well, 
which it might.

It just struck me that with trim on, a bunch of zero-blocks is what you'd 
expect from free-space, which is what a COW filesystem would be 
allocating from when there's a write into a database file like that 
(assuming it's not set NOCOW).  On spinning rust or without trim/discard 
set, unzeroed garbage would accumulate in the free space over time, and a 
full 16 KiB of zeros would be far more interesting, as that would mean 
something's actually zeroing it but that mysql isn't getting data written 
back to it after the zeroing, before the crash.

Of course that begs the question of whether it was a normal COW file or 
if you had it NOCOW.  Setting it NOCOW (of course doing the correct set 
the directory NOCOW, copy the file into it dance, so it's NOCOW from the 
beginning) could be interesting too, and may in fact actually eliminate 
the problem depending on how mysql handles such things.  Presumably it 
has some sort of database resiliency scheme as most filesystems don't do 
the checksumming that btrfs does so it can't rely on that, and my 
argument has always been that in some cases it might actually be better 
to let the database handle it how it normally does with ordinary 
filesystems and not try to get in the way, which is what NOCOW basically 
does, tell btrfs to let the application handle that file and not to 
interfere.  It'd be interesting to see how well my hypothesis holds up, 
anyway.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Unclean shutdowns cause google-chrome profile to be corrupted in various ways

[Marc MERLIN]

On Sat, Aug 23, 2014 at 02:45:25PM +0900, Naohiro Aota wrote:
> On Sat, Aug 23, 2014 at 12:10 PM, Marc MERLIN <marc@merlins.org> wrote:
> > On Sat, Aug 23, 2014 at 02:52:16AM +0000, Duncan wrote:
> >> > For mysql, I got:
> >> > InnoDB: Page directory corruption:
> >> > infimum not pointed to 140708 11:53:58
> >> > InnoDB: Page dump in ascii and hex (16384 bytes):
> >> > len 16384; hex 00000000(16KB of 0's).
> >>
> >> Is that on ssd or spinning rust, and if ssd, do you run with trim/discard
> >> and/or have you filled the device yet if not (since mkfs.btrfs trims the
> >> device as part of the process)?  I'm wondering if that's 4 4 KiB btrfs
> >> data blocks of trimmed and unwritten SSD?
> >
> > It's on SSD, I do have trim/discard, I never filled the device.
> >
> > But I could totally remove trim and see what happens. I'll do that.
> 
> I'm experiencing the same google-chrome profile corruption on my HDD too.
> It almost always happen to me when the power got lost or kernel get panic.

Thanks for that data point, so that rules out discard and SSDs.

Given that, it sounds like btrfs may still have a bug where everything
does not get to disk in the right order before the system stops.

Now, it is possible that google-chrome has a bug where it doesn't fsync
or end up in consistent points. It's also possible btrfs is just not
able to get all that fsync data to disk before the system crashes, and
the inconsistent state is not its fault.
But I'm a bit perplexed by the 16KB of 0's that ended up in the middle
of my mysql database.

What's interesting though is that Chris told me he never go the profile
issue with google-chrome, but he also said his laptop almost never
crashes, so they're most likely related.

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: Unclean shutdowns cause google-chrome profile to be corrupted in various ways

[Marc MERLIN]

On Sat, Aug 23, 2014 at 05:56:28AM +0000, Duncan wrote:
> Of course that begs the question of whether it was a normal COW file or 
> if you had it NOCOW.  Setting it NOCOW (of course doing the correct set 

I had it at the default of COW, both chrome and mysql.

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: Unclean shutdowns cause google-chrome profile to be corrupted in various ways

[Holger Hoffstätte]

On Sat, 23 Aug 2014 05:32:49 -0700, Marc MERLIN wrote:

> [snip]
> Thanks for that data point, so that rules out discard and SSDs.
> 
> Given that, it sounds like btrfs may still have a bug where everything
> does not get to disk in the right order before the system stops.

A popular candidate for this is mmap'ed IO (almost always a bad idea, 
certainly for writes). This almost always goes wrong one way or the 
other, depending on filesystem + block/VFS/mm bugs of the day.

-h