Bug report - f2fs corruption with 4.14 kernel

Bug report - f2fs corruption with 4.14 kernel

[Marcelo "Marc" Ranolfi]

Dear developers,


I've been using f2fs for my root fs since the 4.9 kernel. The
partition is encrypted with dm-crypt (LUKS).

A few weeks back, after upgrading to kernel 4.14.2, I began to
experience corruption of many files, revealed by boot-time fsck. Most
of the corrupted files were temporary, but not all.

I never got to successfully restore them, since fsck.f2fs attempts to
restore files to /.lost_found (instead of lost+found, as it'd normally
be expected), and the folder did not exist in my system.

After isolating the problem to the kernel version, and after it
affected some important system files, I simply reverted to 4.13.9 and
stood by it for a while.

Two days ago I wanted to give it another try. Even though nothing from
the changelogs seemed to give me hope, I upgraded from 4.13.9 to
4.14.7 and rebooted. And rebooted again. No apparent issue, at that
moment.

But then, after a few hours of work, my applications started to
malfunction. First qemu/libvirt (not sure which); I restarted my VM
but it crashed again after a couple minutes.
Then the music player crashed, followed by my text editor, with a
delay of about 5-10 minutes in between.

This is where I noticed something was seriously wrong.

I attempted to close my browser (Firefox) in order to restart the
computer, but it was impossible. Firefox would not close due to some
missing persistent state file(s) that should be stored in
$HOME/.mozilla, but was missing.

I proceeded to shutdown the system nevertheless, but my DE (Cinnamon)
was not accepting commands due to missing or invalid system libraries.

Hard reset. On power on, fsck once again "recovered" many lost files -
only it actually did not recover a thing, for I still hadn't created
lost_found.

After struggling for a while to get my system to a usable state, I had
to reinstall 1146 packages from the package manager. It seems I'm
alright by now.

I'm sure this is a kernel issue. Could be device mapper, I suppose,
but... I use encryption, in the same fashion, on two other volumes
formatted with ext4. I also have ext2 and ntfs filesystems without
encryption. My hardware is alright. The problem is gone by downgrading
to kernel 4.13.x.

For these reasons I'm quite sure this is an f2fs issue.


Let me know if I can provide something more useful to help in sorting this out.


Regards
Marc

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot

Re: Bug report - f2fs corruption with 4.14 kernel

[Marcelo "Marc" Ranolfi]

Hi all,

confirmed it's not device-mapper.

Also happening with kernel 4.14.8 and 4.15rc5.

The bug only happens after some 2-3 hours of uptime or more.

Yesterday it got me pretty bad; corrupted /var/lib/pacman (I use Arch;
this dir is the package manager library) so I had to restore
everything from the logs via script.


Regards
Marc

On Sun, Dec 24, 2017 at 3:35 AM, Marcelo "Marc" Ranolfi
<marc.2377@gmail.com> wrote:
> Dear developers,
>
>
> I've been using f2fs for my root fs since the 4.9 kernel. The
> partition is encrypted with dm-crypt (LUKS).
>
> A few weeks back, after upgrading to kernel 4.14.2, I began to
> experience corruption of many files, revealed by boot-time fsck. Most
> of the corrupted files were temporary, but not all.
>
> I never got to successfully restore them, since fsck.f2fs attempts to
> restore files to /.lost_found (instead of lost+found, as it'd normally
> be expected), and the folder did not exist in my system.
>
> After isolating the problem to the kernel version, and after it
> affected some important system files, I simply reverted to 4.13.9 and
> stood by it for a while.
>
> Two days ago I wanted to give it another try. Even though nothing from
> the changelogs seemed to give me hope, I upgraded from 4.13.9 to
> 4.14.7 and rebooted. And rebooted again. No apparent issue, at that
> moment.
>
> But then, after a few hours of work, my applications started to
> malfunction. First qemu/libvirt (not sure which); I restarted my VM
> but it crashed again after a couple minutes.
> Then the music player crashed, followed by my text editor, with a
> delay of about 5-10 minutes in between.
>
> This is where I noticed something was seriously wrong.
>
> I attempted to close my browser (Firefox) in order to restart the
> computer, but it was impossible. Firefox would not close due to some
> missing persistent state file(s) that should be stored in
> $HOME/.mozilla, but was missing.
>
> I proceeded to shutdown the system nevertheless, but my DE (Cinnamon)
> was not accepting commands due to missing or invalid system libraries.
>
> Hard reset. On power on, fsck once again "recovered" many lost files -
> only it actually did not recover a thing, for I still hadn't created
> lost_found.
>
> After struggling for a while to get my system to a usable state, I had
> to reinstall 1146 packages from the package manager. It seems I'm
> alright by now.
>
> I'm sure this is a kernel issue. Could be device mapper, I suppose,
> but... I use encryption, in the same fashion, on two other volumes
> formatted with ext4. I also have ext2 and ntfs filesystems without
> encryption. My hardware is alright. The problem is gone by downgrading
> to kernel 4.13.x.
>
> For these reasons I'm quite sure this is an f2fs issue.
>
>
> Let me know if I can provide something more useful to help in sorting this out.
>
>
> Regards
> Marc

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot

Re: Bug report - f2fs corruption with 4.14 kernel

[Jaegeuk Kim]

On 12/26, Marcelo "Marc" Ranolfi wrote:
> Hi all,
> 
> confirmed it's not device-mapper.
> 
> Also happening with kernel 4.14.8 and 4.15rc5.
> 
> The bug only happens after some 2-3 hours of uptime or more.
> 
> Yesterday it got me pretty bad; corrupted /var/lib/pacman (I use Arch;
> this dir is the package manager library) so I had to restore
> everything from the logs via script.

Hi Marc,

Thank you for the report. Could you please check the version of f2fs-tools?
I got a report where fsck.f2fs causes breaking the valid checpoint.
If possible, could you please try the latest tools from here?
https://git.kernel.org/pub/scm/linux/kernel/git/jaegeuk/f2fs-tools.git/log/

Thanks,

> 
> 
> Regards
> Marc
> 
> On Sun, Dec 24, 2017 at 3:35 AM, Marcelo "Marc" Ranolfi
> <marc.2377@gmail.com> wrote:
> > Dear developers,
> >
> >
> > I've been using f2fs for my root fs since the 4.9 kernel. The
> > partition is encrypted with dm-crypt (LUKS).
> >
> > A few weeks back, after upgrading to kernel 4.14.2, I began to
> > experience corruption of many files, revealed by boot-time fsck. Most
> > of the corrupted files were temporary, but not all.
> >
> > I never got to successfully restore them, since fsck.f2fs attempts to
> > restore files to /.lost_found (instead of lost+found, as it'd normally
> > be expected), and the folder did not exist in my system.
> >
> > After isolating the problem to the kernel version, and after it
> > affected some important system files, I simply reverted to 4.13.9 and
> > stood by it for a while.
> >
> > Two days ago I wanted to give it another try. Even though nothing from
> > the changelogs seemed to give me hope, I upgraded from 4.13.9 to
> > 4.14.7 and rebooted. And rebooted again. No apparent issue, at that
> > moment.
> >
> > But then, after a few hours of work, my applications started to
> > malfunction. First qemu/libvirt (not sure which); I restarted my VM
> > but it crashed again after a couple minutes.
> > Then the music player crashed, followed by my text editor, with a
> > delay of about 5-10 minutes in between.
> >
> > This is where I noticed something was seriously wrong.
> >
> > I attempted to close my browser (Firefox) in order to restart the
> > computer, but it was impossible. Firefox would not close due to some
> > missing persistent state file(s) that should be stored in
> > $HOME/.mozilla, but was missing.
> >
> > I proceeded to shutdown the system nevertheless, but my DE (Cinnamon)
> > was not accepting commands due to missing or invalid system libraries.
> >
> > Hard reset. On power on, fsck once again "recovered" many lost files -
> > only it actually did not recover a thing, for I still hadn't created
> > lost_found.
> >
> > After struggling for a while to get my system to a usable state, I had
> > to reinstall 1146 packages from the package manager. It seems I'm
> > alright by now.
> >
> > I'm sure this is a kernel issue. Could be device mapper, I suppose,
> > but... I use encryption, in the same fashion, on two other volumes
> > formatted with ext4. I also have ext2 and ntfs filesystems without
> > encryption. My hardware is alright. The problem is gone by downgrading
> > to kernel 4.13.x.
> >
> > For these reasons I'm quite sure this is an f2fs issue.
> >
> >
> > Let me know if I can provide something more useful to help in sorting this out.
> >
> >
> > Regards
> > Marc

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot

Re: Bug report - f2fs corruption with 4.14 kernel

[Chao Yu]

On 2017/12/28 2:17, Jaegeuk Kim wrote:
> On 12/26, Marcelo "Marc" Ranolfi wrote:
>> Hi all,
>>
>> confirmed it's not device-mapper.
>>
>> Also happening with kernel 4.14.8 and 4.15rc5.
>>
>> The bug only happens after some 2-3 hours of uptime or more.
>>
>> Yesterday it got me pretty bad; corrupted /var/lib/pacman (I use Arch;
>> this dir is the package manager library) so I had to restore
>> everything from the logs via script.
> 
> Hi Marc,
> 
> Thank you for the report. Could you please check the version of f2fs-tools?
> I got a report where fsck.f2fs causes breaking the valid checpoint.
> If possible, could you please try the latest tools from here?
> https://git.kernel.org/pub/scm/linux/kernel/git/jaegeuk/f2fs-tools.git/log/

Hi Marc,

Sorry for later response.

In 4.14 and 4.15rc1, we have developed many features which changed inode
disk layout a lot, and during the development, we have considered backward
compatibility in both kernel module and userland tools, so theoretically,
there should not be any problem. In order to make sure whether the issue
you encounter is a fixed problem or compatibility problem, could you try
last kernel(4.14rc1+) and tools(including mkfs & fsck) in your environment?

Thanks,

> 
> Thanks,
> 
>>
>>
>> Regards
>> Marc
>>
>> On Sun, Dec 24, 2017 at 3:35 AM, Marcelo "Marc" Ranolfi
>> <marc.2377@gmail.com> wrote:
>>> Dear developers,
>>>
>>>
>>> I've been using f2fs for my root fs since the 4.9 kernel. The
>>> partition is encrypted with dm-crypt (LUKS).
>>>
>>> A few weeks back, after upgrading to kernel 4.14.2, I began to
>>> experience corruption of many files, revealed by boot-time fsck. Most
>>> of the corrupted files were temporary, but not all.
>>>
>>> I never got to successfully restore them, since fsck.f2fs attempts to
>>> restore files to /.lost_found (instead of lost+found, as it'd normally
>>> be expected), and the folder did not exist in my system.
>>>
>>> After isolating the problem to the kernel version, and after it
>>> affected some important system files, I simply reverted to 4.13.9 and
>>> stood by it for a while.
>>>
>>> Two days ago I wanted to give it another try. Even though nothing from
>>> the changelogs seemed to give me hope, I upgraded from 4.13.9 to
>>> 4.14.7 and rebooted. And rebooted again. No apparent issue, at that
>>> moment.
>>>
>>> But then, after a few hours of work, my applications started to
>>> malfunction. First qemu/libvirt (not sure which); I restarted my VM
>>> but it crashed again after a couple minutes.
>>> Then the music player crashed, followed by my text editor, with a
>>> delay of about 5-10 minutes in between.
>>>
>>> This is where I noticed something was seriously wrong.
>>>
>>> I attempted to close my browser (Firefox) in order to restart the
>>> computer, but it was impossible. Firefox would not close due to some
>>> missing persistent state file(s) that should be stored in
>>> $HOME/.mozilla, but was missing.
>>>
>>> I proceeded to shutdown the system nevertheless, but my DE (Cinnamon)
>>> was not accepting commands due to missing or invalid system libraries.
>>>
>>> Hard reset. On power on, fsck once again "recovered" many lost files -
>>> only it actually did not recover a thing, for I still hadn't created
>>> lost_found.
>>>
>>> After struggling for a while to get my system to a usable state, I had
>>> to reinstall 1146 packages from the package manager. It seems I'm
>>> alright by now.
>>>
>>> I'm sure this is a kernel issue. Could be device mapper, I suppose,
>>> but... I use encryption, in the same fashion, on two other volumes
>>> formatted with ext4. I also have ext2 and ntfs filesystems without
>>> encryption. My hardware is alright. The problem is gone by downgrading
>>> to kernel 4.13.x.
>>>
>>> For these reasons I'm quite sure this is an f2fs issue.
>>>
>>>
>>> Let me know if I can provide something more useful to help in sorting this out.
>>>
>>>
>>> Regards
>>> Marc
> 
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> Linux-f2fs-devel mailing list
> Linux-f2fs-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
> 
> .
> 


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot

Re: Bug report - f2fs corruption with 4.14 kernel

[Jaegeuk Kim]

On 12/28, Chao Yu wrote:
> On 2017/12/28 2:17, Jaegeuk Kim wrote:
> > On 12/26, Marcelo "Marc" Ranolfi wrote:
> >> Hi all,
> >>
> >> confirmed it's not device-mapper.
> >>
> >> Also happening with kernel 4.14.8 and 4.15rc5.
> >>
> >> The bug only happens after some 2-3 hours of uptime or more.
> >>
> >> Yesterday it got me pretty bad; corrupted /var/lib/pacman (I use Arch;
> >> this dir is the package manager library) so I had to restore
> >> everything from the logs via script.
> > 
> > Hi Marc,
> > 
> > Thank you for the report. Could you please check the version of f2fs-tools?
> > I got a report where fsck.f2fs causes breaking the valid checpoint.
> > If possible, could you please try the latest tools from here?
> > https://git.kernel.org/pub/scm/linux/kernel/git/jaegeuk/f2fs-tools.git/log/
> 
> Hi Marc,
> 
> Sorry for later response.
> 
> In 4.14 and 4.15rc1, we have developed many features which changed inode
> disk layout a lot, and during the development, we have considered backward
> compatibility in both kernel module and userland tools, so theoretically,
> there should not be any problem. In order to make sure whether the issue
> you encounter is a fixed problem or compatibility problem, could you try
> last kernel(4.14rc1+) and tools(including mkfs & fsck) in your environment?

I've been looking around every patches merged in 4.14, and feel that the below
patch looks suspicious subtly. Any chance to revert it and test as well?

commit 0abd8e70d24b665d
  ("f2fs: clear radix tree dirty tag of pages whose dirty flag is cleared")

Thanks,

> 
> Thanks,
> 
> > 
> > Thanks,
> > 
> >>
> >>
> >> Regards
> >> Marc
> >>
> >> On Sun, Dec 24, 2017 at 3:35 AM, Marcelo "Marc" Ranolfi
> >> <marc.2377@gmail.com> wrote:
> >>> Dear developers,
> >>>
> >>>
> >>> I've been using f2fs for my root fs since the 4.9 kernel. The
> >>> partition is encrypted with dm-crypt (LUKS).
> >>>
> >>> A few weeks back, after upgrading to kernel 4.14.2, I began to
> >>> experience corruption of many files, revealed by boot-time fsck. Most
> >>> of the corrupted files were temporary, but not all.
> >>>
> >>> I never got to successfully restore them, since fsck.f2fs attempts to
> >>> restore files to /.lost_found (instead of lost+found, as it'd normally
> >>> be expected), and the folder did not exist in my system.
> >>>
> >>> After isolating the problem to the kernel version, and after it
> >>> affected some important system files, I simply reverted to 4.13.9 and
> >>> stood by it for a while.
> >>>
> >>> Two days ago I wanted to give it another try. Even though nothing from
> >>> the changelogs seemed to give me hope, I upgraded from 4.13.9 to
> >>> 4.14.7 and rebooted. And rebooted again. No apparent issue, at that
> >>> moment.
> >>>
> >>> But then, after a few hours of work, my applications started to
> >>> malfunction. First qemu/libvirt (not sure which); I restarted my VM
> >>> but it crashed again after a couple minutes.
> >>> Then the music player crashed, followed by my text editor, with a
> >>> delay of about 5-10 minutes in between.
> >>>
> >>> This is where I noticed something was seriously wrong.
> >>>
> >>> I attempted to close my browser (Firefox) in order to restart the
> >>> computer, but it was impossible. Firefox would not close due to some
> >>> missing persistent state file(s) that should be stored in
> >>> $HOME/.mozilla, but was missing.
> >>>
> >>> I proceeded to shutdown the system nevertheless, but my DE (Cinnamon)
> >>> was not accepting commands due to missing or invalid system libraries.
> >>>
> >>> Hard reset. On power on, fsck once again "recovered" many lost files -
> >>> only it actually did not recover a thing, for I still hadn't created
> >>> lost_found.
> >>>
> >>> After struggling for a while to get my system to a usable state, I had
> >>> to reinstall 1146 packages from the package manager. It seems I'm
> >>> alright by now.
> >>>
> >>> I'm sure this is a kernel issue. Could be device mapper, I suppose,
> >>> but... I use encryption, in the same fashion, on two other volumes
> >>> formatted with ext4. I also have ext2 and ntfs filesystems without
> >>> encryption. My hardware is alright. The problem is gone by downgrading
> >>> to kernel 4.13.x.
> >>>
> >>> For these reasons I'm quite sure this is an f2fs issue.
> >>>
> >>>
> >>> Let me know if I can provide something more useful to help in sorting this out.
> >>>
> >>>
> >>> Regards
> >>> Marc
> > 
> > ------------------------------------------------------------------------------
> > Check out the vibrant tech community on one of the world's most
> > engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> > _______________________________________________
> > Linux-f2fs-devel mailing list
> > Linux-f2fs-devel@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
> > 
> > .
> > 

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot

Re: Bug report - f2fs corruption with 4.14 kernel

[Marcelo "Marc" Ranolfi]

Hi guys,

Now it's my turn to apologize for the late reply!


Well, I eventually switched filesystems (now using ext4), since I
realized that it was not very wise to use such an actively developed
filesystem (f2fs) and expect maximum stability like I did.

But, before doing that, I did upgrade to linux 4.15.2 and f2fs-tools
1.10.0, and tested for a little over 15 hours. It worked alright.

As such, I believe the issue I reported was fixed.


Thanks
Marc

On Thu, Dec 28, 2017 at 1:09 AM, Jaegeuk Kim <jaegeuk@kernel.org> wrote:
> On 12/28, Chao Yu wrote:
>> On 2017/12/28 2:17, Jaegeuk Kim wrote:
>> > On 12/26, Marcelo "Marc" Ranolfi wrote:
>> >> Hi all,
>> >>
>> >> confirmed it's not device-mapper.
>> >>
>> >> Also happening with kernel 4.14.8 and 4.15rc5.
>> >>
>> >> The bug only happens after some 2-3 hours of uptime or more.
>> >>
>> >> Yesterday it got me pretty bad; corrupted /var/lib/pacman (I use Arch;
>> >> this dir is the package manager library) so I had to restore
>> >> everything from the logs via script.
>> >
>> > Hi Marc,
>> >
>> > Thank you for the report. Could you please check the version of f2fs-tools?
>> > I got a report where fsck.f2fs causes breaking the valid checpoint.
>> > If possible, could you please try the latest tools from here?
>> > https://git.kernel.org/pub/scm/linux/kernel/git/jaegeuk/f2fs-tools.git/log/
>>
>> Hi Marc,
>>
>> Sorry for later response.
>>
>> In 4.14 and 4.15rc1, we have developed many features which changed inode
>> disk layout a lot, and during the development, we have considered backward
>> compatibility in both kernel module and userland tools, so theoretically,
>> there should not be any problem. In order to make sure whether the issue
>> you encounter is a fixed problem or compatibility problem, could you try
>> last kernel(4.14rc1+) and tools(including mkfs & fsck) in your environment?
>
> I've been looking around every patches merged in 4.14, and feel that the below
> patch looks suspicious subtly. Any chance to revert it and test as well?
>
> commit 0abd8e70d24b665d
>   ("f2fs: clear radix tree dirty tag of pages whose dirty flag is cleared")
>
> Thanks,
>
>>
>> Thanks,
>>
>> >
>> > Thanks,
>> >
>> >>
>> >>
>> >> Regards
>> >> Marc
>> >>
>> >> On Sun, Dec 24, 2017 at 3:35 AM, Marcelo "Marc" Ranolfi
>> >> <marc.2377@gmail.com> wrote:
>> >>> Dear developers,
>> >>>
>> >>>
>> >>> I've been using f2fs for my root fs since the 4.9 kernel. The
>> >>> partition is encrypted with dm-crypt (LUKS).
>> >>>
>> >>> A few weeks back, after upgrading to kernel 4.14.2, I began to
>> >>> experience corruption of many files, revealed by boot-time fsck. Most
>> >>> of the corrupted files were temporary, but not all.
>> >>>
>> >>> I never got to successfully restore them, since fsck.f2fs attempts to
>> >>> restore files to /.lost_found (instead of lost+found, as it'd normally
>> >>> be expected), and the folder did not exist in my system.
>> >>>
>> >>> After isolating the problem to the kernel version, and after it
>> >>> affected some important system files, I simply reverted to 4.13.9 and
>> >>> stood by it for a while.
>> >>>
>> >>> Two days ago I wanted to give it another try. Even though nothing from
>> >>> the changelogs seemed to give me hope, I upgraded from 4.13.9 to
>> >>> 4.14.7 and rebooted. And rebooted again. No apparent issue, at that
>> >>> moment.
>> >>>
>> >>> But then, after a few hours of work, my applications started to
>> >>> malfunction. First qemu/libvirt (not sure which); I restarted my VM
>> >>> but it crashed again after a couple minutes.
>> >>> Then the music player crashed, followed by my text editor, with a
>> >>> delay of about 5-10 minutes in between.
>> >>>
>> >>> This is where I noticed something was seriously wrong.
>> >>>
>> >>> I attempted to close my browser (Firefox) in order to restart the
>> >>> computer, but it was impossible. Firefox would not close due to some
>> >>> missing persistent state file(s) that should be stored in
>> >>> $HOME/.mozilla, but was missing.
>> >>>
>> >>> I proceeded to shutdown the system nevertheless, but my DE (Cinnamon)
>> >>> was not accepting commands due to missing or invalid system libraries.
>> >>>
>> >>> Hard reset. On power on, fsck once again "recovered" many lost files -
>> >>> only it actually did not recover a thing, for I still hadn't created
>> >>> lost_found.
>> >>>
>> >>> After struggling for a while to get my system to a usable state, I had
>> >>> to reinstall 1146 packages from the package manager. It seems I'm
>> >>> alright by now.
>> >>>
>> >>> I'm sure this is a kernel issue. Could be device mapper, I suppose,
>> >>> but... I use encryption, in the same fashion, on two other volumes
>> >>> formatted with ext4. I also have ext2 and ntfs filesystems without
>> >>> encryption. My hardware is alright. The problem is gone by downgrading
>> >>> to kernel 4.13.x.
>> >>>
>> >>> For these reasons I'm quite sure this is an f2fs issue.
>> >>>
>> >>>
>> >>> Let me know if I can provide something more useful to help in sorting this out.
>> >>>
>> >>>
>> >>> Regards
>> >>> Marc
>> >
>> > ------------------------------------------------------------------------------
>> > Check out the vibrant tech community on one of the world's most
>> > engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>> > _______________________________________________
>> > Linux-f2fs-devel mailing list
>> > Linux-f2fs-devel@lists.sourceforge.net
>> > https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
>> >
>> > .
>> >

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot