Re: f2fs bug: Unable to mount big volumes in kernel 4.5

Re: f2fs bug: Unable to mount big volumes in kernel 4.5

[Chao Yu]

Hi all,

> -----Original Message-----
> From: Jaegeuk Kim [mailto:jaegeuk@kernel.org]
> Sent: Monday, March 21, 2016 6:47 AM
> To: David Gnedt
> Cc: Chao Yu
> Subject: Re: f2fs bug: Unable to mount big volumes in kernel 4.5
> 
> Hello,
> 
> As you pointed out, it seems there is a bug in sanity check routine, which
> doesn't cover the large section case.

Actually, there is a bug in f2fs-tools 1.6.0, it will trigger sanity check failure
in f2fs kernel module since in mkfs.f2fs we will align segment_count and
segment_count_main with different size if parameter -s or -z is configured larger
than 1.

Following commit in dev branch of f2fs-tools has fixed this issue, could you test this
patch firstly?
("mkfs.f2fs: set segment_count in super block correctly")

Thanks,

> 
> Could you test the attached patch?
> 
> Thanks,
> 
> On Sun, Mar 20, 2016 at 04:53:10PM +0100, David Gnedt wrote:
> > Hello,
> >
> > I have a problem with mounting a very big f2fs volume (8TB Seagate SMR drive) in
> > kernel 4.5. It works without any problems in 4.4.6, but with 4.5 I get errors
> > during mount (see attached file smr_log.txt).
> >
> > I have created the volume with f2fs-tools 1.6.0 and also a fsck works without
> > any problems (see attached file smr_fsck.txt).
> >
> > I suspect the following commit to be the cause:
> >
> > commit 9a59b62fd88196844cee5fff851bee2cfd7afb6e
> > f2fs: do more integrity verification for superblock
> >
> >
> https://git.kernel.org/cgit/linux/kernel/git/jaegeuk/f2fs.git/commit/fs/f2fs/super.c?id=9a
> 59b62fd88196844cee5fff851bee2cfd7afb6e
> >
> > I haven't had time to dig deeper, but my guess is that the volume is too big, so
> > that the 32bit integers overflow during the new check routine
> > sanity_check_area_boundary().
> >
> > Please let me know, if you need any more information. F2fs is currently the only
> > filesystem that really works on those SMR drives, therefore it is an important
> > issue to me.
> >
> > Best regards,
> > David Gnedt
> 
> > [437136.843299] F2FS-fs (dm-7): Wrong MAIN_AREA boundary, start(4063232) end(1953505792)
> blocks(1949433856)
> > [437136.843302] F2FS-fs (dm-7): Can't find valid F2FS filesystem in 1th superblock
> > [437136.843418] F2FS-fs (dm-7): Wrong MAIN_AREA boundary, start(4063232) end(1953505792)
> blocks(1949433856)
> > [437136.843419] F2FS-fs (dm-7): Can't find valid F2FS filesystem in 2th superblock
> > [437136.843420] F2FS-fs (dm-7): Wrong MAIN_AREA boundary, start(4063232) end(1953505792)
> blocks(1949433856)
> > [437136.843421] F2FS-fs (dm-7): Can't find valid F2FS filesystem in 1th superblock
> > [437136.843422] F2FS-fs (dm-7): Wrong MAIN_AREA boundary, start(4063232) end(1953505792)
> blocks(1949433856)
> > [437136.843422] F2FS-fs (dm-7): Can't find valid F2FS filesystem in 2th superblock
> 
> > Info: sector size = 512
> > Info: total sectors = 15628046336 (7630882 MB)
> > Info: MKFS version
> >   "Linux version 4.4.0-040400-generic (kernel@gomeisa) (gcc version 5.2.1 20151010 (Ubuntu
> 5.2.1-22ubuntu2) ) #201601101930 SMP Mon Jan 11 00:32:41 UTC 2016"
> > Info: FSCK version
> >   from "Linux version 4.4.0-040400-generic (kernel@gomeisa) (gcc version 5.2.1 20151010
> (Ubuntu 5.2.1-22ubuntu2) ) #201601101930 SMP Mon Jan 11 00:32:41 UTC 2016"
> >     to "Linux version 4.5.0-040500-generic (kernel@tangerine) (gcc version 5.2.1 20151010
> (Ubuntu 5.2.1-22ubuntu2) ) #201603140130 SMP Mon Mar 14 05:32:22 UTC 2016"
> > Info: superblock features = 0 :
> > Info: superblock encrypt level = 0, salt = 00000000000000000000000000000000
> > Info: total FS sectors = 15628046336 (7630882 MB)
> > Info: CKPT version = 122a
> > Info: checkpoint state = 1 :  unmount
> >
> > [FSCK] Unreachable nat entries                        [Ok..] [0x0]
> > [FSCK] SIT valid block bitmap checking                [Ok..]
> > [FSCK] Hard link checking for regular file            [Ok..] [0x0]
> > [FSCK] valid_block_count matching with CP             [Ok..] [0x57cea239]
> > [FSCK] valid_node_count matcing with CP (de lookup)   [Ok..] [0x61b3bd]
> > [FSCK] valid_node_count matcing with CP (nat lookup)  [Ok..] [0x61b3bd]
> > [FSCK] valid_inode_count matched with CP              [Ok..] [0x4c0a61]
> > [FSCK] free segment_count matched with CP             [Ok..] [0xe31ac]
> > [FSCK] next block offset is free                      [Ok..]
> > [FSCK] fixing SIT types
> > [FSCK] other corrupted bugs                           [Ok..]
> >
> > Done.



------------------------------------------------------------------------------
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://pubads.g.doubleclick.net/gampad/clk?id=278785351&iu=/4140

Re: f2fs bug: Unable to mount big volumes in kernel 4.5

[Matthias Prager]

Am 21.03.2016 um 04:18 schrieb Chao Yu:
> Hi all,
> 
>> -----Original Message-----
>> From: Jaegeuk Kim [mailto:jaegeuk@kernel.org]
>> Sent: Monday, March 21, 2016 6:47 AM
>> To: David Gnedt
>> Cc: Chao Yu
>> Subject: Re: f2fs bug: Unable to mount big volumes in kernel 4.5
>>
>> Hello,
>>
>> As you pointed out, it seems there is a bug in sanity check routine, which
>> doesn't cover the large section case.
> 
> Actually, there is a bug in f2fs-tools 1.6.0, it will trigger sanity check failure
> in f2fs kernel module since in mkfs.f2fs we will align segment_count and
> segment_count_main with different size if parameter -s or -z is configured larger
> than 1.
> 
> Following commit in dev branch of f2fs-tools has fixed this issue, could you test this
> patch firstly?
> ("mkfs.f2fs: set segment_count in super block correctly")
I just did (with an unpatched 4.5.0 kernel) and with this f2f2-tools
patch applied a newly created fs with '-s 9' does mount without errors.

There seem to be quite a few interesting commits in the f2fs-tools dev
tree. Is a new release of f2fs-tools planned soon?

> 
> Thanks,
> 
>>
>> Could you test the attached patch?
>>
>> Thanks,
>>
>> On Sun, Mar 20, 2016 at 04:53:10PM +0100, David Gnedt wrote:
>>> Hello,
>>>
>>> I have a problem with mounting a very big f2fs volume (8TB Seagate SMR drive) in
>>> kernel 4.5. It works without any problems in 4.4.6, but with 4.5 I get errors
>>> during mount (see attached file smr_log.txt).
>>>
>>> I have created the volume with f2fs-tools 1.6.0 and also a fsck works without
>>> any problems (see attached file smr_fsck.txt).
>>>
>>> I suspect the following commit to be the cause:
>>>
>>> commit 9a59b62fd88196844cee5fff851bee2cfd7afb6e
>>> f2fs: do more integrity verification for superblock
>>>
>>>
>> https://git.kernel.org/cgit/linux/kernel/git/jaegeuk/f2fs.git/commit/fs/f2fs/super.c?id=9a
>> 59b62fd88196844cee5fff851bee2cfd7afb6e
>>>
>>> I haven't had time to dig deeper, but my guess is that the volume is too big, so
>>> that the 32bit integers overflow during the new check routine
>>> sanity_check_area_boundary().
>>>
>>> Please let me know, if you need any more information. F2fs is currently the only
>>> filesystem that really works on those SMR drives, therefore it is an important
>>> issue to me.
>>>
>>> Best regards,
>>> David Gnedt
>>
>>> [437136.843299] F2FS-fs (dm-7): Wrong MAIN_AREA boundary, start(4063232) end(1953505792)
>> blocks(1949433856)
>>> [437136.843302] F2FS-fs (dm-7): Can't find valid F2FS filesystem in 1th superblock
>>> [437136.843418] F2FS-fs (dm-7): Wrong MAIN_AREA boundary, start(4063232) end(1953505792)
>> blocks(1949433856)
>>> [437136.843419] F2FS-fs (dm-7): Can't find valid F2FS filesystem in 2th superblock
>>> [437136.843420] F2FS-fs (dm-7): Wrong MAIN_AREA boundary, start(4063232) end(1953505792)
>> blocks(1949433856)
>>> [437136.843421] F2FS-fs (dm-7): Can't find valid F2FS filesystem in 1th superblock
>>> [437136.843422] F2FS-fs (dm-7): Wrong MAIN_AREA boundary, start(4063232) end(1953505792)
>> blocks(1949433856)
>>> [437136.843422] F2FS-fs (dm-7): Can't find valid F2FS filesystem in 2th superblock
>>
>>> Info: sector size = 512
>>> Info: total sectors = 15628046336 (7630882 MB)
>>> Info: MKFS version
>>>   "Linux version 4.4.0-040400-generic (kernel@gomeisa) (gcc version 5.2.1 20151010 (Ubuntu
>> 5.2.1-22ubuntu2) ) #201601101930 SMP Mon Jan 11 00:32:41 UTC 2016"
>>> Info: FSCK version
>>>   from "Linux version 4.4.0-040400-generic (kernel@gomeisa) (gcc version 5.2.1 20151010
>> (Ubuntu 5.2.1-22ubuntu2) ) #201601101930 SMP Mon Jan 11 00:32:41 UTC 2016"
>>>     to "Linux version 4.5.0-040500-generic (kernel@tangerine) (gcc version 5.2.1 20151010
>> (Ubuntu 5.2.1-22ubuntu2) ) #201603140130 SMP Mon Mar 14 05:32:22 UTC 2016"
>>> Info: superblock features = 0 :
>>> Info: superblock encrypt level = 0, salt = 00000000000000000000000000000000
>>> Info: total FS sectors = 15628046336 (7630882 MB)
>>> Info: CKPT version = 122a
>>> Info: checkpoint state = 1 :  unmount
>>>
>>> [FSCK] Unreachable nat entries                        [Ok..] [0x0]
>>> [FSCK] SIT valid block bitmap checking                [Ok..]
>>> [FSCK] Hard link checking for regular file            [Ok..] [0x0]
>>> [FSCK] valid_block_count matching with CP             [Ok..] [0x57cea239]
>>> [FSCK] valid_node_count matcing with CP (de lookup)   [Ok..] [0x61b3bd]
>>> [FSCK] valid_node_count matcing with CP (nat lookup)  [Ok..] [0x61b3bd]
>>> [FSCK] valid_inode_count matched with CP              [Ok..] [0x4c0a61]
>>> [FSCK] free segment_count matched with CP             [Ok..] [0xe31ac]
>>> [FSCK] next block offset is free                      [Ok..]
>>> [FSCK] fixing SIT types
>>> [FSCK] other corrupted bugs                           [Ok..]
>>>
>>> Done.
> 


------------------------------------------------------------------------------
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://pubads.g.doubleclick.net/gampad/clk?id=278785351&iu=/4140

Re: f2fs bug: Unable to mount big volumes in kernel 4.5

[Marc Lehmann]

On Mon, Mar 21, 2016 at 11:18:35AM +0800, Chao Yu <chao2.yu@samsung.com> wrote:
> > As you pointed out, it seems there is a bug in sanity check routine, which
> > doesn't cover the large section case.
> 
> Actually, there is a bug in f2fs-tools 1.6.0, it will trigger sanity check failure
> in f2fs kernel module since in mkfs.f2fs we will align segment_count and
> segment_count_main with different size if parameter -s or -z is configured larger
> than 1.
> 
> Following commit in dev branch of f2fs-tools has fixed this issue, could you test this
> patch firstly?
> ("mkfs.f2fs: set segment_count in super block correctly")

Stupid question from my side, does that mean we have to reformat existing
volumes? Because mkfs clearly won't fix existing volumes, so fixing mkfs
will not fix the issue.

A fix in the f2fs or fsck.f2fs would be needed for existing volumes.

-- 
                The choice of a       Deliantra, the free code+content MORPG
      -----==-     _GNU_              http://www.deliantra.net
      ----==-- _       generation
      ---==---(_)__  __ ____  __      Marc Lehmann
      --==---/ / _ \/ // /\ \/ /      schmorp@schmorp.de
      -=====/_/_//_/\_,_/ /_/\_\

------------------------------------------------------------------------------
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://pubads.g.doubleclick.net/gampad/clk?id=278785351&iu=/4140

Re: f2fs bug: Unable to mount big volumes in kernel 4.5

[Jaegeuk Kim]

On Mon, Mar 21, 2016 at 10:58:49AM +0100, Matthias Prager wrote:
> Am 21.03.2016 um 04:18 schrieb Chao Yu:
> > Hi all,
> > 
> >> -----Original Message-----
> >> From: Jaegeuk Kim [mailto:jaegeuk@kernel.org]
> >> Sent: Monday, March 21, 2016 6:47 AM
> >> To: David Gnedt
> >> Cc: Chao Yu
> >> Subject: Re: f2fs bug: Unable to mount big volumes in kernel 4.5
> >>
> >> Hello,
> >>
> >> As you pointed out, it seems there is a bug in sanity check routine, which
> >> doesn't cover the large section case.
> > 
> > Actually, there is a bug in f2fs-tools 1.6.0, it will trigger sanity check failure
> > in f2fs kernel module since in mkfs.f2fs we will align segment_count and
> > segment_count_main with different size if parameter -s or -z is configured larger
> > than 1.
> > 
> > Following commit in dev branch of f2fs-tools has fixed this issue, could you test this
> > patch firstly?
> > ("mkfs.f2fs: set segment_count in super block correctly")
> I just did (with an unpatched 4.5.0 kernel) and with this f2f2-tools
> patch applied a newly created fs with '-s 9' does mount without errors.
> 
> There seem to be quite a few interesting commits in the f2fs-tools dev
> tree. Is a new release of f2fs-tools planned soon?

Hope to release v1.7.0 in the next month, which will introduce resize.f2fs and
sload.f2fs.
I'm waiting for rolling out products shipped by the tools.

Thanks,

> 
> > 
> > Thanks,
> > 
> >>
> >> Could you test the attached patch?
> >>
> >> Thanks,
> >>
> >> On Sun, Mar 20, 2016 at 04:53:10PM +0100, David Gnedt wrote:
> >>> Hello,
> >>>
> >>> I have a problem with mounting a very big f2fs volume (8TB Seagate SMR drive) in
> >>> kernel 4.5. It works without any problems in 4.4.6, but with 4.5 I get errors
> >>> during mount (see attached file smr_log.txt).
> >>>
> >>> I have created the volume with f2fs-tools 1.6.0 and also a fsck works without
> >>> any problems (see attached file smr_fsck.txt).
> >>>
> >>> I suspect the following commit to be the cause:
> >>>
> >>> commit 9a59b62fd88196844cee5fff851bee2cfd7afb6e
> >>> f2fs: do more integrity verification for superblock
> >>>
> >>>
> >> https://git.kernel.org/cgit/linux/kernel/git/jaegeuk/f2fs.git/commit/fs/f2fs/super.c?id=9a
> >> 59b62fd88196844cee5fff851bee2cfd7afb6e
> >>>
> >>> I haven't had time to dig deeper, but my guess is that the volume is too big, so
> >>> that the 32bit integers overflow during the new check routine
> >>> sanity_check_area_boundary().
> >>>
> >>> Please let me know, if you need any more information. F2fs is currently the only
> >>> filesystem that really works on those SMR drives, therefore it is an important
> >>> issue to me.
> >>>
> >>> Best regards,
> >>> David Gnedt
> >>
> >>> [437136.843299] F2FS-fs (dm-7): Wrong MAIN_AREA boundary, start(4063232) end(1953505792)
> >> blocks(1949433856)
> >>> [437136.843302] F2FS-fs (dm-7): Can't find valid F2FS filesystem in 1th superblock
> >>> [437136.843418] F2FS-fs (dm-7): Wrong MAIN_AREA boundary, start(4063232) end(1953505792)
> >> blocks(1949433856)
> >>> [437136.843419] F2FS-fs (dm-7): Can't find valid F2FS filesystem in 2th superblock
> >>> [437136.843420] F2FS-fs (dm-7): Wrong MAIN_AREA boundary, start(4063232) end(1953505792)
> >> blocks(1949433856)
> >>> [437136.843421] F2FS-fs (dm-7): Can't find valid F2FS filesystem in 1th superblock
> >>> [437136.843422] F2FS-fs (dm-7): Wrong MAIN_AREA boundary, start(4063232) end(1953505792)
> >> blocks(1949433856)
> >>> [437136.843422] F2FS-fs (dm-7): Can't find valid F2FS filesystem in 2th superblock
> >>
> >>> Info: sector size = 512
> >>> Info: total sectors = 15628046336 (7630882 MB)
> >>> Info: MKFS version
> >>>   "Linux version 4.4.0-040400-generic (kernel@gomeisa) (gcc version 5.2.1 20151010 (Ubuntu
> >> 5.2.1-22ubuntu2) ) #201601101930 SMP Mon Jan 11 00:32:41 UTC 2016"
> >>> Info: FSCK version
> >>>   from "Linux version 4.4.0-040400-generic (kernel@gomeisa) (gcc version 5.2.1 20151010
> >> (Ubuntu 5.2.1-22ubuntu2) ) #201601101930 SMP Mon Jan 11 00:32:41 UTC 2016"
> >>>     to "Linux version 4.5.0-040500-generic (kernel@tangerine) (gcc version 5.2.1 20151010
> >> (Ubuntu 5.2.1-22ubuntu2) ) #201603140130 SMP Mon Mar 14 05:32:22 UTC 2016"
> >>> Info: superblock features = 0 :
> >>> Info: superblock encrypt level = 0, salt = 00000000000000000000000000000000
> >>> Info: total FS sectors = 15628046336 (7630882 MB)
> >>> Info: CKPT version = 122a
> >>> Info: checkpoint state = 1 :  unmount
> >>>
> >>> [FSCK] Unreachable nat entries                        [Ok..] [0x0]
> >>> [FSCK] SIT valid block bitmap checking                [Ok..]
> >>> [FSCK] Hard link checking for regular file            [Ok..] [0x0]
> >>> [FSCK] valid_block_count matching with CP             [Ok..] [0x57cea239]
> >>> [FSCK] valid_node_count matcing with CP (de lookup)   [Ok..] [0x61b3bd]
> >>> [FSCK] valid_node_count matcing with CP (nat lookup)  [Ok..] [0x61b3bd]
> >>> [FSCK] valid_inode_count matched with CP              [Ok..] [0x4c0a61]
> >>> [FSCK] free segment_count matched with CP             [Ok..] [0xe31ac]
> >>> [FSCK] next block offset is free                      [Ok..]
> >>> [FSCK] fixing SIT types
> >>> [FSCK] other corrupted bugs                           [Ok..]
> >>>
> >>> Done.
> > 

------------------------------------------------------------------------------
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://pubads.g.doubleclick.net/gampad/clk?id=278785351&iu=/4140

Re: f2fs bug: Unable to mount big volumes in kernel 4.5

[Jaegeuk Kim]

On Mon, Mar 21, 2016 at 11:30:38AM +0100, Marc Lehmann wrote:
> On Mon, Mar 21, 2016 at 11:18:35AM +0800, Chao Yu <chao2.yu@samsung.com> wrote:
> > > As you pointed out, it seems there is a bug in sanity check routine, which
> > > doesn't cover the large section case.
> > 
> > Actually, there is a bug in f2fs-tools 1.6.0, it will trigger sanity check failure
> > in f2fs kernel module since in mkfs.f2fs we will align segment_count and
> > segment_count_main with different size if parameter -s or -z is configured larger
> > than 1.
> > 
> > Following commit in dev branch of f2fs-tools has fixed this issue, could you test this
> > patch firstly?
> > ("mkfs.f2fs: set segment_count in super block correctly")
> 
> Stupid question from my side, does that mean we have to reformat existing
> volumes? Because mkfs clearly won't fix existing volumes, so fixing mkfs
> will not fix the issue.

Exactly.
The f2fs patch should be merged to take into account such the backward
compatibility.
And, I don't think wrong segment_count hurts the f2fs behavior.

Thanks,

> 
> A fix in the f2fs or fsck.f2fs would be needed for existing volumes.
> 
> -- 
>                 The choice of a       Deliantra, the free code+content MORPG
>       -----==-     _GNU_              http://www.deliantra.net
>       ----==-- _       generation
>       ---==---(_)__  __ ____  __      Marc Lehmann
>       --==---/ / _ \/ // /\ \/ /      schmorp@schmorp.de
>       -=====/_/_//_/\_,_/ /_/\_\

------------------------------------------------------------------------------
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://pubads.g.doubleclick.net/gampad/clk?id=278785351&iu=/4140

Re: f2fs bug: Unable to mount big volumes in kernel 4.5

[David Gnedt]

Hello,

> Following commit in dev branch of f2fs-tools has fixed this issue, could you
test this
> patch firstly?
> ("mkfs.f2fs: set segment_count in super block correctly")

Sorry, I cannot recreate the volume on the 8TB Seagate drive right now, but I
did a test with a loopback device of equal size.

Should I be worried about the changes in mkfs.f2fs since 1.6.0? Would it be
suggested to recreate the filesystem? Or is it maybe possible to manually fix it?
If it is really only the alignment, I think it shouldn't matter for SMR drives,
as they are not using constant size zones anyway, so misalignment cannot be
avoided with f2fs.

>> Could you test the attached patch?

I did a roundup test of all kernel/f2fs-tools version I thought that make sense.
I hope the patch will be included in upcoming mainline kernels.

Short summary:
--------------
Filesystems created with a recent dev version of f2fs-tools work without any
problems and don't need the f2fs kernel patch.
Filesystems created with older f2fs-tools need the f2fs kernel patch to mount
correctly.
I guess that is exactly what everyone expected to be the outcome.

Full details:
-------------
f2fs-tools D:  release 1.6.0 (Seagate SMR drive)
f2fs-tools L1: release 1.6.0 (Loopback)
f2fs-tools L2: 2016-01-08 mkfs.f2fs: introduce zone align for main area (Loopback)
f2fs-tools L3: 2016-03-16 mkfs.f2fs: set segment_count in super block correctly
(Loopback)

Test procedure:
$ truncate -s 8001559724032 f2fs-tools-1.6.0.img
$ losetup -f f2fs-tools-1.6.0.img
$ mkfs.f2fs -s64 -t0 -a0 /dev/loop0
$ mount -t f2fs -onoinline_data,noatime,flush_merge,no_heap,ro /dev/loop0 /mnt
$ umount /mnt

Kernel: Linux 4.4.6 x86_64
f2fs-tools D:  Works
f2fs-tools L1: Works
f2fs-tools L2: Works
f2fs-tools L3: Works

Kernel: Linux 4.5.0 x86_64
f2fs-tools D:  Failed -> Same errors as before
f2fs-tools L1: Failed -> Same errors as before
f2fs-tools L2: Failed -> Same errors as before
f2fs-tools L3: Works

Kernel: Linux 4.5.0+patch x86_64
f2fs-tools D:  Works
f2fs-tools L1: Works
f2fs-tools L2: Works
f2fs-tools L3: Works

Best regards,
David Gnedt

------------------------------------------------------------------------------
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://pubads.g.doubleclick.net/gampad/clk?id=278785351&iu=/4140

Re: f2fs bug: Unable to mount big volumes in kernel 4.5

[Matthias Prager]

Hello everyone,

if there really is a suboptimal/faulty alignment being done by
f2fs-tools <=1.6.0 (and I haven't done the math, so I'm just assuming),
wouldn't it be prudent to include a modified kernel patch, which allows
for mounting of older filesystems, but puts out a warning in dmesg which
advises the user to run fsck.f2fs some time in the future?

---
Matthias

Am 21.03.2016 um 21:58 schrieb David Gnedt:
> Hello,
> 
>> Following commit in dev branch of f2fs-tools has fixed this issue, could you
> test this
>> patch firstly?
>> ("mkfs.f2fs: set segment_count in super block correctly")
> 
> Sorry, I cannot recreate the volume on the 8TB Seagate drive right now, but I
> did a test with a loopback device of equal size.
> 
> Should I be worried about the changes in mkfs.f2fs since 1.6.0? Would it be
> suggested to recreate the filesystem? Or is it maybe possible to manually fix it?
> If it is really only the alignment, I think it shouldn't matter for SMR drives,
> as they are not using constant size zones anyway, so misalignment cannot be
> avoided with f2fs.
> 
>>> Could you test the attached patch?
> 
> I did a roundup test of all kernel/f2fs-tools version I thought that make sense.
> I hope the patch will be included in upcoming mainline kernels.
> 
> Short summary:
> --------------
> Filesystems created with a recent dev version of f2fs-tools work without any
> problems and don't need the f2fs kernel patch.
> Filesystems created with older f2fs-tools need the f2fs kernel patch to mount
> correctly.
> I guess that is exactly what everyone expected to be the outcome.
> 
> Full details:
> -------------
> f2fs-tools D:  release 1.6.0 (Seagate SMR drive)
> f2fs-tools L1: release 1.6.0 (Loopback)
> f2fs-tools L2: 2016-01-08 mkfs.f2fs: introduce zone align for main area (Loopback)
> f2fs-tools L3: 2016-03-16 mkfs.f2fs: set segment_count in super block correctly
> (Loopback)
> 
> Test procedure:
> $ truncate -s 8001559724032 f2fs-tools-1.6.0.img
> $ losetup -f f2fs-tools-1.6.0.img
> $ mkfs.f2fs -s64 -t0 -a0 /dev/loop0
> $ mount -t f2fs -onoinline_data,noatime,flush_merge,no_heap,ro /dev/loop0 /mnt
> $ umount /mnt
> 
> Kernel: Linux 4.4.6 x86_64
> f2fs-tools D:  Works
> f2fs-tools L1: Works
> f2fs-tools L2: Works
> f2fs-tools L3: Works
> 
> Kernel: Linux 4.5.0 x86_64
> f2fs-tools D:  Failed -> Same errors as before
> f2fs-tools L1: Failed -> Same errors as before
> f2fs-tools L2: Failed -> Same errors as before
> f2fs-tools L3: Works
> 
> Kernel: Linux 4.5.0+patch x86_64
> f2fs-tools D:  Works
> f2fs-tools L1: Works
> f2fs-tools L2: Works
> f2fs-tools L3: Works
> 
> Best regards,
> David Gnedt
> 


------------------------------------------------------------------------------
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://pubads.g.doubleclick.net/gampad/clk?id=278785351&iu=/4140

Re: f2fs bug: Unable to mount big volumes in kernel 4.5

[Chao Yu]

Hi Jaegeuk,

> -----Original Message-----
> From: Jaegeuk Kim [mailto:jaegeuk@kernel.org]
> Sent: Tuesday, March 22, 2016 12:04 AM
> To: Marc Lehmann
> Cc: Chao Yu; linux-f2fs-devel@lists.sourceforge.net
> Subject: Re: [f2fs-dev] f2fs bug: Unable to mount big volumes in kernel 4.5
> 
> On Mon, Mar 21, 2016 at 11:30:38AM +0100, Marc Lehmann wrote:
> > On Mon, Mar 21, 2016 at 11:18:35AM +0800, Chao Yu <chao2.yu@samsung.com> wrote:
> > > > As you pointed out, it seems there is a bug in sanity check routine, which
> > > > doesn't cover the large section case.
> > >
> > > Actually, there is a bug in f2fs-tools 1.6.0, it will trigger sanity check failure
> > > in f2fs kernel module since in mkfs.f2fs we will align segment_count and
> > > segment_count_main with different size if parameter -s or -z is configured larger
> > > than 1.
> > >
> > > Following commit in dev branch of f2fs-tools has fixed this issue, could you test this
> > > patch firstly?
> > > ("mkfs.f2fs: set segment_count in super block correctly")
> >
> > Stupid question from my side, does that mean we have to reformat existing
> > volumes? Because mkfs clearly won't fix existing volumes, so fixing mkfs
> > will not fix the issue.
> 
> Exactly.
> The f2fs patch should be merged to take into account such the backward
> compatibility.

Agreed.

> And, I don't think wrong segment_count hurts the f2fs behavior.

Now, we use segment_count to check our last blkaddr, if we ignore correctness
of segment_count, our boundary checked should be wrong.

So how about enabling fsck.f2fs ability of repairing incorrect value of
segment_count, and for f2fs module, warn our user to run fsck if we detect
such error?

Thanks,

> 
> Thanks,
> 
> >
> > A fix in the f2fs or fsck.f2fs would be needed for existing volumes.
> >
> > --
> >                 The choice of a       Deliantra, the free code+content MORPG
> >       -----==-     _GNU_              http://www.deliantra.net
> >       ----==-- _       generation
> >       ---==---(_)__  __ ____  __      Marc Lehmann
> >       --==---/ / _ \/ // /\ \/ /      schmorp@schmorp.de
> >       -=====/_/_//_/\_,_/ /_/\_\


------------------------------------------------------------------------------
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://pubads.g.doubleclick.net/gampad/clk?id=278785351&iu=/4140

Re: f2fs bug: Unable to mount big volumes in kernel 4.5

[Chao Yu]

Hi,

> -----Original Message-----
> From: David Gnedt [mailto:david.gnedt@davizone.at]
> Sent: Tuesday, March 22, 2016 4:58 AM
> To: Chao Yu; 'Jaegeuk Kim'; 'Matthias Prager'
> Cc: linux-f2fs-devel@lists.sourceforge.net
> Subject: Re: f2fs bug: Unable to mount big volumes in kernel 4.5
> 
> Hello,
> 
> > Following commit in dev branch of f2fs-tools has fixed this issue, could you
> test this
> > patch firstly?
> > ("mkfs.f2fs: set segment_count in super block correctly")
> 
> Sorry, I cannot recreate the volume on the 8TB Seagate drive right now, but I
> did a test with a loopback device of equal size.
> 
> Should I be worried about the changes in mkfs.f2fs since 1.6.0?

I think it would be safe to use 1.6.0 + additional 2 alignment fixing patches before
next version was released.

> Would it be
> suggested to recreate the filesystem? Or is it maybe possible to manually fix it?

We don't need to recreate the filesystem, IMO, it's better to show some dmesg to
reminder us to use fsck to fix the misalignment when detecting such error.

> If it is really only the alignment, I think it shouldn't matter for SMR drives,
> as they are not using constant size zones anyway, so misalignment cannot be
> avoided with f2fs.
> 
> >> Could you test the attached patch?
> 
> I did a roundup test of all kernel/f2fs-tools version I thought that make sense.
> I hope the patch will be included in upcoming mainline kernels.

Thank you for the test. :)

Thanks,

> 
> Short summary:
> --------------
> Filesystems created with a recent dev version of f2fs-tools work without any
> problems and don't need the f2fs kernel patch.
> Filesystems created with older f2fs-tools need the f2fs kernel patch to mount
> correctly.
> I guess that is exactly what everyone expected to be the outcome.
> 
> Full details:
> -------------
> f2fs-tools D:  release 1.6.0 (Seagate SMR drive)
> f2fs-tools L1: release 1.6.0 (Loopback)
> f2fs-tools L2: 2016-01-08 mkfs.f2fs: introduce zone align for main area (Loopback)
> f2fs-tools L3: 2016-03-16 mkfs.f2fs: set segment_count in super block correctly
> (Loopback)
> 
> Test procedure:
> $ truncate -s 8001559724032 f2fs-tools-1.6.0.img
> $ losetup -f f2fs-tools-1.6.0.img
> $ mkfs.f2fs -s64 -t0 -a0 /dev/loop0
> $ mount -t f2fs -onoinline_data,noatime,flush_merge,no_heap,ro /dev/loop0 /mnt
> $ umount /mnt
> 
> Kernel: Linux 4.4.6 x86_64
> f2fs-tools D:  Works
> f2fs-tools L1: Works
> f2fs-tools L2: Works
> f2fs-tools L3: Works
> 
> Kernel: Linux 4.5.0 x86_64
> f2fs-tools D:  Failed -> Same errors as before
> f2fs-tools L1: Failed -> Same errors as before
> f2fs-tools L2: Failed -> Same errors as before
> f2fs-tools L3: Works
> 
> Kernel: Linux 4.5.0+patch x86_64
> f2fs-tools D:  Works
> f2fs-tools L1: Works
> f2fs-tools L2: Works
> f2fs-tools L3: Works
> 
> Best regards,
> David Gnedt


------------------------------------------------------------------------------
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://pubads.g.doubleclick.net/gampad/clk?id=278785351&iu=/4140

Re: f2fs bug: Unable to mount big volumes in kernel 4.5

[Chao Yu]

Hi,

> -----Original Message-----
> From: Matthias Prager [mailto:linux@matthiasprager.de]
> Sent: Tuesday, March 22, 2016 6:56 AM
> To: David Gnedt; Chao Yu; 'Jaegeuk Kim'
> Cc: linux-f2fs-devel@lists.sourceforge.net
> Subject: Re: f2fs bug: Unable to mount big volumes in kernel 4.5
> 
> Hello everyone,
> 
> if there really is a suboptimal/faulty alignment being done by
> f2fs-tools <=1.6.0 (and I haven't done the math, so I'm just assuming),
> wouldn't it be prudent to include a modified kernel patch, which allows
> for mounting of older filesystems, but puts out a warning in dmesg which
> advises the user to run fsck.f2fs some time in the future?

I agree with that.

Thanks,

> 
> ---
> Matthias
> 
> Am 21.03.2016 um 21:58 schrieb David Gnedt:
> > Hello,
> >
> >> Following commit in dev branch of f2fs-tools has fixed this issue, could you
> > test this
> >> patch firstly?
> >> ("mkfs.f2fs: set segment_count in super block correctly")
> >
> > Sorry, I cannot recreate the volume on the 8TB Seagate drive right now, but I
> > did a test with a loopback device of equal size.
> >
> > Should I be worried about the changes in mkfs.f2fs since 1.6.0? Would it be
> > suggested to recreate the filesystem? Or is it maybe possible to manually fix it?
> > If it is really only the alignment, I think it shouldn't matter for SMR drives,
> > as they are not using constant size zones anyway, so misalignment cannot be
> > avoided with f2fs.
> >
> >>> Could you test the attached patch?
> >
> > I did a roundup test of all kernel/f2fs-tools version I thought that make sense.
> > I hope the patch will be included in upcoming mainline kernels.
> >
> > Short summary:
> > --------------
> > Filesystems created with a recent dev version of f2fs-tools work without any
> > problems and don't need the f2fs kernel patch.
> > Filesystems created with older f2fs-tools need the f2fs kernel patch to mount
> > correctly.
> > I guess that is exactly what everyone expected to be the outcome.
> >
> > Full details:
> > -------------
> > f2fs-tools D:  release 1.6.0 (Seagate SMR drive)
> > f2fs-tools L1: release 1.6.0 (Loopback)
> > f2fs-tools L2: 2016-01-08 mkfs.f2fs: introduce zone align for main area (Loopback)
> > f2fs-tools L3: 2016-03-16 mkfs.f2fs: set segment_count in super block correctly
> > (Loopback)
> >
> > Test procedure:
> > $ truncate -s 8001559724032 f2fs-tools-1.6.0.img
> > $ losetup -f f2fs-tools-1.6.0.img
> > $ mkfs.f2fs -s64 -t0 -a0 /dev/loop0
> > $ mount -t f2fs -onoinline_data,noatime,flush_merge,no_heap,ro /dev/loop0 /mnt
> > $ umount /mnt
> >
> > Kernel: Linux 4.4.6 x86_64
> > f2fs-tools D:  Works
> > f2fs-tools L1: Works
> > f2fs-tools L2: Works
> > f2fs-tools L3: Works
> >
> > Kernel: Linux 4.5.0 x86_64
> > f2fs-tools D:  Failed -> Same errors as before
> > f2fs-tools L1: Failed -> Same errors as before
> > f2fs-tools L2: Failed -> Same errors as before
> > f2fs-tools L3: Works
> >
> > Kernel: Linux 4.5.0+patch x86_64
> > f2fs-tools D:  Works
> > f2fs-tools L1: Works
> > f2fs-tools L2: Works
> > f2fs-tools L3: Works
> >
> > Best regards,
> > David Gnedt
> >



------------------------------------------------------------------------------
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://pubads.g.doubleclick.net/gampad/clk?id=278785351&iu=/4140

Re: f2fs bug: Unable to mount big volumes in kernel 4.5

[Jaegeuk Kim]

Hi,

On Tue, Mar 22, 2016 at 11:37:53AM +0800, Chao Yu wrote:
> Hi Jaegeuk,
> 
> > -----Original Message-----
> > From: Jaegeuk Kim [mailto:jaegeuk@kernel.org]
> > Sent: Tuesday, March 22, 2016 12:04 AM
> > To: Marc Lehmann
> > Cc: Chao Yu; linux-f2fs-devel@lists.sourceforge.net
> > Subject: Re: [f2fs-dev] f2fs bug: Unable to mount big volumes in kernel 4.5
> > 
> > On Mon, Mar 21, 2016 at 11:30:38AM +0100, Marc Lehmann wrote:
> > > On Mon, Mar 21, 2016 at 11:18:35AM +0800, Chao Yu <chao2.yu@samsung.com> wrote:
> > > > > As you pointed out, it seems there is a bug in sanity check routine, which
> > > > > doesn't cover the large section case.
> > > >
> > > > Actually, there is a bug in f2fs-tools 1.6.0, it will trigger sanity check failure
> > > > in f2fs kernel module since in mkfs.f2fs we will align segment_count and
> > > > segment_count_main with different size if parameter -s or -z is configured larger
> > > > than 1.
> > > >
> > > > Following commit in dev branch of f2fs-tools has fixed this issue, could you test this
> > > > patch firstly?
> > > > ("mkfs.f2fs: set segment_count in super block correctly")
> > >
> > > Stupid question from my side, does that mean we have to reformat existing
> > > volumes? Because mkfs clearly won't fix existing volumes, so fixing mkfs
> > > will not fix the issue.
> > 
> > Exactly.
> > The f2fs patch should be merged to take into account such the backward
> > compatibility.
> 
> Agreed.
> 
> > And, I don't think wrong segment_count hurts the f2fs behavior.
> 
> Now, we use segment_count to check our last blkaddr, if we ignore correctness
> of segment_count, our boundary checked should be wrong.
> 
> So how about enabling fsck.f2fs ability of repairing incorrect value of
> segment_count, and for f2fs module, warn our user to run fsck if we detect
> such error?

Got it.
IMO, however, it'd good to fix the wrong counts by calling commit_super at the
mount time like this.

Thanks,

>From a23eaeb33076cb97f3200af81bac4c75add63a7d Mon Sep 17 00:00:00 2001
From: Jaegeuk Kim <jaegeuk@kernel.org>
Date: Sun, 20 Mar 2016 15:33:20 -0700
Subject: [PATCH] f2fs: cover large section in sanity check of super

This patch fixes the bug which does not cover a large section case when checking
the sanity of superblock.
If f2fs detects misalignment, it will fix the superblock during the mount time,
so it doesn't need to trigger fsck.f2fs further.

Reported-by: Matthias Prager <linux@matthiasprager.de>
Reported-by: David Gnedt <david.gnedt@davizone.at>
Cc: stable 4.5+ <stable@vger.kernel.org>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
---
 fs/f2fs/super.c | 96 +++++++++++++++++++++++++++++++++++----------------------
 1 file changed, 59 insertions(+), 37 deletions(-)

diff --git a/fs/f2fs/super.c b/fs/f2fs/super.c
index 15bb81f..c88c6c9 100644
--- a/fs/f2fs/super.c
+++ b/fs/f2fs/super.c
@@ -984,9 +984,25 @@ static loff_t max_file_blocks(void)
 	return result;
 }
 
+static int __f2fs_commit_super(struct buffer_head *bh,
+			struct f2fs_super_block *super)
+{
+	lock_buffer(bh);
+	if (super)
+		memcpy(bh->b_data + F2FS_SUPER_OFFSET, super, sizeof(*super));
+	set_buffer_uptodate(bh);
+	set_buffer_dirty(bh);
+	unlock_buffer(bh);
+
+	/* it's rare case, we can do fua all the time */
+	return __sync_dirty_buffer(bh, WRITE_FLUSH_FUA);
+}
+
 static inline bool sanity_check_area_boundary(struct super_block *sb,
-					struct f2fs_super_block *raw_super)
+					struct buffer_head *bh)
 {
+	struct f2fs_super_block *raw_super = (struct f2fs_super_block *)
+					(bh->b_data + F2FS_SUPER_OFFSET);
 	u32 segment0_blkaddr = le32_to_cpu(raw_super->segment0_blkaddr);
 	u32 cp_blkaddr = le32_to_cpu(raw_super->cp_blkaddr);
 	u32 sit_blkaddr = le32_to_cpu(raw_super->sit_blkaddr);
@@ -1000,6 +1016,10 @@ static inline bool sanity_check_area_boundary(struct super_block *sb,
 	u32 segment_count_main = le32_to_cpu(raw_super->segment_count_main);
 	u32 segment_count = le32_to_cpu(raw_super->segment_count);
 	u32 log_blocks_per_seg = le32_to_cpu(raw_super->log_blocks_per_seg);
+	u64 main_end_blkaddr = main_blkaddr +
+				(segment_count_main << log_blocks_per_seg);
+	u64 seg_end_blkaddr = segment0_blkaddr +
+				(segment_count << log_blocks_per_seg);
 
 	if (segment0_blkaddr != cp_blkaddr) {
 		f2fs_msg(sb, KERN_INFO,
@@ -1044,22 +1064,39 @@ static inline bool sanity_check_area_boundary(struct super_block *sb,
 		return true;
 	}
 
-	if (main_blkaddr + (segment_count_main << log_blocks_per_seg) !=
-		segment0_blkaddr + (segment_count << log_blocks_per_seg)) {
+	if (main_end_blkaddr > seg_end_blkaddr) {
 		f2fs_msg(sb, KERN_INFO,
-			"Wrong MAIN_AREA boundary, start(%u) end(%u) blocks(%u)",
+			"Wrong MAIN_AREA boundary, start(%u) end(%u) block(%u)",
 			main_blkaddr,
-			segment0_blkaddr + (segment_count << log_blocks_per_seg),
+			segment0_blkaddr +
+				(segment_count << log_blocks_per_seg),
 			segment_count_main << log_blocks_per_seg);
 		return true;
-	}
+	} else if (main_end_blkaddr < seg_end_blkaddr) {
+		int err;
 
+		raw_super->segment_count = cpu_to_le32((main_end_blkaddr -
+				segment0_blkaddr) >> log_blocks_per_seg);
+
+		err = __f2fs_commit_super(bh, NULL);
+		f2fs_msg(sb, KERN_INFO,
+			"Fix alignment : %s, start(%u) end(%u) block(%u)",
+			err ? "failed": "done",
+			main_blkaddr,
+			segment0_blkaddr +
+				(segment_count << log_blocks_per_seg),
+			segment_count_main << log_blocks_per_seg);
+		if (err)
+			return true;
+	}
 	return false;
 }
 
 static int sanity_check_raw_super(struct super_block *sb,
-			struct f2fs_super_block *raw_super)
+				struct buffer_head *bh)
 {
+	struct f2fs_super_block *raw_super = (struct f2fs_super_block *)
+					(bh->b_data + F2FS_SUPER_OFFSET);
 	unsigned int blocksize;
 
 	if (F2FS_SUPER_MAGIC != le32_to_cpu(raw_super->magic)) {
@@ -1126,7 +1163,7 @@ static int sanity_check_raw_super(struct super_block *sb,
 	}
 
 	/* check CP/SIT/NAT/SSA/MAIN_AREA area boundary */
-	if (sanity_check_area_boundary(sb, raw_super))
+	if (sanity_check_area_boundary(sb, bh))
 		return 1;
 
 	return 0;
@@ -1202,7 +1239,7 @@ static int read_raw_super_block(struct super_block *sb,
 {
 	int block;
 	struct buffer_head *bh;
-	struct f2fs_super_block *super, *buf;
+	struct f2fs_super_block *super;
 	int err = 0;
 
 	super = kzalloc(sizeof(struct f2fs_super_block), GFP_KERNEL);
@@ -1218,11 +1255,8 @@ static int read_raw_super_block(struct super_block *sb,
 			continue;
 		}
 
-		buf = (struct f2fs_super_block *)
-				(bh->b_data + F2FS_SUPER_OFFSET);
-
 		/* sanity checking of raw super */
-		if (sanity_check_raw_super(sb, buf)) {
+		if (sanity_check_raw_super(sb, bh)) {
 			f2fs_msg(sb, KERN_ERR,
 				"Can't find valid F2FS filesystem in %dth superblock",
 				block + 1);
@@ -1232,7 +1266,8 @@ static int read_raw_super_block(struct super_block *sb,
 		}
 
 		if (!*raw_super) {
-			memcpy(super, buf, sizeof(*super));
+			memcpy(super, bh->b_data + F2FS_SUPER_OFFSET,
+							sizeof(*super));
 			*valid_super_block = block;
 			*raw_super = super;
 		}
@@ -1252,42 +1287,29 @@ static int read_raw_super_block(struct super_block *sb,
 	return err;
 }
 
-static int __f2fs_commit_super(struct f2fs_sb_info *sbi, int block)
+int f2fs_commit_super(struct f2fs_sb_info *sbi, bool recover)
 {
-	struct f2fs_super_block *super = F2FS_RAW_SUPER(sbi);
 	struct buffer_head *bh;
 	int err;
 
-	bh = sb_getblk(sbi->sb, block);
+	/* write back-up superblock first */
+	bh = sb_getblk(sbi->sb, sbi->valid_super_block ? 0: 1);
 	if (!bh)
 		return -EIO;
-
-	lock_buffer(bh);
-	memcpy(bh->b_data + F2FS_SUPER_OFFSET, super, sizeof(*super));
-	set_buffer_uptodate(bh);
-	set_buffer_dirty(bh);
-	unlock_buffer(bh);
-
-	/* it's rare case, we can do fua all the time */
-	err = __sync_dirty_buffer(bh, WRITE_FLUSH_FUA);
+	err = __f2fs_commit_super(bh, F2FS_RAW_SUPER(sbi));
 	brelse(bh);
 
-	return err;
-}
-
-int f2fs_commit_super(struct f2fs_sb_info *sbi, bool recover)
-{
-	int err;
-
-	/* write back-up superblock first */
-	err = __f2fs_commit_super(sbi, sbi->valid_super_block ? 0 : 1);
-
 	/* if we are in recovery path, skip writing valid superblock */
 	if (recover || err)
 		return err;
 
 	/* write current valid superblock */
-	return __f2fs_commit_super(sbi, sbi->valid_super_block);
+	bh = sb_getblk(sbi->sb, sbi->valid_super_block);
+	if (!bh)
+		return -EIO;
+	err = __f2fs_commit_super(bh, F2FS_RAW_SUPER(sbi));
+	brelse(bh);
+	return err;
 }
 
 static int f2fs_fill_super(struct super_block *sb, void *data, int silent)
-- 
2.6.3


------------------------------------------------------------------------------
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://pubads.g.doubleclick.net/gampad/clk?id=278785351&iu=/4140

Re: f2fs bug: Unable to mount big volumes in kernel 4.5

[Matthias Prager]

Am 22.03.2016 um 21:36 schrieb Jaegeuk Kim:
> Hi,
> 
> On Mon, Mar 21, 2016 at 11:56:02PM +0100, Matthias Prager wrote:
>> Hello everyone,
>>
>> if there really is a suboptimal/faulty alignment being done by
>> f2fs-tools <=1.6.0 (and I haven't done the math, so I'm just assuming),
>> wouldn't it be prudent to include a modified kernel patch, which allows
>> for mounting of older filesystems, but puts out a warning in dmesg which
>> advises the user to run fsck.f2fs some time in the future?
> 
> So, could you check the patch that I replied or the same patch that I attached
> here?
> That should dynamically fix this issue when mounting f2fs.
I will do so tomorrow and report back.

> 
> Thanks,
> 
>>
>> ---
>> Matthias
>>
>> Am 21.03.2016 um 21:58 schrieb David Gnedt:
>>> Hello,
>>>
>>>> Following commit in dev branch of f2fs-tools has fixed this issue, could you
>>> test this
>>>> patch firstly?
>>>> ("mkfs.f2fs: set segment_count in super block correctly")
>>>
>>> Sorry, I cannot recreate the volume on the 8TB Seagate drive right now, but I
>>> did a test with a loopback device of equal size.
>>>
>>> Should I be worried about the changes in mkfs.f2fs since 1.6.0? Would it be
>>> suggested to recreate the filesystem? Or is it maybe possible to manually fix it?
>>> If it is really only the alignment, I think it shouldn't matter for SMR drives,
>>> as they are not using constant size zones anyway, so misalignment cannot be
>>> avoided with f2fs.
>>>
>>>>> Could you test the attached patch?
>>>
>>> I did a roundup test of all kernel/f2fs-tools version I thought that make sense.
>>> I hope the patch will be included in upcoming mainline kernels.
>>>
>>> Short summary:
>>> --------------
>>> Filesystems created with a recent dev version of f2fs-tools work without any
>>> problems and don't need the f2fs kernel patch.
>>> Filesystems created with older f2fs-tools need the f2fs kernel patch to mount
>>> correctly.
>>> I guess that is exactly what everyone expected to be the outcome.
>>>
>>> Full details:
>>> -------------
>>> f2fs-tools D:  release 1.6.0 (Seagate SMR drive)
>>> f2fs-tools L1: release 1.6.0 (Loopback)
>>> f2fs-tools L2: 2016-01-08 mkfs.f2fs: introduce zone align for main area (Loopback)
>>> f2fs-tools L3: 2016-03-16 mkfs.f2fs: set segment_count in super block correctly
>>> (Loopback)
>>>
>>> Test procedure:
>>> $ truncate -s 8001559724032 f2fs-tools-1.6.0.img
>>> $ losetup -f f2fs-tools-1.6.0.img
>>> $ mkfs.f2fs -s64 -t0 -a0 /dev/loop0
>>> $ mount -t f2fs -onoinline_data,noatime,flush_merge,no_heap,ro /dev/loop0 /mnt
>>> $ umount /mnt
>>>
>>> Kernel: Linux 4.4.6 x86_64
>>> f2fs-tools D:  Works
>>> f2fs-tools L1: Works
>>> f2fs-tools L2: Works
>>> f2fs-tools L3: Works
>>>
>>> Kernel: Linux 4.5.0 x86_64
>>> f2fs-tools D:  Failed -> Same errors as before
>>> f2fs-tools L1: Failed -> Same errors as before
>>> f2fs-tools L2: Failed -> Same errors as before
>>> f2fs-tools L3: Works
>>>
>>> Kernel: Linux 4.5.0+patch x86_64
>>> f2fs-tools D:  Works
>>> f2fs-tools L1: Works
>>> f2fs-tools L2: Works
>>> f2fs-tools L3: Works
>>>
>>> Best regards,
>>> David Gnedt
>>>


------------------------------------------------------------------------------
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://pubads.g.doubleclick.net/gampad/clk?id=278785351&iu=/4140

Re: f2fs bug: Unable to mount big volumes in kernel 4.5

[Jaegeuk Kim]

Hi all,

Just to note that, I decided to release v1.6.1, since I realized that
"fsck.f2fs -a" gives segfault which is fixed by the below patch in the
dev branch.

http://git.kernel.org/cgit/linux/kernel/git/jaegeuk/f2fs-tools.git/commit/?h=dev&id=231fc70f6e9d57658393911c705346b061f23513

Thanks,

On Mon, Mar 21, 2016 at 11:56:02PM +0100, Matthias Prager wrote:
> Hello everyone,
> 
> if there really is a suboptimal/faulty alignment being done by
> f2fs-tools <=1.6.0 (and I haven't done the math, so I'm just assuming),
> wouldn't it be prudent to include a modified kernel patch, which allows
> for mounting of older filesystems, but puts out a warning in dmesg which
> advises the user to run fsck.f2fs some time in the future?
> 
> ---
> Matthias
> 
> Am 21.03.2016 um 21:58 schrieb David Gnedt:
> > Hello,
> > 
> >> Following commit in dev branch of f2fs-tools has fixed this issue, could you
> > test this
> >> patch firstly?
> >> ("mkfs.f2fs: set segment_count in super block correctly")
> > 
> > Sorry, I cannot recreate the volume on the 8TB Seagate drive right now, but I
> > did a test with a loopback device of equal size.
> > 
> > Should I be worried about the changes in mkfs.f2fs since 1.6.0? Would it be
> > suggested to recreate the filesystem? Or is it maybe possible to manually fix it?
> > If it is really only the alignment, I think it shouldn't matter for SMR drives,
> > as they are not using constant size zones anyway, so misalignment cannot be
> > avoided with f2fs.
> > 
> >>> Could you test the attached patch?
> > 
> > I did a roundup test of all kernel/f2fs-tools version I thought that make sense.
> > I hope the patch will be included in upcoming mainline kernels.
> > 
> > Short summary:
> > --------------
> > Filesystems created with a recent dev version of f2fs-tools work without any
> > problems and don't need the f2fs kernel patch.
> > Filesystems created with older f2fs-tools need the f2fs kernel patch to mount
> > correctly.
> > I guess that is exactly what everyone expected to be the outcome.
> > 
> > Full details:
> > -------------
> > f2fs-tools D:  release 1.6.0 (Seagate SMR drive)
> > f2fs-tools L1: release 1.6.0 (Loopback)
> > f2fs-tools L2: 2016-01-08 mkfs.f2fs: introduce zone align for main area (Loopback)
> > f2fs-tools L3: 2016-03-16 mkfs.f2fs: set segment_count in super block correctly
> > (Loopback)
> > 
> > Test procedure:
> > $ truncate -s 8001559724032 f2fs-tools-1.6.0.img
> > $ losetup -f f2fs-tools-1.6.0.img
> > $ mkfs.f2fs -s64 -t0 -a0 /dev/loop0
> > $ mount -t f2fs -onoinline_data,noatime,flush_merge,no_heap,ro /dev/loop0 /mnt
> > $ umount /mnt
> > 
> > Kernel: Linux 4.4.6 x86_64
> > f2fs-tools D:  Works
> > f2fs-tools L1: Works
> > f2fs-tools L2: Works
> > f2fs-tools L3: Works
> > 
> > Kernel: Linux 4.5.0 x86_64
> > f2fs-tools D:  Failed -> Same errors as before
> > f2fs-tools L1: Failed -> Same errors as before
> > f2fs-tools L2: Failed -> Same errors as before
> > f2fs-tools L3: Works
> > 
> > Kernel: Linux 4.5.0+patch x86_64
> > f2fs-tools D:  Works
> > f2fs-tools L1: Works
> > f2fs-tools L2: Works
> > f2fs-tools L3: Works
> > 
> > Best regards,
> > David Gnedt
> > 

------------------------------------------------------------------------------
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://pubads.g.doubleclick.net/gampad/clk?id=278785351&iu=/4140

Re: f2fs bug: Unable to mount big volumes in kernel 4.5

[Jaegeuk Kim]

On Tue, Mar 22, 2016 at 09:50:47PM +0100, Matthias Prager wrote:
> Am 22.03.2016 um 21:36 schrieb Jaegeuk Kim:
> > Hi,
> > 
> > On Mon, Mar 21, 2016 at 11:56:02PM +0100, Matthias Prager wrote:
> >> Hello everyone,
> >>
> >> if there really is a suboptimal/faulty alignment being done by
> >> f2fs-tools <=1.6.0 (and I haven't done the math, so I'm just assuming),
> >> wouldn't it be prudent to include a modified kernel patch, which allows
> >> for mounting of older filesystems, but puts out a warning in dmesg which
> >> advises the user to run fsck.f2fs some time in the future?
> > 
> > So, could you check the patch that I replied or the same patch that I attached
> > here?
> > That should dynamically fix this issue when mounting f2fs.
> I will do so tomorrow and report back.

Thank you~!

> 
> > 
> > Thanks,
> > 
> >>
> >> ---
> >> Matthias
> >>
> >> Am 21.03.2016 um 21:58 schrieb David Gnedt:
> >>> Hello,
> >>>
> >>>> Following commit in dev branch of f2fs-tools has fixed this issue, could you
> >>> test this
> >>>> patch firstly?
> >>>> ("mkfs.f2fs: set segment_count in super block correctly")
> >>>
> >>> Sorry, I cannot recreate the volume on the 8TB Seagate drive right now, but I
> >>> did a test with a loopback device of equal size.
> >>>
> >>> Should I be worried about the changes in mkfs.f2fs since 1.6.0? Would it be
> >>> suggested to recreate the filesystem? Or is it maybe possible to manually fix it?
> >>> If it is really only the alignment, I think it shouldn't matter for SMR drives,
> >>> as they are not using constant size zones anyway, so misalignment cannot be
> >>> avoided with f2fs.
> >>>
> >>>>> Could you test the attached patch?
> >>>
> >>> I did a roundup test of all kernel/f2fs-tools version I thought that make sense.
> >>> I hope the patch will be included in upcoming mainline kernels.
> >>>
> >>> Short summary:
> >>> --------------
> >>> Filesystems created with a recent dev version of f2fs-tools work without any
> >>> problems and don't need the f2fs kernel patch.
> >>> Filesystems created with older f2fs-tools need the f2fs kernel patch to mount
> >>> correctly.
> >>> I guess that is exactly what everyone expected to be the outcome.
> >>>
> >>> Full details:
> >>> -------------
> >>> f2fs-tools D:  release 1.6.0 (Seagate SMR drive)
> >>> f2fs-tools L1: release 1.6.0 (Loopback)
> >>> f2fs-tools L2: 2016-01-08 mkfs.f2fs: introduce zone align for main area (Loopback)
> >>> f2fs-tools L3: 2016-03-16 mkfs.f2fs: set segment_count in super block correctly
> >>> (Loopback)
> >>>
> >>> Test procedure:
> >>> $ truncate -s 8001559724032 f2fs-tools-1.6.0.img
> >>> $ losetup -f f2fs-tools-1.6.0.img
> >>> $ mkfs.f2fs -s64 -t0 -a0 /dev/loop0
> >>> $ mount -t f2fs -onoinline_data,noatime,flush_merge,no_heap,ro /dev/loop0 /mnt
> >>> $ umount /mnt
> >>>
> >>> Kernel: Linux 4.4.6 x86_64
> >>> f2fs-tools D:  Works
> >>> f2fs-tools L1: Works
> >>> f2fs-tools L2: Works
> >>> f2fs-tools L3: Works
> >>>
> >>> Kernel: Linux 4.5.0 x86_64
> >>> f2fs-tools D:  Failed -> Same errors as before
> >>> f2fs-tools L1: Failed -> Same errors as before
> >>> f2fs-tools L2: Failed -> Same errors as before
> >>> f2fs-tools L3: Works
> >>>
> >>> Kernel: Linux 4.5.0+patch x86_64
> >>> f2fs-tools D:  Works
> >>> f2fs-tools L1: Works
> >>> f2fs-tools L2: Works
> >>> f2fs-tools L3: Works
> >>>
> >>> Best regards,
> >>> David Gnedt
> >>>

------------------------------------------------------------------------------
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://pubads.g.doubleclick.net/gampad/clk?id=278785351&iu=/4140

Re: f2fs bug: Unable to mount big volumes in kernel 4.5

[Matthias Prager]

Am 23.03.2016 um 13:54 schrieb David Gnedt:
> Hello,
> 
> On 2016-03-22 21:36, Jaegeuk Kim wrote:
>> So, could you check the patch that I replied or the same patch that I attached
>> here?
>> That should dynamically fix this issue when mounting f2fs.
> 
> I had to manually apply the patch on 4.5 sources. I have attached the modified
> patch. If you want, I can also test based on the recent f2fs kernel tree.
> 
> I have used a similar setup as in my last test. The new patch fixes the old
> volumes as expected:
> [144530.446411] F2FS-fs (loop0): Fix alignment : done, start(4063232)
> end(1953505792) block(1949433856)
> [144530.447700] F2FS-fs (loop0): Fix alignment : done, start(4063232)
> end(1953505792) block(1949433856)
> 
> Additionally, the fixed volumes mount without problems on all kernel versions I
> tested: 4.5.0 x86_64, 4.5.0+patchv2 x86_64
> I haven't tried 4.4.6 x86_64 yet, but if you want I can do that later today.
> 
> The only thing that might not be as expected is that the alignment is also fixed
> when the volume is mounted read-only.
I will get to my tests later today, but wanted to comment on this
detail. Writing on a read-only fs is a no go! There usually is a reason
why someone mounts an filesystem read-only and f2fs should not simply
ignore such a flag.

> 
> Best regards,
> David Gnedt
> 


------------------------------------------------------------------------------
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://pubads.g.doubleclick.net/gampad/clk?id=278785351&iu=/4140

Re: f2fs bug: Unable to mount big volumes in kernel 4.5

[Marc Lehmann]

On Wed, Mar 23, 2016 at 05:41:57PM +0100, Matthias Prager <linux@matthiasprager.de> wrote:
> detail. Writing on a read-only fs is a no go! There usually is a reason
> why someone mounts an filesystem read-only and f2fs should not simply
> ignore such a flag.

Hmm, no.

Even an unusual case is enough, such as mounting it read-only for data
recovery because the underlying device dies on writes (quite likely for
ssds, the primary target for f2fs).

However, other linux filesystems either replay their journal on readonly
mounts, or even *require* a replay (most have a flag to prevent that,
but that might not help data recovery). In fact, my example above can
relatively easily worked around using device mapper and a temporary
snapshot volume for writes.

So, the "ro" mount flag in linux does not mean "do not write to the
backing store", and did not have that meaning for a long time, so it's
fine for f2fs to write even for ro mounts.

That means the correct behaviour for f2fs is to write unless "norecovery"
has been specified, which already exists for this purpose on f2fs.

Different behaviour would put f2fs at odds with other existing
filesystems, which write on "ro" mounts to gain integrity.

Even better would be if there was a "force" or similar option which would
allow me to mount filesystems with possibly bad superblock data. This
could even be rolled into the existing "norecovery" switch, which, when
given, could try to mount even if the superblock has (some amount of) bad
data.

-- 
                The choice of a       Deliantra, the free code+content MORPG
      -----==-     _GNU_              http://www.deliantra.net
      ----==-- _       generation
      ---==---(_)__  __ ____  __      Marc Lehmann
      --==---/ / _ \/ // /\ \/ /      schmorp@schmorp.de
      -=====/_/_//_/\_,_/ /_/\_\

------------------------------------------------------------------------------
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://pubads.g.doubleclick.net/gampad/clk?id=278785351&iu=/4140

Re: f2fs bug: Unable to mount big volumes in kernel 4.5

[Jaegeuk Kim]

On Wed, Mar 23, 2016 at 10:00:10PM +0100, Marc Lehmann wrote:
> On Wed, Mar 23, 2016 at 05:41:57PM +0100, Matthias Prager <linux@matthiasprager.de> wrote:
> > detail. Writing on a read-only fs is a no go! There usually is a reason
> > why someone mounts an filesystem read-only and f2fs should not simply
> > ignore such a flag.
> 
> Hmm, no.
> 
> Even an unusual case is enough, such as mounting it read-only for data
> recovery because the underlying device dies on writes (quite likely for
> ssds, the primary target for f2fs).
> 
> However, other linux filesystems either replay their journal on readonly
> mounts, or even *require* a replay (most have a flag to prevent that,
> but that might not help data recovery). In fact, my example above can
> relatively easily worked around using device mapper and a temporary
> snapshot volume for writes.
> 
> So, the "ro" mount flag in linux does not mean "do not write to the
> backing store", and did not have that meaning for a long time, so it's
> fine for f2fs to write even for ro mounts.
> 
> That means the correct behaviour for f2fs is to write unless "norecovery"
> has been specified, which already exists for this purpose on f2fs.

Well. IMO, we have to deal with fixing inconsistency and doing recovery
separately.

In terms of recovery, I agree that we must do it no matter how fs is mounted.
The only missing part in f2fs is that, we need to throw an error, if there is
something to recover under read-write and norecovery mount likewise ext4.

Regarding to fixing inconsistency dynamically, it seems depending on filesystems
according to how severely it can corrupt the underlying on-disk layout.
For example, I could see that btrfs conducts btrfs_recover_relocation only
when read-write mount, while doing btrfs_replay_log all the time.

Back to our cases, I think fixing superblock just depends on our policy.
Initially, I added f2fs_commit_super to recover the broken superblock
optionally, since we have no problem to mount with the alternate valid
superblock.

In the case of misaligned end address, it also doesn't need to fix it at
read-only mount, since it has nothing to do with read operations.
That is only used to check the boundary bug when writing blocks.

Lastly, when considerting RO to RW case as you pointed out, I think we need to
handle that in f2fs_remount by adding a superblock flag likewise ext4.

Let me submit some patches for them.

Thanks,

> Different behaviour would put f2fs at odds with other existing
> filesystems, which write on "ro" mounts to gain integrity.
> 
> Even better would be if there was a "force" or similar option which would
> allow me to mount filesystems with possibly bad superblock data. This
> could even be rolled into the existing "norecovery" switch, which, when
> given, could try to mount even if the superblock has (some amount of) bad
> data.
> 
> -- 
>                 The choice of a       Deliantra, the free code+content MORPG
>       -----==-     _GNU_              http://www.deliantra.net
>       ----==-- _       generation
>       ---==---(_)__  __ ____  __      Marc Lehmann
>       --==---/ / _ \/ // /\ \/ /      schmorp@schmorp.de
>       -=====/_/_//_/\_,_/ /_/\_\

------------------------------------------------------------------------------
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://pubads.g.doubleclick.net/gampad/clk?id=278785351&iu=/4140