Multidevice f2fs mount after disk rearrangement

Multidevice f2fs mount after disk rearrangement

[Oleksandr Natalenko]

Hello.

I was brave enough to create f2fs filesystem spanning through 2 physical 
device using this command:

# mkfs.f2fs -t 0 /dev/sdc -c /dev/sdd

It worked fine until I removed /dev/sdb from my system, so f2fs devices 
became:

/dev/sdc -> /dev/sdb
/dev/sdd -> /dev/sdc

Now, when I try to mount it, I get the following:

# mount -t f2fs /dev/sdb /mnt/fs
mount: /mnt/fs: mount(2) system call failed: No such file or directory.

In dmesg:

[Jan 4 17:25] F2FS-fs (sdb): Mount Device [ 0]:             /dev/sdc,    
59063,        0 -  1cd6fff
[  +0,000024] F2FS-fs (sdb): Failed to find devices

fsck also fails with the following assertion:

[ASSERT] (init_sb_info: 908) !strcmp((char *)sb->devs[i].path, (char 
*)c.devices[i].path)

Am I doing something obviously stupid, and the device path can be 
(somehow) changed so that the mount succeeds, or this is unfixable, and 
f2fs relies on persistent device naming?

Please suggest.

Thank you.

-- 
   Oleksandr Natalenko (post-factum)

Re: Multidevice f2fs mount after disk rearrangement

[Oleksandr Natalenko]

Hi.

On 04.01.2020 17:29, Oleksandr Natalenko wrote:
> I was brave enough to create f2fs filesystem spanning through 2
> physical device using this command:
> 
> # mkfs.f2fs -t 0 /dev/sdc -c /dev/sdd
> 
> It worked fine until I removed /dev/sdb from my system, so f2fs devices 
> became:
> 
> /dev/sdc -> /dev/sdb
> /dev/sdd -> /dev/sdc
> 
> Now, when I try to mount it, I get the following:
> 
> # mount -t f2fs /dev/sdb /mnt/fs
> mount: /mnt/fs: mount(2) system call failed: No such file or directory.
> 
> In dmesg:
> 
> [Jan 4 17:25] F2FS-fs (sdb): Mount Device [ 0]:             /dev/sdc,
>   59063,        0 -  1cd6fff
> [  +0,000024] F2FS-fs (sdb): Failed to find devices
> 
> fsck also fails with the following assertion:
> 
> [ASSERT] (init_sb_info: 908) !strcmp((char *)sb->devs[i].path, (char
> *)c.devices[i].path)
> 
> Am I doing something obviously stupid, and the device path can be
> (somehow) changed so that the mount succeeds, or this is unfixable,
> and f2fs relies on persistent device naming?
> 
> Please suggest.
> 
> Thank you.

Erm, fine. I studied f2fs-tools code a little bit and discovered that 
superblock indeed had /dev/sdX paths saved as strings. So I fired up 
hexedit and just changed the superblock directly on the first device, 
substituting sdc with sdb and sdd with sdc (I did it twice; I guess 
there are 2 copies of superblock), and after this the mount worked.

Am I really supposed to do this manually ;)?

-- 
   Oleksandr Natalenko (post-factum)

Re: [f2fs-dev] Multidevice f2fs mount after disk rearrangement

[Chao Yu]

Hello,

Thanks for the report. :)

On 2020/1/5 5:52, Oleksandr Natalenko via Linux-f2fs-devel wrote:
> Hi.
> 
> On 04.01.2020 17:29, Oleksandr Natalenko wrote:
>> I was brave enough to create f2fs filesystem spanning through 2
>> physical device using this command:
>>
>> # mkfs.f2fs -t 0 /dev/sdc -c /dev/sdd
>>
>> It worked fine until I removed /dev/sdb from my system, so f2fs devices 
>> became:
>>
>> /dev/sdc -> /dev/sdb
>> /dev/sdd -> /dev/sdc
>>
>> Now, when I try to mount it, I get the following:
>>
>> # mount -t f2fs /dev/sdb /mnt/fs
>> mount: /mnt/fs: mount(2) system call failed: No such file or directory.
>>
>> In dmesg:
>>
>> [Jan 4 17:25] F2FS-fs (sdb): Mount Device [ 0]:             /dev/sdc,
>>   59063,        0 -  1cd6fff
>> [  +0,000024] F2FS-fs (sdb): Failed to find devices
>>
>> fsck also fails with the following assertion:
>>
>> [ASSERT] (init_sb_info: 908) !strcmp((char *)sb->devs[i].path, (char
>> *)c.devices[i].path)
>>
>> Am I doing something obviously stupid, and the device path can be
>> (somehow) changed so that the mount succeeds, or this is unfixable,
>> and f2fs relies on persistent device naming?
>>
>> Please suggest.
>>
>> Thank you.
> 
> Erm, fine. I studied f2fs-tools code a little bit and discovered that 
> superblock indeed had /dev/sdX paths saved as strings. So I fired up 
> hexedit and just changed the superblock directly on the first device, 
> substituting sdc with sdb and sdd with sdc (I did it twice; I guess 
> there are 2 copies of superblock), and after this the mount worked.

Alright, it works if superblock checksum feature is off...

> 
> Am I really supposed to do this manually ;)?

We'd better add that ability in tune.f2fs. And I guess we need to let
kernel/fsck to notice that case, and give hint to run tune.f2fs to
reconfigure primary/secondary/... device paths.

Thanks,

> 

Re: Multidevice f2fs mount after disk rearrangement

[Jaegeuk Kim]

On 01/04, Oleksandr Natalenko wrote:
> Hi.
> 
> On 04.01.2020 17:29, Oleksandr Natalenko wrote:
> > I was brave enough to create f2fs filesystem spanning through 2
> > physical device using this command:
> > 
> > # mkfs.f2fs -t 0 /dev/sdc -c /dev/sdd
> > 
> > It worked fine until I removed /dev/sdb from my system, so f2fs devices
> > became:
> > 
> > /dev/sdc -> /dev/sdb
> > /dev/sdd -> /dev/sdc
> > 
> > Now, when I try to mount it, I get the following:
> > 
> > # mount -t f2fs /dev/sdb /mnt/fs
> > mount: /mnt/fs: mount(2) system call failed: No such file or directory.
> > 
> > In dmesg:
> > 
> > [Jan 4 17:25] F2FS-fs (sdb): Mount Device [ 0]:             /dev/sdc,
> >   59063,        0 -  1cd6fff
> > [  +0,000024] F2FS-fs (sdb): Failed to find devices
> > 
> > fsck also fails with the following assertion:
> > 
> > [ASSERT] (init_sb_info: 908) !strcmp((char *)sb->devs[i].path, (char
> > *)c.devices[i].path)
> > 
> > Am I doing something obviously stupid, and the device path can be
> > (somehow) changed so that the mount succeeds, or this is unfixable,
> > and f2fs relies on persistent device naming?
> > 
> > Please suggest.
> > 
> > Thank you.
> 
> Erm, fine. I studied f2fs-tools code a little bit and discovered that
> superblock indeed had /dev/sdX paths saved as strings. So I fired up hexedit
> and just changed the superblock directly on the first device, substituting
> sdc with sdb and sdd with sdc (I did it twice; I guess there are 2 copies of
> superblock), and after this the mount worked.
> 
> Am I really supposed to do this manually ;)?

Thank you for investigating this ahead of me. :) Yes, the device list is stored
in superblock, so hacking it manually should work.

Let me think about a tool to tune that.

Thanks,

> 
> -- 
>   Oleksandr Natalenko (post-factum)

Re: [f2fs-dev] Multidevice f2fs mount after disk rearrangement

[Jaegeuk Kim]

On 01/06, Chao Yu wrote:
> Hello,
> 
> Thanks for the report. :)
> 
> On 2020/1/5 5:52, Oleksandr Natalenko via Linux-f2fs-devel wrote:
> > Hi.
> > 
> > On 04.01.2020 17:29, Oleksandr Natalenko wrote:
> >> I was brave enough to create f2fs filesystem spanning through 2
> >> physical device using this command:
> >>
> >> # mkfs.f2fs -t 0 /dev/sdc -c /dev/sdd
> >>
> >> It worked fine until I removed /dev/sdb from my system, so f2fs devices 
> >> became:
> >>
> >> /dev/sdc -> /dev/sdb
> >> /dev/sdd -> /dev/sdc
> >>
> >> Now, when I try to mount it, I get the following:
> >>
> >> # mount -t f2fs /dev/sdb /mnt/fs
> >> mount: /mnt/fs: mount(2) system call failed: No such file or directory.
> >>
> >> In dmesg:
> >>
> >> [Jan 4 17:25] F2FS-fs (sdb): Mount Device [ 0]:             /dev/sdc,
> >>   59063,        0 -  1cd6fff
> >> [  +0,000024] F2FS-fs (sdb): Failed to find devices
> >>
> >> fsck also fails with the following assertion:
> >>
> >> [ASSERT] (init_sb_info: 908) !strcmp((char *)sb->devs[i].path, (char
> >> *)c.devices[i].path)
> >>
> >> Am I doing something obviously stupid, and the device path can be
> >> (somehow) changed so that the mount succeeds, or this is unfixable,
> >> and f2fs relies on persistent device naming?
> >>
> >> Please suggest.
> >>
> >> Thank you.
> > 
> > Erm, fine. I studied f2fs-tools code a little bit and discovered that 
> > superblock indeed had /dev/sdX paths saved as strings. So I fired up 
> > hexedit and just changed the superblock directly on the first device, 
> > substituting sdc with sdb and sdd with sdc (I did it twice; I guess 
> > there are 2 copies of superblock), and after this the mount worked.
> 
> Alright, it works if superblock checksum feature is off...
> 
> > 
> > Am I really supposed to do this manually ;)?
> 
> We'd better add that ability in tune.f2fs. And I guess we need to let
> kernel/fsck to notice that case, and give hint to run tune.f2fs to
> reconfigure primary/secondary/... device paths.

I'm thinking to add tunesb.f2fs to edit superblock explicitly, since it has
to edit it without getting superblock/checkpoint and other f2fs metadata.

For example,
# tunesb.f2fs -c /dev/sdb -c /dev/sdc /dev/sda
.. superblock info ..
.. device list ..
.. hot/cold extensions ..

Will modify the device list, if it's different from parameter.

> 
> Thanks,
> 
> > 

Re: Multidevice f2fs mount after disk rearrangement

[Oleksandr Natalenko]

Hi.

On 06.01.2020 19:34, Jaegeuk Kim wrote:
> Thank you for investigating this ahead of me. :) Yes, the device list 
> is stored
> in superblock, so hacking it manually should work.
> 
> Let me think about a tool to tune that.

Thank you both for the replies.

IIUC, tune.f2fs is not there yet. I saw a submission, but I do not see 
it as accepted, right?

Having this in tune.f2fs would be fine (assuming the assertion is 
replaced with some meaningful hint message), but wouldn't it be more 
convenient for an ordinary user to have implemented something like:

# mount -t f2fs /dev/sdb -o nextdev=/dev/sdc /mnt/fs

Hm?

-- 
   Oleksandr Natalenko (post-factum)

Re: [f2fs-dev] Multidevice f2fs mount after disk rearrangement

[Chao Yu]

On 2020/1/7 2:40, Jaegeuk Kim wrote:
> On 01/06, Chao Yu wrote:
>> Hello,
>>
>> Thanks for the report. :)
>>
>> On 2020/1/5 5:52, Oleksandr Natalenko via Linux-f2fs-devel wrote:
>>> Hi.
>>>
>>> On 04.01.2020 17:29, Oleksandr Natalenko wrote:
>>>> I was brave enough to create f2fs filesystem spanning through 2
>>>> physical device using this command:
>>>>
>>>> # mkfs.f2fs -t 0 /dev/sdc -c /dev/sdd
>>>>
>>>> It worked fine until I removed /dev/sdb from my system, so f2fs devices 
>>>> became:
>>>>
>>>> /dev/sdc -> /dev/sdb
>>>> /dev/sdd -> /dev/sdc
>>>>
>>>> Now, when I try to mount it, I get the following:
>>>>
>>>> # mount -t f2fs /dev/sdb /mnt/fs
>>>> mount: /mnt/fs: mount(2) system call failed: No such file or directory.
>>>>
>>>> In dmesg:
>>>>
>>>> [Jan 4 17:25] F2FS-fs (sdb): Mount Device [ 0]:             /dev/sdc,
>>>>   59063,        0 -  1cd6fff
>>>> [  +0,000024] F2FS-fs (sdb): Failed to find devices
>>>>
>>>> fsck also fails with the following assertion:
>>>>
>>>> [ASSERT] (init_sb_info: 908) !strcmp((char *)sb->devs[i].path, (char
>>>> *)c.devices[i].path)
>>>>
>>>> Am I doing something obviously stupid, and the device path can be
>>>> (somehow) changed so that the mount succeeds, or this is unfixable,
>>>> and f2fs relies on persistent device naming?
>>>>
>>>> Please suggest.
>>>>
>>>> Thank you.
>>>
>>> Erm, fine. I studied f2fs-tools code a little bit and discovered that 
>>> superblock indeed had /dev/sdX paths saved as strings. So I fired up 
>>> hexedit and just changed the superblock directly on the first device, 
>>> substituting sdc with sdb and sdd with sdc (I did it twice; I guess 
>>> there are 2 copies of superblock), and after this the mount worked.
>>
>> Alright, it works if superblock checksum feature is off...
>>
>>>
>>> Am I really supposed to do this manually ;)?
>>
>> We'd better add that ability in tune.f2fs. And I guess we need to let
>> kernel/fsck to notice that case, and give hint to run tune.f2fs to
>> reconfigure primary/secondary/... device paths.
> 
> I'm thinking to add tunesb.f2fs to edit superblock explicitly, since it has
> to edit it without getting superblock/checkpoint and other f2fs metadata.
> 
> For example,
> # tunesb.f2fs -c /dev/sdb -c /dev/sdc /dev/sda
> .. superblock info ..
> .. device list ..
> .. hot/cold extensions ..
> 
> Will modify the device list, if it's different from parameter.

Looks good to me.

Thanks,

> 
>>
>> Thanks,
>>
>>>
> .
> 

Re: [f2fs-dev] Multidevice f2fs mount after disk rearrangement

[Chao Yu]

On 2020/1/7 2:40, Oleksandr Natalenko via Linux-f2fs-devel wrote:
> Hi.
> 
> On 06.01.2020 19:34, Jaegeuk Kim wrote:
>> Thank you for investigating this ahead of me. :) Yes, the device list 
>> is stored
>> in superblock, so hacking it manually should work.
>>
>> Let me think about a tool to tune that.
> 
> Thank you both for the replies.
> 
> IIUC, tune.f2fs is not there yet. I saw a submission, but I do not see 
> it as accepted, right?
> 
> Having this in tune.f2fs would be fine (assuming the assertion is 
> replaced with some meaningful hint message), but wouldn't it be more 
> convenient for an ordinary user to have implemented something like:
> 
> # mount -t f2fs /dev/sdb -o nextdev=/dev/sdc /mnt/fs

Hmm... sounds reasonable, however, the risk is obvious, if we mount with wrong
primary device, filesystem can be aware that with metadata sanity check, if we
mount with wrong secondary/... devices by mistake (or intentionally, people
may think filesystem should be aware illegal parameters....), filesystem won't
be aware of that, then metadata/data will be inconsistent...

Although that may also happen when we use tunesb.f2fs, but fsck.f2fs can be
followed to verify the modification of tunesb.f2fs, that would be much safer.

So I suggest we can do that in tools first, maybe implement nextdev mount option
if we have added metadata in secondary/... device.

Thanks,

> 
> Hm?
> 

Re: [f2fs-dev] Multidevice f2fs mount after disk rearrangement

[Jaegeuk Kim]

On 01/08, Chao Yu wrote:
> On 2020/1/7 2:40, Oleksandr Natalenko via Linux-f2fs-devel wrote:
> > Hi.
> > 
> > On 06.01.2020 19:34, Jaegeuk Kim wrote:
> >> Thank you for investigating this ahead of me. :) Yes, the device list 
> >> is stored
> >> in superblock, so hacking it manually should work.
> >>
> >> Let me think about a tool to tune that.
> > 
> > Thank you both for the replies.
> > 
> > IIUC, tune.f2fs is not there yet. I saw a submission, but I do not see 
> > it as accepted, right?
> > 
> > Having this in tune.f2fs would be fine (assuming the assertion is 
> > replaced with some meaningful hint message), but wouldn't it be more 
> > convenient for an ordinary user to have implemented something like:
> > 
> > # mount -t f2fs /dev/sdb -o nextdev=/dev/sdc /mnt/fs
> 
> Hmm... sounds reasonable, however, the risk is obvious, if we mount with wrong
> primary device, filesystem can be aware that with metadata sanity check, if we
> mount with wrong secondary/... devices by mistake (or intentionally, people
> may think filesystem should be aware illegal parameters....), filesystem won't
> be aware of that, then metadata/data will be inconsistent...
> 
> Although that may also happen when we use tunesb.f2fs, but fsck.f2fs can be
> followed to verify the modification of tunesb.f2fs, that would be much safer.
> 
> So I suggest we can do that in tools first, maybe implement nextdev mount option
> if we have added metadata in secondary/... device.

+1, it'd be risky for user to give the device list whenever mounting the
filesystem. There'll be subtle corner cases where f2fs needs to deal with
given ambiguous sets between superblock and mount option.

> 
> Thanks,
> 
> > 
> > Hm?
> >