How to replace a missing device with a smaller one

How to replace a missing device with a smaller one

[Nathan Dehnel]

I have a 10-disk raid10 with a missing device I'm trying to replace. I
get this error when doing it though:

btrfs replace start 1 /dev/bcache0 /mnt
ERROR: target device smaller than source device (required 1000203091968 bytes)

I see that people recommend resizing a disk before replacing it, which
isn't an option for me because it's gone. I'm replacing the drive by
copying from its mirror, so can I resize the mirror and then replace?
How do I do that? Do I need to run "btrfs fi res" on each of the
remaining drives in the array?

Re: How to replace a missing device with a smaller one

[Qu Wenruo]

[-- Attachment #1.1: Type: text/plain, Size: 981 bytes --]



On 2019/11/18 上午10:09, Nathan Dehnel wrote:
> I have a 10-disk raid10 with a missing device I'm trying to replace. I
> get this error when doing it though:
> 
> btrfs replace start 1 /dev/bcache0 /mnt
> ERROR: target device smaller than source device (required 1000203091968 bytes)
> 
> I see that people recommend resizing a disk before replacing it, which
> isn't an option for me because it's gone.

Oh, that's indeed a problem.

We should allow to change missing device's size.

> I'm replacing the drive by
> copying from its mirror, so can I resize the mirror and then replace?
> How do I do that? Do I need to run "btrfs fi res" on each of the
> remaining drives in the array?
> 
As a workaround, you could remove that missing device (which would
relocate all chunks using it, so it can be slow).

Then add the new device to the fs.

With that done, it's recommended to do a convert to take full use the
two added devices.

Thanks,
Qu


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: How to replace a missing device with a smaller one

[Qu Wenruo]

[-- Attachment #1.1: Type: text/plain, Size: 1564 bytes --]



On 2019/11/18 下午1:32, Qu Wenruo wrote:
> 
> 
> On 2019/11/18 上午10:09, Nathan Dehnel wrote:
>> I have a 10-disk raid10 with a missing device I'm trying to replace. I
>> get this error when doing it though:
>>
>> btrfs replace start 1 /dev/bcache0 /mnt
>> ERROR: target device smaller than source device (required 1000203091968 bytes)
>>
>> I see that people recommend resizing a disk before replacing it, which
>> isn't an option for me because it's gone.
> 
> Oh, that's indeed a problem.
> 
> We should allow to change missing device's size.

I have CCed you with a patch to allow user to *shrink* the missing device.

You can also get the patch from patchwork:
https://patchwork.kernel.org/patch/11249009/

Please give a try, since the device size is pretty small, I believe with
that patch, we can go quick shrink, that means "btrfs fi resize" command
should return immediately.

Then you can go regular replace, this should save you a lot of IO by
avoiding the IO/time consuming device removal.

Thanks,
Qu

> 
>> I'm replacing the drive by
>> copying from its mirror, so can I resize the mirror and then replace?
>> How do I do that? Do I need to run "btrfs fi res" on each of the
>> remaining drives in the array?
>>
> As a workaround, you could remove that missing device (which would
> relocate all chunks using it, so it can be slow).
> 
> Then add the new device to the fs.
> 
> With that done, it's recommended to do a convert to take full use the
> two added devices.
> 
> Thanks,
> Qu
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: How to replace a missing device with a smaller one

[David Sterba]

On Mon, Nov 18, 2019 at 03:08:00PM +0800, Qu Wenruo wrote:
> On 2019/11/18 下午1:32, Qu Wenruo wrote:
> > On 2019/11/18 上午10:09, Nathan Dehnel wrote:
> >> I have a 10-disk raid10 with a missing device I'm trying to replace. I
> >> get this error when doing it though:
> >>
> >> btrfs replace start 1 /dev/bcache0 /mnt
> >> ERROR: target device smaller than source device (required 1000203091968 bytes)
> >>
> >> I see that people recommend resizing a disk before replacing it, which
> >> isn't an option for me because it's gone.
> > 
> > Oh, that's indeed a problem.
> > 
> > We should allow to change missing device's size.
> 
> I have CCed you with a patch to allow user to *shrink* the missing device.
> 
> You can also get the patch from patchwork:
> https://patchwork.kernel.org/patch/11249009/
> 
> Please give a try, since the device size is pretty small, I believe with
> that patch, we can go quick shrink, that means "btrfs fi resize" command
> should return immediately.

So it can be recteated eg. on loop devices, where some of them are
slightly smaller, then go missing and replace is started, right?

Re: How to replace a missing device with a smaller one

[Qu Wenruo]

[-- Attachment #1.1: Type: text/plain, Size: 1490 bytes --]



On 2019/11/19 下午10:38, David Sterba wrote:
> On Mon, Nov 18, 2019 at 03:08:00PM +0800, Qu Wenruo wrote:
>> On 2019/11/18 下午1:32, Qu Wenruo wrote:
>>> On 2019/11/18 上午10:09, Nathan Dehnel wrote:
>>>> I have a 10-disk raid10 with a missing device I'm trying to replace. I
>>>> get this error when doing it though:
>>>>
>>>> btrfs replace start 1 /dev/bcache0 /mnt
>>>> ERROR: target device smaller than source device (required 1000203091968 bytes)
>>>>
>>>> I see that people recommend resizing a disk before replacing it, which
>>>> isn't an option for me because it's gone.
>>>
>>> Oh, that's indeed a problem.
>>>
>>> We should allow to change missing device's size.
>>
>> I have CCed you with a patch to allow user to *shrink* the missing device.
>>
>> You can also get the patch from patchwork:
>> https://patchwork.kernel.org/patch/11249009/
>>
>> Please give a try, since the device size is pretty small, I believe with
>> that patch, we can go quick shrink, that means "btrfs fi resize" command
>> should return immediately.
> 
> So it can be recteated eg. on loop devices, where some of them are
> slightly smaller, then go missing and replace is started, right?
> 

Replace will still be rejected, but we can do resize of that missing
dev, then replace, as a workaround.

I haven't do the auto-resize for replace yet, since I'm not sure if that
could make cases like replacing 1T device with 10G driver happening.

Thanks,
Qu


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 520 bytes --]

Re: How to replace a missing device with a smaller one

[Zygo Blaxell]

[-- Attachment #1: Type: text/plain, Size: 1883 bytes --]

On Wed, Nov 20, 2019 at 08:02:43AM +0800, Qu Wenruo wrote:
> 
> 
> On 2019/11/19 下午10:38, David Sterba wrote:
> > On Mon, Nov 18, 2019 at 03:08:00PM +0800, Qu Wenruo wrote:
> >> On 2019/11/18 下午1:32, Qu Wenruo wrote:
> >>> On 2019/11/18 上午10:09, Nathan Dehnel wrote:
> >>>> I have a 10-disk raid10 with a missing device I'm trying to replace. I
> >>>> get this error when doing it though:
> >>>>
> >>>> btrfs replace start 1 /dev/bcache0 /mnt
> >>>> ERROR: target device smaller than source device (required 1000203091968 bytes)
> >>>>
> >>>> I see that people recommend resizing a disk before replacing it, which
> >>>> isn't an option for me because it's gone.
> >>>
> >>> Oh, that's indeed a problem.
> >>>
> >>> We should allow to change missing device's size.
> >>
> >> I have CCed you with a patch to allow user to *shrink* the missing device.
> >>
> >> You can also get the patch from patchwork:
> >> https://patchwork.kernel.org/patch/11249009/
> >>
> >> Please give a try, since the device size is pretty small, I believe with
> >> that patch, we can go quick shrink, that means "btrfs fi resize" command
> >> should return immediately.
> > 
> > So it can be recteated eg. on loop devices, where some of them are
> > slightly smaller, then go missing and replace is started, right?
> > 
> 
> Replace will still be rejected, but we can do resize of that missing
> dev, then replace, as a workaround.
> 
> I haven't do the auto-resize for replace yet, since I'm not sure if that
> could make cases like replacing 1T device with 10G driver happening.

That case would be much more tolerable if device replace/resize/delete would
stop to check for fatal signals at least between block groups, if not more
often.  Currently if you pick the wrong size, the only way to make it stop
is to reboot.

> Thanks,
> Qu
> 




[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: How to replace a missing device with a smaller one

[Chris Murphy]

On Sun, Nov 17, 2019 at 10:32 PM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>
>
>
> On 2019/11/18 上午10:09, Nathan Dehnel wrote:
> > I have a 10-disk raid10 with a missing device I'm trying to replace. I
> > get this error when doing it though:
> >
> > btrfs replace start 1 /dev/bcache0 /mnt
> > ERROR: target device smaller than source device (required 1000203091968 bytes)
> >
> > I see that people recommend resizing a disk before replacing it, which
> > isn't an option for me because it's gone.
>
> Oh, that's indeed a problem.
>
> We should allow to change missing device's size.
>
> > I'm replacing the drive by
> > copying from its mirror, so can I resize the mirror and then replace?
> > How do I do that? Do I need to run "btrfs fi res" on each of the
> > remaining drives in the array?
> >
> As a workaround, you could remove that missing device (which would
> relocate all chunks using it, so it can be slow).
>
> Then add the new device to the fs.
>
> With that done, it's recommended to do a convert to take full use the
> two added devices.

I think I'd advise adding the new device, and then removing the
missing device, rather than the other way around.

remove before add means the redundancy has to be re-established on the
remaining drives, device add then only increases fs capacity, and then
a balance is necessary to actually use the new device. Basically it
will cause two restripe tasks to happen.

add before remove, means the new device is available and will be used
during the redundancy being re-established (chunk replication); a full
balance won't be necessary, just check 'btrfs fi us' to see if there
are any single chunks for some reason, and if so a filtered
convert,soft balance can be done to convert them to raid10. This means
no restripe, just re-establishing the replication, and maybe a
"cleanup" filtered balance at the end to make sure all chunks are in
fact raid10.



-- 
Chris Murphy

Re: How to replace a missing device with a smaller one

[Anand Jain]

On 11/20/19 8:05 AM, Zygo Blaxell wrote:
> On Wed, Nov 20, 2019 at 08:02:43AM +0800, Qu Wenruo wrote:
>>
>>
>> On 2019/11/19 下午10:38, David Sterba wrote:
>>> On Mon, Nov 18, 2019 at 03:08:00PM +0800, Qu Wenruo wrote:
>>>> On 2019/11/18 下午1:32, Qu Wenruo wrote:
>>>>> On 2019/11/18 上午10:09, Nathan Dehnel wrote:
>>>>>> I have a 10-disk raid10 with a missing device I'm trying to replace. I
>>>>>> get this error when doing it though:
>>>>>>
>>>>>> btrfs replace start 1 /dev/bcache0 /mnt
>>>>>> ERROR: target device smaller than source device (required 1000203091968 bytes)
>>>>>>
>>>>>> I see that people recommend resizing a disk before replacing it, which
>>>>>> isn't an option for me because it's gone.
>>>>>
>>>>> Oh, that's indeed a problem.
>>>>>
>>>>> We should allow to change missing device's size.
>>>>
>>>> I have CCed you with a patch to allow user to *shrink* the missing device.
>>>>
>>>> You can also get the patch from patchwork:
>>>> https://patchwork.kernel.org/patch/11249009/
>>>>
>>>> Please give a try, since the device size is pretty small, I believe with
>>>> that patch, we can go quick shrink, that means "btrfs fi resize" command
>>>> should return immediately.
>>>
>>> So it can be recteated eg. on loop devices, where some of them are
>>> slightly smaller, then go missing and replace is started, right?
>>>
>>
>> Replace will still be rejected, but we can do resize of that missing
>> dev, then replace, as a workaround.
>>
>> I haven't do the auto-resize for replace yet, since I'm not sure if that
>> could make cases like replacing 1T device with 10G driver happening.
> 
> That case would be much more tolerable if device replace/resize/delete would
> stop to check for fatal signals at least between block groups, if not more
> often.  Currently if you pick the wrong size, the only way to make it stop
> is to reboot.

  I agree. IMO too
     btrfs replace --<option to skip-check|auto-resize>
  is better as commented in the patch thread.

  Already if you comment out the size checks[1], the replace with
  a smaller device works fine. But But you have to manually estimate
  the disk size for the actual consumption. So the effort here should
  be auto pre-check.

[1]

diff --git a/cmds/replace.c b/cmds/replace.c
index 2321aa156fe2..1a4155d360d6 100644
--- a/cmds/replace.c
+++ b/cmds/replace.c
@@ -246,11 +246,13 @@ static int cmd_replace_start(const struct 
cmd_struct *cmd,
                 goto leave_with_error;

         dstdev_size = get_partition_size(dstdev);
+/*
         if (srcdev_size > dstdev_size) {
                 error("target device smaller than source device 
(required %llu bytes)",
                         srcdev_size);
                 goto leave_with_error;
         }
+*/

         fddstdev = open(dstdev, O_RDWR);
         if (fddstdev < 0) {


diff --git a/fs/btrfs/dev-replace.c b/fs/btrfs/dev-replace.c
index 9a29d6de9017..86fc0f8653be 100644
--- a/fs/btrfs/dev-replace.c
+++ b/fs/btrfs/dev-replace.c
@@ -214,7 +214,7 @@ static int btrfs_init_dev_replace_tgtdev(struct 
btrfs_fs_info *fs_info,
                 }
         }

-
+/*
         if (i_size_read(bdev->bd_inode) <
             btrfs_device_get_total_bytes(srcdev)) {
                 btrfs_err(fs_info,
@@ -222,7 +222,7 @@ static int btrfs_init_dev_replace_tgtdev(struct 
btrfs_fs_info *fs_info,
                 ret = -EINVAL;
                 goto error;
         }
-
+*/

         device = btrfs_alloc_device(NULL, &devid, NULL);
         if (IS_ERR(device)) {





>> Thanks,
>> Qu
>>
> 
> 
> 

Re: How to replace a missing device with a smaller one

[Nathan Dehnel]

The patch worked. Thanks for the help.

On Mon, Nov 18, 2019 at 1:08 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>
>
>
> On 2019/11/18 下午1:32, Qu Wenruo wrote:
> >
> >
> > On 2019/11/18 上午10:09, Nathan Dehnel wrote:
> >> I have a 10-disk raid10 with a missing device I'm trying to replace. I
> >> get this error when doing it though:
> >>
> >> btrfs replace start 1 /dev/bcache0 /mnt
> >> ERROR: target device smaller than source device (required 1000203091968 bytes)
> >>
> >> I see that people recommend resizing a disk before replacing it, which
> >> isn't an option for me because it's gone.
> >
> > Oh, that's indeed a problem.
> >
> > We should allow to change missing device's size.
>
> I have CCed you with a patch to allow user to *shrink* the missing device.
>
> You can also get the patch from patchwork:
> https://patchwork.kernel.org/patch/11249009/
>
> Please give a try, since the device size is pretty small, I believe with
> that patch, we can go quick shrink, that means "btrfs fi resize" command
> should return immediately.
>
> Then you can go regular replace, this should save you a lot of IO by
> avoiding the IO/time consuming device removal.
>
> Thanks,
> Qu
>
> >
> >> I'm replacing the drive by
> >> copying from its mirror, so can I resize the mirror and then replace?
> >> How do I do that? Do I need to run "btrfs fi res" on each of the
> >> remaining drives in the array?
> >>
> > As a workaround, you could remove that missing device (which would
> > relocate all chunks using it, so it can be slow).
> >
> > Then add the new device to the fs.
> >
> > With that done, it's recommended to do a convert to take full use the
> > two added devices.
> >
> > Thanks,
> > Qu
> >
>