Grow set size issue

Grow set size issue

[jes.sorensen]

Hi Neil,

In the below patch you changed the error handling, to make the kernel
not setting the size of the device being an error. However we still have
the code in place to handle the error, except it never triggers.

Question is do you remember the reason for this change? Old kernels not
allowing it, are there any legitimate reasons for the kernel to refuse
the size change?

Cheers,
Jes

commit b0a658ffbcd2104594e8a7a185fa0fe05127723e
Author: NeilBrown <neilb@suse.de>
Date:   Thu May 3 16:18:22 2012 +1000

    Grow: failing the set the per-device size is not an error.
    
    Signed-off-by: NeilBrown <neilb@suse.de>

diff --git a/Grow.c b/Grow.c
index 0b0d718..330e719 100644
--- a/Grow.c
+++ b/Grow.c
@@ -1668,7 +1668,9 @@ int Grow_reshape(char *devname, int fd, int quiet, char *backup_file,
                rv = 0;
                for (mdi = sra->devs; mdi; mdi = mdi->next) {
                        if (sysfs_set_num(sra, mdi, "size", size) < 0) {
-                               rv = 1;
+                               /* Probably kernel refusing to let us
+                                * reduce the size - not an error.
+                                */
                                break;
                        }
                        if (array.not_persistent == 0 &&

Re: Grow set size issue

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 2898 bytes --]

On Thu, Mar 30 2017, jes.sorensen@gmail.com wrote:

> Hi Neil,
>
> In the below patch you changed the error handling, to make the kernel
> not setting the size of the device being an error. However we still have
> the code in place to handle the error, except it never triggers.

So we do.  I should have removed all of that.
I should have just reverted
Commit: 65a9798b58b4 ("FIX: Detect error and rollback metadata")


>
> Question is do you remember the reason for this change? Old kernels not
> allowing it, are there any legitimate reasons for the kernel to refuse
> the size change?

I needed to go further back to remind myself why we do these size change
at all.
The command being run here is "mdadm --grow /dev/mdX --size=foo"
which has a primary purpose of changing the component_size of the array.
What can happen is that someone makes all the components bigger
(E.g. with LVM) and then uses this command to set --size=max, and it
doesn't work.  That is because md doesn't know the devices are bigger.

You can tell md that devices have changes size by writing to the "size"
attribute.
mdadm doesn't have an option for doing that per-device - you need to
poke into sysfs.

To make it a bit easier, when you use "--grow --size=foo", mdadm will
always write "foo" to the "size" attribute of each device, just incase
that will be helpful. In the case where the device is now bigger, it
will.

In the case where the size of the array is being reduced, it is not
permitted to change the "size" of each device until the "component_size"
of the array has changed, so these attempts to change "size" will fail.
But that isn't a problem.

In short, the attempt to change "size" here is a convenience, and
optimization.  It doesn't matter if it fails.

So please just revert all bits of the above commit that are still
present.

Thanks!
NeilBrown


>
> Cheers,
> Jes
>
> commit b0a658ffbcd2104594e8a7a185fa0fe05127723e
> Author: NeilBrown <neilb@suse.de>
> Date:   Thu May 3 16:18:22 2012 +1000
>
>     Grow: failing the set the per-device size is not an error.
>     
>     Signed-off-by: NeilBrown <neilb@suse.de>
>
> diff --git a/Grow.c b/Grow.c
> index 0b0d718..330e719 100644
> --- a/Grow.c
> +++ b/Grow.c
> @@ -1668,7 +1668,9 @@ int Grow_reshape(char *devname, int fd, int quiet, char *backup_file,
>                 rv = 0;
>                 for (mdi = sra->devs; mdi; mdi = mdi->next) {
>                         if (sysfs_set_num(sra, mdi, "size", size) < 0) {
> -                               rv = 1;
> +                               /* Probably kernel refusing to let us
> +                                * reduce the size - not an error.
> +                                */
>                                 break;
>                         }
>                         if (array.not_persistent == 0 &&

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Grow set size issue

[Jes Sorensen]

On 03/29/2017 05:34 PM, NeilBrown wrote:
> On Thu, Mar 30 2017, jes.sorensen@gmail.com wrote:
>
>> Hi Neil,
>>
>> In the below patch you changed the error handling, to make the kernel
>> not setting the size of the device being an error. However we still have
>> the code in place to handle the error, except it never triggers.
>
> So we do.  I should have removed all of that.
> I should have just reverted
> Commit: 65a9798b58b4 ("FIX: Detect error and rollback metadata")
>
>
>>
>> Question is do you remember the reason for this change? Old kernels not
>> allowing it, are there any legitimate reasons for the kernel to refuse
>> the size change?
>
> I needed to go further back to remind myself why we do these size change
> at all.
> The command being run here is "mdadm --grow /dev/mdX --size=foo"
> which has a primary purpose of changing the component_size of the array.
> What can happen is that someone makes all the components bigger
> (E.g. with LVM) and then uses this command to set --size=max, and it
> doesn't work.  That is because md doesn't know the devices are bigger.
>
> You can tell md that devices have changes size by writing to the "size"
> attribute.
> mdadm doesn't have an option for doing that per-device - you need to
> poke into sysfs.
>
> To make it a bit easier, when you use "--grow --size=foo", mdadm will
> always write "foo" to the "size" attribute of each device, just incase
> that will be helpful. In the case where the device is now bigger, it
> will.
>
> In the case where the size of the array is being reduced, it is not
> permitted to change the "size" of each device until the "component_size"
> of the array has changed, so these attempts to change "size" will fail.
> But that isn't a problem.
>
> In short, the attempt to change "size" here is a convenience, and
> optimization.  It doesn't matter if it fails.
>
> So please just revert all bits of the above commit that are still
> present.

Hi Neil,

Thanks for the explanation, I'll take the big hammer to the leftovers 
and get rid of them.

Cheers,
Jes

Re: Grow set size issue

[Tomasz Majchrzak]

On Thu, Mar 30, 2017 at 10:10:32AM -0400, Jes Sorensen wrote:
> On 03/29/2017 05:34 PM, NeilBrown wrote:
> >On Thu, Mar 30 2017, jes.sorensen@gmail.com wrote:
> >
> >>Hi Neil,
> >>
> >>In the below patch you changed the error handling, to make the kernel
> >>not setting the size of the device being an error. However we still have
> >>the code in place to handle the error, except it never triggers.
> >
> >So we do.  I should have removed all of that.
> >I should have just reverted
> >Commit: 65a9798b58b4 ("FIX: Detect error and rollback metadata")
> >
> >
> >>
> >>Question is do you remember the reason for this change? Old kernels not
> >>allowing it, are there any legitimate reasons for the kernel to refuse
> >>the size change?
> >
> >I needed to go further back to remind myself why we do these size change
> >at all.
> >The command being run here is "mdadm --grow /dev/mdX --size=foo"
> >which has a primary purpose of changing the component_size of the array.
> >What can happen is that someone makes all the components bigger
> >(E.g. with LVM) and then uses this command to set --size=max, and it
> >doesn't work.  That is because md doesn't know the devices are bigger.
> >
> >You can tell md that devices have changes size by writing to the "size"
> >attribute.
> >mdadm doesn't have an option for doing that per-device - you need to
> >poke into sysfs.
> >
> >To make it a bit easier, when you use "--grow --size=foo", mdadm will
> >always write "foo" to the "size" attribute of each device, just incase
> >that will be helpful. In the case where the device is now bigger, it
> >will.
> >
> >In the case where the size of the array is being reduced, it is not
> >permitted to change the "size" of each device until the "component_size"
> >of the array has changed, so these attempts to change "size" will fail.
> >But that isn't a problem.
> >
> >In short, the attempt to change "size" here is a convenience, and
> >optimization.  It doesn't matter if it fails.
> >
> >So please just revert all bits of the above commit that are still
> >present.
> 
> Hi Neil,
> 
> Thanks for the explanation, I'll take the big hammer to the
> leftovers and get rid of them.
> 
> Jes

Hi,

The commit 758b327cf5a ("Grow: Remove unnecessary optimization") breaks grow
operation for external metadata.

mdadm --create /dev/md/imsm0 --metadata=imsm --raid-devices=4 /dev/nvme[0-3]n1 --run
mdadm --create /dev/md/r5 --level=5 --chunk 64 --size=1G --raid-devices=3 /dev/nvme[0-2]n1 --run --assume-clean

MDADM_EXPERIMENTAL=1 mdadm --grow /dev/md/r5 --size=2G
mdadm: Cannot set device size for /dev/md/r5: No space left on device

From update_size in MD driver:

	if (avail < num_sectors)
		-ENOSPC;

In the above case 'avail' is set to old value (1GiB) and new size requested
('num_sectors') is 2GiB.

Shall we revert the recent change then?

Regards,

Tomek

Re: Grow set size issue

[Jes Sorensen]

On 06/02/2017 07:01 AM, Tomasz Majchrzak wrote:
> On Thu, Mar 30, 2017 at 10:10:32AM -0400, Jes Sorensen wrote:
>> On 03/29/2017 05:34 PM, NeilBrown wrote:
>>> On Thu, Mar 30 2017, jes.sorensen@gmail.com wrote:
>>>
>>>> Hi Neil,
>>>>
>>>> In the below patch you changed the error handling, to make the kernel
>>>> not setting the size of the device being an error. However we still have
>>>> the code in place to handle the error, except it never triggers.
>>>
>>> So we do.  I should have removed all of that.
>>> I should have just reverted
>>> Commit: 65a9798b58b4 ("FIX: Detect error and rollback metadata")
>>>
>>>
>>>>
>>>> Question is do you remember the reason for this change? Old kernels not
>>>> allowing it, are there any legitimate reasons for the kernel to refuse
>>>> the size change?
>>>
>>> I needed to go further back to remind myself why we do these size change
>>> at all.
>>> The command being run here is "mdadm --grow /dev/mdX --size=foo"
>>> which has a primary purpose of changing the component_size of the array.
>>> What can happen is that someone makes all the components bigger
>>> (E.g. with LVM) and then uses this command to set --size=max, and it
>>> doesn't work.  That is because md doesn't know the devices are bigger.
>>>
>>> You can tell md that devices have changes size by writing to the "size"
>>> attribute.
>>> mdadm doesn't have an option for doing that per-device - you need to
>>> poke into sysfs.
>>>
>>> To make it a bit easier, when you use "--grow --size=foo", mdadm will
>>> always write "foo" to the "size" attribute of each device, just incase
>>> that will be helpful. In the case where the device is now bigger, it
>>> will.
>>>
>>> In the case where the size of the array is being reduced, it is not
>>> permitted to change the "size" of each device until the "component_size"
>>> of the array has changed, so these attempts to change "size" will fail.
>>> But that isn't a problem.
>>>
>>> In short, the attempt to change "size" here is a convenience, and
>>> optimization.  It doesn't matter if it fails.
>>>
>>> So please just revert all bits of the above commit that are still
>>> present.
>>
>> Hi Neil,
>>
>> Thanks for the explanation, I'll take the big hammer to the
>> leftovers and get rid of them.
>>
>> Jes
> 
> Hi,
> 
> The commit 758b327cf5a ("Grow: Remove unnecessary optimization") breaks grow
> operation for external metadata.
> 
> mdadm --create /dev/md/imsm0 --metadata=imsm --raid-devices=4 /dev/nvme[0-3]n1 --run
> mdadm --create /dev/md/r5 --level=5 --chunk 64 --size=1G --raid-devices=3 /dev/nvme[0-2]n1 --run --assume-clean
> 
> MDADM_EXPERIMENTAL=1 mdadm --grow /dev/md/r5 --size=2G
> mdadm: Cannot set device size for /dev/md/r5: No space left on device
> 
>  From update_size in MD driver:
> 
> 	if (avail < num_sectors)
> 		-ENOSPC;
> 
> In the above case 'avail' is set to old value (1GiB) and new size requested
> ('num_sectors') is 2GiB.
> 
> Shall we revert the recent change then?

Rather than just reverting, track down whats wrong and fix it.

Jes

[PATCH] Grow: set component size prior to array size

[Tomasz Majchrzak]

It is a partial revert of commit 758b327cf5a7 ("Grow: Remove unnecessary
optimization"). For native metadata component size is set in kernel for
entire disk space. As external metadata supports multiple arrays within
one disk, the component size is set to array size. If component size is
not updated prior to array size update, the grow operation fails.

Signed-off-by: Tomasz Majchrzak <tomasz.majchrzak@intel.com>
---
 Grow.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/Grow.c b/Grow.c
index ecf5ca0..4ecb1d8 100644
--- a/Grow.c
+++ b/Grow.c
@@ -1977,6 +1977,8 @@ int Grow_reshape(char *devname, int fd,
 		 */
 		min_csize = 0;
 		for (mdi = sra->devs; mdi; mdi = mdi->next) {
+			sysfs_set_num(sra, mdi, "size", s->size == MAX_SIZE ? 0
+				      : s->size);
 			if (array.not_persistent == 0 &&
 			    array.major_version == 0 &&
 			    get_linux_version() < 3001000) {
-- 
1.8.3.1

Re: [PATCH] Grow: set component size prior to array size

[Jes Sorensen]

On 06/05/2017 10:09 AM, Tomasz Majchrzak wrote:
> It is a partial revert of commit 758b327cf5a7 ("Grow: Remove unnecessary
> optimization"). For native metadata component size is set in kernel for
> entire disk space. As external metadata supports multiple arrays within
> one disk, the component size is set to array size. If component size is
> not updated prior to array size update, the grow operation fails.
> 
> Signed-off-by: Tomasz Majchrzak <tomasz.majchrzak@intel.com>
> ---
>   Grow.c | 2 ++
>   1 file changed, 2 insertions(+)

Applied!

Thanks,
Jes

Re: [PATCH] Grow: set component size prior to array size

[Tomasz Majchrzak]

On Mon, Jun 05, 2017 at 11:06:48AM -0400, Jes Sorensen wrote:
> On 06/05/2017 10:09 AM, Tomasz Majchrzak wrote:
> >It is a partial revert of commit 758b327cf5a7 ("Grow: Remove unnecessary
> >optimization"). For native metadata component size is set in kernel for
> >entire disk space. As external metadata supports multiple arrays within
> >one disk, the component size is set to array size. If component size is
> >not updated prior to array size update, the grow operation fails.
> >
> >Signed-off-by: Tomasz Majchrzak <tomasz.majchrzak@intel.com>
> >---
> >  Grow.c | 2 ++
> >  1 file changed, 2 insertions(+)
> 
> Applied!
> 
> Thanks,
> Jes
>

Would you mind pushing it to the repository? I'd like to have it ported into
certain Linux distribution.

Tomek

Re: [PATCH] Grow: set component size prior to array size

[Jes Sorensen]

On 06/06/2017 10:47 AM, Tomasz Majchrzak wrote:
> On Mon, Jun 05, 2017 at 11:06:48AM -0400, Jes Sorensen wrote:
>> On 06/05/2017 10:09 AM, Tomasz Majchrzak wrote:
>>> It is a partial revert of commit 758b327cf5a7 ("Grow: Remove unnecessary
>>> optimization"). For native metadata component size is set in kernel for
>>> entire disk space. As external metadata supports multiple arrays within
>>> one disk, the component size is set to array size. If component size is
>>> not updated prior to array size update, the grow operation fails.
>>>
>>> Signed-off-by: Tomasz Majchrzak <tomasz.majchrzak@intel.com>
>>> ---
>>>   Grow.c | 2 ++
>>>   1 file changed, 2 insertions(+)
>>
>> Applied!
>>
>> Thanks,
>> Jes
>>
> 
> Would you mind pushing it to the repository? I'd like to have it ported into
> certain Linux distribution.

Only because it's you :)

Done!

Jes