raid5 reshape is stuck

raid5 reshape is stuck

[Xiao Ni]

Hi Neil

   I encounter the problem when I reshape a 4-disks raid5 to raid5. It just can
appear with loop devices.

   The steps are:

[root@dhcp-12-158 mdadm-3.3.2]# mdadm -CR /dev/md0 -l5 -n5 /dev/loop[0-4] --assume-clean
mdadm: /dev/loop0 appears to be part of a raid array:
       level=raid5 devices=6 ctime=Fri May 15 13:47:17 2015
mdadm: /dev/loop1 appears to be part of a raid array:
       level=raid5 devices=6 ctime=Fri May 15 13:47:17 2015
mdadm: /dev/loop2 appears to be part of a raid array:
       level=raid5 devices=6 ctime=Fri May 15 13:47:17 2015
mdadm: /dev/loop3 appears to be part of a raid array:
       level=raid5 devices=6 ctime=Fri May 15 13:47:17 2015
mdadm: /dev/loop4 appears to be part of a raid array:
       level=raid5 devices=6 ctime=Fri May 15 13:47:17 2015
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md0 started.
[root@dhcp-12-158 mdadm-3.3.2]# mdadm /dev/md0 -a /dev/loop5
mdadm: added /dev/loop5
[root@dhcp-12-158 mdadm-3.3.2]# mdadm --grow /dev/md0 --raid-devices 6
mdadm: Need to backup 10240K of critical section..
[root@dhcp-12-158 mdadm-3.3.2]# cat /proc/mdstat 
Personalities : [raid6] [raid5] [raid4] 
md0 : active raid5 loop5[5] loop4[4] loop3[3] loop2[2] loop1[1] loop0[0]
      8187904 blocks super 1.2 level 5, 512k chunk, algorithm 2 [6/6] [UUUUUU]
      [>....................]  reshape =  0.0% (0/2046976) finish=6396.8min speed=0K/sec
      
unused devices: <none>

   It because the sync_max is set to 0 when run the command --grow

[root@dhcp-12-158 mdadm-3.3.2]# cd /sys/block/md0/md/
[root@dhcp-12-158 md]# cat sync_max 
0

   I tried reproduce with normal sata devices. The progress of reshape is no problem. Then
I checked the Grow.c. If I use sata devices, in function reshape_array, the return value
of set_new_data_offset is 0. But if I used loop devices, it return 1. Then it call the function
start_reshape. 

   In the function start_reshape it set the sync_max to reshape_progress. But in sysfs_read it
doesn't read reshape_progress. So it's 0 and the sync_max is set to 0. Why it need to set the
sync_max at this? I'm not sure about this. 

   I tried to fix this but I'm not sure whether it's the right way. I'll send the patches in 
other mails.

Best Regards
Xiao

Re: raid5 reshape is stuck

[Xiao Ni]

----- Original Message -----
> From: "Xiao Ni" <xni@redhat.com>
> To: linux-raid@vger.kernel.org
> Sent: Friday, May 15, 2015 3:00:24 PM
> Subject: raid5 reshape is stuck
> 
> Hi Neil
> 
>    I encounter the problem when I reshape a 4-disks raid5 to raid5. It just
>    can
> appear with loop devices.
> 
>    The steps are:
> 
> [root@dhcp-12-158 mdadm-3.3.2]# mdadm -CR /dev/md0 -l5 -n5 /dev/loop[0-4]
> --assume-clean
> mdadm: /dev/loop0 appears to be part of a raid array:
>        level=raid5 devices=6 ctime=Fri May 15 13:47:17 2015
> mdadm: /dev/loop1 appears to be part of a raid array:
>        level=raid5 devices=6 ctime=Fri May 15 13:47:17 2015
> mdadm: /dev/loop2 appears to be part of a raid array:
>        level=raid5 devices=6 ctime=Fri May 15 13:47:17 2015
> mdadm: /dev/loop3 appears to be part of a raid array:
>        level=raid5 devices=6 ctime=Fri May 15 13:47:17 2015
> mdadm: /dev/loop4 appears to be part of a raid array:
>        level=raid5 devices=6 ctime=Fri May 15 13:47:17 2015
> mdadm: Defaulting to version 1.2 metadata
> mdadm: array /dev/md0 started.
> [root@dhcp-12-158 mdadm-3.3.2]# mdadm /dev/md0 -a /dev/loop5
> mdadm: added /dev/loop5
> [root@dhcp-12-158 mdadm-3.3.2]# mdadm --grow /dev/md0 --raid-devices 6
> mdadm: Need to backup 10240K of critical section..
> [root@dhcp-12-158 mdadm-3.3.2]# cat /proc/mdstat
> Personalities : [raid6] [raid5] [raid4]
> md0 : active raid5 loop5[5] loop4[4] loop3[3] loop2[2] loop1[1] loop0[0]
>       8187904 blocks super 1.2 level 5, 512k chunk, algorithm 2 [6/6]
>       [UUUUUU]
>       [>....................]  reshape =  0.0% (0/2046976) finish=6396.8min
>       speed=0K/sec
>       
> unused devices: <none>
> 
>    It because the sync_max is set to 0 when run the command --grow
> 
> [root@dhcp-12-158 mdadm-3.3.2]# cd /sys/block/md0/md/
> [root@dhcp-12-158 md]# cat sync_max
> 0
> 
>    I tried reproduce with normal sata devices. The progress of reshape is no
>    problem. Then
> I checked the Grow.c. If I use sata devices, in function reshape_array, the
> return value
> of set_new_data_offset is 0. But if I used loop devices, it return 1. Then it
> call the function
> start_reshape.
> 
>    In the function start_reshape it set the sync_max to reshape_progress. But
>    in sysfs_read it
> doesn't read reshape_progress. So it's 0 and the sync_max is set to 0. Why it
> need to set the
> sync_max at this? I'm not sure about this.
> 
>    I tried to fix this but I'm not sure whether it's the right way. I'll send
>    the patches in
> other mails.
> 


    If there is no need to set sync_max and sync_min here. The method below also can fix the
problem.

-int start_reshape(struct mdinfo *sra, int already_running,
-                 int before_data_disks, int data_disks)
+int start_reshape(struct mdinfo *sra, int already_running)
 {
        int err;
-       unsigned long long sync_max_to_set;

        sysfs_set_num(sra, NULL, "suspend_lo", 0x7FFFFFFFFFFFFFFFULL);
        err = sysfs_set_num(sra, NULL, "suspend_hi", sra->reshape_progress);
        err = err ?: sysfs_set_num(sra, NULL, "suspend_lo",
                                   sra->reshape_progress);
-       if (before_data_disks <= data_disks)
-               sync_max_to_set = sra->reshape_progress / data_disks;
-       else
-               sync_max_to_set = (sra->component_size * data_disks
-                                  - sra->reshape_progress) / data_disks;
-       if (!already_running)
-               sysfs_set_num(sra, NULL, "sync_min", sync_max_to_set);
-       err = err ?: sysfs_set_num(sra, NULL, "sync_max", sync_max_to_set);
        if (!already_running)
                err = err ?: sysfs_set_str(sra, NULL, "sync_action", "reshape");

@@ -3260,8 +3250,8 @@
                           devname, container, &reshape) < 0)
                goto release;

-       err = start_reshape(sra, restart, reshape.before.data_disks,
-                           reshape.after.data_disks);
+       err = start_reshape(sra, restart);
+                           
        if (err) {
                pr_err("Cannot %s reshape for %s\n",
                       restart ? "continue" : "start",

Re: raid5 reshape is stuck

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 3391 bytes --]

On Fri, 15 May 2015 03:00:24 -0400 (EDT) Xiao Ni <xni@redhat.com> wrote:

> Hi Neil
> 
>    I encounter the problem when I reshape a 4-disks raid5 to raid5. It just can
> appear with loop devices.
> 
>    The steps are:
> 
> [root@dhcp-12-158 mdadm-3.3.2]# mdadm -CR /dev/md0 -l5 -n5 /dev/loop[0-4] --assume-clean
> mdadm: /dev/loop0 appears to be part of a raid array:
>        level=raid5 devices=6 ctime=Fri May 15 13:47:17 2015
> mdadm: /dev/loop1 appears to be part of a raid array:
>        level=raid5 devices=6 ctime=Fri May 15 13:47:17 2015
> mdadm: /dev/loop2 appears to be part of a raid array:
>        level=raid5 devices=6 ctime=Fri May 15 13:47:17 2015
> mdadm: /dev/loop3 appears to be part of a raid array:
>        level=raid5 devices=6 ctime=Fri May 15 13:47:17 2015
> mdadm: /dev/loop4 appears to be part of a raid array:
>        level=raid5 devices=6 ctime=Fri May 15 13:47:17 2015
> mdadm: Defaulting to version 1.2 metadata
> mdadm: array /dev/md0 started.
> [root@dhcp-12-158 mdadm-3.3.2]# mdadm /dev/md0 -a /dev/loop5
> mdadm: added /dev/loop5
> [root@dhcp-12-158 mdadm-3.3.2]# mdadm --grow /dev/md0 --raid-devices 6
> mdadm: Need to backup 10240K of critical section..
> [root@dhcp-12-158 mdadm-3.3.2]# cat /proc/mdstat 
> Personalities : [raid6] [raid5] [raid4] 
> md0 : active raid5 loop5[5] loop4[4] loop3[3] loop2[2] loop1[1] loop0[0]
>       8187904 blocks super 1.2 level 5, 512k chunk, algorithm 2 [6/6] [UUUUUU]
>       [>....................]  reshape =  0.0% (0/2046976) finish=6396.8min speed=0K/sec
>       
> unused devices: <none>
> 
>    It because the sync_max is set to 0 when run the command --grow
> 
> [root@dhcp-12-158 mdadm-3.3.2]# cd /sys/block/md0/md/
> [root@dhcp-12-158 md]# cat sync_max 
> 0
> 
>    I tried reproduce with normal sata devices. The progress of reshape is no problem. Then
> I checked the Grow.c. If I use sata devices, in function reshape_array, the return value
> of set_new_data_offset is 0. But if I used loop devices, it return 1. Then it call the function
> start_reshape. 

set_new_data_offset returns '0' if there is room on the devices to reduce the
data offset so that the reshape starts writing to unused space on the array.
This removes the need for a backup file, or the use of a spare device to
store a temporary backup.
It returns '1' if there was no room for relocating the data_offset.

So on your sata devices (which are presumably larger than your loop devices)
there was room.  On your loop devices there was not.


> 
>    In the function start_reshape it set the sync_max to reshape_progress. But in sysfs_read it
> doesn't read reshape_progress. So it's 0 and the sync_max is set to 0. Why it need to set the
> sync_max at this? I'm not sure about this. 

sync_max is set to 0 so that the reshape does not start until the backup has
been taken.
Once the backup is taken, child_monitor() should set sync_max to "max".

Can you  check if that is happening?

Thanks,
NeilBrown


> 
>    I tried to fix this but I'm not sure whether it's the right way. I'll send the patches in 
> other mails.
> 
> Best Regards
> Xiao
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]

Re: raid5 reshape is stuck

[Xiao Ni]

----- Original Message -----
> From: "NeilBrown" <neilb@suse.de>
> To: "Xiao Ni" <xni@redhat.com>
> Cc: linux-raid@vger.kernel.org
> Sent: Thursday, May 21, 2015 7:48:37 AM
> Subject: Re: raid5 reshape is stuck
> 
> On Fri, 15 May 2015 03:00:24 -0400 (EDT) Xiao Ni <xni@redhat.com> wrote:
> 
> > Hi Neil
> > 
> >    I encounter the problem when I reshape a 4-disks raid5 to raid5. It just
> >    can
> > appear with loop devices.
> > 
> >    The steps are:
> > 
> > [root@dhcp-12-158 mdadm-3.3.2]# mdadm -CR /dev/md0 -l5 -n5 /dev/loop[0-4]
> > --assume-clean
> > mdadm: /dev/loop0 appears to be part of a raid array:
> >        level=raid5 devices=6 ctime=Fri May 15 13:47:17 2015
> > mdadm: /dev/loop1 appears to be part of a raid array:
> >        level=raid5 devices=6 ctime=Fri May 15 13:47:17 2015
> > mdadm: /dev/loop2 appears to be part of a raid array:
> >        level=raid5 devices=6 ctime=Fri May 15 13:47:17 2015
> > mdadm: /dev/loop3 appears to be part of a raid array:
> >        level=raid5 devices=6 ctime=Fri May 15 13:47:17 2015
> > mdadm: /dev/loop4 appears to be part of a raid array:
> >        level=raid5 devices=6 ctime=Fri May 15 13:47:17 2015
> > mdadm: Defaulting to version 1.2 metadata
> > mdadm: array /dev/md0 started.
> > [root@dhcp-12-158 mdadm-3.3.2]# mdadm /dev/md0 -a /dev/loop5
> > mdadm: added /dev/loop5
> > [root@dhcp-12-158 mdadm-3.3.2]# mdadm --grow /dev/md0 --raid-devices 6
> > mdadm: Need to backup 10240K of critical section..
> > [root@dhcp-12-158 mdadm-3.3.2]# cat /proc/mdstat
> > Personalities : [raid6] [raid5] [raid4]
> > md0 : active raid5 loop5[5] loop4[4] loop3[3] loop2[2] loop1[1] loop0[0]
> >       8187904 blocks super 1.2 level 5, 512k chunk, algorithm 2 [6/6]
> >       [UUUUUU]
> >       [>....................]  reshape =  0.0% (0/2046976) finish=6396.8min
> >       speed=0K/sec
> >       
> > unused devices: <none>
> > 
> >    It because the sync_max is set to 0 when run the command --grow
> > 
> > [root@dhcp-12-158 mdadm-3.3.2]# cd /sys/block/md0/md/
> > [root@dhcp-12-158 md]# cat sync_max
> > 0
> > 
> >    I tried reproduce with normal sata devices. The progress of reshape is
> >    no problem. Then
> > I checked the Grow.c. If I use sata devices, in function reshape_array, the
> > return value
> > of set_new_data_offset is 0. But if I used loop devices, it return 1. Then
> > it call the function
> > start_reshape.
> 
> set_new_data_offset returns '0' if there is room on the devices to reduce the
> data offset so that the reshape starts writing to unused space on the array.
> This removes the need for a backup file, or the use of a spare device to
> store a temporary backup.
> It returns '1' if there was no room for relocating the data_offset.
> 
> So on your sata devices (which are presumably larger than your loop devices)
> there was room.  On your loop devices there was not.
> 
> 
> > 
> >    In the function start_reshape it set the sync_max to reshape_progress.
> >    But in sysfs_read it
> > doesn't read reshape_progress. So it's 0 and the sync_max is set to 0. Why
> > it need to set the
> > sync_max at this? I'm not sure about this.
> 
> sync_max is set to 0 so that the reshape does not start until the backup has
> been taken.
> Once the backup is taken, child_monitor() should set sync_max to "max".
> 
> Can you  check if that is happening?
> 
> Thanks,
> NeilBrown
> 
> 

  Thanks very much for the explaining. The problem maybe is fixed. I tried reproduce this with newest
kernel and newest mdadm. Now the problem don't exist. I'll do more tests and give the answer above later.

Best Regards
Xiao

Re: raid5 reshape is stuck

[Xiao Ni]

----- Original Message -----
> From: "Xiao Ni" <xni@redhat.com>
> To: "NeilBrown" <neilb@suse.de>
> Cc: linux-raid@vger.kernel.org
> Sent: Thursday, May 21, 2015 11:37:57 AM
> Subject: Re: raid5 reshape is stuck
> 
> 
> 
> ----- Original Message -----
> > From: "NeilBrown" <neilb@suse.de>
> > To: "Xiao Ni" <xni@redhat.com>
> > Cc: linux-raid@vger.kernel.org
> > Sent: Thursday, May 21, 2015 7:48:37 AM
> > Subject: Re: raid5 reshape is stuck
> > 
> > On Fri, 15 May 2015 03:00:24 -0400 (EDT) Xiao Ni <xni@redhat.com> wrote:
> > 
> > > Hi Neil
> > > 
> > >    I encounter the problem when I reshape a 4-disks raid5 to raid5. It
> > >    just
> > >    can
> > > appear with loop devices.
> > > 
> > >    The steps are:
> > > 
> > > [root@dhcp-12-158 mdadm-3.3.2]# mdadm -CR /dev/md0 -l5 -n5 /dev/loop[0-4]
> > > --assume-clean
> > > mdadm: /dev/loop0 appears to be part of a raid array:
> > >        level=raid5 devices=6 ctime=Fri May 15 13:47:17 2015
> > > mdadm: /dev/loop1 appears to be part of a raid array:
> > >        level=raid5 devices=6 ctime=Fri May 15 13:47:17 2015
> > > mdadm: /dev/loop2 appears to be part of a raid array:
> > >        level=raid5 devices=6 ctime=Fri May 15 13:47:17 2015
> > > mdadm: /dev/loop3 appears to be part of a raid array:
> > >        level=raid5 devices=6 ctime=Fri May 15 13:47:17 2015
> > > mdadm: /dev/loop4 appears to be part of a raid array:
> > >        level=raid5 devices=6 ctime=Fri May 15 13:47:17 2015
> > > mdadm: Defaulting to version 1.2 metadata
> > > mdadm: array /dev/md0 started.
> > > [root@dhcp-12-158 mdadm-3.3.2]# mdadm /dev/md0 -a /dev/loop5
> > > mdadm: added /dev/loop5
> > > [root@dhcp-12-158 mdadm-3.3.2]# mdadm --grow /dev/md0 --raid-devices 6
> > > mdadm: Need to backup 10240K of critical section..
> > > [root@dhcp-12-158 mdadm-3.3.2]# cat /proc/mdstat
> > > Personalities : [raid6] [raid5] [raid4]
> > > md0 : active raid5 loop5[5] loop4[4] loop3[3] loop2[2] loop1[1] loop0[0]
> > >       8187904 blocks super 1.2 level 5, 512k chunk, algorithm 2 [6/6]
> > >       [UUUUUU]
> > >       [>....................]  reshape =  0.0% (0/2046976)
> > >       finish=6396.8min
> > >       speed=0K/sec
> > >       
> > > unused devices: <none>
> > > 
> > >    It because the sync_max is set to 0 when run the command --grow
> > > 
> > > [root@dhcp-12-158 mdadm-3.3.2]# cd /sys/block/md0/md/
> > > [root@dhcp-12-158 md]# cat sync_max
> > > 0
> > > 
> > >    I tried reproduce with normal sata devices. The progress of reshape is
> > >    no problem. Then
> > > I checked the Grow.c. If I use sata devices, in function reshape_array,
> > > the
> > > return value
> > > of set_new_data_offset is 0. But if I used loop devices, it return 1.
> > > Then
> > > it call the function
> > > start_reshape.
> > 
> > set_new_data_offset returns '0' if there is room on the devices to reduce
> > the
> > data offset so that the reshape starts writing to unused space on the
> > array.
> > This removes the need for a backup file, or the use of a spare device to
> > store a temporary backup.
> > It returns '1' if there was no room for relocating the data_offset.
> > 
> > So on your sata devices (which are presumably larger than your loop
> > devices)
> > there was room.  On your loop devices there was not.
> > 
> > 
> > > 
> > >    In the function start_reshape it set the sync_max to reshape_progress.
> > >    But in sysfs_read it
> > > doesn't read reshape_progress. So it's 0 and the sync_max is set to 0.
> > > Why
> > > it need to set the
> > > sync_max at this? I'm not sure about this.
> > 
> > sync_max is set to 0 so that the reshape does not start until the backup
> > has
> > been taken.
> > Once the backup is taken, child_monitor() should set sync_max to "max".
> > 
> > Can you  check if that is happening?
> > 
> > Thanks,
> > NeilBrown
> > 
> > 
> 
>   Thanks very much for the explaining. The problem maybe is fixed. I tried
>   reproduce this with newest
> kernel and newest mdadm. Now the problem don't exist. I'll do more tests and
> give the answer above later.
> 

Hi Neil

   As you said, it doesn't enter child monitor. The problem still exist.

The kernel version :
[root@intel-canoepass-02 tmp]# uname -r
4.0.4

mdadm I used is the newest git code from git://git.neil.brown.name/mdadm.git

   
   In the function continue_via_systemd the parent find pid is bigger than 0 and
status is 0. So it return 1. So it have no opportunity to call child_monitor.


   And if it want to set sync_max to 0 until the backup has been taken. Why does not 
set sync_max to 0 directly, but use the value reshape_progress? There is a little confused.

Best Regards
Xiao

Re: raid5 reshape is stuck

[Xiao Ni]

----- Original Message -----
> From: "Xiao Ni" <xni@redhat.com>
> To: "NeilBrown" <neilb@suse.de>
> Cc: linux-raid@vger.kernel.org
> Sent: Thursday, May 21, 2015 8:31:58 PM
> Subject: Re: raid5 reshape is stuck
> 
> 
> 
> ----- Original Message -----
> > From: "Xiao Ni" <xni@redhat.com>
> > To: "NeilBrown" <neilb@suse.de>
> > Cc: linux-raid@vger.kernel.org
> > Sent: Thursday, May 21, 2015 11:37:57 AM
> > Subject: Re: raid5 reshape is stuck
> > 
> > 
> > 
> > ----- Original Message -----
> > > From: "NeilBrown" <neilb@suse.de>
> > > To: "Xiao Ni" <xni@redhat.com>
> > > Cc: linux-raid@vger.kernel.org
> > > Sent: Thursday, May 21, 2015 7:48:37 AM
> > > Subject: Re: raid5 reshape is stuck
> > > 
> > > On Fri, 15 May 2015 03:00:24 -0400 (EDT) Xiao Ni <xni@redhat.com> wrote:
> > > 
> > > > Hi Neil
> > > > 
> > > >    I encounter the problem when I reshape a 4-disks raid5 to raid5. It
> > > >    just
> > > >    can
> > > > appear with loop devices.
> > > > 
> > > >    The steps are:
> > > > 
> > > > [root@dhcp-12-158 mdadm-3.3.2]# mdadm -CR /dev/md0 -l5 -n5
> > > > /dev/loop[0-4]
> > > > --assume-clean
> > > > mdadm: /dev/loop0 appears to be part of a raid array:
> > > >        level=raid5 devices=6 ctime=Fri May 15 13:47:17 2015
> > > > mdadm: /dev/loop1 appears to be part of a raid array:
> > > >        level=raid5 devices=6 ctime=Fri May 15 13:47:17 2015
> > > > mdadm: /dev/loop2 appears to be part of a raid array:
> > > >        level=raid5 devices=6 ctime=Fri May 15 13:47:17 2015
> > > > mdadm: /dev/loop3 appears to be part of a raid array:
> > > >        level=raid5 devices=6 ctime=Fri May 15 13:47:17 2015
> > > > mdadm: /dev/loop4 appears to be part of a raid array:
> > > >        level=raid5 devices=6 ctime=Fri May 15 13:47:17 2015
> > > > mdadm: Defaulting to version 1.2 metadata
> > > > mdadm: array /dev/md0 started.
> > > > [root@dhcp-12-158 mdadm-3.3.2]# mdadm /dev/md0 -a /dev/loop5
> > > > mdadm: added /dev/loop5
> > > > [root@dhcp-12-158 mdadm-3.3.2]# mdadm --grow /dev/md0 --raid-devices 6
> > > > mdadm: Need to backup 10240K of critical section..
> > > > [root@dhcp-12-158 mdadm-3.3.2]# cat /proc/mdstat
> > > > Personalities : [raid6] [raid5] [raid4]
> > > > md0 : active raid5 loop5[5] loop4[4] loop3[3] loop2[2] loop1[1]
> > > > loop0[0]
> > > >       8187904 blocks super 1.2 level 5, 512k chunk, algorithm 2 [6/6]
> > > >       [UUUUUU]
> > > >       [>....................]  reshape =  0.0% (0/2046976)
> > > >       finish=6396.8min
> > > >       speed=0K/sec
> > > >       
> > > > unused devices: <none>
> > > > 
> > > >    It because the sync_max is set to 0 when run the command --grow
> > > > 
> > > > [root@dhcp-12-158 mdadm-3.3.2]# cd /sys/block/md0/md/
> > > > [root@dhcp-12-158 md]# cat sync_max
> > > > 0
> > > > 
> > > >    I tried reproduce with normal sata devices. The progress of reshape
> > > >    is
> > > >    no problem. Then
> > > > I checked the Grow.c. If I use sata devices, in function reshape_array,
> > > > the
> > > > return value
> > > > of set_new_data_offset is 0. But if I used loop devices, it return 1.
> > > > Then
> > > > it call the function
> > > > start_reshape.
> > > 
> > > set_new_data_offset returns '0' if there is room on the devices to reduce
> > > the
> > > data offset so that the reshape starts writing to unused space on the
> > > array.
> > > This removes the need for a backup file, or the use of a spare device to
> > > store a temporary backup.
> > > It returns '1' if there was no room for relocating the data_offset.
> > > 
> > > So on your sata devices (which are presumably larger than your loop
> > > devices)
> > > there was room.  On your loop devices there was not.
> > > 
> > > 
> > > > 
> > > >    In the function start_reshape it set the sync_max to
> > > >    reshape_progress.
> > > >    But in sysfs_read it
> > > > doesn't read reshape_progress. So it's 0 and the sync_max is set to 0.
> > > > Why
> > > > it need to set the
> > > > sync_max at this? I'm not sure about this.
> > > 
> > > sync_max is set to 0 so that the reshape does not start until the backup
> > > has
> > > been taken.
> > > Once the backup is taken, child_monitor() should set sync_max to "max".
> > > 
> > > Can you  check if that is happening?
> > > 
> > > Thanks,
> > > NeilBrown
> > > 
> > > 
> > 
> >   Thanks very much for the explaining. The problem maybe is fixed. I tried
> >   reproduce this with newest
> > kernel and newest mdadm. Now the problem don't exist. I'll do more tests
> > and
> > give the answer above later.
> > 
> 
> Hi Neil
> 
>    As you said, it doesn't enter child monitor. The problem still exist.
> 
> The kernel version :
> [root@intel-canoepass-02 tmp]# uname -r
> 4.0.4
> 
> mdadm I used is the newest git code from git://git.neil.brown.name/mdadm.git
> 
>    
>    In the function continue_via_systemd the parent find pid is bigger than 0
>    and
> status is 0. So it return 1. So it have no opportunity to call child_monitor.

    Does it should return 1 when pid > 0 and status is not zero?

diff --git a/Grow.c b/Grow.c
index 44ee8a7..e96465a 100644
--- a/Grow.c
+++ b/Grow.c
@@ -2755,7 +2755,7 @@ static int continue_via_systemd(char *devnm)
      break;
   default: /* parent - good */
      pid = wait(&status);
-     if (pid >= 0 && status == 0)
+     if (pid >= 0 && status != 0)
         return 1;
   }   
   return 0;

> 
> 
>    And if it want to set sync_max to 0 until the backup has been taken. Why
>    does not
> set sync_max to 0 directly, but use the value reshape_progress? There is a
> little confused.
> 
> Best Regards
> Xiao
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: raid5 reshape is stuck

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 6778 bytes --]

On Thu, 21 May 2015 08:31:58 -0400 (EDT) Xiao Ni <xni@redhat.com> wrote:

> 
> 
> ----- Original Message -----
> > From: "Xiao Ni" <xni@redhat.com>
> > To: "NeilBrown" <neilb@suse.de>
> > Cc: linux-raid@vger.kernel.org
> > Sent: Thursday, May 21, 2015 11:37:57 AM
> > Subject: Re: raid5 reshape is stuck
> > 
> > 
> > 
> > ----- Original Message -----
> > > From: "NeilBrown" <neilb@suse.de>
> > > To: "Xiao Ni" <xni@redhat.com>
> > > Cc: linux-raid@vger.kernel.org
> > > Sent: Thursday, May 21, 2015 7:48:37 AM
> > > Subject: Re: raid5 reshape is stuck
> > > 
> > > On Fri, 15 May 2015 03:00:24 -0400 (EDT) Xiao Ni <xni@redhat.com> wrote:
> > > 
> > > > Hi Neil
> > > > 
> > > >    I encounter the problem when I reshape a 4-disks raid5 to raid5. It
> > > >    just
> > > >    can
> > > > appear with loop devices.
> > > > 
> > > >    The steps are:
> > > > 
> > > > [root@dhcp-12-158 mdadm-3.3.2]# mdadm -CR /dev/md0 -l5 -n5 /dev/loop[0-4]
> > > > --assume-clean
> > > > mdadm: /dev/loop0 appears to be part of a raid array:
> > > >        level=raid5 devices=6 ctime=Fri May 15 13:47:17 2015
> > > > mdadm: /dev/loop1 appears to be part of a raid array:
> > > >        level=raid5 devices=6 ctime=Fri May 15 13:47:17 2015
> > > > mdadm: /dev/loop2 appears to be part of a raid array:
> > > >        level=raid5 devices=6 ctime=Fri May 15 13:47:17 2015
> > > > mdadm: /dev/loop3 appears to be part of a raid array:
> > > >        level=raid5 devices=6 ctime=Fri May 15 13:47:17 2015
> > > > mdadm: /dev/loop4 appears to be part of a raid array:
> > > >        level=raid5 devices=6 ctime=Fri May 15 13:47:17 2015
> > > > mdadm: Defaulting to version 1.2 metadata
> > > > mdadm: array /dev/md0 started.
> > > > [root@dhcp-12-158 mdadm-3.3.2]# mdadm /dev/md0 -a /dev/loop5
> > > > mdadm: added /dev/loop5
> > > > [root@dhcp-12-158 mdadm-3.3.2]# mdadm --grow /dev/md0 --raid-devices 6
> > > > mdadm: Need to backup 10240K of critical section..
> > > > [root@dhcp-12-158 mdadm-3.3.2]# cat /proc/mdstat
> > > > Personalities : [raid6] [raid5] [raid4]
> > > > md0 : active raid5 loop5[5] loop4[4] loop3[3] loop2[2] loop1[1] loop0[0]
> > > >       8187904 blocks super 1.2 level 5, 512k chunk, algorithm 2 [6/6]
> > > >       [UUUUUU]
> > > >       [>....................]  reshape =  0.0% (0/2046976)
> > > >       finish=6396.8min
> > > >       speed=0K/sec
> > > >       
> > > > unused devices: <none>
> > > > 
> > > >    It because the sync_max is set to 0 when run the command --grow
> > > > 
> > > > [root@dhcp-12-158 mdadm-3.3.2]# cd /sys/block/md0/md/
> > > > [root@dhcp-12-158 md]# cat sync_max
> > > > 0
> > > > 
> > > >    I tried reproduce with normal sata devices. The progress of reshape is
> > > >    no problem. Then
> > > > I checked the Grow.c. If I use sata devices, in function reshape_array,
> > > > the
> > > > return value
> > > > of set_new_data_offset is 0. But if I used loop devices, it return 1.
> > > > Then
> > > > it call the function
> > > > start_reshape.
> > > 
> > > set_new_data_offset returns '0' if there is room on the devices to reduce
> > > the
> > > data offset so that the reshape starts writing to unused space on the
> > > array.
> > > This removes the need for a backup file, or the use of a spare device to
> > > store a temporary backup.
> > > It returns '1' if there was no room for relocating the data_offset.
> > > 
> > > So on your sata devices (which are presumably larger than your loop
> > > devices)
> > > there was room.  On your loop devices there was not.
> > > 
> > > 
> > > > 
> > > >    In the function start_reshape it set the sync_max to reshape_progress.
> > > >    But in sysfs_read it
> > > > doesn't read reshape_progress. So it's 0 and the sync_max is set to 0.
> > > > Why
> > > > it need to set the
> > > > sync_max at this? I'm not sure about this.
> > > 
> > > sync_max is set to 0 so that the reshape does not start until the backup
> > > has
> > > been taken.
> > > Once the backup is taken, child_monitor() should set sync_max to "max".
> > > 
> > > Can you  check if that is happening?
> > > 
> > > Thanks,
> > > NeilBrown
> > > 
> > > 
> > 
> >   Thanks very much for the explaining. The problem maybe is fixed. I tried
> >   reproduce this with newest
> > kernel and newest mdadm. Now the problem don't exist. I'll do more tests and
> > give the answer above later.
> > 
> 
> Hi Neil
> 
>    As you said, it doesn't enter child monitor. The problem still exist.
> 
> The kernel version :
> [root@intel-canoepass-02 tmp]# uname -r
> 4.0.4
> 
> mdadm I used is the newest git code from git://git.neil.brown.name/mdadm.git
> 
>    
>    In the function continue_via_systemd the parent find pid is bigger than 0 and
> status is 0. So it return 1. So it have no opportunity to call child_monitor.

If continue_via_systemd succeeded, that implies that 
  systemctl start mdadm-grow-continue@mdXXX.service

succeeded.  So 
   mdadm --grow --continue /dev/mdXXX

was run, so that mdadm should call 'child_monitor' and update sync_max when
appropriate.  Can you check if it does?


> 
> 
>    And if it want to set sync_max to 0 until the backup has been taken. Why does not 
> set sync_max to 0 directly, but use the value reshape_progress? There is a little confused.

When reshaping an array to a different array of the same size, such as a
4-driver RAID5 to a 5-driver RAID6, then mdadm needs to backup, one piece at
a time, the entire array (unless it can change data_offset, which is a
relatively new ability).

If you stop an array when it is in the middle of such a reshape, and then
reassemble the array, the backup process need to recommence where it left off.
So it tells the kernel that the reshape can progress as far as where it was
up to before.  So 'sync_max' is set based on the value of 'reshape_progress'.
(This will happen almost instantly).

Then the background mdadm (or the mdadm started by systemd) will backup the
next few stripes, update sync_max, wait for those stripes to be reshaped, then
discard the old backup, create a new one of the few stripes after that, and
continue.

Does that make it a little clearer?

And in response to your other email:
>     Does it should return 1 when pid > 0 and status is not zero?

No.  continue_via_systemd should return 1 precisely when the 'systemctl'
command was successfully run.  So 'status' must be zero.


Thanks,
NeilBrown




> 
> Best Regards
> Xiao
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]

Re: raid5 reshape is stuck

[Xiao Ni]

----- Original Message -----
> From: "NeilBrown" <neilb@suse.de>
> To: "Xiao Ni" <xni@redhat.com>
> Cc: linux-raid@vger.kernel.org
> Sent: Monday, May 25, 2015 11:50:01 AM
> Subject: Re: raid5 reshape is stuck
> 
> On Thu, 21 May 2015 08:31:58 -0400 (EDT) Xiao Ni <xni@redhat.com> wrote:
> 
> > 
> > 
> > ----- Original Message -----
> > > From: "Xiao Ni" <xni@redhat.com>
> > > To: "NeilBrown" <neilb@suse.de>
> > > Cc: linux-raid@vger.kernel.org
> > > Sent: Thursday, May 21, 2015 11:37:57 AM
> > > Subject: Re: raid5 reshape is stuck
> > > 
> > > 
> > > 
> > > ----- Original Message -----
> > > > From: "NeilBrown" <neilb@suse.de>
> > > > To: "Xiao Ni" <xni@redhat.com>
> > > > Cc: linux-raid@vger.kernel.org
> > > > Sent: Thursday, May 21, 2015 7:48:37 AM
> > > > Subject: Re: raid5 reshape is stuck
> > > > 
> > > > On Fri, 15 May 2015 03:00:24 -0400 (EDT) Xiao Ni <xni@redhat.com>
> > > > wrote:
> > > > 
> > > > > Hi Neil
> > > > > 
> > > > >    I encounter the problem when I reshape a 4-disks raid5 to raid5.
> > > > >    It
> > > > >    just
> > > > >    can
> > > > > appear with loop devices.
> > > > > 
> > > > >    The steps are:
> > > > > 
> > > > > [root@dhcp-12-158 mdadm-3.3.2]# mdadm -CR /dev/md0 -l5 -n5
> > > > > /dev/loop[0-4]
> > > > > --assume-clean
> > > > > mdadm: /dev/loop0 appears to be part of a raid array:
> > > > >        level=raid5 devices=6 ctime=Fri May 15 13:47:17 2015
> > > > > mdadm: /dev/loop1 appears to be part of a raid array:
> > > > >        level=raid5 devices=6 ctime=Fri May 15 13:47:17 2015
> > > > > mdadm: /dev/loop2 appears to be part of a raid array:
> > > > >        level=raid5 devices=6 ctime=Fri May 15 13:47:17 2015
> > > > > mdadm: /dev/loop3 appears to be part of a raid array:
> > > > >        level=raid5 devices=6 ctime=Fri May 15 13:47:17 2015
> > > > > mdadm: /dev/loop4 appears to be part of a raid array:
> > > > >        level=raid5 devices=6 ctime=Fri May 15 13:47:17 2015
> > > > > mdadm: Defaulting to version 1.2 metadata
> > > > > mdadm: array /dev/md0 started.
> > > > > [root@dhcp-12-158 mdadm-3.3.2]# mdadm /dev/md0 -a /dev/loop5
> > > > > mdadm: added /dev/loop5
> > > > > [root@dhcp-12-158 mdadm-3.3.2]# mdadm --grow /dev/md0 --raid-devices
> > > > > 6
> > > > > mdadm: Need to backup 10240K of critical section..
> > > > > [root@dhcp-12-158 mdadm-3.3.2]# cat /proc/mdstat
> > > > > Personalities : [raid6] [raid5] [raid4]
> > > > > md0 : active raid5 loop5[5] loop4[4] loop3[3] loop2[2] loop1[1]
> > > > > loop0[0]
> > > > >       8187904 blocks super 1.2 level 5, 512k chunk, algorithm 2 [6/6]
> > > > >       [UUUUUU]
> > > > >       [>....................]  reshape =  0.0% (0/2046976)
> > > > >       finish=6396.8min
> > > > >       speed=0K/sec
> > > > >       
> > > > > unused devices: <none>
> > > > > 
> > > > >    It because the sync_max is set to 0 when run the command --grow
> > > > > 
> > > > > [root@dhcp-12-158 mdadm-3.3.2]# cd /sys/block/md0/md/
> > > > > [root@dhcp-12-158 md]# cat sync_max
> > > > > 0
> > > > > 
> > > > >    I tried reproduce with normal sata devices. The progress of
> > > > >    reshape is
> > > > >    no problem. Then
> > > > > I checked the Grow.c. If I use sata devices, in function
> > > > > reshape_array,
> > > > > the
> > > > > return value
> > > > > of set_new_data_offset is 0. But if I used loop devices, it return 1.
> > > > > Then
> > > > > it call the function
> > > > > start_reshape.
> > > > 
> > > > set_new_data_offset returns '0' if there is room on the devices to
> > > > reduce
> > > > the
> > > > data offset so that the reshape starts writing to unused space on the
> > > > array.
> > > > This removes the need for a backup file, or the use of a spare device
> > > > to
> > > > store a temporary backup.
> > > > It returns '1' if there was no room for relocating the data_offset.
> > > > 
> > > > So on your sata devices (which are presumably larger than your loop
> > > > devices)
> > > > there was room.  On your loop devices there was not.
> > > > 
> > > > 
> > > > > 
> > > > >    In the function start_reshape it set the sync_max to
> > > > >    reshape_progress.
> > > > >    But in sysfs_read it
> > > > > doesn't read reshape_progress. So it's 0 and the sync_max is set to
> > > > > 0.
> > > > > Why
> > > > > it need to set the
> > > > > sync_max at this? I'm not sure about this.
> > > > 
> > > > sync_max is set to 0 so that the reshape does not start until the
> > > > backup
> > > > has
> > > > been taken.
> > > > Once the backup is taken, child_monitor() should set sync_max to "max".
> > > > 
> > > > Can you  check if that is happening?
> > > > 
> > > > Thanks,
> > > > NeilBrown
> > > > 
> > > > 
> > > 
> > >   Thanks very much for the explaining. The problem maybe is fixed. I
> > >   tried
> > >   reproduce this with newest
> > > kernel and newest mdadm. Now the problem don't exist. I'll do more tests
> > > and
> > > give the answer above later.
> > > 
> > 
> > Hi Neil
> > 
> >    As you said, it doesn't enter child monitor. The problem still exist.
> > 
> > The kernel version :
> > [root@intel-canoepass-02 tmp]# uname -r
> > 4.0.4
> > 
> > mdadm I used is the newest git code from
> > git://git.neil.brown.name/mdadm.git
> > 
> >    
> >    In the function continue_via_systemd the parent find pid is bigger than
> >    0 and
> > status is 0. So it return 1. So it have no opportunity to call
> > child_monitor.
> 
> If continue_via_systemd succeeded, that implies that
>   systemctl start mdadm-grow-continue@mdXXX.service
> 
> succeeded.  So
>    mdadm --grow --continue /dev/mdXXX
> 
> was run, so that mdadm should call 'child_monitor' and update sync_max when
> appropriate.  Can you check if it does?


[root@intel-waimeabay-hedt-01 create_assemble]# systemctl start mdadm-grow-continue@md0.service
[root@intel-waimeabay-hedt-01 create_assemble]# echo $?
0
[root@intel-waimeabay-hedt-01 create_assemble]# systemctl status mdadm-grow-continue@md0.service
mdadm-grow-continue@md0.service - Manage MD Reshape on /dev/md0
   Loaded: loaded (/usr/lib/systemd/system/mdadm-grow-continue@.service; static)
   Active: failed (Result: exit-code) since Tue 2015-05-26 05:33:59 EDT; 21s ago
  Process: 5374 ExecStart=/usr/sbin/mdadm --grow --continue /dev/%I (code=exited, status=1/FAILURE)
 Main PID: 5374 (code=exited, status=1/FAILURE)

May 26 05:33:59 intel-waimeabay-hedt-01.lab.eng.rdu.redhat.com systemd[1]: Started Manage MD Reshape on /dev/md0.
May 26 05:33:59 intel-waimeabay-hedt-01.lab.eng.rdu.redhat.com systemd[1]: mdadm-grow-continue@md0.service: main process exited, ...URE
May 26 05:33:59 intel-waimeabay-hedt-01.lab.eng.rdu.redhat.com systemd[1]: Unit mdadm-grow-continue@md0.service entered failed state.
Hint: Some lines were ellipsized, use -l to show in full.
[root@intel-waimeabay-hedt-01 create_assemble]# mdadm --grow --continue /dev/md0 --backup-file=tmp0
mdadm: Need to backup 6144K of critical section..

Now the reshape start. 

   
> 
> 
> > 
> > 
> >    And if it want to set sync_max to 0 until the backup has been taken. Why
> >    does not
> > set sync_max to 0 directly, but use the value reshape_progress? There is a
> > little confused.
> 
> When reshaping an array to a different array of the same size, such as a
> 4-driver RAID5 to a 5-driver RAID6, then mdadm needs to backup, one piece at
> a time, the entire array (unless it can change data_offset, which is a
> relatively new ability).
> 
> If you stop an array when it is in the middle of such a reshape, and then
> reassemble the array, the backup process need to recommence where it left
> off.
> So it tells the kernel that the reshape can progress as far as where it was
> up to before.  So 'sync_max' is set based on the value of 'reshape_progress'.
> (This will happen almost instantly).
> 
> Then the background mdadm (or the mdadm started by systemd) will backup the
> next few stripes, update sync_max, wait for those stripes to be reshaped,
> then
> discard the old backup, create a new one of the few stripes after that, and
> continue.
> 
> Does that make it a little clearer?

This is a big dinner for me. I need digest this for a while. Thanks very much
for this. What's the "backup process"?

Could you explain backup in detail. I read the man about backup file.

When  relocating the first few stripes on a RAID5 or RAID6, it is not possible to keep the data on disk completely
consistent and crash-proof.  To provide the required safety, mdadm disables writes to the array while this "critical  
section"  is reshaped, and takes a backup of the data that is in that section.  

What's the reason about data consistent when relocate data? 

> 
> And in response to your other email:
> >     Does it should return 1 when pid > 0 and status is not zero?
> 
> No.  continue_via_systemd should return 1 precisely when the 'systemctl'
> command was successfully run.  So 'status' must be zero.
> 
> 

I got this. So reshape_array should return when continue_via_systemd return 1. Then the
reshape is going on when run the command mdadm --grow --continue. Now the child_monitor
is called and sync_max is set to max.

Best Regards
Xiao

Re: raid5 reshape is stuck

[Xiao Ni]

----- Original Message -----
> From: "NeilBrown" <neilb@suse.de>
> To: "Xiao Ni" <xni@redhat.com>
> Cc: linux-raid@vger.kernel.org
> Sent: Monday, May 25, 2015 11:50:01 AM
> Subject: Re: raid5 reshape is stuck
>
> On Thu, 21 May 2015 08:31:58 -0400 (EDT) Xiao Ni <xni@redhat.com> wrote:
>
> >
> >
> > ----- Original Message -----
> > > From: "Xiao Ni" <xni@redhat.com>
> > > To: "NeilBrown" <neilb@suse.de>
> > > Cc: linux-raid@vger.kernel.org
> > > Sent: Thursday, May 21, 2015 11:37:57 AM
> > > Subject: Re: raid5 reshape is stuck
> > >
> > >
> > >
> > > ----- Original Message -----
> > > > From: "NeilBrown" <neilb@suse.de>
> > > > To: "Xiao Ni" <xni@redhat.com>
> > > > Cc: linux-raid@vger.kernel.org
> > > > Sent: Thursday, May 21, 2015 7:48:37 AM
> > > > Subject: Re: raid5 reshape is stuck
> > > >
> > > > On Fri, 15 May 2015 03:00:24 -0400 (EDT) Xiao Ni <xni@redhat.com>
> > > > wrote:
> > > >
> > > > > Hi Neil
> > > > >
> > > > >    I encounter the problem when I reshape a 4-disks raid5 to raid5.
> > > > >    It
> > > > >    just
> > > > >    can
> > > > > appear with loop devices.
> > > > >
> > > > >    The steps are:
> > > > >
> > > > > [root@dhcp-12-158 mdadm-3.3.2]# mdadm -CR /dev/md0 -l5 -n5
> > > > > /dev/loop[0-4]
> > > > > --assume-clean
> > > > > mdadm: /dev/loop0 appears to be part of a raid array:
> > > > >        level=raid5 devices=6 ctime=Fri May 15 13:47:17 2015
> > > > > mdadm: /dev/loop1 appears to be part of a raid array:
> > > > >        level=raid5 devices=6 ctime=Fri May 15 13:47:17 2015
> > > > > mdadm: /dev/loop2 appears to be part of a raid array:
> > > > >        level=raid5 devices=6 ctime=Fri May 15 13:47:17 2015
> > > > > mdadm: /dev/loop3 appears to be part of a raid array:
> > > > >        level=raid5 devices=6 ctime=Fri May 15 13:47:17 2015
> > > > > mdadm: /dev/loop4 appears to be part of a raid array:
> > > > >        level=raid5 devices=6 ctime=Fri May 15 13:47:17 2015
> > > > > mdadm: Defaulting to version 1.2 metadata
> > > > > mdadm: array /dev/md0 started.
> > > > > [root@dhcp-12-158 mdadm-3.3.2]# mdadm /dev/md0 -a /dev/loop5
> > > > > mdadm: added /dev/loop5
> > > > > [root@dhcp-12-158 mdadm-3.3.2]# mdadm --grow /dev/md0 --raid-devices
> > > > > 6
> > > > > mdadm: Need to backup 10240K of critical section..
> > > > > [root@dhcp-12-158 mdadm-3.3.2]# cat /proc/mdstat
> > > > > Personalities : [raid6] [raid5] [raid4]
> > > > > md0 : active raid5 loop5[5] loop4[4] loop3[3] loop2[2] loop1[1]
> > > > > loop0[0]
> > > > >       8187904 blocks super 1.2 level 5, 512k chunk, algorithm 2 [6/6]
> > > > >       [UUUUUU]
> > > > >       [>....................]  reshape =  0.0% (0/2046976)
> > > > >       finish=6396.8min
> > > > >       speed=0K/sec
> > > > >      
> > > > > unused devices: <none>
> > > > >
> > > > >    It because the sync_max is set to 0 when run the command --grow
> > > > >
> > > > > [root@dhcp-12-158 mdadm-3.3.2]# cd /sys/block/md0/md/
> > > > > [root@dhcp-12-158 md]# cat sync_max
> > > > > 0
> > > > >
> > > > >    I tried reproduce with normal sata devices. The progress of
> > > > >    reshape is
> > > > >    no problem. Then
> > > > > I checked the Grow.c. If I use sata devices, in function
> > > > > reshape_array,
> > > > > the
> > > > > return value
> > > > > of set_new_data_offset is 0. But if I used loop devices, it return 1.
> > > > > Then
> > > > > it call the function
> > > > > start_reshape.
> > > >
> > > > set_new_data_offset returns '0' if there is room on the devices to
> > > > reduce
> > > > the
> > > > data offset so that the reshape starts writing to unused space on the
> > > > array.
> > > > This removes the need for a backup file, or the use of a spare device
> > > > to
> > > > store a temporary backup.
> > > > It returns '1' if there was no room for relocating the data_offset.
> > > >
> > > > So on your sata devices (which are presumably larger than your loop
> > > > devices)
> > > > there was room.  On your loop devices there was not.
> > > >
> > > >
> > > > >
> > > > >    In the function start_reshape it set the sync_max to
> > > > >    reshape_progress.
> > > > >    But in sysfs_read it
> > > > > doesn't read reshape_progress. So it's 0 and the sync_max is set to
> > > > > 0.
> > > > > Why
> > > > > it need to set the
> > > > > sync_max at this? I'm not sure about this.
> > > >
> > > > sync_max is set to 0 so that the reshape does not start until the
> > > > backup
> > > > has
> > > > been taken.
> > > > Once the backup is taken, child_monitor() should set sync_max to "max".
> > > >
> > > > Can you  check if that is happening?
> > > >
> > > > Thanks,
> > > > NeilBrown
> > > >
> > > >
> > >
> > >   Thanks very much for the explaining. The problem maybe is fixed. I
> > >   tried
> > >   reproduce this with newest
> > > kernel and newest mdadm. Now the problem don't exist. I'll do more tests
> > > and
> > > give the answer above later.
> > >
> >
> > Hi Neil
> >
> >    As you said, it doesn't enter child monitor. The problem still exist.
> >
> > The kernel version :
> > [root@intel-canoepass-02 tmp]# uname -r
> > 4.0.4
> >
> > mdadm I used is the newest git code from
> > git://git.neil.brown.name/mdadm.git
> >
> >    
> >    In the function continue_via_systemd the parent find pid is bigger than
> >    0 and
> > status is 0. So it return 1. So it have no opportunity to call
> > child_monitor.
>
> If continue_via_systemd succeeded, that implies that
>   systemctl start mdadm-grow-continue@mdXXX.service
>
> succeeded.  So
>    mdadm --grow --continue /dev/mdXXX
>
> was run, so that mdadm should call 'child_monitor' and update sync_max when
> appropriate.  Can you check if it does?

The service is not running.

[root@intel-waimeabay-hedt-01 create_assemble]# systemctl start mdadm-grow-continue@md0.service
[root@intel-waimeabay-hedt-01 create_assemble]# echo $?
0
[root@intel-waimeabay-hedt-01 create_assemble]# systemctl status mdadm-grow-continue@md0.service
mdadm-grow-continue@md0.service - Manage MD Reshape on /dev/md0
   Loaded: loaded (/usr/lib/systemd/system/mdadm-grow-continue@.service; static)
   Active: failed (Result: exit-code) since Tue 2015-05-26 05:33:59 EDT; 21s ago
  Process: 5374 ExecStart=/usr/sbin/mdadm --grow --continue /dev/%I (code=exited, status=1/FAILURE)
 Main PID: 5374 (code=exited, status=1/FAILURE)

May 26 05:33:59 intel-waimeabay-hedt-01.lab.eng.rdu.redhat.com systemd[1]: Started Manage MD Reshape on /dev/md0.
May 26 05:33:59 intel-waimeabay-hedt-01.lab.eng.rdu.redhat.com systemd[1]: mdadm-grow-continue@md0.service: main process exited, ...URE
May 26 05:33:59 intel-waimeabay-hedt-01.lab.eng.rdu.redhat.com systemd[1]: Unit mdadm-grow-continue@md0.service entered failed state.
Hint: Some lines were ellipsized, use -l to show in full.

[root@intel-waimeabay-hedt-01 create_assemble]# mdadm --grow --continue /dev/md0 --backup-file=tmp0
mdadm: Need to backup 6144K of critical section..

Now the reshape start.

Try modify the service file :
ExecStart=/usr/sbin/mdadm --grow --continue /dev/%I --backup-file=/root/tmp0

It doesn't work too.

[root@intel-waimeabay-hedt-01 ~]# systemctl daemon-reload
[root@intel-waimeabay-hedt-01 ~]# systemctl start mdadm-grow-continue@md0.service
[root@intel-waimeabay-hedt-01 ~]# systemctl status mdadm-grow-continue@md0.service
mdadm-grow-continue@md0.service - Manage MD Reshape on /dev/md0
   Loaded: loaded (/usr/lib/systemd/system/mdadm-grow-continue@.service; static)
   Active: failed (Result: exit-code) since Tue 2015-05-26 05:50:22 EDT; 10s ago
  Process: 6475 ExecStart=/usr/sbin/mdadm --grow --continue /dev/%I --backup-file=/root/tmp0 (code=exited, status=1/FAILURE)
 Main PID: 6475 (code=exited, status=1/FAILURE)

May 26 05:50:22 intel-waimeabay-hedt-01.lab.eng.rdu.redhat.com systemd[1]: Started Manage MD Reshape on /dev/md0.
May 26 05:50:22 intel-waimeabay-hedt-01.lab.eng.rdu.redhat.com systemd[1]: mdadm-grow-continue@md0.service: main process exited, ...URE
May 26 05:50:22 intel-waimeabay-hedt-01.lab.eng.rdu.redhat.com systemd[1]: Unit mdadm-grow-continue@md0.service entered failed state.
Hint: Some lines were ellipsized, use -l to show in full.


  
>
>
> >
> >
> >    And if it want to set sync_max to 0 until the backup has been taken. Why
> >    does not
> > set sync_max to 0 directly, but use the value reshape_progress? There is a
> > little confused.
>
> When reshaping an array to a different array of the same size, such as a
> 4-driver RAID5 to a 5-driver RAID6, then mdadm needs to backup, one piece at
> a time, the entire array (unless it can change data_offset, which is a
> relatively new ability).
>
> If you stop an array when it is in the middle of such a reshape, and then
> reassemble the array, the backup process need to recommence where it left
> off.
> So it tells the kernel that the reshape can progress as far as where it was
> up to before.  So 'sync_max' is set based on the value of 'reshape_progress'.
> (This will happen almost instantly).
>
> Then the background mdadm (or the mdadm started by systemd) will backup the
> next few stripes, update sync_max, wait for those stripes to be reshaped,
> then
> discard the old backup, create a new one of the few stripes after that, and
> continue.
>
> Does that make it a little clearer?

This is a big dinner for me. I need digest this for a while. Thanks very much
for this. What's the "backup process"?

Could you explain backup in detail. I read the man about backup file.

When  relocating the first few stripes on a RAID5 or RAID6, it is not possible to keep the data on disk completely
consistent and crash-proof.  To provide the required safety, mdadm disables writes to the array while this "critical  
section"  is reshaped, and takes a backup of the data that is in that section.  

What's the reason about data consistent when relocate data?

>
> And in response to your other email:
> >     Does it should return 1 when pid > 0 and status is not zero?
>
> No.  continue_via_systemd should return 1 precisely when the 'systemctl'
> command was successfully run.  So 'status' must be zero.
>
>

I got this. So reshape_array should return when continue_via_systemd return 1. Then the
reshape is going on when run the command mdadm --grow --continue. Now the child_monitor
is called and sync_max is set to max.

Best Regards
Xiao

Re: raid5 reshape is stuck

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 6987 bytes --]

On Tue, 26 May 2015 06:48:23 -0400 (EDT) Xiao Ni <xni@redhat.com> wrote:


> > >    
> > >    In the function continue_via_systemd the parent find pid is bigger than
> > >    0 and
> > > status is 0. So it return 1. So it have no opportunity to call
> > > child_monitor.
> >
> > If continue_via_systemd succeeded, that implies that
> >   systemctl start mdadm-grow-continue@mdXXX.service
> >
> > succeeded.  So
> >    mdadm --grow --continue /dev/mdXXX
> >
> > was run, so that mdadm should call 'child_monitor' and update sync_max when
> > appropriate.  Can you check if it does?
> 
> The service is not running.
> 
> [root@intel-waimeabay-hedt-01 create_assemble]# systemctl start mdadm-grow-continue@md0.service
> [root@intel-waimeabay-hedt-01 create_assemble]# echo $?
> 0
> [root@intel-waimeabay-hedt-01 create_assemble]# systemctl status mdadm-grow-continue@md0.service
> mdadm-grow-continue@md0.service - Manage MD Reshape on /dev/md0
>    Loaded: loaded (/usr/lib/systemd/system/mdadm-grow-continue@.service; static)
>    Active: failed (Result: exit-code) since Tue 2015-05-26 05:33:59 EDT; 21s ago
>   Process: 5374 ExecStart=/usr/sbin/mdadm --grow --continue /dev/%I (code=exited, status=1/FAILURE)
>  Main PID: 5374 (code=exited, status=1/FAILURE)
> 
> May 26 05:33:59 intel-waimeabay-hedt-01.lab.eng.rdu.redhat.com systemd[1]: Started Manage MD Reshape on /dev/md0.
> May 26 05:33:59 intel-waimeabay-hedt-01.lab.eng.rdu.redhat.com systemd[1]: mdadm-grow-continue@md0.service: main process exited, ...URE
> May 26 05:33:59 intel-waimeabay-hedt-01.lab.eng.rdu.redhat.com systemd[1]: Unit mdadm-grow-continue@md0.service entered failed state.
> Hint: Some lines were ellipsized, use -l to show in full.

Hmm.. I wonder why systemctl isn't reporting the error message from mdadm.


> 
> [root@intel-waimeabay-hedt-01 create_assemble]# mdadm --grow --continue /dev/md0 --backup-file=tmp0
> mdadm: Need to backup 6144K of critical section..
> 
> Now the reshape start.
> 
> Try modify the service file :
> ExecStart=/usr/sbin/mdadm --grow --continue /dev/%I --backup-file=/root/tmp0
> 
> It doesn't work too.

I tried that change and it make it work.

> 
> [root@intel-waimeabay-hedt-01 ~]# systemctl daemon-reload
> [root@intel-waimeabay-hedt-01 ~]# systemctl start mdadm-grow-continue@md0.service
> [root@intel-waimeabay-hedt-01 ~]# systemctl status mdadm-grow-continue@md0.service
> mdadm-grow-continue@md0.service - Manage MD Reshape on /dev/md0
>    Loaded: loaded (/usr/lib/systemd/system/mdadm-grow-continue@.service; static)
>    Active: failed (Result: exit-code) since Tue 2015-05-26 05:50:22 EDT; 10s ago
>   Process: 6475 ExecStart=/usr/sbin/mdadm --grow --continue /dev/%I --backup-file=/root/tmp0 (code=exited, status=1/FAILURE)
>  Main PID: 6475 (code=exited, status=1/FAILURE)
> 
> May 26 05:50:22 intel-waimeabay-hedt-01.lab.eng.rdu.redhat.com systemd[1]: Started Manage MD Reshape on /dev/md0.
> May 26 05:50:22 intel-waimeabay-hedt-01.lab.eng.rdu.redhat.com systemd[1]: mdadm-grow-continue@md0.service: main process exited, ...URE
> May 26 05:50:22 intel-waimeabay-hedt-01.lab.eng.rdu.redhat.com systemd[1]: Unit mdadm-grow-continue@md0.service entered failed state.
> Hint: Some lines were ellipsized, use -l to show in full.
> 
> 
>   
> >
> >
> > >
> > >
> > >    And if it want to set sync_max to 0 until the backup has been taken. Why
> > >    does not
> > > set sync_max to 0 directly, but use the value reshape_progress? There is a
> > > little confused.
> >
> > When reshaping an array to a different array of the same size, such as a
> > 4-driver RAID5 to a 5-driver RAID6, then mdadm needs to backup, one piece at
> > a time, the entire array (unless it can change data_offset, which is a
> > relatively new ability).
> >
> > If you stop an array when it is in the middle of such a reshape, and then
> > reassemble the array, the backup process need to recommence where it left
> > off.
> > So it tells the kernel that the reshape can progress as far as where it was
> > up to before.  So 'sync_max' is set based on the value of 'reshape_progress'.
> > (This will happen almost instantly).
> >
> > Then the background mdadm (or the mdadm started by systemd) will backup the
> > next few stripes, update sync_max, wait for those stripes to be reshaped,
> > then
> > discard the old backup, create a new one of the few stripes after that, and
> > continue.
> >
> > Does that make it a little clearer?
> 
> This is a big dinner for me. I need digest this for a while. Thanks very much
> for this. What's the "backup process"?
> 
> Could you explain backup in detail. I read the man about backup file.
> 
> When  relocating the first few stripes on a RAID5 or RAID6, it is not possible to keep the data on disk completely
> consistent and crash-proof.  To provide the required safety, mdadm disables writes to the array while this "critical  
> section"  is reshaped, and takes a backup of the data that is in that section.  
> 
> What's the reason about data consistent when relocate data?

If you are reshaping a RAID5 from 3 drives to 4 drives, then the first stripe
will start out as:

   D0  D1   P   -

and you want to change it to

   D0  D1   D2  P

If the system crashes while that is happening, you won't know if either or
both of D2 and P were written, but it is fairly safe just to assume they
weren't and recalculate the parity.
However the second stripe will initially be:

   P  D2  D3 

and you want to change it to

   P  D3  D4  D5

If you crash in the middle of doing that you cannot know which block is D3
- if either.  D4 might have been written, and D3 not yet written.  So D3 is
lost.  

So mdadm takes a copy of a whole stripe, allows the kernel to reshape that
one stripe, updates the metadata to record that the stripe has been fully
reshaped, and then discards the backup.
So if you crash in the middle of reshaping the second stripe above, mdadm
will restore it from the backup.

The backup can be stored in a separate file, or in a device which is being
added to the array.


The reason why "mdadm --grow --continue" doesn't work unless you add the
"--backup=...." is because it doesn't find the "device  being added" - it
looks for a spare, but there aren't any spares any more.   That should be
easy enough to fix.

Thanks,
NeilBrown

> 
> >
> > And in response to your other email:
> > >     Does it should return 1 when pid > 0 and status is not zero?
> >
> > No.  continue_via_systemd should return 1 precisely when the 'systemctl'
> > command was successfully run.  So 'status' must be zero.
> >
> >
> 
> I got this. So reshape_array should return when continue_via_systemd return 1. Then the
> reshape is going on when run the command mdadm --grow --continue. Now the child_monitor
> is called and sync_max is set to max.
> 
> Best Regards
> Xiao
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]

Re: raid5 reshape is stuck

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 7996 bytes --]

On Wed, 27 May 2015 10:02:53 +1000 NeilBrown <neilb@suse.de> wrote:

> On Tue, 26 May 2015 06:48:23 -0400 (EDT) Xiao Ni <xni@redhat.com> wrote:
> 
> 
> > > >    
> > > >    In the function continue_via_systemd the parent find pid is bigger than
> > > >    0 and
> > > > status is 0. So it return 1. So it have no opportunity to call
> > > > child_monitor.
> > >
> > > If continue_via_systemd succeeded, that implies that
> > >   systemctl start mdadm-grow-continue@mdXXX.service
> > >
> > > succeeded.  So
> > >    mdadm --grow --continue /dev/mdXXX
> > >
> > > was run, so that mdadm should call 'child_monitor' and update sync_max when
> > > appropriate.  Can you check if it does?
> > 
> > The service is not running.
> > 
> > [root@intel-waimeabay-hedt-01 create_assemble]# systemctl start mdadm-grow-continue@md0.service
> > [root@intel-waimeabay-hedt-01 create_assemble]# echo $?
> > 0
> > [root@intel-waimeabay-hedt-01 create_assemble]# systemctl status mdadm-grow-continue@md0.service
> > mdadm-grow-continue@md0.service - Manage MD Reshape on /dev/md0
> >    Loaded: loaded (/usr/lib/systemd/system/mdadm-grow-continue@.service; static)
> >    Active: failed (Result: exit-code) since Tue 2015-05-26 05:33:59 EDT; 21s ago
> >   Process: 5374 ExecStart=/usr/sbin/mdadm --grow --continue /dev/%I (code=exited, status=1/FAILURE)
> >  Main PID: 5374 (code=exited, status=1/FAILURE)
> > 
> > May 26 05:33:59 intel-waimeabay-hedt-01.lab.eng.rdu.redhat.com systemd[1]: Started Manage MD Reshape on /dev/md0.
> > May 26 05:33:59 intel-waimeabay-hedt-01.lab.eng.rdu.redhat.com systemd[1]: mdadm-grow-continue@md0.service: main process exited, ...URE
> > May 26 05:33:59 intel-waimeabay-hedt-01.lab.eng.rdu.redhat.com systemd[1]: Unit mdadm-grow-continue@md0.service entered failed state.
> > Hint: Some lines were ellipsized, use -l to show in full.
> 
> Hmm.. I wonder why systemctl isn't reporting the error message from mdadm.
> 
> 
> > 
> > [root@intel-waimeabay-hedt-01 create_assemble]# mdadm --grow --continue /dev/md0 --backup-file=tmp0
> > mdadm: Need to backup 6144K of critical section..
> > 
> > Now the reshape start.
> > 
> > Try modify the service file :
> > ExecStart=/usr/sbin/mdadm --grow --continue /dev/%I --backup-file=/root/tmp0
> > 
> > It doesn't work too.
> 
> I tried that change and it make it work.
> 
> > 
> > [root@intel-waimeabay-hedt-01 ~]# systemctl daemon-reload
> > [root@intel-waimeabay-hedt-01 ~]# systemctl start mdadm-grow-continue@md0.service
> > [root@intel-waimeabay-hedt-01 ~]# systemctl status mdadm-grow-continue@md0.service
> > mdadm-grow-continue@md0.service - Manage MD Reshape on /dev/md0
> >    Loaded: loaded (/usr/lib/systemd/system/mdadm-grow-continue@.service; static)
> >    Active: failed (Result: exit-code) since Tue 2015-05-26 05:50:22 EDT; 10s ago
> >   Process: 6475 ExecStart=/usr/sbin/mdadm --grow --continue /dev/%I --backup-file=/root/tmp0 (code=exited, status=1/FAILURE)
> >  Main PID: 6475 (code=exited, status=1/FAILURE)
> > 
> > May 26 05:50:22 intel-waimeabay-hedt-01.lab.eng.rdu.redhat.com systemd[1]: Started Manage MD Reshape on /dev/md0.
> > May 26 05:50:22 intel-waimeabay-hedt-01.lab.eng.rdu.redhat.com systemd[1]: mdadm-grow-continue@md0.service: main process exited, ...URE
> > May 26 05:50:22 intel-waimeabay-hedt-01.lab.eng.rdu.redhat.com systemd[1]: Unit mdadm-grow-continue@md0.service entered failed state.
> > Hint: Some lines were ellipsized, use -l to show in full.
> > 
> > 
> >   
> > >
> > >
> > > >
> > > >
> > > >    And if it want to set sync_max to 0 until the backup has been taken. Why
> > > >    does not
> > > > set sync_max to 0 directly, but use the value reshape_progress? There is a
> > > > little confused.
> > >
> > > When reshaping an array to a different array of the same size, such as a
> > > 4-driver RAID5 to a 5-driver RAID6, then mdadm needs to backup, one piece at
> > > a time, the entire array (unless it can change data_offset, which is a
> > > relatively new ability).
> > >
> > > If you stop an array when it is in the middle of such a reshape, and then
> > > reassemble the array, the backup process need to recommence where it left
> > > off.
> > > So it tells the kernel that the reshape can progress as far as where it was
> > > up to before.  So 'sync_max' is set based on the value of 'reshape_progress'.
> > > (This will happen almost instantly).
> > >
> > > Then the background mdadm (or the mdadm started by systemd) will backup the
> > > next few stripes, update sync_max, wait for those stripes to be reshaped,
> > > then
> > > discard the old backup, create a new one of the few stripes after that, and
> > > continue.
> > >
> > > Does that make it a little clearer?
> > 
> > This is a big dinner for me. I need digest this for a while. Thanks very much
> > for this. What's the "backup process"?
> > 
> > Could you explain backup in detail. I read the man about backup file.
> > 
> > When  relocating the first few stripes on a RAID5 or RAID6, it is not possible to keep the data on disk completely
> > consistent and crash-proof.  To provide the required safety, mdadm disables writes to the array while this "critical  
> > section"  is reshaped, and takes a backup of the data that is in that section.  
> > 
> > What's the reason about data consistent when relocate data?
> 
> If you are reshaping a RAID5 from 3 drives to 4 drives, then the first stripe
> will start out as:
> 
>    D0  D1   P   -
> 
> and you want to change it to
> 
>    D0  D1   D2  P
> 
> If the system crashes while that is happening, you won't know if either or
> both of D2 and P were written, but it is fairly safe just to assume they
> weren't and recalculate the parity.
> However the second stripe will initially be:
> 
>    P  D2  D3 
> 
> and you want to change it to
> 
>    P  D3  D4  D5
> 
> If you crash in the middle of doing that you cannot know which block is D3
> - if either.  D4 might have been written, and D3 not yet written.  So D3 is
> lost.  
> 
> So mdadm takes a copy of a whole stripe, allows the kernel to reshape that
> one stripe, updates the metadata to record that the stripe has been fully
> reshaped, and then discards the backup.
> So if you crash in the middle of reshaping the second stripe above, mdadm
> will restore it from the backup.
> 
> The backup can be stored in a separate file, or in a device which is being
> added to the array.
> 
> 
> The reason why "mdadm --grow --continue" doesn't work unless you add the
> "--backup=...." is because it doesn't find the "device  being added" - it
> looks for a spare, but there aren't any spares any more.   That should be
> easy enough to fix.

That wasn't too painful - I think this fixes the problem.
Could you confirm?

Thanks,
NeilBrown


diff --git a/Grow.c b/Grow.c
index a20ff3e70142..85de1d27f03a 100644
--- a/Grow.c
+++ b/Grow.c
@@ -850,7 +850,8 @@ int reshape_prepare_fdlist(char *devname,
 	for (sd = sra->devs; sd; sd = sd->next) {
 		if (sd->disk.state & (1<<MD_DISK_FAULTY))
 			continue;
-		if (sd->disk.state & (1<<MD_DISK_SYNC)) {
+		if (sd->disk.state & (1<<MD_DISK_SYNC) &&
+		    sd->disk.raid_disk < raid_disks) {
 			char *dn = map_dev(sd->disk.major,
 					   sd->disk.minor, 1);
 			fdlist[sd->disk.raid_disk]
@@ -3184,7 +3185,7 @@ started:
 	d = reshape_prepare_fdlist(devname, sra, odisks,
 				   nrdisks, blocks, backup_file,
 				   fdlist, offsets);
-	if (d < 0) {
+	if (d < odisks) {
 		goto release;
 	}
 	if ((st->ss->manage_reshape == NULL) ||
@@ -3196,7 +3197,7 @@ started:
 				       devname);
 				pr_err(" Please provide one with \"--backup=...\"\n");
 				goto release;
-			} else if (sra->array.spare_disks == 0) {
+			} else if (d == odisks) {
 				pr_err("%s: Cannot grow - need a spare or backup-file to backup critical section\n", devname);
 				goto release;
 			}


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]

Re: raid5 reshape is stuck

[Xiao Ni]

----- Original Message -----
> From: "NeilBrown" <neilb@suse.de>
> To: "Xiao Ni" <xni@redhat.com>
> Cc: linux-raid@vger.kernel.org
> Sent: Wednesday, May 27, 2015 9:10:04 AM
> Subject: Re: raid5 reshape is stuck
> 
> On Wed, 27 May 2015 10:02:53 +1000 NeilBrown <neilb@suse.de> wrote:
> 
> > On Tue, 26 May 2015 06:48:23 -0400 (EDT) Xiao Ni <xni@redhat.com> wrote:
> > 
> > 
> > > > >    
> > > > >    In the function continue_via_systemd the parent find pid is bigger
> > > > >    than
> > > > >    0 and
> > > > > status is 0. So it return 1. So it have no opportunity to call
> > > > > child_monitor.
> > > >
> > > > If continue_via_systemd succeeded, that implies that
> > > >   systemctl start mdadm-grow-continue@mdXXX.service
> > > >
> > > > succeeded.  So
> > > >    mdadm --grow --continue /dev/mdXXX
> > > >
> > > > was run, so that mdadm should call 'child_monitor' and update sync_max
> > > > when
> > > > appropriate.  Can you check if it does?
> > > 
> > > The service is not running.
> > > 
> > > [root@intel-waimeabay-hedt-01 create_assemble]# systemctl start
> > > mdadm-grow-continue@md0.service
> > > [root@intel-waimeabay-hedt-01 create_assemble]# echo $?
> > > 0
> > > [root@intel-waimeabay-hedt-01 create_assemble]# systemctl status
> > > mdadm-grow-continue@md0.service
> > > mdadm-grow-continue@md0.service - Manage MD Reshape on /dev/md0
> > >    Loaded: loaded (/usr/lib/systemd/system/mdadm-grow-continue@.service;
> > >    static)
> > >    Active: failed (Result: exit-code) since Tue 2015-05-26 05:33:59 EDT;
> > >    21s ago
> > >   Process: 5374 ExecStart=/usr/sbin/mdadm --grow --continue /dev/%I
> > >   (code=exited, status=1/FAILURE)
> > >  Main PID: 5374 (code=exited, status=1/FAILURE)
> > > 
> > > May 26 05:33:59 intel-waimeabay-hedt-01.lab.eng.rdu.redhat.com
> > > systemd[1]: Started Manage MD Reshape on /dev/md0.
> > > May 26 05:33:59 intel-waimeabay-hedt-01.lab.eng.rdu.redhat.com
> > > systemd[1]: mdadm-grow-continue@md0.service: main process exited, ...URE
> > > May 26 05:33:59 intel-waimeabay-hedt-01.lab.eng.rdu.redhat.com
> > > systemd[1]: Unit mdadm-grow-continue@md0.service entered failed state.
> > > Hint: Some lines were ellipsized, use -l to show in full.
> > 
> > Hmm.. I wonder why systemctl isn't reporting the error message from mdadm.

I don't know the reason too. The return value $? is 0 after run systemctl start.
But the status is failed.
 
> > 
> > 
> > > 
> > > [root@intel-waimeabay-hedt-01 create_assemble]# mdadm --grow --continue
> > > /dev/md0 --backup-file=tmp0
> > > mdadm: Need to backup 6144K of critical section..
> > > 
> > > Now the reshape start.
> > > 
> > > Try modify the service file :
> > > ExecStart=/usr/sbin/mdadm --grow --continue /dev/%I
> > > --backup-file=/root/tmp0
> > > 
> > > It doesn't work too.
> > 
> > I tried that change and it make it work.

[root@intel-waimeabay-hedt-01 mdadm]# cat /usr/lib/systemd/system/mdadm-grow-continue\@.service 
#  This file is part of mdadm.
#
#  mdadm is free software; you can redistribute it and/or modify it
#  under the terms of the GNU General Public License as published by
#  the Free Software Foundation; either version 2 of the License, or
#  (at your option) any later version.

[Unit]
Description=Manage MD Reshape on /dev/%I
DefaultDependencies=no

[Service]
ExecStart=/usr/sbin/mdadm --grow --continue /dev/%I --backup-file=/root/tmp0
StandardInput=null
StandardOutput=null
StandardError=null
KillMode=none
[root@intel-waimeabay-hedt-01 mdadm]# cat /proc/mdstat 
Personalities : [raid6] [raid5] [raid4] 
md0 : active raid5 loop4[4] loop3[3] loop2[2] loop1[1] loop0[0]
      1532928 blocks super 1.2 level 5, 512k chunk, algorithm 2 [5/5] [UUUUU]
      [>....................]  reshape =  0.0% (0/510976) finish=532.2min speed=0K/sec
      
unused devices: <none>
[root@intel-waimeabay-hedt-01 mdadm]# systemctl start mdadm-grow-continue@md0.service
[root@intel-waimeabay-hedt-01 mdadm]# systemctl status mdadm-grow-continue@md0.service
mdadm-grow-continue@md0.service - Manage MD Reshape on /dev/md0
   Loaded: loaded (/usr/lib/systemd/system/mdadm-grow-continue@.service; static)
   Active: failed (Result: exit-code) since Wed 2015-05-27 02:45:40 EDT; 12s ago
  Process: 24596 ExecStart=/usr/sbin/mdadm --grow --continue /dev/%I --backup-file=/root/tmp0 (code=exited, status=1/FAILURE)
 Main PID: 24596 (code=exited, status=1/FAILURE)

May 27 02:45:40 intel-waimeabay-hedt-01.lab.eng.rdu.redhat.com systemd[1]: Started Manage MD Reshape on /dev/md0.
May 27 02:45:40 intel-waimeabay-hedt-01.lab.eng.rdu.redhat.com systemd[1]: mdadm-grow-continue@md0.service: main process exited, ...URE
May 27 02:45:40 intel-waimeabay-hedt-01.lab.eng.rdu.redhat.com systemd[1]: Unit mdadm-grow-continue@md0.service entered failed state.
Hint: Some lines were ellipsized, use -l to show in full.

It's still failed after changing the file. 


> > 
> > > 
> > > [root@intel-waimeabay-hedt-01 ~]# systemctl daemon-reload
> > > [root@intel-waimeabay-hedt-01 ~]# systemctl start
> > > mdadm-grow-continue@md0.service
> > > [root@intel-waimeabay-hedt-01 ~]# systemctl status
> > > mdadm-grow-continue@md0.service
> > > mdadm-grow-continue@md0.service - Manage MD Reshape on /dev/md0
> > >    Loaded: loaded (/usr/lib/systemd/system/mdadm-grow-continue@.service;
> > >    static)
> > >    Active: failed (Result: exit-code) since Tue 2015-05-26 05:50:22 EDT;
> > >    10s ago
> > >   Process: 6475 ExecStart=/usr/sbin/mdadm --grow --continue /dev/%I
> > >   --backup-file=/root/tmp0 (code=exited, status=1/FAILURE)
> > >  Main PID: 6475 (code=exited, status=1/FAILURE)
> > > 
> > > May 26 05:50:22 intel-waimeabay-hedt-01.lab.eng.rdu.redhat.com
> > > systemd[1]: Started Manage MD Reshape on /dev/md0.
> > > May 26 05:50:22 intel-waimeabay-hedt-01.lab.eng.rdu.redhat.com
> > > systemd[1]: mdadm-grow-continue@md0.service: main process exited, ...URE
> > > May 26 05:50:22 intel-waimeabay-hedt-01.lab.eng.rdu.redhat.com
> > > systemd[1]: Unit mdadm-grow-continue@md0.service entered failed state.
> > > Hint: Some lines were ellipsized, use -l to show in full.
> > > 
> > > 
> > >   
> > > >
> > > >
> > > > >
> > > > >
> > > > >    And if it want to set sync_max to 0 until the backup has been
> > > > >    taken. Why
> > > > >    does not
> > > > > set sync_max to 0 directly, but use the value reshape_progress? There
> > > > > is a
> > > > > little confused.
> > > >
> > > > When reshaping an array to a different array of the same size, such as
> > > > a
> > > > 4-driver RAID5 to a 5-driver RAID6, then mdadm needs to backup, one
> > > > piece at
> > > > a time, the entire array (unless it can change data_offset, which is a
> > > > relatively new ability).
> > > >
> > > > If you stop an array when it is in the middle of such a reshape, and
> > > > then
> > > > reassemble the array, the backup process need to recommence where it
> > > > left
> > > > off.
> > > > So it tells the kernel that the reshape can progress as far as where it
> > > > was
> > > > up to before.  So 'sync_max' is set based on the value of
> > > > 'reshape_progress'.
> > > > (This will happen almost instantly).
> > > >
> > > > Then the background mdadm (or the mdadm started by systemd) will backup
> > > > the
> > > > next few stripes, update sync_max, wait for those stripes to be
> > > > reshaped,
> > > > then
> > > > discard the old backup, create a new one of the few stripes after that,
> > > > and
> > > > continue.
> > > >
> > > > Does that make it a little clearer?
> > > 
> > > This is a big dinner for me. I need digest this for a while. Thanks very
> > > much
> > > for this. What's the "backup process"?
> > > 
> > > Could you explain backup in detail. I read the man about backup file.
> > > 
> > > When  relocating the first few stripes on a RAID5 or RAID6, it is not
> > > possible to keep the data on disk completely
> > > consistent and crash-proof.  To provide the required safety, mdadm
> > > disables writes to the array while this "critical
> > > section"  is reshaped, and takes a backup of the data that is in that
> > > section.
> > > 
> > > What's the reason about data consistent when relocate data?
> > 
> > If you are reshaping a RAID5 from 3 drives to 4 drives, then the first
> > stripe
> > will start out as:
> > 
> >    D0  D1   P   -
> > 
> > and you want to change it to
> > 
> >    D0  D1   D2  P
> > 
> > If the system crashes while that is happening, you won't know if either or
> > both of D2 and P were written, but it is fairly safe just to assume they
> > weren't and recalculate the parity.
> > However the second stripe will initially be:
> > 
> >    P  D2  D3
> > 
> > and you want to change it to
> > 
> >    P  D3  D4  D5
> > 
> > If you crash in the middle of doing that you cannot know which block is D3
> > - if either.  D4 might have been written, and D3 not yet written.  So D3 is
> > lost.
> > 
> > So mdadm takes a copy of a whole stripe, allows the kernel to reshape that
> > one stripe, updates the metadata to record that the stripe has been fully
> > reshaped, and then discards the backup.
> > So if you crash in the middle of reshaping the second stripe above, mdadm
> > will restore it from the backup.
> > 
> > The backup can be stored in a separate file, or in a device which is being
> > added to the array.
> > 
> > 
> > The reason why "mdadm --grow --continue" doesn't work unless you add the
> > "--backup=...." is because it doesn't find the "device  being added" - it
> > looks for a spare, but there aren't any spares any more.   That should be
> > easy enough to fix.


   :) I got this. Thanks for the details 
> 
> That wasn't too painful - I think this fixes the problem.
> Could you confirm?
> 
> Thanks,
> NeilBrown
> 
> 
> diff --git a/Grow.c b/Grow.c
> index a20ff3e70142..85de1d27f03a 100644
> --- a/Grow.c
> +++ b/Grow.c
> @@ -850,7 +850,8 @@ int reshape_prepare_fdlist(char *devname,
>  	for (sd = sra->devs; sd; sd = sd->next) {
>  		if (sd->disk.state & (1<<MD_DISK_FAULTY))
>  			continue;
> -		if (sd->disk.state & (1<<MD_DISK_SYNC)) {
> +		if (sd->disk.state & (1<<MD_DISK_SYNC) &&
> +		    sd->disk.raid_disk < raid_disks) {
>  			char *dn = map_dev(sd->disk.major,
>  					   sd->disk.minor, 1);
>  			fdlist[sd->disk.raid_disk]
> @@ -3184,7 +3185,7 @@ started:
>  	d = reshape_prepare_fdlist(devname, sra, odisks,
>  				   nrdisks, blocks, backup_file,
>  				   fdlist, offsets);
> -	if (d < 0) {
> +	if (d < odisks) {
>  		goto release;
>  	}
>  	if ((st->ss->manage_reshape == NULL) ||
> @@ -3196,7 +3197,7 @@ started:
>  				       devname);
>  				pr_err(" Please provide one with \"--backup=...\"\n");
>  				goto release;
> -			} else if (sra->array.spare_disks == 0) {
> +			} else if (d == odisks) {
>  				pr_err("%s: Cannot grow - need a spare or backup-file to backup critical
>  				section\n", devname);
>  				goto release;
>  			}
> 
> 

  I tried this, it doesn't work. 

Re: raid5 reshape is stuck

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 1004 bytes --]

On Wed, 27 May 2015 07:28:04 -0400 (EDT) Xiao Ni <xni@redhat.com> wrote:


> [root@intel-waimeabay-hedt-01 mdadm]# cat /usr/lib/systemd/system/mdadm-grow-continue\@.service 
> #  This file is part of mdadm.
> #
> #  mdadm is free software; you can redistribute it and/or modify it
> #  under the terms of the GNU General Public License as published by
> #  the Free Software Foundation; either version 2 of the License, or
> #  (at your option) any later version.
> 
> [Unit]
> Description=Manage MD Reshape on /dev/%I
> DefaultDependencies=no
> 
> [Service]
> ExecStart=/usr/sbin/mdadm --grow --continue /dev/%I --backup-file=/root/tmp0

Please remove the ---backup-file=/root/tmp0 for further testing.  The patch I
provided should make that unnecessary.

> StandardInput=null
> StandardOutput=null
> StandardError=null

Could you try removing these - that might allow error messages to appear.
I wonder why I included them - they shouldn't be needed.

Thanks,
NeilBrown


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]

Re: raid5 reshape is stuck

[Xiao Ni]

----- Original Message -----
> From: "NeilBrown" <neilb@suse.de>
> To: "Xiao Ni" <xni@redhat.com>
> Cc: linux-raid@vger.kernel.org
> Sent: Wednesday, May 27, 2015 7:34:49 PM
> Subject: Re: raid5 reshape is stuck
> 
> On Wed, 27 May 2015 07:28:04 -0400 (EDT) Xiao Ni <xni@redhat.com> wrote:
> 
> 
> > [root@intel-waimeabay-hedt-01 mdadm]# cat
> > /usr/lib/systemd/system/mdadm-grow-continue\@.service
> > #  This file is part of mdadm.
> > #
> > #  mdadm is free software; you can redistribute it and/or modify it
> > #  under the terms of the GNU General Public License as published by
> > #  the Free Software Foundation; either version 2 of the License, or
> > #  (at your option) any later version.
> > 
> > [Unit]
> > Description=Manage MD Reshape on /dev/%I
> > DefaultDependencies=no
> > 
> > [Service]
> > ExecStart=/usr/sbin/mdadm --grow --continue /dev/%I
> > --backup-file=/root/tmp0
> 
> Please remove the ---backup-file=/root/tmp0 for further testing.  The patch I
> provided should make that unnecessary.
> 
> > StandardInput=null
> > StandardOutput=null
> > StandardError=null
> 
> Could you try removing these - that might allow error messages to appear.
> I wonder why I included them - they shouldn't be needed.
> 
> Thanks,
> NeilBrown
> 
> 

[root@intel-waimeabay-hedt-01 mdadm]# mdadm -CR /dev/md0 -l5 -n4 /dev/loop[0-3] --assume-clean
mdadm: /dev/loop0 appears to be part of a raid array:
       level=raid5 devices=5 ctime=Wed May 27 02:45:08 2015
mdadm: /dev/loop1 appears to be part of a raid array:
       level=raid5 devices=5 ctime=Wed May 27 02:45:08 2015
mdadm: /dev/loop2 appears to be part of a raid array:
       level=raid5 devices=5 ctime=Wed May 27 02:45:08 2015
mdadm: /dev/loop3 appears to be part of a raid array:
       level=raid5 devices=5 ctime=Wed May 27 02:45:08 2015
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md0 started.
[root@intel-waimeabay-hedt-01 mdadm]# mdadm /dev/md0 -a /dev/loop4 
mdadm: added /dev/loop4
[root@intel-waimeabay-hedt-01 mdadm]# mdadm --grow /dev/md0 --raid-devices=5
mdadm: Need to backup 6144K of critical section..
[root@intel-waimeabay-hedt-01 mdadm]# cat /proc/mdstat 
Personalities : [raid6] [raid5] [raid4] 
md0 : active raid5 loop4[4] loop3[3] loop2[2] loop1[1] loop0[0]
      1532928 blocks super 1.2 level 5, 512k chunk, algorithm 2 [5/5] [UUUUU]
      [>....................]  reshape =  0.0% (0/510976) finish=532.2min speed=0K/sec
      
unused devices: <none>
[root@intel-waimeabay-hedt-01 mdadm]# cat /usr/lib/systemd/system/mdadm-grow-continue\@.service 
#  This file is part of mdadm.
#
#  mdadm is free software; you can redistribute it and/or modify it
#  under the terms of the GNU General Public License as published by
#  the Free Software Foundation; either version 2 of the License, or
#  (at your option) any later version.

[Unit]
Description=Manage MD Reshape on /dev/%I
DefaultDependencies=no

[Service]
ExecStart=/usr/sbin/mdadm --grow --continue /dev/%I 
#StandardInput=null
#StandardOutput=null
#StandardError=null
KillMode=none


The problem still exist. And there are messages in /var/log/messages

May 27 08:03:29 intel-waimeabay-hedt-01 systemd: mdadm-grow-continue@md0.service: main process exited, code=exited, status=1/FAILURE
May 27 08:03:29 intel-waimeabay-hedt-01 systemd: Unit mdadm-grow-continue@md0.service entered failed state.

Xiao

Re: raid5 reshape is stuck

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 3958 bytes --]

On Wed, 27 May 2015 08:04:24 -0400 (EDT) Xiao Ni <xni@redhat.com> wrote:

> 
> 
> ----- Original Message -----
> > From: "NeilBrown" <neilb@suse.de>
> > To: "Xiao Ni" <xni@redhat.com>
> > Cc: linux-raid@vger.kernel.org
> > Sent: Wednesday, May 27, 2015 7:34:49 PM
> > Subject: Re: raid5 reshape is stuck
> > 
> > On Wed, 27 May 2015 07:28:04 -0400 (EDT) Xiao Ni <xni@redhat.com> wrote:
> > 
> > 
> > > [root@intel-waimeabay-hedt-01 mdadm]# cat
> > > /usr/lib/systemd/system/mdadm-grow-continue\@.service
> > > #  This file is part of mdadm.
> > > #
> > > #  mdadm is free software; you can redistribute it and/or modify it
> > > #  under the terms of the GNU General Public License as published by
> > > #  the Free Software Foundation; either version 2 of the License, or
> > > #  (at your option) any later version.
> > > 
> > > [Unit]
> > > Description=Manage MD Reshape on /dev/%I
> > > DefaultDependencies=no
> > > 
> > > [Service]
> > > ExecStart=/usr/sbin/mdadm --grow --continue /dev/%I
> > > --backup-file=/root/tmp0
> > 
> > Please remove the ---backup-file=/root/tmp0 for further testing.  The patch I
> > provided should make that unnecessary.
> > 
> > > StandardInput=null
> > > StandardOutput=null
> > > StandardError=null
> > 
> > Could you try removing these - that might allow error messages to appear.
> > I wonder why I included them - they shouldn't be needed.
> > 
> > Thanks,
> > NeilBrown
> > 
> > 
> 
> [root@intel-waimeabay-hedt-01 mdadm]# mdadm -CR /dev/md0 -l5 -n4 /dev/loop[0-3] --assume-clean
> mdadm: /dev/loop0 appears to be part of a raid array:
>        level=raid5 devices=5 ctime=Wed May 27 02:45:08 2015
> mdadm: /dev/loop1 appears to be part of a raid array:
>        level=raid5 devices=5 ctime=Wed May 27 02:45:08 2015
> mdadm: /dev/loop2 appears to be part of a raid array:
>        level=raid5 devices=5 ctime=Wed May 27 02:45:08 2015
> mdadm: /dev/loop3 appears to be part of a raid array:
>        level=raid5 devices=5 ctime=Wed May 27 02:45:08 2015
> mdadm: Defaulting to version 1.2 metadata
> mdadm: array /dev/md0 started.
> [root@intel-waimeabay-hedt-01 mdadm]# mdadm /dev/md0 -a /dev/loop4 
> mdadm: added /dev/loop4
> [root@intel-waimeabay-hedt-01 mdadm]# mdadm --grow /dev/md0 --raid-devices=5
> mdadm: Need to backup 6144K of critical section..
> [root@intel-waimeabay-hedt-01 mdadm]# cat /proc/mdstat 
> Personalities : [raid6] [raid5] [raid4] 
> md0 : active raid5 loop4[4] loop3[3] loop2[2] loop1[1] loop0[0]
>       1532928 blocks super 1.2 level 5, 512k chunk, algorithm 2 [5/5] [UUUUU]
>       [>....................]  reshape =  0.0% (0/510976) finish=532.2min speed=0K/sec
>       
> unused devices: <none>
> [root@intel-waimeabay-hedt-01 mdadm]# cat /usr/lib/systemd/system/mdadm-grow-continue\@.service 
> #  This file is part of mdadm.
> #
> #  mdadm is free software; you can redistribute it and/or modify it
> #  under the terms of the GNU General Public License as published by
> #  the Free Software Foundation; either version 2 of the License, or
> #  (at your option) any later version.
> 
> [Unit]
> Description=Manage MD Reshape on /dev/%I
> DefaultDependencies=no
> 
> [Service]
> ExecStart=/usr/sbin/mdadm --grow --continue /dev/%I 
> #StandardInput=null
> #StandardOutput=null
> #StandardError=null
> KillMode=none
> 
> 
> The problem still exist. And there are messages in /var/log/messages
> 
> May 27 08:03:29 intel-waimeabay-hedt-01 systemd: mdadm-grow-continue@md0.service: main process exited, code=exited, status=1/FAILURE
> May 27 08:03:29 intel-waimeabay-hedt-01 systemd: Unit mdadm-grow-continue@md0.service entered failed state.
> 

Does
  systemctl status -l mdadm-grow-continue@md0.service

report anything different.  That was the result I expected from removing the
Standard*=null lines.

I assume the new mdadm is installed in /usr/sbin/mdadm.

Thanks,
NeilBrown

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]

Re: raid5 reshape is stuck

[Xiao Ni]

----- Original Message -----
> From: "NeilBrown" <neilb@suse.de>
> To: "Xiao Ni" <xni@redhat.com>
> Cc: linux-raid@vger.kernel.org
> Sent: Thursday, May 28, 2015 6:59:58 AM
> Subject: Re: raid5 reshape is stuck
> 
> On Wed, 27 May 2015 08:04:24 -0400 (EDT) Xiao Ni <xni@redhat.com> wrote:
> 
> > 
> > 
> > ----- Original Message -----
> > > From: "NeilBrown" <neilb@suse.de>
> > > To: "Xiao Ni" <xni@redhat.com>
> > > Cc: linux-raid@vger.kernel.org
> > > Sent: Wednesday, May 27, 2015 7:34:49 PM
> > > Subject: Re: raid5 reshape is stuck
> > > 
> > > On Wed, 27 May 2015 07:28:04 -0400 (EDT) Xiao Ni <xni@redhat.com> wrote:
> > > 
> > > 
> > > > [root@intel-waimeabay-hedt-01 mdadm]# cat
> > > > /usr/lib/systemd/system/mdadm-grow-continue\@.service
> > > > #  This file is part of mdadm.
> > > > #
> > > > #  mdadm is free software; you can redistribute it and/or modify it
> > > > #  under the terms of the GNU General Public License as published by
> > > > #  the Free Software Foundation; either version 2 of the License, or
> > > > #  (at your option) any later version.
> > > > 
> > > > [Unit]
> > > > Description=Manage MD Reshape on /dev/%I
> > > > DefaultDependencies=no
> > > > 
> > > > [Service]
> > > > ExecStart=/usr/sbin/mdadm --grow --continue /dev/%I
> > > > --backup-file=/root/tmp0
> > > 
> > > Please remove the ---backup-file=/root/tmp0 for further testing.  The
> > > patch I
> > > provided should make that unnecessary.
> > > 
> > > > StandardInput=null
> > > > StandardOutput=null
> > > > StandardError=null
> > > 
> > > Could you try removing these - that might allow error messages to appear.
> > > I wonder why I included them - they shouldn't be needed.
> > > 
> > > Thanks,
> > > NeilBrown
> > > 
> > > 
> > 
> > [root@intel-waimeabay-hedt-01 mdadm]# mdadm -CR /dev/md0 -l5 -n4
> > /dev/loop[0-3] --assume-clean
> > mdadm: /dev/loop0 appears to be part of a raid array:
> >        level=raid5 devices=5 ctime=Wed May 27 02:45:08 2015
> > mdadm: /dev/loop1 appears to be part of a raid array:
> >        level=raid5 devices=5 ctime=Wed May 27 02:45:08 2015
> > mdadm: /dev/loop2 appears to be part of a raid array:
> >        level=raid5 devices=5 ctime=Wed May 27 02:45:08 2015
> > mdadm: /dev/loop3 appears to be part of a raid array:
> >        level=raid5 devices=5 ctime=Wed May 27 02:45:08 2015
> > mdadm: Defaulting to version 1.2 metadata
> > mdadm: array /dev/md0 started.
> > [root@intel-waimeabay-hedt-01 mdadm]# mdadm /dev/md0 -a /dev/loop4
> > mdadm: added /dev/loop4
> > [root@intel-waimeabay-hedt-01 mdadm]# mdadm --grow /dev/md0
> > --raid-devices=5
> > mdadm: Need to backup 6144K of critical section..
> > [root@intel-waimeabay-hedt-01 mdadm]# cat /proc/mdstat
> > Personalities : [raid6] [raid5] [raid4]
> > md0 : active raid5 loop4[4] loop3[3] loop2[2] loop1[1] loop0[0]
> >       1532928 blocks super 1.2 level 5, 512k chunk, algorithm 2 [5/5]
> >       [UUUUU]
> >       [>....................]  reshape =  0.0% (0/510976) finish=532.2min
> >       speed=0K/sec
> >       
> > unused devices: <none>
> > [root@intel-waimeabay-hedt-01 mdadm]# cat
> > /usr/lib/systemd/system/mdadm-grow-continue\@.service
> > #  This file is part of mdadm.
> > #
> > #  mdadm is free software; you can redistribute it and/or modify it
> > #  under the terms of the GNU General Public License as published by
> > #  the Free Software Foundation; either version 2 of the License, or
> > #  (at your option) any later version.
> > 
> > [Unit]
> > Description=Manage MD Reshape on /dev/%I
> > DefaultDependencies=no
> > 
> > [Service]
> > ExecStart=/usr/sbin/mdadm --grow --continue /dev/%I
> > #StandardInput=null
> > #StandardOutput=null
> > #StandardError=null
> > KillMode=none
> > 
> > 
> > The problem still exist. And there are messages in /var/log/messages
> > 
> > May 27 08:03:29 intel-waimeabay-hedt-01 systemd:
> > mdadm-grow-continue@md0.service: main process exited, code=exited,
> > status=1/FAILURE
> > May 27 08:03:29 intel-waimeabay-hedt-01 systemd: Unit
> > mdadm-grow-continue@md0.service entered failed state.
> > 
> 
> Does
>   systemctl status -l mdadm-grow-continue@md0.service
> 
> report anything different.  That was the result I expected from removing the
> Standard*=null lines.
> 
> I assume the new mdadm is installed in /usr/sbin/mdadm.
> 
> Thanks,
> NeilBrown
> 

Yes! There are some new messages:
[root@intel-waimeabay-hedt-01 ~]# systemctl status -l mdadm-grow-continue@md0.service
mdadm-grow-continue@md0.service - Manage MD Reshape on /dev/md0
   Loaded: loaded (/usr/lib/systemd/system/mdadm-grow-continue@.service; static)
   Active: failed (Result: exit-code) since Thu 2015-05-28 02:30:50 EDT; 2s ago
  Process: 26618 ExecStart=/usr/sbin/mdadm --grow --continue /dev/%I (code=exited, status=1/FAILURE)
 Main PID: 26618 (code=exited, status=1/FAILURE)

May 28 02:30:50 intel-waimeabay-hedt-01.lab.eng.rdu.redhat.com systemd[1]: Started Manage MD Reshape on /dev/md0.
May 28 02:30:50 intel-waimeabay-hedt-01.lab.eng.rdu.redhat.com mdadm[26618]: mdadm: Need to backup 6144K of critical section..
May 28 02:30:50 intel-waimeabay-hedt-01.lab.eng.rdu.redhat.com mdadm[26618]: mdadm: array: cannot open component /dev/vcs6
May 28 02:30:50 intel-waimeabay-hedt-01.lab.eng.rdu.redhat.com systemd[1]: mdadm-grow-continue@md0.service: main process exited, code=exited, status=1/FAILURE
May 28 02:30:50 intel-waimeabay-hedt-01.lab.eng.rdu.redhat.com systemd[1]: Unit mdadm-grow-continue@md0.service entered failed state.

Xiao

Re: raid5 reshape is stuck

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 6364 bytes --]

On Thu, 28 May 2015 02:32:51 -0400 (EDT) Xiao Ni <xni@redhat.com> wrote:

> 
> 
> ----- Original Message -----
> > From: "NeilBrown" <neilb@suse.de>
> > To: "Xiao Ni" <xni@redhat.com>
> > Cc: linux-raid@vger.kernel.org
> > Sent: Thursday, May 28, 2015 6:59:58 AM
> > Subject: Re: raid5 reshape is stuck
> > 
> > On Wed, 27 May 2015 08:04:24 -0400 (EDT) Xiao Ni <xni@redhat.com> wrote:
> > 
> > > 
> > > 
> > > ----- Original Message -----
> > > > From: "NeilBrown" <neilb@suse.de>
> > > > To: "Xiao Ni" <xni@redhat.com>
> > > > Cc: linux-raid@vger.kernel.org
> > > > Sent: Wednesday, May 27, 2015 7:34:49 PM
> > > > Subject: Re: raid5 reshape is stuck
> > > > 
> > > > On Wed, 27 May 2015 07:28:04 -0400 (EDT) Xiao Ni <xni@redhat.com> wrote:
> > > > 
> > > > 
> > > > > [root@intel-waimeabay-hedt-01 mdadm]# cat
> > > > > /usr/lib/systemd/system/mdadm-grow-continue\@.service
> > > > > #  This file is part of mdadm.
> > > > > #
> > > > > #  mdadm is free software; you can redistribute it and/or modify it
> > > > > #  under the terms of the GNU General Public License as published by
> > > > > #  the Free Software Foundation; either version 2 of the License, or
> > > > > #  (at your option) any later version.
> > > > > 
> > > > > [Unit]
> > > > > Description=Manage MD Reshape on /dev/%I
> > > > > DefaultDependencies=no
> > > > > 
> > > > > [Service]
> > > > > ExecStart=/usr/sbin/mdadm --grow --continue /dev/%I
> > > > > --backup-file=/root/tmp0
> > > > 
> > > > Please remove the ---backup-file=/root/tmp0 for further testing.  The
> > > > patch I
> > > > provided should make that unnecessary.
> > > > 
> > > > > StandardInput=null
> > > > > StandardOutput=null
> > > > > StandardError=null
> > > > 
> > > > Could you try removing these - that might allow error messages to appear.
> > > > I wonder why I included them - they shouldn't be needed.
> > > > 
> > > > Thanks,
> > > > NeilBrown
> > > > 
> > > > 
> > > 
> > > [root@intel-waimeabay-hedt-01 mdadm]# mdadm -CR /dev/md0 -l5 -n4
> > > /dev/loop[0-3] --assume-clean
> > > mdadm: /dev/loop0 appears to be part of a raid array:
> > >        level=raid5 devices=5 ctime=Wed May 27 02:45:08 2015
> > > mdadm: /dev/loop1 appears to be part of a raid array:
> > >        level=raid5 devices=5 ctime=Wed May 27 02:45:08 2015
> > > mdadm: /dev/loop2 appears to be part of a raid array:
> > >        level=raid5 devices=5 ctime=Wed May 27 02:45:08 2015
> > > mdadm: /dev/loop3 appears to be part of a raid array:
> > >        level=raid5 devices=5 ctime=Wed May 27 02:45:08 2015
> > > mdadm: Defaulting to version 1.2 metadata
> > > mdadm: array /dev/md0 started.
> > > [root@intel-waimeabay-hedt-01 mdadm]# mdadm /dev/md0 -a /dev/loop4
> > > mdadm: added /dev/loop4
> > > [root@intel-waimeabay-hedt-01 mdadm]# mdadm --grow /dev/md0
> > > --raid-devices=5
> > > mdadm: Need to backup 6144K of critical section..
> > > [root@intel-waimeabay-hedt-01 mdadm]# cat /proc/mdstat
> > > Personalities : [raid6] [raid5] [raid4]
> > > md0 : active raid5 loop4[4] loop3[3] loop2[2] loop1[1] loop0[0]
> > >       1532928 blocks super 1.2 level 5, 512k chunk, algorithm 2 [5/5]
> > >       [UUUUU]
> > >       [>....................]  reshape =  0.0% (0/510976) finish=532.2min
> > >       speed=0K/sec
> > >       
> > > unused devices: <none>
> > > [root@intel-waimeabay-hedt-01 mdadm]# cat
> > > /usr/lib/systemd/system/mdadm-grow-continue\@.service
> > > #  This file is part of mdadm.
> > > #
> > > #  mdadm is free software; you can redistribute it and/or modify it
> > > #  under the terms of the GNU General Public License as published by
> > > #  the Free Software Foundation; either version 2 of the License, or
> > > #  (at your option) any later version.
> > > 
> > > [Unit]
> > > Description=Manage MD Reshape on /dev/%I
> > > DefaultDependencies=no
> > > 
> > > [Service]
> > > ExecStart=/usr/sbin/mdadm --grow --continue /dev/%I
> > > #StandardInput=null
> > > #StandardOutput=null
> > > #StandardError=null
> > > KillMode=none
> > > 
> > > 
> > > The problem still exist. And there are messages in /var/log/messages
> > > 
> > > May 27 08:03:29 intel-waimeabay-hedt-01 systemd:
> > > mdadm-grow-continue@md0.service: main process exited, code=exited,
> > > status=1/FAILURE
> > > May 27 08:03:29 intel-waimeabay-hedt-01 systemd: Unit
> > > mdadm-grow-continue@md0.service entered failed state.
> > > 
> > 
> > Does
> >   systemctl status -l mdadm-grow-continue@md0.service
> > 
> > report anything different.  That was the result I expected from removing the
> > Standard*=null lines.
> > 
> > I assume the new mdadm is installed in /usr/sbin/mdadm.
> > 
> > Thanks,
> > NeilBrown
> > 
> 
> Yes! There are some new messages:
> [root@intel-waimeabay-hedt-01 ~]# systemctl status -l mdadm-grow-continue@md0.service
> mdadm-grow-continue@md0.service - Manage MD Reshape on /dev/md0
>    Loaded: loaded (/usr/lib/systemd/system/mdadm-grow-continue@.service; static)
>    Active: failed (Result: exit-code) since Thu 2015-05-28 02:30:50 EDT; 2s ago
>   Process: 26618 ExecStart=/usr/sbin/mdadm --grow --continue /dev/%I (code=exited, status=1/FAILURE)
>  Main PID: 26618 (code=exited, status=1/FAILURE)
> 
> May 28 02:30:50 intel-waimeabay-hedt-01.lab.eng.rdu.redhat.com systemd[1]: Started Manage MD Reshape on /dev/md0.
> May 28 02:30:50 intel-waimeabay-hedt-01.lab.eng.rdu.redhat.com mdadm[26618]: mdadm: Need to backup 6144K of critical section..
> May 28 02:30:50 intel-waimeabay-hedt-01.lab.eng.rdu.redhat.com mdadm[26618]: mdadm: array: cannot open component /dev/vcs6
> May 28 02:30:50 intel-waimeabay-hedt-01.lab.eng.rdu.redhat.com systemd[1]: mdadm-grow-continue@md0.service: main process exited, code=exited, status=1/FAILURE
> May 28 02:30:50 intel-waimeabay-hedt-01.lab.eng.rdu.redhat.com systemd[1]: Unit mdadm-grow-continue@md0.service entered failed state.

any idea why it cannot open it?

The message is probably coming from reshape_prepare_fdlist()
Could you get those "pr_err"s to print out errno as well?
The device really has to exist, because mdadm has managed to find that name
in /dev.  Could this be a 'selinux' related issue?  I can only think that it
might be a permission problem but root shouldn't have those.

Thanks,
NeilBrown

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]

Re: raid5 reshape is stuck

[XiaoNi]

On 05/28/2015 02:49 PM, NeilBrown wrote:
> On Thu, 28 May 2015 02:32:51 -0400 (EDT) Xiao Ni <xni@redhat.com> wrote:
>
>>
>> ----- Original Message -----
>>> From: "NeilBrown" <neilb@suse.de>
>>> To: "Xiao Ni" <xni@redhat.com>
>>> Cc: linux-raid@vger.kernel.org
>>> Sent: Thursday, May 28, 2015 6:59:58 AM
>>> Subject: Re: raid5 reshape is stuck
>>>
>>> On Wed, 27 May 2015 08:04:24 -0400 (EDT) Xiao Ni <xni@redhat.com> wrote:
>>>
>>>>
>>>> ----- Original Message -----
>>>>> From: "NeilBrown" <neilb@suse.de>
>>>>> To: "Xiao Ni" <xni@redhat.com>
>>>>> Cc: linux-raid@vger.kernel.org
>>>>> Sent: Wednesday, May 27, 2015 7:34:49 PM
>>>>> Subject: Re: raid5 reshape is stuck
>>>>>
>>>>> On Wed, 27 May 2015 07:28:04 -0400 (EDT) Xiao Ni <xni@redhat.com> wrote:
>>>>>
>>>>>
>>>>>> [root@intel-waimeabay-hedt-01 mdadm]# cat
>>>>>> /usr/lib/systemd/system/mdadm-grow-continue\@.service
>>>>>> #  This file is part of mdadm.
>>>>>> #
>>>>>> #  mdadm is free software; you can redistribute it and/or modify it
>>>>>> #  under the terms of the GNU General Public License as published by
>>>>>> #  the Free Software Foundation; either version 2 of the License, or
>>>>>> #  (at your option) any later version.
>>>>>>
>>>>>> [Unit]
>>>>>> Description=Manage MD Reshape on /dev/%I
>>>>>> DefaultDependencies=no
>>>>>>
>>>>>> [Service]
>>>>>> ExecStart=/usr/sbin/mdadm --grow --continue /dev/%I
>>>>>> --backup-file=/root/tmp0
>>>>> Please remove the ---backup-file=/root/tmp0 for further testing.  The
>>>>> patch I
>>>>> provided should make that unnecessary.
>>>>>
>>>>>> StandardInput=null
>>>>>> StandardOutput=null
>>>>>> StandardError=null
>>>>> Could you try removing these - that might allow error messages to appear.
>>>>> I wonder why I included them - they shouldn't be needed.
>>>>>
>>>>> Thanks,
>>>>> NeilBrown
>>>>>
>>>>>
>>>> [root@intel-waimeabay-hedt-01 mdadm]# mdadm -CR /dev/md0 -l5 -n4
>>>> /dev/loop[0-3] --assume-clean
>>>> mdadm: /dev/loop0 appears to be part of a raid array:
>>>>         level=raid5 devices=5 ctime=Wed May 27 02:45:08 2015
>>>> mdadm: /dev/loop1 appears to be part of a raid array:
>>>>         level=raid5 devices=5 ctime=Wed May 27 02:45:08 2015
>>>> mdadm: /dev/loop2 appears to be part of a raid array:
>>>>         level=raid5 devices=5 ctime=Wed May 27 02:45:08 2015
>>>> mdadm: /dev/loop3 appears to be part of a raid array:
>>>>         level=raid5 devices=5 ctime=Wed May 27 02:45:08 2015
>>>> mdadm: Defaulting to version 1.2 metadata
>>>> mdadm: array /dev/md0 started.
>>>> [root@intel-waimeabay-hedt-01 mdadm]# mdadm /dev/md0 -a /dev/loop4
>>>> mdadm: added /dev/loop4
>>>> [root@intel-waimeabay-hedt-01 mdadm]# mdadm --grow /dev/md0
>>>> --raid-devices=5
>>>> mdadm: Need to backup 6144K of critical section..
>>>> [root@intel-waimeabay-hedt-01 mdadm]# cat /proc/mdstat
>>>> Personalities : [raid6] [raid5] [raid4]
>>>> md0 : active raid5 loop4[4] loop3[3] loop2[2] loop1[1] loop0[0]
>>>>        1532928 blocks super 1.2 level 5, 512k chunk, algorithm 2 [5/5]
>>>>        [UUUUU]
>>>>        [>....................]  reshape =  0.0% (0/510976) finish=532.2min
>>>>        speed=0K/sec
>>>>        
>>>> unused devices: <none>
>>>> [root@intel-waimeabay-hedt-01 mdadm]# cat
>>>> /usr/lib/systemd/system/mdadm-grow-continue\@.service
>>>> #  This file is part of mdadm.
>>>> #
>>>> #  mdadm is free software; you can redistribute it and/or modify it
>>>> #  under the terms of the GNU General Public License as published by
>>>> #  the Free Software Foundation; either version 2 of the License, or
>>>> #  (at your option) any later version.
>>>>
>>>> [Unit]
>>>> Description=Manage MD Reshape on /dev/%I
>>>> DefaultDependencies=no
>>>>
>>>> [Service]
>>>> ExecStart=/usr/sbin/mdadm --grow --continue /dev/%I
>>>> #StandardInput=null
>>>> #StandardOutput=null
>>>> #StandardError=null
>>>> KillMode=none
>>>>
>>>>
>>>> The problem still exist. And there are messages in /var/log/messages
>>>>
>>>> May 27 08:03:29 intel-waimeabay-hedt-01 systemd:
>>>> mdadm-grow-continue@md0.service: main process exited, code=exited,
>>>> status=1/FAILURE
>>>> May 27 08:03:29 intel-waimeabay-hedt-01 systemd: Unit
>>>> mdadm-grow-continue@md0.service entered failed state.
>>>>
>>> Does
>>>    systemctl status -l mdadm-grow-continue@md0.service
>>>
>>> report anything different.  That was the result I expected from removing the
>>> Standard*=null lines.
>>>
>>> I assume the new mdadm is installed in /usr/sbin/mdadm.
>>>
>>> Thanks,
>>> NeilBrown
>>>
>> Yes! There are some new messages:
>> [root@intel-waimeabay-hedt-01 ~]# systemctl status -l mdadm-grow-continue@md0.service
>> mdadm-grow-continue@md0.service - Manage MD Reshape on /dev/md0
>>     Loaded: loaded (/usr/lib/systemd/system/mdadm-grow-continue@.service; static)
>>     Active: failed (Result: exit-code) since Thu 2015-05-28 02:30:50 EDT; 2s ago
>>    Process: 26618 ExecStart=/usr/sbin/mdadm --grow --continue /dev/%I (code=exited, status=1/FAILURE)
>>   Main PID: 26618 (code=exited, status=1/FAILURE)
>>
>> May 28 02:30:50 intel-waimeabay-hedt-01.lab.eng.rdu.redhat.com systemd[1]: Started Manage MD Reshape on /dev/md0.
>> May 28 02:30:50 intel-waimeabay-hedt-01.lab.eng.rdu.redhat.com mdadm[26618]: mdadm: Need to backup 6144K of critical section..
>> May 28 02:30:50 intel-waimeabay-hedt-01.lab.eng.rdu.redhat.com mdadm[26618]: mdadm: array: cannot open component /dev/vcs6
>> May 28 02:30:50 intel-waimeabay-hedt-01.lab.eng.rdu.redhat.com systemd[1]: mdadm-grow-continue@md0.service: main process exited, code=exited, status=1/FAILURE
>> May 28 02:30:50 intel-waimeabay-hedt-01.lab.eng.rdu.redhat.com systemd[1]: Unit mdadm-grow-continue@md0.service entered failed state.
> any idea why it cannot open it?
>
> The message is probably coming from reshape_prepare_fdlist()
> Could you get those "pr_err"s to print out errno as well?
> The device really has to exist, because mdadm has managed to find that name
> in /dev.  Could this be a 'selinux' related issue?  I can only think that it
> might be a permission problem but root shouldn't have those.
>
> Thanks,
> NeilBrown
Sorry for late reply. It can get so much knowledge from one problem. As 
you said, it's really the permission problem.

May 29 06:47:41 intel-waimeabay-hedt-01.lab.eng.rdu.redhat.com 
mdadm[28636]: mdadm: array: cannot open component /dev/vcs6
May 29 06:47:41 intel-waimeabay-hedt-01.lab.eng.rdu.redhat.com 
mdadm[28636]: mdadm: errno is 13, err is Permission denied

And it's really the problem about selinux. The patch works after the 
command "setenforce 0". It need the patch you gave.  Is it right
to setenforce 0 every time? I'll read the doc about selinux.

Best Regards
Xiao

Re: raid5 reshape is stuck

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 639 bytes --]

On Fri, 29 May 2015 19:13:38 +0800 XiaoNi <xni@redhat.com> wrote:


> And it's really the problem about selinux. The patch works after the 
> command "setenforce 0". It need the patch you gave.  Is it right
> to setenforce 0 every time? I'll read the doc about selinux.

I know nothing about selinux, and don't really want to.
Presumably selinux needs to be told that mdadm, when run from systemd, can
access all block devices.
If there is something I can add to the system unit file to make this happen
I am happy to, but I would need someone to tell me what.

Glad you are making progress and learning a lot!!

NeilBrown

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]

Re: raid5 reshape is stuck

[XiaoNi]

On 05/29/2015 07:19 PM, NeilBrown wrote:
> On Fri, 29 May 2015 19:13:38 +0800 XiaoNi <xni@redhat.com> wrote:
>
>
>> And it's really the problem about selinux. The patch works after the
>> command "setenforce 0". It need the patch you gave.  Is it right
>> to setenforce 0 every time? I'll read the doc about selinux.
> I know nothing about selinux, and don't really want to.
> Presumably selinux needs to be told that mdadm, when run from systemd, can
> access all block devices.
> If there is something I can add to the system unit file to make this happen
> I am happy to, but I would need someone to tell me what.
>
> Glad you are making progress and learning a lot!!
>
> NeilBrown

But it really need the patch you gave several days before. I already 
tried without the patch, reshape can be
stuck even selinux is set to Permissive.

And it's a problem with systemd and selinux too. If I run mdadm --grow 
--continue /dev/md0, the reshape can start.
If I run systemctl start mdadm-grow-continue@md0.service, it can't 
start. I'll look for the answer.

Best Regards
Xiao