Problem with array raid10 array resync on 4.4.0 (keeps reyncing each reboot)

Problem with array raid10 array resync on 4.4.0 (keeps reyncing each reboot)

[Eric Valette]

Hi,

My raid 10 array (5 disk with one spare) was doing a  resync after an 
upgrade to 4.4.0 from 4.1.15. The resync progress was steady and at the 
end the /proc/mdstat was apparently complete but when rebooting, it 
started resycing over and over. I noticed my dmesg was totally filled 
with raid10 conf printout message so it was impossible to trace anything 
else.

Did a resync test with 3.14.58 (because I knew it had worked for resync 
before and was still available as a boot option)  and the array was 
correctly rebuild.
Runs fine with 4.1.16 now.

Please CC me as I'm not subscribed.

-- eric

Re: Problem with array raid10 array resync on 4.4.0 (keeps reyncing each reboot)

[Shaohua Li]

On Tue, Jan 26, 2016 at 03:49:42PM +0100, Eric Valette wrote:
> Hi,
> 
> My raid 10 array (5 disk with one spare) was doing a  resync after an
> upgrade to 4.4.0 from 4.1.15. The resync progress was steady and at the end
> the /proc/mdstat was apparently complete but when rebooting, it started
> resycing over and over. I noticed my dmesg was totally filled with raid10
> conf printout message so it was impossible to trace anything else.
> 
> Did a resync test with 3.14.58 (because I knew it had worked for resync
> before and was still available as a boot option)  and the array was
> correctly rebuild.
> Runs fine with 4.1.16 now.
> 
> Please CC me as I'm not subscribed.

Could you please provide more info, like mdadm -D /dev/md0 in v4.4? If you run
a stop/reassemble, does the resync start?

Thanks,
Shaohua

Re: Problem with array raid10 array resync on 4.4.0 (keeps reyncing each reboot)

[Eric Valette]

On 27/01/2016 00:31, Shaohua Li wrote:
> On Tue, Jan 26, 2016 at 03:49:42PM +0100, Eric Valette wrote:
>> Hi,
>>
>> My raid 10 array (5 disk with one spare) was doing a  resync after an
>> upgrade to 4.4.0 from 4.1.15. The resync progress was steady and at the end
>> the /proc/mdstat was apparently complete but when rebooting, it started
>> resycing over and over. I noticed my dmesg was totally filled with raid10
>> conf printout message so it was impossible to trace anything else.
>>
>> Did a resync test with 3.14.58 (because I knew it had worked for resync
>> before and was still available as a boot option)  and the array was
>> correctly rebuild.
>> Runs fine with 4.1.16 now.
>>
>> Please CC me as I'm not subscribed.
>
> Could you please provide more info, like mdadm -D /dev/md0 in v4.4? If you run
> a stop/reassemble, does the resync start?

I'm not going to retry kernel 4.4.0 on this device as I'm no more 
confident about raid10 support with this 4.4 version.

So on 4.1.16 :

mdadm -D /dev/md0
/dev/md0:
         Version : 1.2
   Creation Time : Wed Jun 20 23:56:59 2012
      Raid Level : raid10
      Array Size : 5860268032 (5588.79 GiB 6000.91 GB)
   Used Dev Size : 2930134016 (2794.39 GiB 3000.46 GB)
    Raid Devices : 4
   Total Devices : 5
     Persistence : Superblock is persistent

     Update Time : Sun Jan 24 17:15:58 2016
           State : clean
  Active Devices : 4
Working Devices : 5
  Failed Devices : 0
   Spare Devices : 1

          Layout : near=2
      Chunk Size : 512K

            Name : nas2:0  (local to host nas2)
            UUID : 6abe1f20:90c629de:fadd8dc0:ca14c928
          Events : 480
 
 

     Number   Major   Minor   RaidDevice State 
 

        0       8       17        0      active sync set-A   /dev/sdb1 
 

        1       8       33        1      active sync set-B   /dev/sdc1 
 

        2       8       49        2      active sync set-A   /dev/sdd1 
 

        3       8       65        3      active sync set-B   /dev/sde1 
 

 
 

        4       8       81        -      spare   /dev/sdf1

uname -a
Linux nas2 4.1.16 #1 SMP Sat Jan 23 19:29:59 CET 2016 x86_64 GNU/Linux

And if I stop the array manually and reboot, its no more resynced with 
this 4.1.16 kernel.

Re: Problem with array raid10 array resync on 4.4.0 (keeps reyncing each reboot)

[Shaohua Li]

On Wed, Jan 27, 2016 at 08:24:29AM +0100, Eric Valette wrote:
> On 27/01/2016 00:31, Shaohua Li wrote:
> >On Tue, Jan 26, 2016 at 03:49:42PM +0100, Eric Valette wrote:
> >>Hi,
> >>
> >>My raid 10 array (5 disk with one spare) was doing a  resync after an
> >>upgrade to 4.4.0 from 4.1.15. The resync progress was steady and at the end
> >>the /proc/mdstat was apparently complete but when rebooting, it started
> >>resycing over and over. I noticed my dmesg was totally filled with raid10
> >>conf printout message so it was impossible to trace anything else.
> >>
> >>Did a resync test with 3.14.58 (because I knew it had worked for resync
> >>before and was still available as a boot option)  and the array was
> >>correctly rebuild.
> >>Runs fine with 4.1.16 now.
> >>
> >>Please CC me as I'm not subscribed.
> >
> >Could you please provide more info, like mdadm -D /dev/md0 in v4.4? If you run
> >a stop/reassemble, does the resync start?
> 
> I'm not going to retry kernel 4.4.0 on this device as I'm no more confident
> about raid10 support with this 4.4 version.

Thanks! I know switching to a kernel which is not working well is risky, but
the info from 4.1 doesn't have enough info for debuging. I also tried to
reproduce the issue locally, but no success. Did you have any other info which
could help debugging, for example special config?

Thanks,
Shaohua

Re: Problem with array raid10 array resync on 4.4.0 (keeps reyncing each reboot)

[Eric Valette]

On 28/01/2016 19:25, Shaohua Li wrote:
> On Wed, Jan 27, 2016 at 08:24:29AM +0100, Eric Valette wrote:
>> On 27/01/2016 00:31, Shaohua Li wrote:
>>> On Tue, Jan 26, 2016 at 03:49:42PM +0100, Eric Valette wrote:
>>>> Hi,
>>>>
>>>> My raid 10 array (5 disk with one spare) was doing a  resync after an
>>>> upgrade to 4.4.0 from 4.1.15. The resync progress was steady and at the end
>>>> the /proc/mdstat was apparently complete but when rebooting, it started
>>>> resycing over and over. I noticed my dmesg was totally filled with raid10
>>>> conf printout message so it was impossible to trace anything else.
>>>>
>>>> Did a resync test with 3.14.58 (because I knew it had worked for resync
>>>> before and was still available as a boot option)  and the array was
>>>> correctly rebuild.
>>>> Runs fine with 4.1.16 now.
>>>>
>>>> Please CC me as I'm not subscribed.
>>>
>>> Could you please provide more info, like mdadm -D /dev/md0 in v4.4? If you run
>>> a stop/reassemble, does the resync start?
>>
>> I'm not going to retry kernel 4.4.0 on this device as I'm no more confident
>> about raid10 support with this 4.4 version.
>
> Thanks! I know switching to a kernel which is not working well is risky, but
> the info from 4.1 doesn't have enough info for debuging. I also tried to
> reproduce the issue locally, but no success. Did you have any other info which
> could help debugging, for example special config?


If really needed can rebuild a 4.4 kernel and boot it. What do you mean 
by special config? I can send you my kernel .config once rebuild 
(oldconfig from 4.1.16)

But I doubt the problem will show up now that the array is correctly 
rebuild. From memory I used 4.4 kernel without problem for a few days, 
until it decided to resync the array for an unknown reason (standard 
periodic rebuild, disk block read/write error detected, other?).

NB: I have a disk on the array with 56 sectors in error reported by 
smart and noticed read error/SATA reset sequence during first array 
rebuild sequence, but not in any later resync afterward. It did not even 
decided to use the spare disk...

Let me know how I can help debugging further...

-- eric

Re: Problem with array raid10 array resync on 4.4.0 (keeps reyncing each reboot)

[Eric Valette]

On 28/01/2016 20:52, Eric Valette wrote:
> On 28/01/2016 19:25, Shaohua Li wrote:
>
> If really needed can rebuild a 4.4 kernel and boot it. What do you mean
> by special config? I can send you my kernel .config once rebuild
> (oldconfig from 4.1.16)

Linux nas2 4.4.1 #1 SMP Sun Jan 31 23:46:26 CET 2016 x86_64 GNU/Linux


mdadm -D /dev/md0
/dev/md0:
         Version : 1.2
   Creation Time : Wed Jun 20 23:56:59 2012
      Raid Level : raid10
      Array Size : 5860268032 (5588.79 GiB 6000.91 GB)
   Used Dev Size : 2930134016 (2794.39 GiB 3000.46 GB)
    Raid Devices : 4
   Total Devices : 5
     Persistence : Superblock is persistent

     Update Time : Sun Jan 31 23:51:44 2016
           State : active, resyncing
  Active Devices : 4
Working Devices : 5
  Failed Devices : 0
   Spare Devices : 1

          Layout : near=2
      Chunk Size : 512K

   Resync Status : 0% complete

            Name : nas2:0  (local to host nas2)
            UUID : 6abe1f20:90c629de:fadd8dc0:ca14c928
          Events : 488

     Number   Major   Minor   RaidDevice State
        0       8       17        0      active sync set-A   /dev/sdb1
        1       8       33        1      active sync set-B   /dev/sdc1
        2       8       49        2      active sync set-A   /dev/sdd1
        3       8       65        3      active sync set-B   /dev/sde1

        4       8       81        -      spare   /dev/sdf1

So it starts again rechecking the array

mdadm -E /dev/sd[b-f]1
/dev/sdb1:
           Magic : a92b4efc
         Version : 1.2
     Feature Map : 0x0
      Array UUID : 6abe1f20:90c629de:fadd8dc0:ca14c928
            Name : nas2:0  (local to host nas2)
   Creation Time : Wed Jun 20 23:56:59 2012
      Raid Level : raid10
    Raid Devices : 4

  Avail Dev Size : 5860268943 (2794.39 GiB 3000.46 GB)
      Array Size : 5860268032 (5588.79 GiB 6000.91 GB)
   Used Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
     Data Offset : 262144 sectors
    Super Offset : 8 sectors
    Unused Space : before=262064 sectors, after=911 sectors
           State : active
     Device UUID : ce9d878a:37a4f3a3:936bd905:c4ed9970

     Update Time : Sun Jan 31 23:56:38 2016
        Checksum : c92206ec - correct
          Events : 489

          Layout : near=2
      Chunk Size : 512K

    Device Role : Active device 0
    Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdc1:
           Magic : a92b4efc
         Version : 1.2
     Feature Map : 0x0
      Array UUID : 6abe1f20:90c629de:fadd8dc0:ca14c928
            Name : nas2:0  (local to host nas2)
   Creation Time : Wed Jun 20 23:56:59 2012
      Raid Level : raid10
    Raid Devices : 4

  Avail Dev Size : 5860268943 (2794.39 GiB 3000.46 GB)
      Array Size : 5860268032 (5588.79 GiB 6000.91 GB)
   Used Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
     Data Offset : 262144 sectors
    Super Offset : 8 sectors
    Unused Space : before=262064 sectors, after=911 sectors
           State : active
     Device UUID : 8c89bdf8:4f3f8ace:c15b5634:7a874071

     Update Time : Sun Jan 31 23:56:38 2016
        Checksum : 911217a8 - correct
          Events : 489

          Layout : near=2
      Chunk Size : 512K

    Device Role : Active device 1
    Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdd1:
           Magic : a92b4efc
         Version : 1.2
     Feature Map : 0x0
      Array UUID : 6abe1f20:90c629de:fadd8dc0:ca14c928
            Name : nas2:0  (local to host nas2)
   Creation Time : Wed Jun 20 23:56:59 2012
      Raid Level : raid10
    Raid Devices : 4

  Avail Dev Size : 5860268943 (2794.39 GiB 3000.46 GB)
      Array Size : 5860268032 (5588.79 GiB 6000.91 GB)
   Used Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
     Data Offset : 262144 sectors
    Super Offset : 8 sectors
    Unused Space : before=262064 sectors, after=911 sectors
           State : active
     Device UUID : 9c0655c3:dca7722e:8ecc1a17:fb4d9089

     Update Time : Sun Jan 31 23:56:38 2016
        Checksum : b6a6bcd3 - correct
          Events : 489

          Layout : near=2
      Chunk Size : 512K

    Device Role : Active device 2
    Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sde1:
           Magic : a92b4efc
         Version : 1.2
     Feature Map : 0x0
      Array UUID : 6abe1f20:90c629de:fadd8dc0:ca14c928
            Name : nas2:0  (local to host nas2)
   Creation Time : Wed Jun 20 23:56:59 2012
      Raid Level : raid10
    Raid Devices : 4

  Avail Dev Size : 5860268943 (2794.39 GiB 3000.46 GB)
      Array Size : 5860268032 (5588.79 GiB 6000.91 GB)
   Used Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
     Data Offset : 262144 sectors
    Super Offset : 8 sectors
    Unused Space : before=262064 sectors, after=911 sectors
           State : active
     Device UUID : c97b767a:84d2e7e2:52557d30:51c39784

     Update Time : Sun Jan 31 23:56:38 2016
        Checksum : 36a6d284 - correct
          Events : 489

          Layout : near=2
      Chunk Size : 512K

    Device Role : Active device 3
    Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdf1:
           Magic : a92b4efc
         Version : 1.2
     Feature Map : 0x0
      Array UUID : 6abe1f20:90c629de:fadd8dc0:ca14c928
            Name : nas2:0  (local to host nas2)
   Creation Time : Wed Jun 20 23:56:59 2012
      Raid Level : raid10
    Raid Devices : 4

  Avail Dev Size : 5860268943 (2794.39 GiB 3000.46 GB)
      Array Size : 5860268032 (5588.79 GiB 6000.91 GB)
   Used Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
     Data Offset : 262144 sectors
    Super Offset : 8 sectors
    Unused Space : before=262064 sectors, after=911 sectors
           State : active
     Device UUID : 82667e81:a6158319:85e0282e:845eec1c

     Update Time : Sun Jan 31 23:51:44 2016
        Checksum : a4a259e - correct
          Events : 488

          Layout : near=2
      Chunk Size : 512K

    Device Role : spare
    Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
root@nas2:~#

-- eric

Re: Problem with array raid10 array resync on 4.4.0 (keeps reyncing each reboot)

[Shaohua Li]

On Mon, Feb 01, 2016 at 12:00:36AM +0100, Eric Valette wrote:
> On 28/01/2016 20:52, Eric Valette wrote:
> >On 28/01/2016 19:25, Shaohua Li wrote:
> >
> >If really needed can rebuild a 4.4 kernel and boot it. What do you mean
> >by special config? I can send you my kernel .config once rebuild
> >(oldconfig from 4.1.16)
> 
> Linux nas2 4.4.1 #1 SMP Sun Jan 31 23:46:26 CET 2016 x86_64 GNU/Linux

Nothing useful. Did you try stop the array and reassemble it in 4.4 and check
if resync runs? I can only imagine the reboot script didn't clean shutdown the
array and trigger the issue. A stop/reassemble circle will tell us if this is
the issue.

Thanks,
Shaohua

Re: Problem with array raid10 array resync on 4.4.0 (keeps reyncing each reboot)

[Eric Valette]

On 04/02/2016 02:17, Shaohua Li wrote:
> On Mon, Feb 01, 2016 at 12:00:36AM +0100, Eric Valette wrote:
>> On 28/01/2016 20:52, Eric Valette wrote:
>>> On 28/01/2016 19:25, Shaohua Li wrote:
>>>
>>> If really needed can rebuild a 4.4 kernel and boot it. What do you mean
>>> by special config? I can send you my kernel .config once rebuild
>>> (oldconfig from 4.1.16)
>>
>> Linux nas2 4.4.1 #1 SMP Sun Jan 31 23:46:26 CET 2016 x86_64 GNU/Linux
>
> Nothing useful. Did you try stop the array and reassemble it in 4.4 and check
> if resync runs? I can only imagine the reboot script didn't clean shutdown the
> array and trigger the issue. A stop/reassemble circle will tell us if this is
> the issue.

If the issue was in the script it would happen with all the kernel 
version no? Si far 3.14, 3.18, 4.1 works correctly (when upgrading). 
Only 4.4 fails.

4.4 in fact seems to fails to boot the first time I boot it. As I have 
no console normally, I have to shut down via power button and then it 
resync. I can understand this behavior but not
	1) that the resync fails each time and
	2) that it fails to boot the first time I change for a 4.4 kernel.

-- eric