* Problem with array raid10 array resync on 4.4.0 (keeps reyncing each reboot)
@ 2016-01-26 14:49 Eric Valette
2016-01-26 23:31 ` Shaohua Li
0 siblings, 1 reply; 8+ messages in thread
From: Eric Valette @ 2016-01-26 14:49 UTC (permalink / raw)
To: linux-raid
Hi,
My raid 10 array (5 disk with one spare) was doing a resync after an
upgrade to 4.4.0 from 4.1.15. The resync progress was steady and at the
end the /proc/mdstat was apparently complete but when rebooting, it
started resycing over and over. I noticed my dmesg was totally filled
with raid10 conf printout message so it was impossible to trace anything
else.
Did a resync test with 3.14.58 (because I knew it had worked for resync
before and was still available as a boot option) and the array was
correctly rebuild.
Runs fine with 4.1.16 now.
Please CC me as I'm not subscribed.
-- eric
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Problem with array raid10 array resync on 4.4.0 (keeps reyncing each reboot)
2016-01-26 14:49 Problem with array raid10 array resync on 4.4.0 (keeps reyncing each reboot) Eric Valette
@ 2016-01-26 23:31 ` Shaohua Li
2016-01-27 7:24 ` Eric Valette
0 siblings, 1 reply; 8+ messages in thread
From: Shaohua Li @ 2016-01-26 23:31 UTC (permalink / raw)
To: Eric Valette; +Cc: linux-raid
On Tue, Jan 26, 2016 at 03:49:42PM +0100, Eric Valette wrote:
> Hi,
>
> My raid 10 array (5 disk with one spare) was doing a resync after an
> upgrade to 4.4.0 from 4.1.15. The resync progress was steady and at the end
> the /proc/mdstat was apparently complete but when rebooting, it started
> resycing over and over. I noticed my dmesg was totally filled with raid10
> conf printout message so it was impossible to trace anything else.
>
> Did a resync test with 3.14.58 (because I knew it had worked for resync
> before and was still available as a boot option) and the array was
> correctly rebuild.
> Runs fine with 4.1.16 now.
>
> Please CC me as I'm not subscribed.
Could you please provide more info, like mdadm -D /dev/md0 in v4.4? If you run
a stop/reassemble, does the resync start?
Thanks,
Shaohua
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Problem with array raid10 array resync on 4.4.0 (keeps reyncing each reboot)
2016-01-26 23:31 ` Shaohua Li
@ 2016-01-27 7:24 ` Eric Valette
2016-01-28 18:25 ` Shaohua Li
0 siblings, 1 reply; 8+ messages in thread
From: Eric Valette @ 2016-01-27 7:24 UTC (permalink / raw)
To: Shaohua Li; +Cc: linux-raid
On 27/01/2016 00:31, Shaohua Li wrote:
> On Tue, Jan 26, 2016 at 03:49:42PM +0100, Eric Valette wrote:
>> Hi,
>>
>> My raid 10 array (5 disk with one spare) was doing a resync after an
>> upgrade to 4.4.0 from 4.1.15. The resync progress was steady and at the end
>> the /proc/mdstat was apparently complete but when rebooting, it started
>> resycing over and over. I noticed my dmesg was totally filled with raid10
>> conf printout message so it was impossible to trace anything else.
>>
>> Did a resync test with 3.14.58 (because I knew it had worked for resync
>> before and was still available as a boot option) and the array was
>> correctly rebuild.
>> Runs fine with 4.1.16 now.
>>
>> Please CC me as I'm not subscribed.
>
> Could you please provide more info, like mdadm -D /dev/md0 in v4.4? If you run
> a stop/reassemble, does the resync start?
I'm not going to retry kernel 4.4.0 on this device as I'm no more
confident about raid10 support with this 4.4 version.
So on 4.1.16 :
mdadm -D /dev/md0
/dev/md0:
Version : 1.2
Creation Time : Wed Jun 20 23:56:59 2012
Raid Level : raid10
Array Size : 5860268032 (5588.79 GiB 6000.91 GB)
Used Dev Size : 2930134016 (2794.39 GiB 3000.46 GB)
Raid Devices : 4
Total Devices : 5
Persistence : Superblock is persistent
Update Time : Sun Jan 24 17:15:58 2016
State : clean
Active Devices : 4
Working Devices : 5
Failed Devices : 0
Spare Devices : 1
Layout : near=2
Chunk Size : 512K
Name : nas2:0 (local to host nas2)
UUID : 6abe1f20:90c629de:fadd8dc0:ca14c928
Events : 480
Number Major Minor RaidDevice State
0 8 17 0 active sync set-A /dev/sdb1
1 8 33 1 active sync set-B /dev/sdc1
2 8 49 2 active sync set-A /dev/sdd1
3 8 65 3 active sync set-B /dev/sde1
4 8 81 - spare /dev/sdf1
uname -a
Linux nas2 4.1.16 #1 SMP Sat Jan 23 19:29:59 CET 2016 x86_64 GNU/Linux
And if I stop the array manually and reboot, its no more resynced with
this 4.1.16 kernel.
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Problem with array raid10 array resync on 4.4.0 (keeps reyncing each reboot)
2016-01-27 7:24 ` Eric Valette
@ 2016-01-28 18:25 ` Shaohua Li
2016-01-28 19:52 ` Eric Valette
0 siblings, 1 reply; 8+ messages in thread
From: Shaohua Li @ 2016-01-28 18:25 UTC (permalink / raw)
To: Eric Valette; +Cc: linux-raid
On Wed, Jan 27, 2016 at 08:24:29AM +0100, Eric Valette wrote:
> On 27/01/2016 00:31, Shaohua Li wrote:
> >On Tue, Jan 26, 2016 at 03:49:42PM +0100, Eric Valette wrote:
> >>Hi,
> >>
> >>My raid 10 array (5 disk with one spare) was doing a resync after an
> >>upgrade to 4.4.0 from 4.1.15. The resync progress was steady and at the end
> >>the /proc/mdstat was apparently complete but when rebooting, it started
> >>resycing over and over. I noticed my dmesg was totally filled with raid10
> >>conf printout message so it was impossible to trace anything else.
> >>
> >>Did a resync test with 3.14.58 (because I knew it had worked for resync
> >>before and was still available as a boot option) and the array was
> >>correctly rebuild.
> >>Runs fine with 4.1.16 now.
> >>
> >>Please CC me as I'm not subscribed.
> >
> >Could you please provide more info, like mdadm -D /dev/md0 in v4.4? If you run
> >a stop/reassemble, does the resync start?
>
> I'm not going to retry kernel 4.4.0 on this device as I'm no more confident
> about raid10 support with this 4.4 version.
Thanks! I know switching to a kernel which is not working well is risky, but
the info from 4.1 doesn't have enough info for debuging. I also tried to
reproduce the issue locally, but no success. Did you have any other info which
could help debugging, for example special config?
Thanks,
Shaohua
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Problem with array raid10 array resync on 4.4.0 (keeps reyncing each reboot)
2016-01-28 18:25 ` Shaohua Li
@ 2016-01-28 19:52 ` Eric Valette
2016-01-31 23:00 ` Eric Valette
0 siblings, 1 reply; 8+ messages in thread
From: Eric Valette @ 2016-01-28 19:52 UTC (permalink / raw)
To: Shaohua Li; +Cc: linux-raid
On 28/01/2016 19:25, Shaohua Li wrote:
> On Wed, Jan 27, 2016 at 08:24:29AM +0100, Eric Valette wrote:
>> On 27/01/2016 00:31, Shaohua Li wrote:
>>> On Tue, Jan 26, 2016 at 03:49:42PM +0100, Eric Valette wrote:
>>>> Hi,
>>>>
>>>> My raid 10 array (5 disk with one spare) was doing a resync after an
>>>> upgrade to 4.4.0 from 4.1.15. The resync progress was steady and at the end
>>>> the /proc/mdstat was apparently complete but when rebooting, it started
>>>> resycing over and over. I noticed my dmesg was totally filled with raid10
>>>> conf printout message so it was impossible to trace anything else.
>>>>
>>>> Did a resync test with 3.14.58 (because I knew it had worked for resync
>>>> before and was still available as a boot option) and the array was
>>>> correctly rebuild.
>>>> Runs fine with 4.1.16 now.
>>>>
>>>> Please CC me as I'm not subscribed.
>>>
>>> Could you please provide more info, like mdadm -D /dev/md0 in v4.4? If you run
>>> a stop/reassemble, does the resync start?
>>
>> I'm not going to retry kernel 4.4.0 on this device as I'm no more confident
>> about raid10 support with this 4.4 version.
>
> Thanks! I know switching to a kernel which is not working well is risky, but
> the info from 4.1 doesn't have enough info for debuging. I also tried to
> reproduce the issue locally, but no success. Did you have any other info which
> could help debugging, for example special config?
If really needed can rebuild a 4.4 kernel and boot it. What do you mean
by special config? I can send you my kernel .config once rebuild
(oldconfig from 4.1.16)
But I doubt the problem will show up now that the array is correctly
rebuild. From memory I used 4.4 kernel without problem for a few days,
until it decided to resync the array for an unknown reason (standard
periodic rebuild, disk block read/write error detected, other?).
NB: I have a disk on the array with 56 sectors in error reported by
smart and noticed read error/SATA reset sequence during first array
rebuild sequence, but not in any later resync afterward. It did not even
decided to use the spare disk...
Let me know how I can help debugging further...
-- eric
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Problem with array raid10 array resync on 4.4.0 (keeps reyncing each reboot)
2016-01-28 19:52 ` Eric Valette
@ 2016-01-31 23:00 ` Eric Valette
2016-02-04 1:17 ` Shaohua Li
0 siblings, 1 reply; 8+ messages in thread
From: Eric Valette @ 2016-01-31 23:00 UTC (permalink / raw)
To: Shaohua Li; +Cc: linux-raid
On 28/01/2016 20:52, Eric Valette wrote:
> On 28/01/2016 19:25, Shaohua Li wrote:
>
> If really needed can rebuild a 4.4 kernel and boot it. What do you mean
> by special config? I can send you my kernel .config once rebuild
> (oldconfig from 4.1.16)
Linux nas2 4.4.1 #1 SMP Sun Jan 31 23:46:26 CET 2016 x86_64 GNU/Linux
mdadm -D /dev/md0
/dev/md0:
Version : 1.2
Creation Time : Wed Jun 20 23:56:59 2012
Raid Level : raid10
Array Size : 5860268032 (5588.79 GiB 6000.91 GB)
Used Dev Size : 2930134016 (2794.39 GiB 3000.46 GB)
Raid Devices : 4
Total Devices : 5
Persistence : Superblock is persistent
Update Time : Sun Jan 31 23:51:44 2016
State : active, resyncing
Active Devices : 4
Working Devices : 5
Failed Devices : 0
Spare Devices : 1
Layout : near=2
Chunk Size : 512K
Resync Status : 0% complete
Name : nas2:0 (local to host nas2)
UUID : 6abe1f20:90c629de:fadd8dc0:ca14c928
Events : 488
Number Major Minor RaidDevice State
0 8 17 0 active sync set-A /dev/sdb1
1 8 33 1 active sync set-B /dev/sdc1
2 8 49 2 active sync set-A /dev/sdd1
3 8 65 3 active sync set-B /dev/sde1
4 8 81 - spare /dev/sdf1
So it starts again rechecking the array
mdadm -E /dev/sd[b-f]1
/dev/sdb1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 6abe1f20:90c629de:fadd8dc0:ca14c928
Name : nas2:0 (local to host nas2)
Creation Time : Wed Jun 20 23:56:59 2012
Raid Level : raid10
Raid Devices : 4
Avail Dev Size : 5860268943 (2794.39 GiB 3000.46 GB)
Array Size : 5860268032 (5588.79 GiB 6000.91 GB)
Used Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=911 sectors
State : active
Device UUID : ce9d878a:37a4f3a3:936bd905:c4ed9970
Update Time : Sun Jan 31 23:56:38 2016
Checksum : c92206ec - correct
Events : 489
Layout : near=2
Chunk Size : 512K
Device Role : Active device 0
Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdc1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 6abe1f20:90c629de:fadd8dc0:ca14c928
Name : nas2:0 (local to host nas2)
Creation Time : Wed Jun 20 23:56:59 2012
Raid Level : raid10
Raid Devices : 4
Avail Dev Size : 5860268943 (2794.39 GiB 3000.46 GB)
Array Size : 5860268032 (5588.79 GiB 6000.91 GB)
Used Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=911 sectors
State : active
Device UUID : 8c89bdf8:4f3f8ace:c15b5634:7a874071
Update Time : Sun Jan 31 23:56:38 2016
Checksum : 911217a8 - correct
Events : 489
Layout : near=2
Chunk Size : 512K
Device Role : Active device 1
Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdd1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 6abe1f20:90c629de:fadd8dc0:ca14c928
Name : nas2:0 (local to host nas2)
Creation Time : Wed Jun 20 23:56:59 2012
Raid Level : raid10
Raid Devices : 4
Avail Dev Size : 5860268943 (2794.39 GiB 3000.46 GB)
Array Size : 5860268032 (5588.79 GiB 6000.91 GB)
Used Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=911 sectors
State : active
Device UUID : 9c0655c3:dca7722e:8ecc1a17:fb4d9089
Update Time : Sun Jan 31 23:56:38 2016
Checksum : b6a6bcd3 - correct
Events : 489
Layout : near=2
Chunk Size : 512K
Device Role : Active device 2
Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sde1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 6abe1f20:90c629de:fadd8dc0:ca14c928
Name : nas2:0 (local to host nas2)
Creation Time : Wed Jun 20 23:56:59 2012
Raid Level : raid10
Raid Devices : 4
Avail Dev Size : 5860268943 (2794.39 GiB 3000.46 GB)
Array Size : 5860268032 (5588.79 GiB 6000.91 GB)
Used Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=911 sectors
State : active
Device UUID : c97b767a:84d2e7e2:52557d30:51c39784
Update Time : Sun Jan 31 23:56:38 2016
Checksum : 36a6d284 - correct
Events : 489
Layout : near=2
Chunk Size : 512K
Device Role : Active device 3
Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdf1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 6abe1f20:90c629de:fadd8dc0:ca14c928
Name : nas2:0 (local to host nas2)
Creation Time : Wed Jun 20 23:56:59 2012
Raid Level : raid10
Raid Devices : 4
Avail Dev Size : 5860268943 (2794.39 GiB 3000.46 GB)
Array Size : 5860268032 (5588.79 GiB 6000.91 GB)
Used Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=911 sectors
State : active
Device UUID : 82667e81:a6158319:85e0282e:845eec1c
Update Time : Sun Jan 31 23:51:44 2016
Checksum : a4a259e - correct
Events : 488
Layout : near=2
Chunk Size : 512K
Device Role : spare
Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
root@nas2:~#
-- eric
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Problem with array raid10 array resync on 4.4.0 (keeps reyncing each reboot)
2016-01-31 23:00 ` Eric Valette
@ 2016-02-04 1:17 ` Shaohua Li
2016-02-04 6:57 ` Eric Valette
0 siblings, 1 reply; 8+ messages in thread
From: Shaohua Li @ 2016-02-04 1:17 UTC (permalink / raw)
To: Eric Valette; +Cc: linux-raid
On Mon, Feb 01, 2016 at 12:00:36AM +0100, Eric Valette wrote:
> On 28/01/2016 20:52, Eric Valette wrote:
> >On 28/01/2016 19:25, Shaohua Li wrote:
> >
> >If really needed can rebuild a 4.4 kernel and boot it. What do you mean
> >by special config? I can send you my kernel .config once rebuild
> >(oldconfig from 4.1.16)
>
> Linux nas2 4.4.1 #1 SMP Sun Jan 31 23:46:26 CET 2016 x86_64 GNU/Linux
Nothing useful. Did you try stop the array and reassemble it in 4.4 and check
if resync runs? I can only imagine the reboot script didn't clean shutdown the
array and trigger the issue. A stop/reassemble circle will tell us if this is
the issue.
Thanks,
Shaohua
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Problem with array raid10 array resync on 4.4.0 (keeps reyncing each reboot)
2016-02-04 1:17 ` Shaohua Li
@ 2016-02-04 6:57 ` Eric Valette
0 siblings, 0 replies; 8+ messages in thread
From: Eric Valette @ 2016-02-04 6:57 UTC (permalink / raw)
To: Shaohua Li; +Cc: linux-raid
On 04/02/2016 02:17, Shaohua Li wrote:
> On Mon, Feb 01, 2016 at 12:00:36AM +0100, Eric Valette wrote:
>> On 28/01/2016 20:52, Eric Valette wrote:
>>> On 28/01/2016 19:25, Shaohua Li wrote:
>>>
>>> If really needed can rebuild a 4.4 kernel and boot it. What do you mean
>>> by special config? I can send you my kernel .config once rebuild
>>> (oldconfig from 4.1.16)
>>
>> Linux nas2 4.4.1 #1 SMP Sun Jan 31 23:46:26 CET 2016 x86_64 GNU/Linux
>
> Nothing useful. Did you try stop the array and reassemble it in 4.4 and check
> if resync runs? I can only imagine the reboot script didn't clean shutdown the
> array and trigger the issue. A stop/reassemble circle will tell us if this is
> the issue.
If the issue was in the script it would happen with all the kernel
version no? Si far 3.14, 3.18, 4.1 works correctly (when upgrading).
Only 4.4 fails.
4.4 in fact seems to fails to boot the first time I boot it. As I have
no console normally, I have to shut down via power button and then it
resync. I can understand this behavior but not
1) that the resync fails each time and
2) that it fails to boot the first time I change for a 4.4 kernel.
-- eric
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2016-02-04 6:57 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-01-26 14:49 Problem with array raid10 array resync on 4.4.0 (keeps reyncing each reboot) Eric Valette
2016-01-26 23:31 ` Shaohua Li
2016-01-27 7:24 ` Eric Valette
2016-01-28 18:25 ` Shaohua Li
2016-01-28 19:52 ` Eric Valette
2016-01-31 23:00 ` Eric Valette
2016-02-04 1:17 ` Shaohua Li
2016-02-04 6:57 ` Eric Valette
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.