* failed reshape!
@ 2011-12-09 13:53 Gavin Peters (蓋文彼德斯)
2011-12-12 3:39 ` NeilBrown
0 siblings, 1 reply; 3+ messages in thread
From: Gavin Peters (蓋文彼德斯) @ 2011-12-09 13:53 UTC (permalink / raw)
To: linux-raid
I tried to reshape today, a raid6 array from seven devices up to
eight. I ran mdadm 3.2.2, something like
# mdadm /dev/md2 --grow -n 8 --layout=preserve
and then, blammo!
Dec 8 22:30:10 avclub kernel: [ 527.094708] RAID5 conf printout:
Dec 8 22:30:10 avclub kernel: [ 527.094712] --- rd:8 wd:8
Dec 8 22:30:10 avclub kernel: [ 527.094714] disk 0, o:1, dev:sdc6
Dec 8 22:30:10 avclub kernel: [ 527.094715] disk 1, o:1, dev:sdf6
Dec 8 22:30:10 avclub kernel: [ 527.094717] disk 2, o:1, dev:sda6
Dec 8 22:30:10 avclub kernel: [ 527.094718] disk 3, o:1, dev:sdd6
Dec 8 22:30:10 avclub kernel: [ 527.094719] disk 4, o:1, dev:sdb6
Dec 8 22:30:10 avclub kernel: [ 527.094720] disk 5, o:1, dev:sde6
Dec 8 22:30:10 avclub kernel: [ 527.094721] disk 6, o:1, dev:sdg6
Dec 8 22:30:10 avclub kernel: [ 527.094722] disk 7, o:1, dev:sdh6
Dec 8 22:30:10 avclub kernel: [ 527.094876] md: reshape of RAID array md2
Dec 8 22:30:10 avclub kernel: [ 527.094886] md: minimum _guaranteed_
speed: 40000 KB/sec/disk.
Dec 8 22:30:10 avclub kernel: [ 527.094892] md: using maximum
available idle IO bandwidth (but not more than 200000 KB/sec) for
reshape.
Dec 8 22:30:10 avclub kernel: [ 527.094912] md: using 128k window,
over a total of 1371476928 blocks.
Dec 8 22:30:11 avclub mdadm[2959]: RebuildStarted event detected on
md device /dev/md2
Dec 8 22:30:11 avclub kernel: [ 527.515359] general protection
fault: 0000 [#1] SMP
Dec 8 22:30:11 avclub kernel: [ 527.515370] last sysfs file:
/sys/devices/virtual/block/md2/md/sync_speed
Dec 8 22:30:11 avclub kernel: [ 527.515376] CPU 5
Dec 8 22:30:11 avclub kernel: [ 527.515381] Modules linked in:
binfmt_misc nfsd exportfs nfs lockd nfs_acl auth_rpcgss sunrpc
snd_usb_audio snd_usb_lib snd_hda_codec_atihdmi fbcon tileblit font
bitblit softcursor
vga16fb vgastate snd_hda_codec_via snd_hda_intel snd_pcm_oss
snd_hda_codec snd_mixer_
oss snd_hwdep snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi
snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device radeon
ttm asus_atk0110 drm_kms_helper ppdev snd drm i2c_algo_bit parport_pc
edac_core edac_mce_amd gspca_zc3xx gspca_main videodev v4l1_compat
v4l2_compat_ioctl32 soundcore snd_page_alloc i2c_piix4 shpchp lp
parport tcp_vegas raid10 raid456 async_pq async_xor xor async_memcpy
usbhid async_raid6_recov hid raid6_pq async_tx raid1 raid0 pata_atiixp
r8169 mii multipath ahci linear [last unloaded: kvm]
Dec 8 22:30:11 avclub kernel: [ 527.515500] Pid: 528, comm:
md2_raid6 Not tainted 2.6.32-32-generic #62-Ubuntu System Product Name
Dec 8 22:30:11 avclub kernel: [ 527.515507] RIP:
0010:[<ffffffff812be15b>] [<ffffffff812be15b>] memcpy_c+0xb/0x20
Dec 8 22:30:11 avclub kernel: [ 527.515526] RSP:
0018:ffff880408985c18 EFLAGS: 00010246
Dec 8 22:30:11 avclub kernel: [ 527.515531] RAX: db73880000000000
RBX: ffff880408984000 RCX: 0000000000000200
Dec 8 22:30:11 avclub kernel: [ 527.515537] RDX: 0000000000000000
RSI: ffff880369717000 RDI: db73880000000000
Dec 8 22:30:11 avclub kernel: [ 527.515543] RBP: ffff880408985c80
R08: 0000000000001000 R09: ffff880408985ca0
Dec 8 22:30:11 avclub kernel: [ 527.515548] R10: 0000000000000000
R11: 0000000000000000 R12: ffff880408985ca0
Dec 8 22:30:11 avclub kernel: [ 527.515553] R13: ffff880369741290
R14: 0000000000000000 R15: 0000000000000000
Dec 8 22:30:11 avclub kernel: [ 527.515560] FS:
00007f465923d7a0(0000) GS:ffff880028340000(0000)
knlGS:00000000f6990760
Dec 8 22:30:11 avclub kernel: [ 527.515566] CS: 0010 DS: 0018 ES:
0018 CR0: 000000008005003b
Dec 8 22:30:11 avclub kernel: [ 527.515571] CR2: 00007fe6aaf92000
CR3: 00000003c3a5e000 CR4: 00000000000006e0
Dec 8 22:30:11 avclub kernel: [ 527.515576] DR0: 0000000000000000
DR1: 0000000000000000 DR2: 0000000000000000
Dec 8 22:30:11 avclub kernel: [ 527.515582] DR3: 0000000000000000
DR6: 00000000ffff0ff0 DR7: 0000000000000400
Dec 8 22:30:11 avclub kernel: [ 527.515589] Process md2_raid6 (pid:
528, threadinfo ffff880408984000, task ffff88040b210000)
Dec 8 22:30:11 avclub kernel: [ 527.515593] Stack:
Dec 8 22:30:11 avclub kernel: [ 527.515596] ffffffffa004a0e7
ffff880408985c50 0000000000000000 0000000000000000
Dec 8 22:30:11 avclub kernel: [ 527.515604] <0> ffffea000bf10d08
0000000000000000 0000000000001000 ffff880408985c80
Dec 8 22:30:11 avclub kernel: [ 527.515614] <0> 0000000000000000
ffff8803696a6930 ffff880369741290 ffff880408985d70
Dec 8 22:30:11 avclub kernel: [ 527.515624] Call Trace:
Dec 8 22:30:11 avclub kernel: [ 527.515639] [<ffffffffa004a0e7>] ?
async_memcpy+0xe7/0x25c [async_memcpy]
Dec 8 22:30:11 avclub kernel: [ 527.515654] [<ffffffffa00aaabb>]
handle_stripe_expansion+0x14b/0x1e0 [raid456]
Dec 8 22:30:11 avclub kernel: [ 527.515668] [<ffffffffa00ab113>]
handle_stripe6+0x5c3/0xb40 [raid456]
Dec 8 22:30:11 avclub kernel: [ 527.515680] [<ffffffffa00a794c>] ?
__release_stripe+0xcc/0x1c0 [raid456]
Dec 8 22:30:11 avclub kernel: [ 527.515692] [<ffffffffa00ac055>]
handle_stripe+0x25/0x30 [raid456]
Dec 8 22:30:11 avclub kernel: [ 527.515703] [<ffffffffa00ac452>]
raid5d+0x202/0x320 [raid456]
Dec 8 22:30:11 avclub kernel: [ 527.515716] [<ffffffff815416b9>] ?
_spin_unlock_irqrestore+0x19/0x30
Dec 8 22:30:11 avclub kernel: [ 527.515725] [<ffffffff8141704c>]
md_thread+0x5c/0x130
Dec 8 22:30:11 avclub kernel: [ 527.515735] [<ffffffff81084cb0>] ?
autoremove_wake_function+0x0/0x40
Dec 8 22:30:11 avclub kernel: [ 527.515743] [<ffffffff81416ff0>] ?
md_thread+0x0/0x130
Dec 8 22:30:11 avclub kernel: [ 527.515750] [<ffffffff81084936>]
kthread+0x96/0xa0
Dec 8 22:30:11 avclub kernel: [ 527.515758] [<ffffffff810131ea>]
child_rip+0xa/0x20
Dec 8 22:30:11 avclub kernel: [ 527.515766] [<ffffffff810848a0>] ?
kthread+0x0/0xa0
Dec 8 22:30:11 avclub kernel: [ 527.515772] [<ffffffff810131e0>] ?
child_rip+0x0/0x20
Dec 8 22:30:11 avclub kernel: [ 527.515776] Code: 81 ea d8 1f 00 00
48 3b 42 20 73 07 48 8b 50 f9 31 c0 c3 31 d2 48 c7 c0 f2 ff ff ff c3
90 90 90 48 89 f8 89 d1 c1 e9 03 83 e2 07 <f3> 48 a5 89 d1 f3 a4 c3 66
66 66 66 2e 0f 1f 84 00 00 00 00 00
Dec 8 22:30:11 avclub kernel: [ 527.515842] RIP
[<ffffffff812be15b>] memcpy_c+0xb/0x20
Dec 8 22:30:11 avclub kernel: [ 527.515850] RSP <ffff880408985c18>
Dec 8 22:30:11 avclub kernel: [ 527.515857] ---[ end trace
5146b1cc8ebe8dc1 ]---
Dec 8 22:30:11 avclub kernel: [ 527.515865] note: md2_raid6[528]
exited with preempt_count 2
Dec 8 22:32:52 avclub kernel: Kernel logging (proc) stopped.
I believe that last line shows me giving up. I am sad.
Thankfully, after rebooting into single user mode, I was able to mdadm
--assemble the array, and it appears to be working. Boy that was a
rush!
$ uname -aLinux avclub 2.6.32-32-generic #62-Ubuntu SMP Wed Apr 20
21:52:38 UTC 2011 x86_64 GNU/Linux
Let me know if I can provide any other information.
- Gavin
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: failed reshape!
2011-12-09 13:53 failed reshape! Gavin Peters (蓋文彼德斯)
@ 2011-12-12 3:39 ` NeilBrown
2011-12-12 12:12 ` Gavin Peters (蓋文彼德斯)
0 siblings, 1 reply; 3+ messages in thread
From: NeilBrown @ 2011-12-12 3:39 UTC (permalink / raw)
To: Gavin Peters (蓋文彼德斯); +Cc: linux-raid
[-- Attachment #1: Type: text/plain, Size: 8092 bytes --]
On Fri, 9 Dec 2011 08:53:42 -0500 Gavin Peters (蓋文彼德斯) <gavin@ytz.ca>
wrote:
> I tried to reshape today, a raid6 array from seven devices up to
> eight. I ran mdadm 3.2.2, something like
>
> # mdadm /dev/md2 --grow -n 8 --layout=preserve
>
> and then, blammo!
>
> Dec 8 22:30:10 avclub kernel: [ 527.094708] RAID5 conf printout:
> Dec 8 22:30:10 avclub kernel: [ 527.094712] --- rd:8 wd:8
> Dec 8 22:30:10 avclub kernel: [ 527.094714] disk 0, o:1, dev:sdc6
> Dec 8 22:30:10 avclub kernel: [ 527.094715] disk 1, o:1, dev:sdf6
> Dec 8 22:30:10 avclub kernel: [ 527.094717] disk 2, o:1, dev:sda6
> Dec 8 22:30:10 avclub kernel: [ 527.094718] disk 3, o:1, dev:sdd6
> Dec 8 22:30:10 avclub kernel: [ 527.094719] disk 4, o:1, dev:sdb6
> Dec 8 22:30:10 avclub kernel: [ 527.094720] disk 5, o:1, dev:sde6
> Dec 8 22:30:10 avclub kernel: [ 527.094721] disk 6, o:1, dev:sdg6
> Dec 8 22:30:10 avclub kernel: [ 527.094722] disk 7, o:1, dev:sdh6
> Dec 8 22:30:10 avclub kernel: [ 527.094876] md: reshape of RAID array md2
> Dec 8 22:30:10 avclub kernel: [ 527.094886] md: minimum _guaranteed_
> speed: 40000 KB/sec/disk.
> Dec 8 22:30:10 avclub kernel: [ 527.094892] md: using maximum
> available idle IO bandwidth (but not more than 200000 KB/sec) for
> reshape.
> Dec 8 22:30:10 avclub kernel: [ 527.094912] md: using 128k window,
> over a total of 1371476928 blocks.
> Dec 8 22:30:11 avclub mdadm[2959]: RebuildStarted event detected on
> md device /dev/md2
> Dec 8 22:30:11 avclub kernel: [ 527.515359] general protection
> fault: 0000 [#1] SMP
> Dec 8 22:30:11 avclub kernel: [ 527.515370] last sysfs file:
> /sys/devices/virtual/block/md2/md/sync_speed
> Dec 8 22:30:11 avclub kernel: [ 527.515376] CPU 5
> Dec 8 22:30:11 avclub kernel: [ 527.515381] Modules linked in:
> binfmt_misc nfsd exportfs nfs lockd nfs_acl auth_rpcgss sunrpc
> snd_usb_audio snd_usb_lib snd_hda_codec_atihdmi fbcon tileblit font
> bitblit softcursor
> vga16fb vgastate snd_hda_codec_via snd_hda_intel snd_pcm_oss
> snd_hda_codec snd_mixer_
> oss snd_hwdep snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi
> snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device radeon
> ttm asus_atk0110 drm_kms_helper ppdev snd drm i2c_algo_bit parport_pc
> edac_core edac_mce_amd gspca_zc3xx gspca_main videodev v4l1_compat
> v4l2_compat_ioctl32 soundcore snd_page_alloc i2c_piix4 shpchp lp
> parport tcp_vegas raid10 raid456 async_pq async_xor xor async_memcpy
> usbhid async_raid6_recov hid raid6_pq async_tx raid1 raid0 pata_atiixp
> r8169 mii multipath ahci linear [last unloaded: kvm]
> Dec 8 22:30:11 avclub kernel: [ 527.515500] Pid: 528, comm:
> md2_raid6 Not tainted 2.6.32-32-generic #62-Ubuntu System Product Name
> Dec 8 22:30:11 avclub kernel: [ 527.515507] RIP:
> 0010:[<ffffffff812be15b>] [<ffffffff812be15b>] memcpy_c+0xb/0x20
> Dec 8 22:30:11 avclub kernel: [ 527.515526] RSP:
> 0018:ffff880408985c18 EFLAGS: 00010246
> Dec 8 22:30:11 avclub kernel: [ 527.515531] RAX: db73880000000000
> RBX: ffff880408984000 RCX: 0000000000000200
> Dec 8 22:30:11 avclub kernel: [ 527.515537] RDX: 0000000000000000
> RSI: ffff880369717000 RDI: db73880000000000
> Dec 8 22:30:11 avclub kernel: [ 527.515543] RBP: ffff880408985c80
> R08: 0000000000001000 R09: ffff880408985ca0
> Dec 8 22:30:11 avclub kernel: [ 527.515548] R10: 0000000000000000
> R11: 0000000000000000 R12: ffff880408985ca0
> Dec 8 22:30:11 avclub kernel: [ 527.515553] R13: ffff880369741290
> R14: 0000000000000000 R15: 0000000000000000
> Dec 8 22:30:11 avclub kernel: [ 527.515560] FS:
> 00007f465923d7a0(0000) GS:ffff880028340000(0000)
> knlGS:00000000f6990760
> Dec 8 22:30:11 avclub kernel: [ 527.515566] CS: 0010 DS: 0018 ES:
> 0018 CR0: 000000008005003b
> Dec 8 22:30:11 avclub kernel: [ 527.515571] CR2: 00007fe6aaf92000
> CR3: 00000003c3a5e000 CR4: 00000000000006e0
> Dec 8 22:30:11 avclub kernel: [ 527.515576] DR0: 0000000000000000
> DR1: 0000000000000000 DR2: 0000000000000000
> Dec 8 22:30:11 avclub kernel: [ 527.515582] DR3: 0000000000000000
> DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Dec 8 22:30:11 avclub kernel: [ 527.515589] Process md2_raid6 (pid:
> 528, threadinfo ffff880408984000, task ffff88040b210000)
> Dec 8 22:30:11 avclub kernel: [ 527.515593] Stack:
> Dec 8 22:30:11 avclub kernel: [ 527.515596] ffffffffa004a0e7
> ffff880408985c50 0000000000000000 0000000000000000
> Dec 8 22:30:11 avclub kernel: [ 527.515604] <0> ffffea000bf10d08
> 0000000000000000 0000000000001000 ffff880408985c80
> Dec 8 22:30:11 avclub kernel: [ 527.515614] <0> 0000000000000000
> ffff8803696a6930 ffff880369741290 ffff880408985d70
> Dec 8 22:30:11 avclub kernel: [ 527.515624] Call Trace:
> Dec 8 22:30:11 avclub kernel: [ 527.515639] [<ffffffffa004a0e7>] ?
> async_memcpy+0xe7/0x25c [async_memcpy]
> Dec 8 22:30:11 avclub kernel: [ 527.515654] [<ffffffffa00aaabb>]
> handle_stripe_expansion+0x14b/0x1e0 [raid456]
> Dec 8 22:30:11 avclub kernel: [ 527.515668] [<ffffffffa00ab113>]
> handle_stripe6+0x5c3/0xb40 [raid456]
> Dec 8 22:30:11 avclub kernel: [ 527.515680] [<ffffffffa00a794c>] ?
> __release_stripe+0xcc/0x1c0 [raid456]
> Dec 8 22:30:11 avclub kernel: [ 527.515692] [<ffffffffa00ac055>]
> handle_stripe+0x25/0x30 [raid456]
> Dec 8 22:30:11 avclub kernel: [ 527.515703] [<ffffffffa00ac452>]
> raid5d+0x202/0x320 [raid456]
> Dec 8 22:30:11 avclub kernel: [ 527.515716] [<ffffffff815416b9>] ?
> _spin_unlock_irqrestore+0x19/0x30
> Dec 8 22:30:11 avclub kernel: [ 527.515725] [<ffffffff8141704c>]
> md_thread+0x5c/0x130
> Dec 8 22:30:11 avclub kernel: [ 527.515735] [<ffffffff81084cb0>] ?
> autoremove_wake_function+0x0/0x40
> Dec 8 22:30:11 avclub kernel: [ 527.515743] [<ffffffff81416ff0>] ?
> md_thread+0x0/0x130
> Dec 8 22:30:11 avclub kernel: [ 527.515750] [<ffffffff81084936>]
> kthread+0x96/0xa0
> Dec 8 22:30:11 avclub kernel: [ 527.515758] [<ffffffff810131ea>]
> child_rip+0xa/0x20
> Dec 8 22:30:11 avclub kernel: [ 527.515766] [<ffffffff810848a0>] ?
> kthread+0x0/0xa0
> Dec 8 22:30:11 avclub kernel: [ 527.515772] [<ffffffff810131e0>] ?
> child_rip+0x0/0x20
> Dec 8 22:30:11 avclub kernel: [ 527.515776] Code: 81 ea d8 1f 00 00
> 48 3b 42 20 73 07 48 8b 50 f9 31 c0 c3 31 d2 48 c7 c0 f2 ff ff ff c3
> 90 90 90 48 89 f8 89 d1 c1 e9 03 83 e2 07 <f3> 48 a5 89 d1 f3 a4 c3 66
> 66 66 66 2e 0f 1f 84 00 00 00 00 00
> Dec 8 22:30:11 avclub kernel: [ 527.515842] RIP
> [<ffffffff812be15b>] memcpy_c+0xb/0x20
> Dec 8 22:30:11 avclub kernel: [ 527.515850] RSP <ffff880408985c18>
> Dec 8 22:30:11 avclub kernel: [ 527.515857] ---[ end trace
> 5146b1cc8ebe8dc1 ]---
> Dec 8 22:30:11 avclub kernel: [ 527.515865] note: md2_raid6[528]
> exited with preempt_count 2
> Dec 8 22:32:52 avclub kernel: Kernel logging (proc) stopped.
>
> I believe that last line shows me giving up. I am sad.
> Thankfully, after rebooting into single user mode, I was able to mdadm
> --assemble the array, and it appears to be working. Boy that was a
> rush!
> $ uname -aLinux avclub 2.6.32-32-generic #62-Ubuntu SMP Wed Apr 20
> 21:52:38 UTC 2011 x86_64 GNU/Linux
> Let me know if I can provide any other information.
>
Thanks for the report.
It seems that as part of the reshape, md is trying to copy to an invalid
memory address.
It copies from 0xffff880369717000 (RSI) to 0xdb73880000000000 (rdi).
The latter is clearly invalid.
I have no idea how this might be happening. My best guess is that 'ddidx' in
handle_stripe_expansion is getting a bad value but I cannot see how that
would happen.
If you have reasonable backups you could try again and see if it still fails.
Maybe it was a one-off.
Not sure what else to suggest. It might be fixed in a newer kernel, or it
might not...
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: failed reshape!
2011-12-12 3:39 ` NeilBrown
@ 2011-12-12 12:12 ` Gavin Peters (蓋文彼德斯)
0 siblings, 0 replies; 3+ messages in thread
From: Gavin Peters (蓋文彼德斯) @ 2011-12-12 12:12 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
2011/12/11 NeilBrown <neilb@suse.de>:
> On Fri, 9 Dec 2011 08:53:42 -0500 Gavin Peters (蓋文彼德斯) <gavin@ytz.ca>
> wrote:
>
>> I tried to reshape today, a raid6 array from seven devices up to
>> eight. I ran mdadm 3.2.2, something like
>>
>> # mdadm /dev/md2 --grow -n 8 --layout=preserve
>>
>> and then, blammo!
>>
>> Dec 8 22:30:10 avclub kernel: [ 527.094708] RAID5 conf printout:
>> Dec 8 22:30:10 avclub kernel: [ 527.094712] --- rd:8 wd:8
>> Dec 8 22:30:10 avclub kernel: [ 527.094714] disk 0, o:1, dev:sdc6
>> Dec 8 22:30:10 avclub kernel: [ 527.094715] disk 1, o:1, dev:sdf6
>> Dec 8 22:30:10 avclub kernel: [ 527.094717] disk 2, o:1, dev:sda6
>> Dec 8 22:30:10 avclub kernel: [ 527.094718] disk 3, o:1, dev:sdd6
>> Dec 8 22:30:10 avclub kernel: [ 527.094719] disk 4, o:1, dev:sdb6
>> Dec 8 22:30:10 avclub kernel: [ 527.094720] disk 5, o:1, dev:sde6
>> Dec 8 22:30:10 avclub kernel: [ 527.094721] disk 6, o:1, dev:sdg6
>> Dec 8 22:30:10 avclub kernel: [ 527.094722] disk 7, o:1, dev:sdh6
>> Dec 8 22:30:10 avclub kernel: [ 527.094876] md: reshape of RAID array md2
>> Dec 8 22:30:10 avclub kernel: [ 527.094886] md: minimum _guaranteed_
>> speed: 40000 KB/sec/disk.
>> Dec 8 22:30:10 avclub kernel: [ 527.094892] md: using maximum
>> available idle IO bandwidth (but not more than 200000 KB/sec) for
>> reshape.
>> Dec 8 22:30:10 avclub kernel: [ 527.094912] md: using 128k window,
>> over a total of 1371476928 blocks.
>> Dec 8 22:30:11 avclub mdadm[2959]: RebuildStarted event detected on
>> md device /dev/md2
>> Dec 8 22:30:11 avclub kernel: [ 527.515359] general protection
>> fault: 0000 [#1] SMP
>> Dec 8 22:30:11 avclub kernel: [ 527.515370] last sysfs file:
>> /sys/devices/virtual/block/md2/md/sync_speed
>> Dec 8 22:30:11 avclub kernel: [ 527.515376] CPU 5
>> Dec 8 22:30:11 avclub kernel: [ 527.515381] Modules linked in:
>> binfmt_misc nfsd exportfs nfs lockd nfs_acl auth_rpcgss sunrpc
>> snd_usb_audio snd_usb_lib snd_hda_codec_atihdmi fbcon tileblit font
>> bitblit softcursor
>> vga16fb vgastate snd_hda_codec_via snd_hda_intel snd_pcm_oss
>> snd_hda_codec snd_mixer_
>> oss snd_hwdep snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi
>> snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device radeon
>> ttm asus_atk0110 drm_kms_helper ppdev snd drm i2c_algo_bit parport_pc
>> edac_core edac_mce_amd gspca_zc3xx gspca_main videodev v4l1_compat
>> v4l2_compat_ioctl32 soundcore snd_page_alloc i2c_piix4 shpchp lp
>> parport tcp_vegas raid10 raid456 async_pq async_xor xor async_memcpy
>> usbhid async_raid6_recov hid raid6_pq async_tx raid1 raid0 pata_atiixp
>> r8169 mii multipath ahci linear [last unloaded: kvm]
>> Dec 8 22:30:11 avclub kernel: [ 527.515500] Pid: 528, comm:
>> md2_raid6 Not tainted 2.6.32-32-generic #62-Ubuntu System Product Name
>> Dec 8 22:30:11 avclub kernel: [ 527.515507] RIP:
>> 0010:[<ffffffff812be15b>] [<ffffffff812be15b>] memcpy_c+0xb/0x20
>> Dec 8 22:30:11 avclub kernel: [ 527.515526] RSP:
>> 0018:ffff880408985c18 EFLAGS: 00010246
>> Dec 8 22:30:11 avclub kernel: [ 527.515531] RAX: db73880000000000
>> RBX: ffff880408984000 RCX: 0000000000000200
>> Dec 8 22:30:11 avclub kernel: [ 527.515537] RDX: 0000000000000000
>> RSI: ffff880369717000 RDI: db73880000000000
>> Dec 8 22:30:11 avclub kernel: [ 527.515543] RBP: ffff880408985c80
>> R08: 0000000000001000 R09: ffff880408985ca0
>> Dec 8 22:30:11 avclub kernel: [ 527.515548] R10: 0000000000000000
>> R11: 0000000000000000 R12: ffff880408985ca0
>> Dec 8 22:30:11 avclub kernel: [ 527.515553] R13: ffff880369741290
>> R14: 0000000000000000 R15: 0000000000000000
>> Dec 8 22:30:11 avclub kernel: [ 527.515560] FS:
>> 00007f465923d7a0(0000) GS:ffff880028340000(0000)
>> knlGS:00000000f6990760
>> Dec 8 22:30:11 avclub kernel: [ 527.515566] CS: 0010 DS: 0018 ES:
>> 0018 CR0: 000000008005003b
>> Dec 8 22:30:11 avclub kernel: [ 527.515571] CR2: 00007fe6aaf92000
>> CR3: 00000003c3a5e000 CR4: 00000000000006e0
>> Dec 8 22:30:11 avclub kernel: [ 527.515576] DR0: 0000000000000000
>> DR1: 0000000000000000 DR2: 0000000000000000
>> Dec 8 22:30:11 avclub kernel: [ 527.515582] DR3: 0000000000000000
>> DR6: 00000000ffff0ff0 DR7: 0000000000000400
>> Dec 8 22:30:11 avclub kernel: [ 527.515589] Process md2_raid6 (pid:
>> 528, threadinfo ffff880408984000, task ffff88040b210000)
>> Dec 8 22:30:11 avclub kernel: [ 527.515593] Stack:
>> Dec 8 22:30:11 avclub kernel: [ 527.515596] ffffffffa004a0e7
>> ffff880408985c50 0000000000000000 0000000000000000
>> Dec 8 22:30:11 avclub kernel: [ 527.515604] <0> ffffea000bf10d08
>> 0000000000000000 0000000000001000 ffff880408985c80
>> Dec 8 22:30:11 avclub kernel: [ 527.515614] <0> 0000000000000000
>> ffff8803696a6930 ffff880369741290 ffff880408985d70
>> Dec 8 22:30:11 avclub kernel: [ 527.515624] Call Trace:
>> Dec 8 22:30:11 avclub kernel: [ 527.515639] [<ffffffffa004a0e7>] ?
>> async_memcpy+0xe7/0x25c [async_memcpy]
>> Dec 8 22:30:11 avclub kernel: [ 527.515654] [<ffffffffa00aaabb>]
>> handle_stripe_expansion+0x14b/0x1e0 [raid456]
>> Dec 8 22:30:11 avclub kernel: [ 527.515668] [<ffffffffa00ab113>]
>> handle_stripe6+0x5c3/0xb40 [raid456]
>> Dec 8 22:30:11 avclub kernel: [ 527.515680] [<ffffffffa00a794c>] ?
>> __release_stripe+0xcc/0x1c0 [raid456]
>> Dec 8 22:30:11 avclub kernel: [ 527.515692] [<ffffffffa00ac055>]
>> handle_stripe+0x25/0x30 [raid456]
>> Dec 8 22:30:11 avclub kernel: [ 527.515703] [<ffffffffa00ac452>]
>> raid5d+0x202/0x320 [raid456]
>> Dec 8 22:30:11 avclub kernel: [ 527.515716] [<ffffffff815416b9>] ?
>> _spin_unlock_irqrestore+0x19/0x30
>> Dec 8 22:30:11 avclub kernel: [ 527.515725] [<ffffffff8141704c>]
>> md_thread+0x5c/0x130
>> Dec 8 22:30:11 avclub kernel: [ 527.515735] [<ffffffff81084cb0>] ?
>> autoremove_wake_function+0x0/0x40
>> Dec 8 22:30:11 avclub kernel: [ 527.515743] [<ffffffff81416ff0>] ?
>> md_thread+0x0/0x130
>> Dec 8 22:30:11 avclub kernel: [ 527.515750] [<ffffffff81084936>]
>> kthread+0x96/0xa0
>> Dec 8 22:30:11 avclub kernel: [ 527.515758] [<ffffffff810131ea>]
>> child_rip+0xa/0x20
>> Dec 8 22:30:11 avclub kernel: [ 527.515766] [<ffffffff810848a0>] ?
>> kthread+0x0/0xa0
>> Dec 8 22:30:11 avclub kernel: [ 527.515772] [<ffffffff810131e0>] ?
>> child_rip+0x0/0x20
>> Dec 8 22:30:11 avclub kernel: [ 527.515776] Code: 81 ea d8 1f 00 00
>> 48 3b 42 20 73 07 48 8b 50 f9 31 c0 c3 31 d2 48 c7 c0 f2 ff ff ff c3
>> 90 90 90 48 89 f8 89 d1 c1 e9 03 83 e2 07 <f3> 48 a5 89 d1 f3 a4 c3 66
>> 66 66 66 2e 0f 1f 84 00 00 00 00 00
>> Dec 8 22:30:11 avclub kernel: [ 527.515842] RIP
>> [<ffffffff812be15b>] memcpy_c+0xb/0x20
>> Dec 8 22:30:11 avclub kernel: [ 527.515850] RSP <ffff880408985c18>
>> Dec 8 22:30:11 avclub kernel: [ 527.515857] ---[ end trace
>> 5146b1cc8ebe8dc1 ]---
>> Dec 8 22:30:11 avclub kernel: [ 527.515865] note: md2_raid6[528]
>> exited with preempt_count 2
>> Dec 8 22:32:52 avclub kernel: Kernel logging (proc) stopped.
>>
>> I believe that last line shows me giving up. I am sad.
>> Thankfully, after rebooting into single user mode, I was able to mdadm
>> --assemble the array, and it appears to be working. Boy that was a
>> rush!
>> $ uname -aLinux avclub 2.6.32-32-generic #62-Ubuntu SMP Wed Apr 20
>> 21:52:38 UTC 2011 x86_64 GNU/Linux
>> Let me know if I can provide any other information.
>>
>
> Thanks for the report.
>
> It seems that as part of the reshape, md is trying to copy to an invalid
> memory address.
> It copies from 0xffff880369717000 (RSI) to 0xdb73880000000000 (rdi).
> The latter is clearly invalid.
>
> I have no idea how this might be happening. My best guess is that 'ddidx' in
> handle_stripe_expansion is getting a bad value but I cannot see how that
> would happen.
>
> If you have reasonable backups you could try again and see if it still fails.
> Maybe it was a one-off.
That appears to be the case. After I rebooted, I was able to assemble
the raid, and it continued the sync with no apparent loss of data.
Naturally, during that resync, I lost a drive, which I'm now doing a
series of resyncs to recover from (the drive reappeared on reboot, so
I can't quite figure out what's up with it)....
Hopefully, sometime in the next day or two I'll have a computer that
works, hasn't lost data, and is not running a RAID resync!
If I can be of any help debugging the problem I had, let me know.
Otherwise, I hope you don't hear from me again. :-D
- Gavin
>
> Not sure what else to suggest. It might be fixed in a newer kernel, or it
> might not...
>
> NeilBrown
>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2011-12-12 12:12 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2011-12-09 13:53 failed reshape! Gavin Peters (蓋文彼德斯)
2011-12-12 3:39 ` NeilBrown
2011-12-12 12:12 ` Gavin Peters (蓋文彼德斯)
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.