From: Harald Braumann <harry@unheit.net>
To: linux-kernel@vger.kernel.org
Subject: Deadlock on concurrent remount-ro and mdadm --stop (2.6.37)
Date: Fri, 21 Jan 2011 13:47:54 +0100 [thread overview]
Message-ID: <20110121124754.GA5233@nn.nn> (raw)
(please CC)
Hi,
my system sometimes completely locks up on shutdown and leaves the raid array
in an unclean state.
On shutdown `umountroot' and `mdadm-raid' are executed concurrently. The
actual commands are:
mount -o remount,ro /
mdadm --stop --scan
This seems to trigger a deadlock in the kernel. Dumping blocked tasks (sysrq-W)
gives:
md126_raid5: call trace:
md_super_wait
autoremove_wake_function
bitmap_unplug
...
mount: call trace:
sync_page
io_schedule
sync_page
__wait_on_bit
wake_bit_function
pagevec_lookup_tag
filemap_fdatawait_range
filemap_write_and_wait
dquot_quota_sync
__sync_filesystem
do_remount_sb
do_mount
memdup_user
sys_mount
system_call_fastpath
Sorry for not supplying a better trace, but while I could trigger the deadlock
some more times, it then always locked up so badly, that not even sysrq-W
would work. If more information is needed I could try again to get a full trace
and dump it over a serial console.
NB: md126_raid5's trace says `bitmap_unplug'. I've added a write-intent bitmap
to the array to speed up the resync. But I also had lockups before, without any
bitmap.
My disk setup is as follows:
sd{a,b,c}1 - raid1 (md127) - ext4
sd{a,b,c}2 - raid5 (md126) - LVM - dm-crypt - LVM - containing root-fs
Some more information, that might be useful:
scripts/ver_linux:
------------------
Gnu C 4.4.5
Gnu make 3.81
binutils 2.20.1
util-linux scripts/ver_linux: 23: fdformat: not found
mount support
module-init-tools found
Linux C Library 2.11.2
Dynamic linker (ldd) 2.11.2
Procps 3.2.8
Console-tools 0.2.3
Sh-utils 8.5
Modules Loaded vboxnetadp vboxnetflt vboxdrv kvm_amd kvm
deflate zlib_deflate ctr twofish_generic twofish_x86_64 twofish_common
camellia serpent blowfish cast5 des_generic cbc xcbc rmd160
sha512_generic sha256_generic sha1_generic hmac crypto_null af_key
fuse nfsd exportfs nfs lockd fscache nfs_acl auth_rpcgss sunrpc bridge
stp ipt_REJECT ipt_LOG xt_state xt_tcpudp xt_physdev iptable_mangle
iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4
iptable_filter ip_tables x_tables ext2 powernow_k8 mperf it87
hwmon_vid snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_intel
snd_hda_codec snd_hwdep snd_pcm_oss snd_mixer_oss radeon snd_pcm
snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi ttm
snd_seq_midi_event snd_seq drm_kms_helper drm agpgart snd_timer
snd_seq_device i2c_algo_bit tpm_tis tpm parport_pc tpm_bios snd
parport shpchp processor button i2c_piix4 soundcore k8temp pci_hotplug
i2c_core pcspkr snd_page_alloc evdev edac_core thermal_sys ext4
mbcache jbd2 crc16 cryptd aes_x86_64 aes_generic xts gf128mul dm_crypt
dm_mod raid456 async_raid6_recov async_pq raid6_pq async_xor xor
async_memcpy async_tx raid1 md_mod lzo sd_mod usbhid crc_t10dif
ata_generic hid pata_acpi ahci libahci ohci_hcd pata_atiixp
firewire_ohci libata ehci_hcd firewire_core ssb usbcore scsi_mod
mmc_core r8169 crc_itu_t mii pcmcia pcmcia_core
/proc/cpuinfo:
--------------
processor : 0
vendor_id : AuthenticAMD
cpu family : 15
model : 107
model name : AMD Athlon(tm) 64 X2 Dual Core Processor 4800+
stepping : 2
cpu MHz : 1000.000
cache size : 512 KB
physical id : 0
siblings : 2
core id : 0
cpu cores : 2
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 1
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt rdtscp lm 3dnowext 3dnow rep_good nopl extd_apicid pni cx16 lahf_lm cmp_legacy svm extapic cr8_legacy 3dnowprefetch lbrv
bogomips : 2005.26
TLB size : 1024 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management: ts fid vid ttp tm stc 100mhzsteps
processor : 1
vendor_id : AuthenticAMD
cpu family : 15
model : 107
model name : AMD Athlon(tm) 64 X2 Dual Core Processor 4800+
stepping : 2
cpu MHz : 1000.000
cache size : 512 KB
physical id : 0
siblings : 2
core id : 1
cpu cores : 2
apicid : 1
initial apicid : 1
fpu : yes
fpu_exception : yes
cpuid level : 1
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt rdtscp lm 3dnowext 3dnow rep_good nopl extd_apicid pni cx16 lahf_lm cmp_legacy svm extapic cr8_legacy 3dnowprefetch lbrv
bogomips : 2005.26
TLB size : 1024 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management: ts fid vid ttp tm stc 100mhzsteps
lspci:
------
00:00.0 Host bridge: ATI Technologies Inc RS690 Host Bridge
00:02.0 PCI bridge: ATI Technologies Inc RS690 PCI to PCI Bridge (PCI Express Graphics Port 0)
00:12.0 SATA controller: ATI Technologies Inc SB600 Non-Raid-5 SATA
00:13.0 USB Controller: ATI Technologies Inc SB600 USB (OHCI0)
00:13.1 USB Controller: ATI Technologies Inc SB600 USB (OHCI1)
00:13.2 USB Controller: ATI Technologies Inc SB600 USB (OHCI2)
00:13.3 USB Controller: ATI Technologies Inc SB600 USB (OHCI3)
00:13.4 USB Controller: ATI Technologies Inc SB600 USB (OHCI4)
00:13.5 USB Controller: ATI Technologies Inc SB600 USB Controller (EHCI)
00:14.0 SMBus: ATI Technologies Inc SBx00 SMBus Controller (rev 14)
00:14.1 IDE interface: ATI Technologies Inc SB600 IDE
00:14.2 Audio device: ATI Technologies Inc SBx00 Azalia (Intel HDA)
00:14.3 ISA bridge: ATI Technologies Inc SB600 PCI to LPC Bridge
00:14.4 PCI bridge: ATI Technologies Inc SBx00 PCI to PCI Bridge
00:18.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration
00:18.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map
00:18.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller
00:18.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control
01:00.0 VGA compatible controller: ATI Technologies Inc Radeon HD 3870
01:00.1 Audio device: ATI Technologies Inc Radeon HD 3870 Audio device
02:0e.0 FireWire (IEEE 1394): Texas Instruments TSB43AB23 IEEE-1394a-2000 Controller (PHY/Link)
02:0f.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL-8110SC/8169SC Gigabit Ethernet (rev 10)
Cheers,
harry
next reply other threads:[~2011-01-21 12:58 UTC|newest]
Thread overview: 5+ messages / expand[flat|nested] mbox.gz Atom feed top
2011-01-21 12:47 Harald Braumann [this message]
2011-01-23 16:15 ` Deadlock on concurrent remount-ro and mdadm --stop (2.6.37) Harald Braumann
2011-01-24 22:57 ` Neil Brown
2011-01-26 0:05 ` Harald Braumann
2011-01-26 21:31 ` Harald Braumann
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20110121124754.GA5233@nn.nn \
--to=harry@unheit.net \
--cc=linux-kernel@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.