[Ocfs2-devel] [PATCH] ocfs2: Do not downconvert if the lock level is already compatible

[Ocfs2-devel] [PATCH] ocfs2: Do not downconvert if the lock level is already compatible

[Sunil Mushran]

David Teigland wrote:
> With this patch I ran alternate and make_panic for about 2.5 hours, and
> then one node hit this BUG.  /var/log/messages didn't catch any of it, so
> no additional info this time.
>
> kernel BUG at fs/ocfs2/dlmglue.c:3395

David,

Please could you re-run with this debug patch.

http://oss.oracle.com/~smushran/.dlmglue/0001-ocfs2-Patch-to-debug-hang-in-dlmglue-when-running-dl.patch

Thanks
Sunil

[Ocfs2-devel] [PATCH] ocfs2: Do not downconvert if the lock level is already compatible

[David Teigland]

On Fri, Jan 29, 2010 at 04:16:39PM -0800, Sunil Mushran wrote:
> David Teigland wrote:
> >With this patch I ran alternate and make_panic for about 2.5 hours, and
> >then one node hit this BUG.  /var/log/messages didn't catch any of it, so
> >no additional info this time.
> >
> >kernel BUG at fs/ocfs2/dlmglue.c:3395
> 
> David,
> 
> Please could you re-run with this debug patch.
> 
> http://oss.oracle.com/~smushran/.dlmglue/0001-ocfs2-Patch-to-debug-hang-in-dlmglue-when-running-dl.patch

I'm working to compress the full logs, but until then here is what
appeared just before the oops on the second node:


Feb  1 13:25:28 bull-02 kernel: (7072000003f00400000000, level 3, inc holders, ex 0, ro 1
Feb  1 13:25:28 bull-02 kernel: (70000003f00400000000, level 3, inc holders, ex 0, ro 1
Feb  1 13:25:28 bull-02 kernel: (7072,3,000003f00400000000, level 3, inc holders, ex 0, ro 1
Feb  1 13:25:28 bull-02 kernel: (7072,3,000003f00400000000, level 3, inc holders, ex 0, ro 1
Feb  1 13:25:28 bull-02 kernel: (7072,3,al000003f00400000000, level 3, inc holders, ex 0, ro 1
Feb  1 13:25:28 bull-02 kernel: (707000003f00400000000, level 3, inc holders, ex 0, ro 1
Feb  1 13:25:28 bull-02 kernel: (7072,3,000003f00400000000, level 3, inc holders, ex 0, ro 1000003f00400000000, level 3, inc holders, ex 0, ro 1
Feb  1 13:25:28 bull-02 kernel: (7072,3,alter000003f00400000000, level 3, inc holders, ex 0, ro 1
Feb  1 13:25:28 bull-02 kernel: (7072,3,000003f00400000000, level 3, inc holders, ex 0, ro 1
Feb  1 13:25:28 bull-02 kernel: (7072,3,alter000003f00400000000, level 3, inc holders, ex 0, ro 1
Feb  1 13:25:28 bull-02 kernel: (7072,3,alte000003f00400000000, level 3, inc holders, ex 0, ro 1
Feb  1 13:25:28 bull-02 kernel: (7072,3,a000003f00400000000, level 3, inc holders, ex 0, ro 1
Feb  1 13:25:28 bull-02 kernel: (7072,3,alter000003f00400000000, level 3, inc holders, ex 0, ro 1
Feb  1 13:25:28 bull-02 kernel: (7072,000003f00400000000, level 3, inc holders, ex 0, ro 1
Feb  1 13:25:28 bull-02 kernel: (70000003f00400000000, level 3, inc holders, ex 0, ro 1
Feb  1 13:25:28 bull-02 kernel: (7072,3,000003f00400000000, level 3, inc holders, ex 0, ro 1
Feb  1 13:25:28 bull-02 kernel: <5000003f00400000000, level 3, inc holders, ex 0, ro 1
Feb  1 13:25:28 bull-02 kernel: (7072,3,000003f00400000000, level 3, inc holders, ex 0, ro 1
Feb  1 13:25:28 bull-02 kernel: (7072,3,al000003f00400000000, level 3, inc holders, ex 0, ro 1
Feb  1 13:25:28 bull-02 kernel: (707000003f00400000000, level 3, inc holders, ex 0, ro 1
Feb  1 13:25:28 bull-02 kernel: (7072,3,000003f00400000000, level 3, inc holders, ex 0, ro 1
Feb  1 13:25:28 bull-02 kernel: (70000003f00400000000, level 3, inc holders, ex 0, ro 1
Feb  1 13:25:28 bull-02 kernel: (7072,3,000003f00400000000, level 3, inc holders, ex 0, ro 1
Feb  1 13:25:28 bull-02 kernel: (7072,3000003f00400000000, level 3, inc holders, ex 0, ro 1
Feb  1 13:25:28 bull-02 kernel: (7072,3,alt000003f00400000000, level 3, inc holders, ex 0, ro 1
Feb  1 13:25:28 bull-02 kernel: (7072,3,000003f00400000000, level 3, inc holders, ex 0, ro 1
Feb  1 13:25:28 bull-02 kernel: (7072,000003f00400000000, level 3, inc holders, ex 0, ro 1
Feb  1 13:25:28 bull-02 kernel: (7072,3,al000003f00400000000, level 3, inc holders, ex 0000003f00400000000, level 3, inc holders, ex 0, ro 1
Feb  1 13:25:28 bull-02 kernel: (7072000003f00400000000, level 3, inc holders, ex 0, ro 1
Feb  1 13:25:28 bull-02 kernel: (7072,3,al000003f00400000000, level 3, inc holders, ex 0, ro 1
Feb  1 13:25:28 bull-02 kernel: (7072,3,al000003f00400000000, level 3, inc holders, ex 0, ro 1
Feb  1 13:25:28 bull-02 kernel: (7072,3,al000003f00400000000, level 3, inc holders, ex 0, ro 1
Feb  1 13:25:28 bull-02 kernel: (7072,3,al000003f00400000000, level 3, inc holders, ex 0, ro 1
Feb  1 13:25:28 bull-02 kernel: (7072,3,al000003f00400000000, level 3, inc holders, ex 0, ro 1
Feb  1 13:25:28 bull-02 kernel: (7072,3,al000003f00400000000, level 3, inc holders, ex 0, ro 1
Feb  1 13:25:28 bull-02 kernel: (7072,3,al000003f00400000000, level 3, inc holders, ex 0, ro 1
Feb  1 13:25:28 bull-02 kernel: (7072,3,al000003f00400000000, level 3, inc holders, ex 0, ro 1
Feb  1 13:25:28 bull-02 kernel: (7072,3,al000003f00400000000, level 3, inc holders, ex 0, ro 1
Feb  1 13:25:28 bull-02 kernel: (7072,3,al000003f00400000000, level 3, inc holders, ex 0, ro 1
Feb  1 13:25:28 bull-02 kernel: (7072,3,al000003f00400000000, level 3, inc holders, ex 0, ro 1
Feb  1 13:25:28 bull-02 kernel: (7072,3,a000003f00400000000, level 3, inc holders, ex 0, ro 1
Feb  1 13:25:28 bull-02 kernel: (7072,3,al000003f00400000000, level 3, inc holders, ex 0, ro 1
Feb  1 13:25:28 bull-02 kernel: (7072,3,al000003f00400000000, level 3, inc holders, ex 0, ro 1
Feb  1 13:25:28 bull-02 kernel: (7072,3,al000003f00400000000, level 3, inc holders, ex 0, ro 1
Feb  1 13:25:28 bull-02 kernel: (7072,3,al000003f00400000000, level 3, inc holders, ex 0, ro 1
Feb  1 13:25:28 bull-02 kernel: (7072,3,al000003f00400000000, level 3, inc holders, ex 0, ro 1
Feb  1 13:25:28 bull-02 kernel: (7072,3,al000003f00400000000, level 3, inc holders, ex 0, ro 1
Feb  1 13:25:28 bull-02 kernel: (7072,3,al000003f00400000000, level 3, inc holders, ex 0, ro 1
Feb  1 13:25:28 bull-02 kernel: (7072,3,al000003f00400000000, level 3, inc holders, ex 0, ro 1
Feb  1 13:25:28 bull-02 kernel: (7072,3,al000003f00400000000, level 3, inc holders, ex 0, ro 1
Feb  1 13:25:28 bull-02 kernel: (7072,3,a000003f00400000000, level 3, inc holders, ex 0, ro 1
Feb  1 13:25:28 bull-02 kernel: (7072,3,al000003f00400000000, level 3, inc holders, ex 0, ro 1
Feb  1 13:25:28 bull-02 kernel: (7072,3,altern000003f00400000000, level 3, inc holders, ex 0, ro 1
Feb  1 13:25:28 bull-02 kernel: (7072,3,a000003f00400000000, level 3, inc holders, ex 0, ro 1
Feb  1 13:25:28 bull-02 kernel: (7072,3,al000003f00400000000, level 3, inc holders, ex 0, ro 1
Feb  1 13:25:28 bull-02 kernel: (7072,3,al000003f00400000000, level 3, inc holders, ex 0, ro 1
Feb  1 13:25:28 bull-02 kernel: (7072,3,al000003f00400000000, level 3, inc holders, ex 0, ro 1
Feb  1 13:25:28 bull-02 kernel: (7072,3,000003f00400000000, level 3, inc holders, ex 0, ro 1
Feb  1 13:25:28 bull-02 kernel: (7072,3,al000003f00400000000, level 3, inc holders, ex 0, ro 1
Feb  1 13:25:28 bull-02 kernel: (7072,3,alt000003f00400000000, level 3, inc holders, ex 0, ro 1
Feb  1 13:25:28 bull-02 kernel: (7072,3,al000003f00400000000, level 3, inc holders, ex 0, ro 1
Feb  1 13:25:28 bull-02 kernel: (7072,3,al000003f00400000000, level 3, inc holders, ex 0, ro 1
Feb  1 13:25:28 bull-02 kernel: (7072,3,al000003f00400000000, level 3, inc holders, ex 0, ro 1
Feb  1 13:25:28 bull-02 kernel: (7072,3,a000003f00400000000, level 3, inc holders, ex 0, ro 1
Feb  1 13:25:28 bull-02 kernel: (7072,3,al000003f00400000000, level 3, inc holders, ex 0, ro 1
Feb  1 13:25:28 bull-02 kernel: (7072,3,a000003f00400000000, level 3, inc holders, ex 0, ro 1
Feb  1 13:25:28 bull-02 kernel: (7072,3,al000003f00400000000, level 3, inc holders, ex 0, ro 1
Feb  1 13:25:28 bull-02 kernel: (7072,3,al000003f00400000000, level 3, inc holders, ex 0, ro 1
Feb  1 13:25:28 bull-02 kernel: (7072,3,al000003f00400000000, level 3, inc holders, ex 0, ro 1
Feb  1 13:25:28 bull-02 kernel: (7072,3,al000003f00400000000, level 3, inc holders, ex 0, ro 1
Feb  1 13:25:28 bull-02 kernel: (7072,3,alt000003f00400000000, level 3, inc holders, ex 0, ro 1
Feb  1 13:25:28 bull-02 kernel: (70000003f00400000000, level 3, inc holders, ex 0, ro 1
Feb  1 13:25:28 bull-02 kernel: (7072,3000003f00400000000, level 3, inc holders,

kernel BUG@fs/ocfs2/dlmglue.c:3420!
invalid opcode: 0000 [#1] SMP
last sysfs file: /sys/devices/pci0000:80/0000:80:02.0/0000:86:01.0/local_cpus
CPU 3
Modules linked in: ocfs2_stack_user dlm ocfs2 ocfs2_nodemanager configfs ocfs2_stackglue sunrpc ipv6 cpufreq_ondemand powernow_k8 freq_table dm_multipath i2c_nforce2 amd64_edac_mod i2c_core shpchp tg3 k8temp serio_raw edac_core qla2xxx mptspi mptscsih ata_generic scsi_transport_fc pata_acpi mptbase scsi_transport_spi scsi_tgt sata_nv pata_amd [last unloaded: scsi_wait_scan]
Pid: 7077, comm: ocfs2dc Not tainted 2.6.32.3 #2 ProLiant DL145 G2
RIP: 0010:[<ffffffffa01eae36>]  [<ffffffffa01eae36>] ocfs2_downconvert_thread+0x4cb/0xdad [ocfs2]
RSP: 0018:ffff88007ce91d90  EFLAGS: 00010046
RAX: 00000000000000b8 RBX: ffff88007c222e50 RCX: 0000000000002784
RDX: 0000000000000000 RSI: 0000000000000046 RDI: 0000000000000046
RBP: ffff88007ce91ee0 R08: 00000000ffffffff R09: 0000000000000000
R10: 0000000000000003 R11: 000000107ce91900 R12: 0000000000000282
R13: 0000000000000000 R14: ffff88007bb15000 R15: ffff88007c222e68
FS:  00007ffabdf1a700(0000) GS:ffff880082100000(0000) knlGS:0000000000000000
CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 00007fe3e403b1c8 CR3: 000000013cd84000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process ocfs2dc (pid: 7077, threadinfo ffff88007ce90000, task ffff8800365b1740)
Stack:
 ffff88007c222e98 ffffffff00000000 ffff880000000001 ffffffff00000001
<0> 0000000000000000 0000000000000041 0000000000000000 ffff880000000000
<0> ffffffff00000000 ffff880000000000 ffff880000000000 ffffffff00000000
Call Trace:
 [<ffffffff81074f6b>] ? autoremove_wake_function+0x0/0x39
 [<ffffffffa01ea96b>] ? ocfs2_downconvert_thread+0x0/0xdad [ocfs2]
 [<ffffffff81074c7e>] kthread+0x7f/0x87
 [<ffffffff81012cea>] child_rip+0xa/0x20
 [<ffffffff81074bff>] ? kthread+0x0/0x87
 [<ffffffff81012ce0>] ? child_rip+0x0/0x20
Code: 24 10 8b 43 68 89 44 24 08 48 8d 43 48 48 89 04 24 31 c0 e8 d0 5e 24 e1 f6 43 40 04 74 0d 4c 8d 63 48 c7 45 8c 00 00 00 00 eb 04 <0f> 0b eb fe 48 8b 4b 40 f6 c1 02 0f 84 2d 01 00 00 80 e5 04 74
RIP  [<ffffffffa01eae36>] ocfs2_downconvert_thread+0x4cb/0xdad [ocfs2]
 RSP <ffff88007ce91d90>

[Ocfs2-devel] [PATCH] ocfs2: Do not downconvert if the lock level is already compatible

[David Teigland]

On Mon, Feb 01, 2010 at 02:19:30PM -0600, David Teigland wrote:
> On Fri, Jan 29, 2010 at 04:16:39PM -0800, Sunil Mushran wrote:
> > David Teigland wrote:
> > >With this patch I ran alternate and make_panic for about 2.5 hours, and
> > >then one node hit this BUG.  /var/log/messages didn't catch any of it, so
> > >no additional info this time.
> > >
> > >kernel BUG at fs/ocfs2/dlmglue.c:3395
> > 
> > David,
> > 
> > Please could you re-run with this debug patch.
> > 
> > http://oss.oracle.com/~smushran/.dlmglue/0001-ocfs2-Patch-to-debug-hang-in-dlmglue-when-running-dl.patch

Using the update to reduce the volume of log messages...

Ran alternate by itself for 10 minutes with no problem, then added
make_panic and got the panic on second node within a couple minutes.

from console:

 ------------[ cut here ]------------
(6930,2,dlm_astd):ocfs2_locking_ast:1104 lock W000000000000000003f00435e4a660, action 2, unlock 0, level 0, newlevel 5
(6930,1,dlm_astd):ocfs2_blocking_ast:1059 BAST fired for lockres M000000000000000004eefc00000000, blocking 3, level 5 type Meta
(6930,1,dlm_astd):ocfs2_generic_handle_bast:932 lockres M000000000000000004eefc00000000, block 3, level 5, l_block 3, dwn 1
(6930,1,dlm_astd):ocfs2_blocking_ast:1059 BAST fired for lockres M000000000000000000000b6354ea08, blocking 5, level 5 type Meta
(6930,1,dlm_astd):ocfs2_generic_handle_bast:932 lockres M000000000000000000000b6354ea08, block 5, level 5, l_block 5, dwn 1
(6930,1,dlm_astd):ocfs2_locking_ast:1104 lock N000000000003f016, action 1, unlock 0, level -1, newlevel 3
(7073,1,make_panic):__ocfs2_cluster_lock:1424 lock M000000000000000003f01500000000, convert from 0 to 3
(6930,1,dlm_astd):ocfs2_locking_ast:1104 lock M000000000000000003f01500000000, action 2, unlock 0, level 0, newlevel 3
(6930,1,dlm_astd):ocfs2_blocking_ast:1059 BAST fired for lockres M000000000000000003f01800000000, blocking 3, level 5 type Meta
(6930,1,dlm_astd):ocfs2_generic_handle_bast:932 lockres M000000000000000003f01800000000, block 3, level 5, l_block 3, dwn 1
kernel BUG at fs/ocfs2/dlmglue.c:3417!
invalid opcode: 0000 [#1] SMP
last sysfs file: /sys/devices/pci0000:80/0000:80:02.0/0000:86:01.0/local_cpus
CPU 0
Modules linked in: ocfs2_stack_user dlm ocfs2 ocfs2_nodemanager configfs ocfs2_stackglue sunrpc ipv6 cpufreq_ondemand powernow_k8 freq_table dm_multipath a
md64_edac_mod edac_core tg3 shpchp serio_raw i2c_nforce2 i2c_core k8temp qla2xxx ata_generic mptspi scsi_transport_fc pata_acpi mptscsih mptbase scsi_trans
port_spi scsi_tgt pata_amd sata_nv [last unloaded: scsi_wait_scan]
Pid: 6957, comm: ocfs2dc Not tainted 2.6.32.3 #2 ProLiant DL145 G2
RIP: 0010:[<ffffffffa020de29>]  [<ffffffffa020de29>] ocfs2_downconvert_thread+0x4cb/0xdad [ocfs2]
RSP: 0018:ffff88007abb7d90  EFLAGS: 00010046
RAX: 00000000000000b9 RBX: ffff88007b99bd50 RCX: 0000000000008506
RDX: 0000000000000000 RSI: 0000000000000046 RDI: 0000000000000046
RBP: ffff88007abb7ee0 R08: ffff88007abb7c80 R09: 0000000000000000
R10: 0000000000000004 R11: 000000107abb7900 R12: 0000000000000282
R13: 0000000000000000 R14: ffff88013a000000 R15: ffff88007b99bd68
FS:  00007f11c9de2710(0000) GS:ffff880028200000(0000) knlGS:0000000000000000
CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 00007f7c6df2f000 CR3: 000000013d11c000 CR4: 00000000000006f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process ocfs2dc (pid: 6957, threadinfo ffff88007abb6000, task ffff88007d3cc5c0)
Stack:
 ffff88007b99bd98 ffffffff00000000 ffff880000000001 ffffffff00000001
<0> 0000000000000000 0000000000000041 0000000000000000 ffff880000000000
<0> ffffffff00000000 ffff880100000000 ffff880000000000 ffffffff00000000
Call Trace:
 [<ffffffff81074f6b>] ? autoremove_wake_function+0x0/0x39
 [<ffffffffa020d95e>] ? ocfs2_downconvert_thread+0x0/0xdad [ocfs2]
 [<ffffffff81074c7e>] kthread+0x7f/0x87
 [<ffffffff81012cea>] child_rip+0xa/0x20
 [<ffffffff81074bff>] ? kthread+0x0/0x87
 [<ffffffff81012ce0>] ? child_rip+0x0/0x20
Code: 24 10 8b 43 68 89 44 24 08 48 8d 43 48 48 89 04 24 31 c0 e8 dd 2e 22 e1 f6 43 40 04 74 0d 4c 8d 63 48 c7 45 8c 00 00 00 00 eb 04 <0f> 0b eb fe 48 8b 
4b 40 f6 c1 02 0f 84 2d 01 00 00 80 e5 04 74
RIP  [<ffffffffa020de29>] ocfs2_downconvert_thread+0x4cb/0xdad [ocfs2]
 RSP <ffff88007abb7d90>
---[ end trace 692271ffad8ccf0d ]---
(7073,1,make_panic):__ocfs2_cluster_lock:1424 lock M000000000000000003f03100000000, convert from 3 to 5
(6930,1,dlm_astd):ocfs2_locking_ast:1104 lock M000000000000000003f03100000000, action 2, unlock 0, level 3, newlevel 5
(6930,1,dlm_astd):ocfs2_locking_ast:1104 lock W000000000000000004f181f5dfe12d, action 1, unlock 0, level -1, newlevel 5
(7073,0,make_panic):__ocfs2_cluster_lock:1424 lock N000000000003f031, convert from -1 to 3
(6930,1,dlm_astd):ocfs2_locking_ast:1104 lock M000000000000000004f18100000000, action 1, unlock 0, level -1, newlevel 5
(6930,1,dlm_astd):ocfs2_locking_ast:1104 lock O000000000000000004f18100000000, action 1, unlock 0, level -1, newlevel 3
(6930,1,dlm_astd):ocfs2_locking_ast:1104 lock N000000000003f031, action 1, unlock 0, level -1, newlevel 3
(7073,0,make_panic):__ocfs2_cluster_lock:1424 lock M000000000000000003f01f00000000, convert from 0 to 3
(6930,1,dlm_astd):ocfs2_locking_ast:1104 lock M000000000000000003f01f00000000, action 2, unlock 0, level 0, newlevel 3
(7073,0,make_panic):__ocfs2_cluster_lock:1424 lock M000000000000000003f00900000000, convert from 0 to 3
(6930,1,dlm_astd):ocfs2_locking_ast:1104 lock M000000000000000003f00900000000, action 2, unlock 0, level 0, newlevel 3
(7073,0,make_panic):__ocfs2_cluster_lock:1424 lock M000000000000000003f01e00000000, convert from 0 to 3
(6930,1,dlm_astd):ocfs2_locking_ast:1104 lock M000000000000000003f01e00000000, action 2, unlock 0, level 0, newlevel 3
(7073,0,make_panic):__ocfs2_cluster_lock:1424 lock M000000000000000003f02c00000000, convert from 3 to 5
(6930,1,dlm_astd):ocfs2_locking_ast:1104 lock M000000000000000003f02c00000000, action 2, unlock 0, level 3, newlevel 5
(7073,0,make_panic):__ocfs2_cluster_lock:1424 lock N000000000003f02c, convert from -1 to 3
(6930,1,dlm_astd):ocfs2_locking_ast:1104 lock W000000000000000004f182f5dfe12e, action 1, unlock 0, level -1, newlevel 5
(6930,1,dlm_astd):ocfs2_locking_ast:1104 lock O000000000000000004f18200000000, action 1, unlock 0, level -1, newlevel 3
(6930,1,dlm_astd):ocfs2_locking_ast:1104 lock M000000000000000004f18200000000, action 1, unlock 0, level -1, newlevel 5
(6930,1,dlm_astd):ocfs2_locking_ast:1104 lock N000000000003f02c, action 1, unlock 0, level -1, newlevel 3
(6930,1,dlm_astd):ocfs2_locking_ast:1104 lock O000000000000000004f18300000000, action 1, unlock 0, level -1, newlevel 3
(7073,0,make_panic):__ocfs2_cluster_lock:1424 lock N000000000003f016, convert from -1 to 3
(6930,1,dlm_astd):ocfs2_locking_ast:1104 lock N000000000003f016, action 1, unlock 0, level -1, newlevel 3
(6930,1,dlm_astd):ocfs2_locking_ast:1104 lock W000000000000000004f183f5dfe12f, action 1, unlock 0, level -1, newlevel 5
(6930,1,dlm_astd):ocfs2_locking_ast:1104 lock M000000000000000004f18300000000, action 1, unlock 0, level -1, newlevel 5
(7073,0,make_panic):__ocfs2_cluster_lock:1424 lock M000000000000000003f00500000000, convert from 0 to 3
(6930,1,dlm_astd):ocfs2_locking_ast:1104 lock M000000000000000003f00500000000, action 2, unlock 0, level 0, newlevel 3
(7073,0,make_panic):__ocfs2_cluster_lock:1424 lock M000000000000000003f03600000000, convert from 3 to 5
(6930,1,dlm_astd):ocfs2_locking_ast:1104 lock M000000000000000003f03600000000, action 2, unlock 0, level 3, newlevel 5
(6930,1,dlm_astd):ocfs2_locking_ast:1104 lock M000000000000000004f18400000000, action 1, unlock 0, level -1, newlevel 5
(7073,0,make_panic):__ocfs2_cluster_lock:1424 lock N000000000003f036, convert from -1 to 3
(6930,1,dlm_astd):ocfs2_locking_ast:1104 lock O000000000000000004f18400000000, action 1, unlock 0, level -1, newlevel 3
(6930,1,dlm_astd):ocfs2_locking_ast:1104 lock W000000000000000004f184f5dfe130, action 1, unlock 0, level -1, newlevel 5
(6930,1,dlm_astd):ocfs2_locking_ast:1104 lock N000000000003f036, action 1, unlock 0, level -1, newlevel 3
(7073,0,make_panic):__ocfs2_cluster_lock:1424 lock M000000000000000003f00f00000000, convert from 0 to 3
(6930,1,dlm_astd):ocfs2_locking_ast:1104 lock M000000000000000003f00f00000000, action 2, unlock 0, level 0, newlevel 3
(7073,0,make_panic):__ocfs2_cluster_lock:1424 lock M000000000000000003f02700000000, convert from 0 to 3
(6930,1,dlm_astd):ocfs2_locking_ast:1104 lock M000000000000000003f02700000000, action 2, unlock 0, level 0, newlevel 3
(7073,0,make_panic):__ocfs2_cluster_lock:1424 lock M000000000000000003f02100000000, convert from 0 to 3
(6930,1,dlm_astd):ocfs2_locking_ast:1104 lock M000000000000000003f02100000000, action 2, unlock 0, level 0, newlevel 3
(7073,0,make_panic):__ocfs2_cluster_lock:1424 lock M000000000000000003f01a00000000, convert from 3 to 5
(6930,1,dlm_astd):ocfs2_locking_ast:1104 lock M000000000000000003f01a00000000, action 2, unlock 0, level 3, newlevel 5
(6930,1,dlm_astd):ocfs2_locking_ast:1104 lock W000000000000000004f185f5dfe131, action 1, unlock 0, level -1, newlevel 5
(7073,0,make_panic):__ocfs2_cluster_lock:1424 lock N000000000003f01a, convert from -1 to 3
(6930,1,dlm_astd):ocfs2_locking_ast:1104 lock M000000000000000004f18500000000, action 1, unlock 0, level -1, newlevel 5
(6930,1,dlm_astd):ocfs2_locking_ast:1104 lock N000000000003f01a, action 1, unlock 0, level -1, newlevel 3
(6930,1,dlm_astd):ocfs2_locking_ast:1104 lock O000000000000000004f18500000000, action 1, unlock 0, level -1, newlevel 3
(7073,0,make_panic):__ocfs2_cluster_lock:1424 lock M000000000000000003f02600000000, convert from 0 to 3
(6930,1,dlm_astd):ocfs2_locking_ast:1104 lock M000000000000000003f02600000000, action 2, unlock 0, level 0, newlevel 3
(7073,0,make_panic):__ocfs2_cluster_lock:1424 lock M000000000000000003f00700000000, convert from 0 to 3
(6930,1,dlm_astd):ocfs2_locking_ast:1104 lock M000000000000000003f00700000000, action 2, unlock 0, level 0, newlevel 3
(7073,0,make_panic):__ocfs2_cluster_lock:1424 lock M000000000000000003f01200000000, convert from 0 to 3
(6930,1,dlm_astd):ocfs2_locking_ast:1104 lock M000000000000000003f01200000000, action 2, unlock 0, level 0, newlevel 3
(7073,0,make_panic):__ocfs2_cluster_lock:1424 lock M000000000000000003f01100000000, convert from 0 to 3
(6930,1,dlm_astd):ocfs2_locking_ast:1104 lock M000000000000000003f01100000000, action 2, unlock 0, level 0, newlevel 3
(7073,0,make_panic):__ocfs2_cluster_lock:1424 lock M000000000000000003f02500000000, convert from 0 to 3
(6930,1,dlm_astd):ocfs2_locking_ast:1104 lock M000000000000000003f02500000000, action 2, unlock 0, level 0, newlevel 3
(7073,0,make_panic):__ocfs2_cluster_lock:1424 lock M000000000000000003f01c00000000, convert from 3 to 5
(6930,1,dlm_astd):ocfs2_locking_ast:1104 lock M000000000000000003f01c00000000, action 2, unlock 0, level 3, newlevel 5
(6930,1,dlm_astd):ocfs2_locking_ast:1104 lock W000000000000000004f186f5dfe132, action 1, unlock 0, level -1, newlevel 5
(7073,0,make_panic):__ocfs2_cluster_lock:1424 lock N000000000003f01c, convert from -1 to 3
(6930,1,dlm_astd):ocfs2_locking_ast:1104 lock M000000000000000004f18600000000, action 1, unlock 0, level -1, newlevel 5
(6930,1,dlm_astd):ocfs2_locking_ast:1104 lock O000000000000000004f18600000000, action 1, unlock 0, level -1, newlevel 3
(6930,1,dlm_astd):ocfs2_locking_ast:1104 lock N000000000003f01c, action 1, unlock 0, level -1, newlevel 3
(7073,0,make_panic):__ocfs2_cluster_lock:1424 lock M000000000000000003f01300000000, convert from 3 to 5
(6930,1,dlm_astd):ocfs2_locking_ast:1104 lock M000000000000000003f01300000000, action 2, unlock 0, level 3, newlevel 5
(7073,0,make_panic):__ocfs2_cluster_lock:1424 lock N000000000003f013, convert from -1 to 3
(6930,1,dlm_astd):ocfs2_locking_ast:1104 lock W000000000000000004f187f5dfe133, action 1, unlock 0, level -1, newlevel 5
(6930,1,dlm_astd):ocfs2_locking_ast:1104 lock M000000000000000004f18700000000, action 1, unlock 0, level -1, newlevel 5
(6930,1,dlm_astd):ocfs2_locking_ast:1104 lock O000000000000000004f18700000000, action 1, unlock 0, level -1, newlevel 3
(6930,1,dlm_astd):ocfs2_locking_ast:1104 lock N000000000003f013, action 1, unlock 0, level -1, newlevel 3
(7073,0,make_panic):__ocfs2_cluster_lock:1424 lock M000000000000000003f02300000000, convert from 3 to 5


from another window doing a tail -f /var/log/messages:
(can go back further if needed)

Feb  1 15:12:49 bull-02 kernel: (6930,2,dlm_astd):ocfs2_locking_ast:1104 lock M000000000000000004f17f00000000, action 1, unlock 0, level -1, newlevel 5
Feb  1 15:12:49 bull-02 kernel: (7073,0,make_panic):__ocfs2_cluster_lock:1424 lock N000000000003f018, convert from -1 to 3
Feb  1 15:12:49 bull-02 kernel: (6930,2,dlm_astd):ocfs2_locking_ast:1104 lock N000000000003f018, action 1, unlock 0, level -1, newlevel 3
Feb  1 15:12:49 bull-02 kernel: (7073,0,make_panic):__ocfs2_cluster_lock:1424 lock M000000000000000003f01000000000, convert from 0 to 3
Feb  1 15:12:49 bull-02 kernel: (6930,2,dlm_astd):ocfs2_blocking_ast:1059 BAST fired for lockres M000000000000000003f00400000000, blocking 5, level 3 type Meta
Feb  1 15:12:49 bull-02 kernel: (6930,2,dlm_astd):ocfs2_generic_handle_bast:932 lockres M000000000000000003f00400000000, block 5, level 3, l_block 5, dwn 1
Feb  1 15:12:49 bull-02 kernel: (6930,2,dlm_astd):ocfs2_blocking_ast:1059 BAST fired for lockres M000000000000000003f02000000000, blocking 5, level 3 type Meta
Feb  1 15:12:49 bull-02 kernel: (6930,2,dlm_astd):ocfs2_generic_handle_bast:932 lockres M000000000000000003f02000000000, block 5, level 3, l_block 5, dwn 1
Feb  1 15:12:49 bull-02 kernel: (6957,0,ocfs2dc):ocfs2_process_blocked_lock:3996 lockres M000000000000000003f00400000000 blocked.
Feb  1 15:12:49 bull-02 kernel: (6957,0,ocfs2dc):ocfs2_prepare_downconvert:3293 lock M000000000000000003f00400000000, new_level = 0, l_blocking = 5
Feb  1 15:12:49 bull-02 kernel: (6957,0,ocfs2dc):ocfs2_downconvert_lock:3313 lock M000000000000000003f00400000000, level 3 => 0
Feb  1 15:12:49 bull-02 kernel: (6957,0,ocfs2dc):ocfs2_process_blocked_lock:4020 lockres M000000000000000003f00400000000, requeue = no.
Feb  1 15:12:49 bull-02 kernel: (6957,0,ocfs2dc):ocfs2_process_blocked_lock:3996 lockres M000000000000000003f02000000000 blocked.
Feb  1 15:12:49 bull-02 kernel: (6957,0,ocfs2dc):ocfs2_prepare_downconvert:3293 lock M000000000000000003f02000000000, new_level = 0, l_blocking = 5
Feb  1 15:12:49 bull-02 kernel: (6930,2,dlm_astd):ocfs2_locking_ast:1104 lock M000000000000000003f00400000000, action 3, unlock 0, level 3, newlevel 0
Feb  1 15:12:49 bull-02 kernel: (6957,0,ocfs2dc):ocfs2_downconvert_lock:3313 lock M000000000000000003f02000000000, level 3 => 0
Feb  1 15:12:49 bull-02 kernel: (6957,0,ocfs2dc):ocfs2_process_blocked_lock:4020 lockres M000000000000000003f02000000000, requeue = no.
Feb  1 15:12:49 bull-02 kernel: (7014,0,alternate):__ocfs2_cluster_lock:1424 lock M000000000000000003f00400000000, convert from 0 to 3
Feb  1 15:12:49 bull-02 kernel: (6930,2,dlm_astd):ocfs2_locking_ast:1104 lock M000000000000000003f02000000000, action 3, unlock 0, level 3, newlevel 0
Feb  1 15:12:49 bull-02 kernel: (6930,2,dlm_astd):ocfs2_locking_ast:1104 lock O000000000000000004f17f00000000, action 1, unlock 0, level -1, newlevel 3
Feb  1 15:12:49 bull-02 kernel: (6930,2,dlm_astd):ocfs2_locking_ast:1104 lock M000000000000000003f01000000000, action 2, unlock 0, level 0, newlevel 3
Feb  1 15:12:49 bull-02 kernel: (7073,0,make_panic):__ocfs2_cluster_lock:1424 lock M000000000000000003f01600000000, convert from 0 to 3
Feb  1 15:12:49 bull-02 kernel: (6930,2,dlm_astd):ocfs2_locking_ast:1104 lock M000000000000000003f01600000000, action 2, unlock 0, level 0, newlevel 3
Feb  1 15:12:49 bull-02 kernel: (7073,0,make_panic):__ocfs2_cluster_lock:1424 lock M000000000000000003f01600000000, convert from 3 to 5
Feb  1 15:12:49 bull-02 kernel: (6930,2,dlm_astd):ocfs2_blocking_ast:1059 BAST fired for lockres M000000000000000006e59d00000000, blocking 3, level 5 type Meta
Feb  1 15:12:49 bull-02 kernel: (6930,2,dlm_astd):ocfs2_generic_handle_bast:932 lockres M000000000000000006e59d00000000, block 3, level 5, l_block 3, dwn 1
Feb  1 15:12:49 bull-02 kernel: (6930,2,dlm_astd):ocfs2_locking_ast:1104 lock M000000000000000003f00400000000, action 2, unlock 0, level 0, newlevel 3
Feb  1 15:12:49 bull-02 kernel: (6930,2,dlm_astd):ocfs2_blocking_ast:1059 BAST fired for lockres M000000000000000003f00400000000, blocking 5, level 3 type Meta
Feb  1 15:12:49 bull-02 kernel: (6957,0,ocfs2dc):ocfs2_process_blocked_lock:3996 lockres M000000000000000006e59d00000000 blocked.
Feb  1 15:12:49 bull-02 kernel: (6930,2,dlm_astd):ocfs2_generic_handle_bast:932 lockres M000000000000000003f00400000000, block 5, level 3, l_block 5, dwn 1
Feb  1 15:12:49 bull-02 kernel: (6957,0,ocfs2dc):ocfs2_prepare_downconvert:3293 lock M000000000000000006e59d00000000, new_level = 3, l_blocking = 3
Feb  1 15:12:49 bull-02 kernel: (6957,0,ocfs2dc):ocfs2_downconvert_lock:3313 lock M000000000000000006e59d00000000, level 5 => 3
Feb  1 15:12:49 bull-02 kernel: (6930,2,dlm_astd):ocfs2_locking_ast:1104 lock M000000000000000003f01600000000, action 2, unlock 0, level 3, newlevel 5
Feb  1 15:12:49 bull-02 kernel: (6957,0,ocfs2dc):ocfs2_process_blocked_lock:4020 lockres M000000000000000006e59d00000000, requeue = no.
Feb  1 15:12:49 bull-02 kernel: (6930,2,dlm_astd):ocfs2_locking_ast:1104 lock M000000000000000006e59d00000000, action 3, unlock 0, level 5, newlevel 3
Feb  1 15:12:49 bull-02 kernel: (6957,0,ocfs2dc):ocfs2_process_blocked_lock:3996 lockres M000000000000000003f00400000000 blocked.
Feb  1 15:12:49 bull-02 kernel: (6957,0,ocfs2dc):ocfs2_process_blocked_lock:4020 lockres M000000000000000003f00400000000, requeue = yes.
Feb  1 15:12:49 bull-02 kernel: (6957,0,ocfs2dc):ocfs2_process_blocked_lock:3996 lockres M000000000000000003f00400000000 blocked.
Feb  1 15:12:49 bull-02 kernel: (6957,0,ocfs2dc):ocfs2_prepare_downconvert:3293 lock M000000000000000003f00400000000, new_level = 0, l_blocking = 5
Feb  1 15:12:49 bull-02 kernel: (6957,0,ocfs2dc):ocfs2_downconvert_lock:3313 lock M000000000000000003f00400000000, level 3 => 0
Feb  1 15:12:49 bull-02 kernel: (6957,0,ocfs2dc):ocfs2_process_blocked_lock:4020 lockres M000000000000000003f00400000000, requeue = no.
Feb  1 15:12:49 bull-02 kernel: (6930,2,dlm_astd):ocfs2_locking_ast:1104 lock M000000000000000003f00400000000, action 3, unlock 0, level 3, newlevel 0
Feb  1 15:12:49 bull-02 kernel: (7014,2,alternate):__ocfs2_cluster_lock:1424 lock M000000000000000003f00400000000, convert from 0 to 5
Feb  1 15:12:49 bull-02 kernel: (6930,2,dlm_astd):ocfs2_locking_ast:1104 lock O000000000000000004f18000000000, action 1, unlock 0, level -1, newlevel 3
Feb  1 15:12:49 bull-02 kernel: (7073,0,make_panic):__ocfs2_cluster_lock:1424 lock N000000000003f016, convert from -1 to 3
Feb  1 15:12:49 bull-02 kernel: (6930,2,dlm_astd):ocfs2_blocking_ast:1059 BAST fired for lockres M000000000000000006e59d00000000, blocking 5, level 3 type Meta
Feb  1 15:12:49 bull-02 kernel: (6930,2,dlm_astd):ocfs2_generic_handle_bast:932 lockres M000000000000000006e59d00000000, block 5, level 3, l_block 5, dwn 1
Feb  1 15:12:49 bull-02 kernel: (6957,0,ocfs2dc):ocfs2_process_blocked_lock:3996 lockres M000000000000000006e59d00000000 blocked.
Feb  1 15:12:49 bull-02 kernel: (6957,0,ocfs2dc):ocfs2_prepare_downconvert:3293 lock M000000000000000006e59d00000000, new_level = 0, l_blocking = 5
Feb  1 15:12:49 bull-02 kernel: (6957,0,ocfs2dc):ocfs2_downconvert_lock:3313 lock M000000000000000006e59d00000000, level 3 => 0
Feb  1 15:12:49 bull-02 kernel: (6957,0,ocfs2dc):ocfs2_process_blocked_lock:4020 lockres M000000000000000006e59d00000000, requeue = no.
Feb  1 15:12:49 bull-02 kernel: (6930,2,dlm_astd):ocfs2_locking_ast:1104 lock M000000000000000006e59d00000000, action 3, unlock 0, level 3, newlevel 0
Feb  1 15:12:49 bull-02 kernel: (6930,2,dlm_astd):ocfs2_locking_ast:1104 lock W000000000000000004f180f5dfe12c, action 1, unlock 0, level -1, newlevel 5
Feb  1 15:12:49 bull-02 kernel: (6930,2,dlm_astd):ocfs2_locking_ast:1104 lock M000000000000000004f18000000000, action 1, unlock 0, level -1, newlevel 5
Feb  1 15:12:50 bull-02 kernel: (6930,2,dlm_astd):ocfs2_locking_ast:1104 lock M000000000000000003f00400000000, action 2, unlock 0, level 0, newlevel 5
Feb  1 15:12:50 bull-02 kernel: (7014,2,alternate):__ocfs2_cluster_lock:1424 lock W000000000000000003f00435e4a660, convert from 0 to 5
Feb  1 15:12:50 bull-02 kernel: (6930,2,dlm_astd):ocfs2_blocking_ast:1059 BAST fired for lockres M000000000000000003f00400000000, blocking 3, level 5 type Meta
Feb  1 15:12:50 bull-02 kernel: (6930,2,dlm_astd):ocfs2_generic_handle_bast:932 lockres M000000000000000003f00400000000, block 3, level 5, l_block 3, dwn 1
Feb  1 15:12:50 bull-02 kernel: (6930,2,dlm_astd):ocfs2_blocking_ast:1059 BAST fired for lockres M000000000000000003f01900000000, blocking 5, level 3 type Meta
Feb  1 15:12:50 bull-02 kernel: (6930,2,dlm_astd):ocfs2_generic_handle_bast:932 lockres M000000000000000003f01900000000, block 5, level 3, l_block 5, dwn 1
Feb  1 15:12:50 bull-02 kernel: (6957,0,ocfs2dc):ocfs2_process_blocked_lock:3996 lockres M000000000000000003f00400000000 blocked.
Feb  1 15:12:50 bull-02 kernel: (6930,2,dlm_astd):ocfs2_blocking_ast:1059 BAST fired for lockres M000000000000000003f00400000000, blocking 5, level 5 type Meta
Feb  1 15:12:50 bull-02 kernel: (6930,2,dlm_astd):ocfs2_generic_handle_bast:932 lockres M000000000000000003f00400000000, block 5, level 5, l_block 5, dwn 1
Feb  1 15:12:50 bull-02 kernel: (6957,0,ocfs2dc):ocfs2_prepare_downconvert:3293 lock M000000000000000003f00400000000, new_level = 0, l_blocking = 5
Feb  1 15:12:50 bull-02 kernel: (6957,0,ocfs2dc):ocfs2_downconvert_lock:3313 lock M000000000000000003f00400000000, level 5 => 0
Feb  1 15:12:50 bull-02 kernel: (6957,0,ocfs2dc):ocfs2_process_blocked_lock:4020 lockres M000000000000000003f00400000000, requeue = no.
Feb  1 15:12:50 bull-02 kernel: (6957,0,ocfs2dc):ocfs2_process_blocked_lock:3996 lockres M000000000000000003f01900000000 blocked.
Feb  1 15:12:50 bull-02 kernel: (6957,0,ocfs2dc):ocfs2_prepare_downconvert:3293 lock M000000000000000003f01900000000, new_level = 0, l_blocking = 5
Feb  1 15:12:50 bull-02 kernel: (6957,0,ocfs2dc):ocfs2_downconvert_lock:3313 lock M000000000000000003f01900000000, level 3 => 0
Feb  1 15:12:50 bull-02 kernel: (6930,2,dlm_astd):ocfs2_locking_ast:1104 lock M000000000000000003f00400000000, action 3, unlock 0, level 5, newlevel 0
Feb  1 15:12:50 bull-02 kernel: (6957,0,ocfs2dc):ocfs2_process_blocked_lock:4020 lockres M000000000000000003f01900000000, requeue = no.
Feb  1 15:12:50 bull-02 kernel: (6930,2,dlm_astd):ocfs2_locking_ast:1104 lock M000000000000000003f01900000000, action 3, unlock 0, level 3, newlevel 0
Feb  1 15:12:50 bull-02 kernel: (6957,0,ocfs2dc):ocfs2_process_blocked_lock:3996 lockres M000000000000000003f00400000000 blocked.
Feb  1 15:12:50 bull-02 kernel: (6957,0,ocfs2dc):ocfs2_unblock_lock:3414 ERROR: lock M000000000000000003f00400000000, lvl 0 , blcklst 1, mask 1, type 0, flags 0x41, hold 0 0, act 0 0, req 0, block 0, pgen 24429
Feb  1 15:12:50 bull-02 kernel: ------------[ cut here ]------------
Message from syslogd at bull-02 at Feb  1 15:12:50 ...
 kernel:------------[ cut here ]------------
Feb  1 15:12:50 bull-02 kernel: (6930,2,dlm_astd):ocfs2_locking_ast:1104 lock W000000000000000003f00435e4a660, action 2, unlock 0, level 0, newlevel 5
Feb  1 15:12:50 bull-02 kernel: (6930,1,dlm_astd):ocfs2_blocking_ast:1059 BAST fired for lockres M000000000000000004eefc00000000, blocking 3, level 5 type Meta
Feb  1 15:12:50 bull-02 kernel: (6930,1,dlm_astd):ocfs2_generic_handle_bast:932 lockres M000000000000000004eefc00000000, block 3, level 5, l_block 3, dwn 1
Feb  1 15:12:50 bull-02 kernel: (6930,1,dlm_astd):ocfs2_blocking_ast:1059 BAST fired for lockres M000000000000000000000b6354ea08, blocking 5, level 5 type Meta
Feb  1 15:12:50 bull-02 kernel: (6930,1,dlm_astd):ocfs2_generic_handle_bast:932 lockres M000000000000000000000b6354ea08, block 5, level 5, l_block 5, dwn 1
Feb  1 15:12:50 bull-02 kernel: (6930,1,dlm_astd):ocfs2_locking_ast:1104 lock N000000000003f016, action 1, unlock 0, level -1, newlevel 3
Feb  1 15:12:50 bull-02 kernel: (7073,1,make_panic):__ocfs2_cluster_lock:1424 lock M000000000000000003f01500000000, convert from 0 to 3
Feb  1 15:12:50 bull-02 kernel: (6930,1,dlm_astd):ocfs2_locking_ast:1104 lock M000000000000000003f01500000000, action 2, unlock 0, level 0, newlevel 3
Feb  1 15:12:50 bull-02 kernel: (6930,1,dlm_astd):ocfs2_blocking_ast:1059 BAST fired for lockres M000000000000000003f01800000000, blocking 3, level 5 type Meta
Feb  1 15:12:50 bull-02 kernel: (6930,1,dlm_astd):ocfs2_generic_handle_bast:932 lockres M000000000000000003f01800000000, block 3, level 5, l_block 3, dwn 1
Feb  1 15:12:50 bull-02 kernel: kernel BUG at fs/ocfs2/dlmglue.c:3417!
Feb  1 15:12:50 bull-02 kernel: invalid opcode: 0000 [#1] SMP
Feb  1 15:12:50 bull-02 kernel: last sysfs file: /sys/devices/pci0000:80/0000:80:02.0/0000:86:01.0/local_cpus
Feb  1 15:12:50 bull-02 kernel: CPU 0
Feb  1 15:12:50 bull-02 kernel: Modules linked in: ocfs2_stack_user dlm ocfs2 ocfs2_nodemanager configfs ocfs2_stackglue sunrpc ipv6 cpufreq_ondemand powernow_k8 freq_table dm_multipath amd64_edac_mod edac_core tg3 shpchp serio_raw i2c_nforce2 i2c_core k8temp qla2xxx ata_generic mptspi scsi_transport_fc pata_acpi mptscsih mptbase scsi_transport_spi scsi_tgt pata_amd sata_nv [last unloaded: scsi_wait_scan]
Feb  1 15:12:50 bull-02 kernel: Pid: 6957, comm: ocfs2dc Not tainted 2.6.32.3 #2 ProLiant DL145 G2
Feb  1 15:12:50 bull-02 kernel: RIP: 0010:[<ffffffffa020de29>]  [<ffffffffa020de29>] ocfs2_downconvert_thread+0x4cb/0xdad [ocfs2]
Feb  1 15:12:50 bull-02 kernel: RSP: 0018:ffff88007abb7d90  EFLAGS: 00010046
Feb  1 15:12:50 bull-02 kernel: RAX: 00000000000000b9 RBX: ffff88007b99bd50 RCX: 0000000000008506
Feb  1 15:12:50 bull-02 kernel: RDX: 0000000000000000 RSI: 0000000000000046 RDI: 0000000000000046
Feb  1 15:12:50 bull-02 kernel: RBP: ffff88007abb7ee0 R08: ffff88007abb7c80 R09: 0000000000000000
Feb  1 15:12:50 bull-02 kernel: R10: 0000000000000004 R11: 000000107abb7900 R12: 0000000000000282
Feb  1 15:12:50 bull-02 kernel: R13: 0000000000000000 R14: ffff88013a000000 R15: ffff88007b99bd68
Feb  1 15:12:50 bull-02 kernel: FS:  00007f11c9de2710(0000) GS:ffff880028200000(0000) knlGS:0000000000000000
Feb  1 15:12:50 bull-02 kernel: CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
Feb  1 15:12:50 bull-02 kernel: CR2: 00007f7c6df2f000 CR3: 000000013d11c000 CR4: 00000000000006f0
Feb  1 15:12:50 bull-02 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Feb  1 15:12:50 bull-02 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Feb  1 15:12:50 bull-02 kernel: Process ocfs2dc (pid: 6957, threadinfo ffff88007abb6000, task ffff88007d3cc5c0)
Feb  1 15:12:50 bull-02 kernel: Stack:
Feb  1 15:12:50 bull-02 kernel: ffff88007b99bd98 ffffffff00000000 ffff880000000001 ffffffff00000001
Feb  1 15:12:50 bull-02 kernel: <0> 0000000000000000 0000000000000041 0000000000000000 ffff880000000000
Feb  1 15:12:50 bull-02 kernel: <0> ffffffff00000000 ffff880100000000 ffff880000000000 ffffffff00000000
Feb  1 15:12:50 bull-02 kernel: Call Trace:
Feb  1 15:12:50 bull-02 kernel: [<ffffffff81074f6b>] ? autoremove_wake_function+0x0/0x39
Feb  1 15:12:50 bull-02 kernel: [<ffffffffa020d95e>] ? ocfs2_downconvert_thread+0x0/0xdad [ocfs2]
Feb  1 15:12:50 bull-02 kernel: [<ffffffff81074c7e>] kthread+0x7f/0x87
Feb  1 15:12:50 bull-02 kernel: [<ffffffff81012cea>] child_rip+0xa/0x20
Feb  1 15:12:50 bull-02 kernel: [<ffffffff81074bff>] ? kthread+0x0/0x87
Feb  1 15:12:50 bull-02 kernel: [<ffffffff81012ce0>] ? child_rip+0x0/0x20
Feb  1 15:12:50 bull-02 kernel: Code: 24 10 8b 43 68 89 44 24 08 48 8d 43 48 48 89 04 24 31 c0 e8 dd 2e 22 e1 f6 43 40 04 74 0d 4c 8d 63 48 c7 45 8c 00 00 00 00 eb 04 <0f> 0b eb fe 48 8b 4b 40 f6 c1 02 0f 84 2d 01 00 00 80 e5 04 74
Feb  1 15:12:50 bull-02 kernel: RIP  [<ffffffffa020de29>] ocfs2_downconvert_thread+0x4cb/0xdad [ocfs2]
Feb  1 15:12:50 bull-02 kernel: RSP <ffff88007abb7d90>
Feb  1 15:12:50 bull-02 kernel: ---[ end trace 692271ffad8ccf0d ]---

[Ocfs2-devel] [PATCH] ocfs2: Do not downconvert if the lock level is already compatible

[Sunil Mushran]

Joel Becker wrote:
> On Fri, Jan 29, 2010 at 10:14:09AM -0800, Mark Fasheh wrote:
>> Hmm, looks like it's the month of dlmglue fixes?  :)
>
> 	fs/dlm is far more asynchronous than o2dlm, and boy it shows us
> up ;-)

True. In this case, it is bast-ing even before the ast. ;)

[Ocfs2-devel] [PATCH] ocfs2: Do not downconvert if the lock level is already compatible

[Joel Becker]

On Fri, Jan 29, 2010 at 10:14:09AM -0800, Mark Fasheh wrote:
> Hmm, looks like it's the month of dlmglue fixes?  :)

	fs/dlm is far more asynchronous than o2dlm, and boy it shows us
up ;-)

Joel

-- 

Life's Little Instruction Book #396

	"Never give anyone a fruitcake."

Joel Becker
Principal Software Developer
Oracle
E-mail: joel.becker at oracle.com
Phone: (650) 506-8127

[Ocfs2-devel] [PATCH] ocfs2: Do not downconvert if the lock level is already compatible

[Mark Fasheh]

On Fri, Jan 29, 2010 at 09:44:11AM -0800, Sunil Mushran wrote:
> During upconvert, if the master were to send a BAST, dlmglue will detect the
> upconversion in process and send a cancel convert to the master. Upon receiving
> the AST for the cancel convert, it will re-process the lock resource to determine
> whether it needs downconverting. Say, the up was from PR to EX and the BAST was
> for EX. After the cancel convert, it will need to downconvert to NL.
> 
> However, if the node was originally upconverting from NL to EX, then there would
> be no reason to downconvert (assuming the same message sequence).
> 
> This patch makes dlmglue consider the possibility that the current lock level
> is already compatible and that downconverting is not required.
> 
> Joel Becker <joel.becker@oracle.com> assisted in fixing this issue.
> 
> Fixes ossbz#1178
> http://oss.oracle.com/bugzilla/show_bug.cgi?id=1178
> 
> Reported-by: Coly Li <coly.li@suse.de>
> Signed-off-by: Sunil Mushran <sunil.mushran@oracle.com>

Hmm, looks like it's the month of dlmglue fixes?  :)

Acked-by: Mark Fasheh <mfasheh@suse.com>
	--Mark

--
Mark Fasheh

[Ocfs2-devel] [PATCH] ocfs2: Do not downconvert if the lock level is already compatible

[Sunil Mushran]

During upconvert, if the master were to send a BAST, dlmglue will detect the
upconversion in process and send a cancel convert to the master. Upon receiving
the AST for the cancel convert, it will re-process the lock resource to determine
whether it needs downconverting. Say, the up was from PR to EX and the BAST was
for EX. After the cancel convert, it will need to downconvert to NL.

However, if the node was originally upconverting from NL to EX, then there would
be no reason to downconvert (assuming the same message sequence).

This patch makes dlmglue consider the possibility that the current lock level
is already compatible and that downconverting is not required.

Joel Becker <joel.becker@oracle.com> assisted in fixing this issue.

Fixes ossbz#1178
http://oss.oracle.com/bugzilla/show_bug.cgi?id=1178

Reported-by: Coly Li <coly.li@suse.de>
Signed-off-by: Sunil Mushran <sunil.mushran@oracle.com>
---
 fs/ocfs2/dlmglue.c |   13 +++++++++++++
 1 files changed, 13 insertions(+), 0 deletions(-)

diff --git a/fs/ocfs2/dlmglue.c b/fs/ocfs2/dlmglue.c
index f7b9f8f..2918c2c 100644
--- a/fs/ocfs2/dlmglue.c
+++ b/fs/ocfs2/dlmglue.c
@@ -3445,6 +3445,19 @@ recheck:
 	if (lockres->l_flags & OCFS2_LOCK_UPCONVERT_FINISHING)
 		goto leave_requeue;
 
+	/*
+	 * How can we block and yet be at NL?  We were trying to upconvert
+	 * from NL and got canceled.  The code comes back here, and now
+	 * we notice and clear BLOCKING.
+	 */
+	if (lockres->l_level == DLM_LOCK_NL) {
+		BUG_ON(lockres->l_ex_holders || lockres->l_ro_holders);
+		lockres->l_blocking = DLM_LOCK_NL;
+		lockres_clear_flags(lockres, OCFS2_LOCK_BLOCKED);
+		spin_unlock_irqrestore(&lockres->l_lock, flags);
+		goto leave;
+	}
+
 	/* if we're blocking an exclusive and we have *any* holders,
 	 * then requeue. */
 	if ((lockres->l_blocking == DLM_LOCK_EX)
-- 
1.6.3.3