From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mx2.suse.de ([195.135.220.15]:35252 "EHLO mx1.suse.de" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1725855AbeJIO4C (ORCPT ); Tue, 9 Oct 2018 10:56:02 -0400 Date: Tue, 9 Oct 2018 09:40:23 +0200 From: Jan Kara To: Paul Moore Cc: jack@suse.cz, viro@zeniv.linux.org.uk, linux-audit@redhat.com, linux-fsdevel@vger.kernel.org, rgb@redhat.com, amir73il@gmail.com Subject: Re: [PATCH 12/11 TESTSUITE] audit_testsuite: Add stress test for tree watches Message-ID: <20181009074023.GD11150@quack2.suse.cz> References: <20180904160632.21210-1-jack@suse.cz> <20180904160632.21210-13-jack@suse.cz> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Sender: linux-fsdevel-owner@vger.kernel.org List-ID: On Fri 05-10-18 17:06:22, Paul Moore wrote: > On Tue, Sep 4, 2018 at 12:06 PM Jan Kara wrote: > > Add stress test for stressing audit tree watches by adding and deleting > > rules while events are generated and watched filesystems are mounted and > > unmounted in parallel. > > > > Signed-off-by: Jan Kara > > --- > > tests/stress_tree/Makefile | 8 +++ > > tests/stress_tree/test | 171 +++++++++++++++++++++++++++++++++++++++++++++ > > 2 files changed, 179 insertions(+) > > create mode 100644 tests/stress_tree/Makefile > > create mode 100755 tests/stress_tree/test > > No commentary on the test itself, other than perhaps it should live > under test_manual/, but in running the tests in a loop today I am > reliably able to panic my test kernel after ~30m or so. Interesting. How do you run the test? > I'm using the kernel linked below which is Fedora Rawhide + > selinux/next + audit/next + audit/working-fsnotify_fixes; a link to > the patches added to the Rawhide kernel can be found in the list > archive linked below. > > * https://groups.google.com/forum/#!topic/kernel-secnext/SFv0d-ij3z8 > * https://copr.fedorainfracloud.org/coprs/pcmoore/kernel-secnext/build/805949 > > The initial panic dump is below: I can try to reproduce this but could you perhaps find out which of the list manipulations in audit_remove_tree_rule() hit the corrution? Honza > > [ 139.619065] list_del corruption. prev->next should be > ffff985fa98d4100, but was ffff985fae91e370 > [ 139.622504] ------------[ cut here ]------------ > [ 139.623402] kernel BUG at lib/list_debug.c:53! > [ 139.624294] invalid opcode: 0000 [#1] SMP PTI > [ 139.625439] CPU: 1 PID: 3248 Comm: auditctl Not tainted > 4.19.0-0.rc6.git2.1.1.secnext.fc30.x86_64 #1 > [ 139.630761] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 > [ 139.631812] RIP: 0010:__list_del_entry_valid.cold.1+0x34/0x4c > [ 139.632853] Code: ce 34 ac e8 b8 f7 c0 ff 0f 0b 48 c7 c7 a8 cf 34 > ac e8 aa f7 c0 ff 0f 0b 48 89 f2 48 89 fe 48 c7 c7 68 cf 34 ac e8 96 > f7 c0 ff <0f> 0b 48 89 fe 48 c7 c7 30 cf 34 ac e8 85 f7 c0 ff 0f 0b 90 > 90 90 > [ 139.636347] RSP: 0018:ffffb4890293fbb8 EFLAGS: 00010246 > [ 139.637295] RAX: 0000000000000054 RBX: ffff985fae91e620 RCX: 0000000000000000 > [ 139.638573] RDX: 0000000000000000 RSI: ffff985fb77d6be8 RDI: ffff985fb77d6be8 > [ 139.639855] RBP: ffff985fa98d40c0 R08: 00046c1ac1fe8d00 R09: 0000000000000000 > [ 139.641136] R10: 0000000000000000 R11: 0000000000000000 R12: ffff985fa98d4100 > [ 139.642416] R13: 0000000000000000 R14: ffff985faf00fe20 R15: 00000000000003f4 > [ 139.643699] FS: 00007f9898252b80(0000) GS:ffff985fb7600000(0000) > knlGS:0000000000000000 > [ 139.645199] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 139.646244] CR2: 0000561d333b8218 CR3: 000000023110a000 CR4: 00000000001406e0 > [ 139.647533] Call Trace: > [ 139.647997] audit_remove_tree_rule+0xad/0x160 > [ 139.648819] audit_del_rule+0x90/0x190 > [ 139.649507] audit_rule_change+0x98e/0xbf0 > [ 139.650259] audit_receive_msg+0x142/0x1160 > [ 139.651030] ? netlink_deliver_tap+0x99/0x410 > [ 139.651832] audit_receive+0x54/0xb0 > [ 139.652495] netlink_unicast+0x181/0x210 > [ 139.653211] netlink_sendmsg+0x218/0x3e0 > [ 139.653936] sock_sendmsg+0x36/0x40 > [ 139.654583] __sys_sendto+0xf1/0x160 > [ 139.655244] ? syscall_trace_enter+0x1d3/0x330 > [ 139.656055] ? trace_hardirqs_off_thunk+0x1a/0x1c > [ 139.656912] __x64_sys_sendto+0x24/0x30 > [ 139.657615] do_syscall_64+0x60/0x1f0 > [ 139.658285] entry_SYSCALL_64_after_hwframe+0x49/0xbe > [ 139.659192] RIP: 0033:0x7f989835a7fb > [ 139.659847] Code: 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 > 0f 1e fa 48 8d 05 25 6f 0c 00 41 89 ca 8b 00 85 c0 75 14 b8 2c 00 00 > 00 0f 05 <48> 3d 00 f0 ff ff 77 75 c3 0f 1f 40 00 41 57 4d 89 c7 41 56 > 41 89 > [ 139.663170] RSP: 002b:00007ffc95ae6b48 EFLAGS: 00000246 ORIG_RAX: > 000000000000002c > [ 139.664531] RAX: ffffffffffffffda RBX: 0000000000000460 RCX: 00007f989835a7fb > [ 139.665809] RDX: 0000000000000460 RSI: 00007ffc95ae6b80 RDI: 0000000000000003 > [ 139.667087] RBP: 0000000000000003 R08: 00007ffc95ae6b6c R09: 000000000000000c > [ 139.668370] R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffc95ae6b80 > [ 139.669648] R13: 00007ffc95ae6b6c R14: 0000000000000002 R15: 000000000000044f > [ 139.670935] Modules linked in: ib_isert iscsi_target_mod ib_srpt > target_core_mod ib_srp scsi_transport_srp rpcrdma ib_umad rdma_ucm > ib_iser ib_ipoib rdma_cm iw_cm libiscsi scsi_transport_iscsi ib_cm > mlx5_ib ib_uverbs ib_core crct10dif_pclmul crc32_pclmul > ghash_clmulni_intel joydev virtio_balloon i2c_piix4 sunrpc > drm_kms_helper ttm crc32c_intel mlx5_core drm virtio_console mlxfw > serio_raw devlink virtio_blk virtio_net net_failover failover > ata_generic pata_acpi qemu_fw_cfg > [ 139.678676] ---[ end trace be98f2acb1e536e4 ]--- -- Jan Kara SUSE Labs, CR From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jan Kara Subject: Re: [PATCH 12/11 TESTSUITE] audit_testsuite: Add stress test for tree watches Date: Tue, 9 Oct 2018 09:40:23 +0200 Message-ID: <20181009074023.GD11150@quack2.suse.cz> References: <20180904160632.21210-1-jack@suse.cz> <20180904160632.21210-13-jack@suse.cz> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: Content-Disposition: inline In-Reply-To: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: linux-audit-bounces@redhat.com Errors-To: linux-audit-bounces@redhat.com To: Paul Moore Cc: jack@suse.cz, rgb@redhat.com, amir73il@gmail.com, linux-audit@redhat.com, viro@zeniv.linux.org.uk, linux-fsdevel@vger.kernel.org List-Id: linux-audit@redhat.com On Fri 05-10-18 17:06:22, Paul Moore wrote: > On Tue, Sep 4, 2018 at 12:06 PM Jan Kara wrote: > > Add stress test for stressing audit tree watches by adding and deleting > > rules while events are generated and watched filesystems are mounted and > > unmounted in parallel. > > > > Signed-off-by: Jan Kara > > --- > > tests/stress_tree/Makefile | 8 +++ > > tests/stress_tree/test | 171 +++++++++++++++++++++++++++++++++++++++++++++ > > 2 files changed, 179 insertions(+) > > create mode 100644 tests/stress_tree/Makefile > > create mode 100755 tests/stress_tree/test > > No commentary on the test itself, other than perhaps it should live > under test_manual/, but in running the tests in a loop today I am > reliably able to panic my test kernel after ~30m or so. Interesting. How do you run the test? > I'm using the kernel linked below which is Fedora Rawhide + > selinux/next + audit/next + audit/working-fsnotify_fixes; a link to > the patches added to the Rawhide kernel can be found in the list > archive linked below. > > * https://groups.google.com/forum/#!topic/kernel-secnext/SFv0d-ij3z8 > * https://copr.fedorainfracloud.org/coprs/pcmoore/kernel-secnext/build/805949 > > The initial panic dump is below: I can try to reproduce this but could you perhaps find out which of the list manipulations in audit_remove_tree_rule() hit the corrution? Honza > > [ 139.619065] list_del corruption. prev->next should be > ffff985fa98d4100, but was ffff985fae91e370 > [ 139.622504] ------------[ cut here ]------------ > [ 139.623402] kernel BUG at lib/list_debug.c:53! > [ 139.624294] invalid opcode: 0000 [#1] SMP PTI > [ 139.625439] CPU: 1 PID: 3248 Comm: auditctl Not tainted > 4.19.0-0.rc6.git2.1.1.secnext.fc30.x86_64 #1 > [ 139.630761] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 > [ 139.631812] RIP: 0010:__list_del_entry_valid.cold.1+0x34/0x4c > [ 139.632853] Code: ce 34 ac e8 b8 f7 c0 ff 0f 0b 48 c7 c7 a8 cf 34 > ac e8 aa f7 c0 ff 0f 0b 48 89 f2 48 89 fe 48 c7 c7 68 cf 34 ac e8 96 > f7 c0 ff <0f> 0b 48 89 fe 48 c7 c7 30 cf 34 ac e8 85 f7 c0 ff 0f 0b 90 > 90 90 > [ 139.636347] RSP: 0018:ffffb4890293fbb8 EFLAGS: 00010246 > [ 139.637295] RAX: 0000000000000054 RBX: ffff985fae91e620 RCX: 0000000000000000 > [ 139.638573] RDX: 0000000000000000 RSI: ffff985fb77d6be8 RDI: ffff985fb77d6be8 > [ 139.639855] RBP: ffff985fa98d40c0 R08: 00046c1ac1fe8d00 R09: 0000000000000000 > [ 139.641136] R10: 0000000000000000 R11: 0000000000000000 R12: ffff985fa98d4100 > [ 139.642416] R13: 0000000000000000 R14: ffff985faf00fe20 R15: 00000000000003f4 > [ 139.643699] FS: 00007f9898252b80(0000) GS:ffff985fb7600000(0000) > knlGS:0000000000000000 > [ 139.645199] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 139.646244] CR2: 0000561d333b8218 CR3: 000000023110a000 CR4: 00000000001406e0 > [ 139.647533] Call Trace: > [ 139.647997] audit_remove_tree_rule+0xad/0x160 > [ 139.648819] audit_del_rule+0x90/0x190 > [ 139.649507] audit_rule_change+0x98e/0xbf0 > [ 139.650259] audit_receive_msg+0x142/0x1160 > [ 139.651030] ? netlink_deliver_tap+0x99/0x410 > [ 139.651832] audit_receive+0x54/0xb0 > [ 139.652495] netlink_unicast+0x181/0x210 > [ 139.653211] netlink_sendmsg+0x218/0x3e0 > [ 139.653936] sock_sendmsg+0x36/0x40 > [ 139.654583] __sys_sendto+0xf1/0x160 > [ 139.655244] ? syscall_trace_enter+0x1d3/0x330 > [ 139.656055] ? trace_hardirqs_off_thunk+0x1a/0x1c > [ 139.656912] __x64_sys_sendto+0x24/0x30 > [ 139.657615] do_syscall_64+0x60/0x1f0 > [ 139.658285] entry_SYSCALL_64_after_hwframe+0x49/0xbe > [ 139.659192] RIP: 0033:0x7f989835a7fb > [ 139.659847] Code: 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 > 0f 1e fa 48 8d 05 25 6f 0c 00 41 89 ca 8b 00 85 c0 75 14 b8 2c 00 00 > 00 0f 05 <48> 3d 00 f0 ff ff 77 75 c3 0f 1f 40 00 41 57 4d 89 c7 41 56 > 41 89 > [ 139.663170] RSP: 002b:00007ffc95ae6b48 EFLAGS: 00000246 ORIG_RAX: > 000000000000002c > [ 139.664531] RAX: ffffffffffffffda RBX: 0000000000000460 RCX: 00007f989835a7fb > [ 139.665809] RDX: 0000000000000460 RSI: 00007ffc95ae6b80 RDI: 0000000000000003 > [ 139.667087] RBP: 0000000000000003 R08: 00007ffc95ae6b6c R09: 000000000000000c > [ 139.668370] R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffc95ae6b80 > [ 139.669648] R13: 00007ffc95ae6b6c R14: 0000000000000002 R15: 000000000000044f > [ 139.670935] Modules linked in: ib_isert iscsi_target_mod ib_srpt > target_core_mod ib_srp scsi_transport_srp rpcrdma ib_umad rdma_ucm > ib_iser ib_ipoib rdma_cm iw_cm libiscsi scsi_transport_iscsi ib_cm > mlx5_ib ib_uverbs ib_core crct10dif_pclmul crc32_pclmul > ghash_clmulni_intel joydev virtio_balloon i2c_piix4 sunrpc > drm_kms_helper ttm crc32c_intel mlx5_core drm virtio_console mlxfw > serio_raw devlink virtio_blk virtio_net net_failover failover > ata_generic pata_acpi qemu_fw_cfg > [ 139.678676] ---[ end trace be98f2acb1e536e4 ]--- -- Jan Kara SUSE Labs, CR