* Re: io_submit with slab free object overwritten [not found] <4a56fc9f-27f7-5cb5-feed-a4e33f05a5d1@lca.pw> @ 2019-02-22 21:07 ` Qian Cai 2019-02-22 21:42 ` Eric Sandeen 0 siblings, 1 reply; 5+ messages in thread From: Qian Cai @ 2019-02-22 21:07 UTC (permalink / raw) To: hch Cc: axboe, viro, hare, bcrl, linux-aio, Linux-MM, jthumshirn, linux-fsdevel, Christoph Lameter Reverted the commit 75374d062756 ("fs: add an iopoll method to struct file_operations") fixed the problem. Christoph mentioned that the field can be calculated by the offset (40 bytes). struct kmem_cache { struct kmem_cache_cpu __percpu *cpu_slab; (8 bytes) slab_flags_t flags; (4) unsigned long min_partial; (8) unsigned int size; (4) unsigned int object_size; (4) unsigned int offset; (4) unsigned int cpu_partial; (4) struct kmem_cache_order_objects oo; (4) /* Allocation and freeing of slabs */ struct kmem_cache_order_objects max; So, it looks like "max" was overwritten after freed. # cat /opt/ltp/runtest/syscalls fgetxattr02 fgetxattr02 io_submit01 io_submit01 # /opt/ltp/runltp -f syscalls uname: Linux 5.0.0-rc7-next-20190222+ #11 SMP Fri Feb 22 14:57:10 EST 2019 ppc64le ppc64le ppc64le GNU/Linux /proc/cmdline BOOT_IMAGE=/vmlinuz-5.0.0-rc7-next-20190222+ root=/dev/mapper/rhel_ibm--p8--01--lp5-root ro rd.lvm.lv=rhel_ibm-p8-01-lp5/root rd.lvm.lv=rhel_ibm-p8-01-lp5/swap crashkernel=768M numa_balancing=enable earlyprintk free reports: total used free shared buff/cache available Mem: 24305408 919552 23120832 12032 265024 22976896 Swap: 8388544 0 8388544 cpuinfo: Architecture: ppc64le Byte Order: Little Endian CPU(s): 16 On-line CPU(s) list: 0-15 Thread(s) per core: 8 Core(s) per socket: 1 Socket(s): 2 NUMA node(s): 2 Model: 2.1 (pvr 004b 0201) Model name: POWER8 (architected), altivec supported Hypervisor vendor: pHyp Virtualization type: para L1d cache: 64K L1i cache: 32K L2 cache: 512K L3 cache: 8192K NUMA node0 CPU(s): NUMA node1 CPU(s): 0-15 Running tests....... <<<test_start>>> tag=fgetxattr02 stime=1550865820 cmdline="fgetxattr02" contacts="" analysis=exit <<<test_output>>> tst_test.c:1096: INFO: Timeout per run is 0h 05m 00s fgetxattr02.c:174: PASS: fgetxattr(2) on testfile passed fgetxattr02.c:188: PASS: fgetxattr(2) on testfile got the right value fgetxattr02.c:201: PASS: fgetxattr(2) on testfile passed: SUCCESS fgetxattr02.c:174: PASS: fgetxattr(2) on testdir passed fgetxattr02.c:188: PASS: fgetxattr(2) on testdir got the right value fgetxattr02.c:201: PASS: fgetxattr(2) on testdir passed: SUCCESS fgetxattr02.c:174: PASS: fgetxattr(2) on symlink passed fgetxattr02.c:188: PASS: fgetxattr(2) on symlink got the right value fgetxattr02.c:201: PASS: fgetxattr(2) on symlink passed: SUCCESS fgetxattr02.c:201: PASS: fgetxattr(2) on fifo passed: ENODATA fgetxattr02.c:201: PASS: fgetxattr(2) on chr passed: ENODATA fgetxattr02.c:201: PASS: fgetxattr(2) on blk passed: ENODATA fgetxattr02.c:201: PASS: fgetxattr(2) on sock passed: ENODATA Summary: passed 13 failed 0 skipped 0 warnings 0 <<<execution_status>>> initiation_status="ok" duration=0 termination_type=exited termination_id=0 corefile=no cutime=0 cstime=1 <<<test_end>>> <<<test_start>>> tag=io_submit01 stime=1550865820 cmdline="io_submit01" contacts="" analysis=exit <<<test_output>>> incrementing stop tst_test.c:1096: INFO: Timeout per run is 0h 05m 00s io_submit01.c:125: PASS: io_submit() with invalid ctx failed with EINVAL io_submit01.c:125: PASS: io_submit() with invalid nr failed with EINVAL io_submit01.c:125: PASS: io_submit() with invalid iocbpp pointer failed with EFAULT io_submit01.c:125: PASS: io_submit() with NULL iocb pointers failed with EFAULT io_submit01.c:125: PASS: io_submit() with invalid fd failed with EBADF io_submit01.c:125: PASS: io_submit() with readonly fd for write failed with EBADF io_submit01.c:125: PASS: io_submit() with writeonly fd for read failed with EBADF io_submit01.c:125: PASS: io_submit() with zero buf size failed with SUCCESS io_submit01.c:125: PASS: io_submit() with zero nr failed with SUCCESS Summary: passed 9 failed 0 skipped 0 warnings 0 On 2/22/19 12:40 AM, Qian Cai wrote: > This is only reproducible on linux-next (20190221), as v5.0-rc7 is fine. Running > two LTP tests and then reboot will trigger this on ppc64le (CONFIG_IO_URING=n > and CONFIG_SHUFFLE_PAGE_ALLOCATOR=y). > > # fgetxattr02 > # io_submit01 > # systemctl reboot > > There is a 32-bit (with all ones) overwritten of free slab objects (poisoned). > > [23424.121182] BUG aio_kiocb (Tainted: G B W L ): Poison overwritten > [23424.121189] > ----------------------------------------------------------------------------- > [23424.121189] > [23424.121197] INFO: 0x000000009f1f5145-0x00000000841e301b. First byte 0xff > instead of 0x6b > [23424.121205] INFO: Allocated in io_submit_one+0x9c/0xb20 age=0 cpu=7 pid=12174 > [23424.121212] __slab_alloc+0x34/0x60 > [23424.121217] kmem_cache_alloc+0x504/0x5c0 > [23424.121221] io_submit_one+0x9c/0xb20 > [23424.121224] sys_io_submit+0xe0/0x350 > [23424.121227] system_call+0x5c/0x70 > [23424.121231] INFO: Freed in aio_complete+0x31c/0x410 age=0 cpu=7 pid=12174 > [23424.121234] kmem_cache_free+0x4bc/0x540 > [23424.121237] aio_complete+0x31c/0x410 > [23424.121240] blkdev_bio_end_io+0x238/0x3e0 > [23424.121243] bio_endio.part.3+0x214/0x330 > [23424.121247] brd_make_request+0x2d8/0x314 [brd] > [23424.121250] generic_make_request+0x220/0x510 > [23424.121254] submit_bio+0xc8/0x1f0 > [23424.121256] blkdev_direct_IO+0x36c/0x610 > [23424.121260] generic_file_read_iter+0xbc/0x230 > [23424.121263] blkdev_read_iter+0x50/0x80 > [23424.121266] aio_read+0x138/0x200 > [23424.121269] io_submit_one+0x7c4/0xb20 > [23424.121272] sys_io_submit+0xe0/0x350 > [23424.121275] system_call+0x5c/0x70 > [23424.121278] INFO: Slab 0x00000000841158ec objects=85 used=85 fp=0x > (null) flags=0x13fffc000000200 > [23424.121282] INFO: Object 0x000000007e677ed8 @offset=5504 fp=0x00000000e42bdf6f > [23424.121282] > [23424.121287] Redzone 000000005483b8fc: bb bb bb bb bb bb bb bb bb bb bb bb bb > bb bb bb ................ > [23424.121291] Redzone 00000000b842fe53: bb bb bb bb bb bb bb bb bb bb bb bb bb > bb bb bb ................ > [23424.121295] Redzone 00000000deb0d052: bb bb bb bb bb bb bb bb bb bb bb bb bb > bb bb bb ................ > [23424.121299] Redzone 0000000014045233: bb bb bb bb bb bb bb bb bb bb bb bb bb > bb bb bb ................ > [23424.121302] Redzone 00000000dd5d6c16: bb bb bb bb bb bb bb bb bb bb bb bb bb > bb bb bb ................ > [23424.121306] Redzone 00000000538b5478: bb bb bb bb bb bb bb bb bb bb bb bb bb > bb bb bb ................ > [23424.121310] Redzone 000000001f7fb704: bb bb bb bb bb bb bb bb bb bb bb bb bb > bb bb bb ................ > [23424.121314] Redzone 0000000000e0484d: bb bb bb bb bb bb bb bb bb bb bb bb bb > bb bb bb ................ > [23424.121318] Object 000000007e677ed8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b > 6b 6b 6b kkkkkkkkkkkkkkkk > [23424.121322] Object 00000000e207f30b: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b > 6b 6b 6b kkkkkkkkkkkkkkkk > [23424.121326] Object 00000000a7a45634: 6b 6b 6b 6b 6b 6b 6b 6b ff ff ff ff 6b > 6b 6b 6b kkkkkkkk....kkkk > [23424.121330] Object 00000000c85d951d: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b > 6b 6b 6b kkkkkkkkkkkkkkkk > [23424.121334] Object 000000003104522f: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b > 6b 6b 6b kkkkkkkkkkkkkkkk > [23424.121338] Object 00000000cfcdd820: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b > 6b 6b 6b kkkkkkkkkkkkkkkk > [23424.121342] Object 00000000dded4924: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b > 6b 6b 6b kkkkkkkkkkkkkkkk > [23424.121346] Object 00000000ff6687a4: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b > 6b 6b 6b kkkkkkkkkkkkkkkk > [23424.121350] Object 00000000df3d67f6: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b > 6b 6b 6b kkkkkkkkkkkkkkkk > [23424.121354] Object 00000000ddc188d1: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b > 6b 6b 6b kkkkkkkkkkkkkkkk > [23424.121358] Object 000000002cee751a: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b > 6b 6b 6b kkkkkkkkkkkkkkkk > [23424.121362] Object 00000000a994f007: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b > 6b 6b a5 kkkkkkkkkkkkkkk. > [23424.121366] Redzone 000000009f3d62e2: bb bb bb bb bb bb bb bb > ........ > [23424.121370] Padding 00000000e5ccead8: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a > 5a 5a 5a ZZZZZZZZZZZZZZZZ > [23424.121374] Padding 000000002b0c1778: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a > 5a 5a 5a ZZZZZZZZZZZZZZZZ > [23424.121378] Padding 00000000c67656c7: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a > 5a 5a 5a ZZZZZZZZZZZZZZZZ > [23424.121382] Padding 0000000078348c5a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a > 5a 5a 5a ZZZZZZZZZZZZZZZZ > [23424.121386] Padding 00000000f3297820: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a > 5a 5a 5a ZZZZZZZZZZZZZZZZ > [23424.121390] Padding 00000000e55789f4: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a > 5a 5a 5a ZZZZZZZZZZZZZZZZ > [23424.121394] Padding 00000000d0fbb94c: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a > 5a 5a 5a ZZZZZZZZZZZZZZZZ > [23424.121397] Padding 00000000bcb27a87: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a > 5a 5a 5a ZZZZZZZZZZZZZZZZ > [23424.121743] CPU: 7 PID: 12174 Comm: vgs Tainted: G B W L > 5.0.0-rc7-next-20190221+ #7 > [23424.121758] Call Trace: > [23424.121762] [c0000004ce5bf7b0] [c0000000007deb8c] dump_stack+0xb0/0xf4 > (unreliable) > [23424.121770] [c0000004ce5bf7f0] [c00000000037d310] print_trailer+0x250/0x278 > [23424.121775] [c0000004ce5bf880] [c00000000036d578] > check_bytes_and_report+0x138/0x160 > [23424.121779] [c0000004ce5bf920] [c00000000036fac8] check_object+0x348/0x3e0 > [23424.121784] [c0000004ce5bf990] [c00000000036fd18] > alloc_debug_processing+0x1b8/0x2c0 > [23424.121788] [c0000004ce5bfa30] [c000000000372d14] ___slab_alloc+0xbb4/0xfa0 > [23424.121792] [c0000004ce5bfb60] [c000000000373134] __slab_alloc+0x34/0x60 > [23424.121802] [c0000004ce5bfb90] [c000000000373664] kmem_cache_alloc+0x504/0x5c0 > [23424.121812] [c0000004ce5bfc20] [c000000000476a9c] io_submit_one+0x9c/0xb20 > [23424.121824] [c0000004ce5bfd50] [c000000000477f10] sys_io_submit+0xe0/0x350 > [23424.121832] [c0000004ce5bfe20] [c00000000000b000] system_call+0x5c/0x70 > [23424.121836] FIX aio_kiocb: Restoring 0x000000009f1f5145-0x00000000841e301b=0x6b > [23424.121836] > [23424.121840] FIX aio_kiocb: Marking all objects used > ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: io_submit with slab free object overwritten 2019-02-22 21:07 ` io_submit with slab free object overwritten Qian Cai @ 2019-02-22 21:42 ` Eric Sandeen 2019-02-22 21:48 ` Qian Cai 0 siblings, 1 reply; 5+ messages in thread From: Eric Sandeen @ 2019-02-22 21:42 UTC (permalink / raw) To: Qian Cai, hch Cc: axboe, viro, hare, bcrl, linux-aio, Linux-MM, jthumshirn, linux-fsdevel, Christoph Lameter On 2/22/19 3:07 PM, Qian Cai wrote: > Reverted the commit 75374d062756 ("fs: add an iopoll method to struct > file_operations") fixed the problem. Christoph mentioned that the field can be > calculated by the offset (40 bytes). I'm a little confused, you can't revert just that patch, right, because others in the iopoll series depend on it. Is the above commit really the culprit, or do you mean you backed out the whole series? thanks, -Eric > struct kmem_cache { > struct kmem_cache_cpu __percpu *cpu_slab; (8 bytes) > slab_flags_t flags; (4) > unsigned long min_partial; (8) > unsigned int size; (4) > unsigned int object_size; (4) > unsigned int offset; (4) > unsigned int cpu_partial; (4) > struct kmem_cache_order_objects oo; (4) > > /* Allocation and freeing of slabs */ > struct kmem_cache_order_objects max; > > So, it looks like "max" was overwritten after freed. > > # cat /opt/ltp/runtest/syscalls > fgetxattr02 fgetxattr02 > io_submit01 io_submit01 > > # /opt/ltp/runltp -f syscalls > > uname: > Linux 5.0.0-rc7-next-20190222+ #11 SMP Fri Feb 22 14:57:10 EST 2019 ppc64le > ppc64le ppc64le GNU/Linux > > /proc/cmdline > BOOT_IMAGE=/vmlinuz-5.0.0-rc7-next-20190222+ > root=/dev/mapper/rhel_ibm--p8--01--lp5-root ro rd.lvm.lv=rhel_ibm-p8-01-lp5/root > rd.lvm.lv=rhel_ibm-p8-01-lp5/swap crashkernel=768M numa_balancing=enable earlyprintk > > free reports: > total used free shared buff/cache available > Mem: 24305408 919552 23120832 12032 265024 22976896 > Swap: 8388544 0 8388544 > > cpuinfo: > Architecture: ppc64le > Byte Order: Little Endian > CPU(s): 16 > On-line CPU(s) list: 0-15 > Thread(s) per core: 8 > Core(s) per socket: 1 > Socket(s): 2 > NUMA node(s): 2 > Model: 2.1 (pvr 004b 0201) > Model name: POWER8 (architected), altivec supported > Hypervisor vendor: pHyp > Virtualization type: para > L1d cache: 64K > L1i cache: 32K > L2 cache: 512K > L3 cache: 8192K > NUMA node0 CPU(s): > NUMA node1 CPU(s): 0-15 > > Running tests....... > <<<test_start>>> > tag=fgetxattr02 stime=1550865820 > cmdline="fgetxattr02" > contacts="" > analysis=exit > <<<test_output>>> > tst_test.c:1096: INFO: Timeout per run is 0h 05m 00s > fgetxattr02.c:174: PASS: fgetxattr(2) on testfile passed > fgetxattr02.c:188: PASS: fgetxattr(2) on testfile got the right value > fgetxattr02.c:201: PASS: fgetxattr(2) on testfile passed: SUCCESS > fgetxattr02.c:174: PASS: fgetxattr(2) on testdir passed > fgetxattr02.c:188: PASS: fgetxattr(2) on testdir got the right value > fgetxattr02.c:201: PASS: fgetxattr(2) on testdir passed: SUCCESS > fgetxattr02.c:174: PASS: fgetxattr(2) on symlink passed > fgetxattr02.c:188: PASS: fgetxattr(2) on symlink got the right value > fgetxattr02.c:201: PASS: fgetxattr(2) on symlink passed: SUCCESS > fgetxattr02.c:201: PASS: fgetxattr(2) on fifo passed: ENODATA > fgetxattr02.c:201: PASS: fgetxattr(2) on chr passed: ENODATA > fgetxattr02.c:201: PASS: fgetxattr(2) on blk passed: ENODATA > fgetxattr02.c:201: PASS: fgetxattr(2) on sock passed: ENODATA > > Summary: > passed 13 > failed 0 > skipped 0 > warnings 0 > <<<execution_status>>> > initiation_status="ok" > duration=0 termination_type=exited termination_id=0 corefile=no > cutime=0 cstime=1 > <<<test_end>>> > <<<test_start>>> > tag=io_submit01 stime=1550865820 > cmdline="io_submit01" > contacts="" > analysis=exit > <<<test_output>>> > incrementing stop > tst_test.c:1096: INFO: Timeout per run is 0h 05m 00s > io_submit01.c:125: PASS: io_submit() with invalid ctx failed with EINVAL > io_submit01.c:125: PASS: io_submit() with invalid nr failed with EINVAL > io_submit01.c:125: PASS: io_submit() with invalid iocbpp pointer failed with EFAULT > io_submit01.c:125: PASS: io_submit() with NULL iocb pointers failed with EFAULT > io_submit01.c:125: PASS: io_submit() with invalid fd failed with EBADF > io_submit01.c:125: PASS: io_submit() with readonly fd for write failed with EBADF > io_submit01.c:125: PASS: io_submit() with writeonly fd for read failed with EBADF > io_submit01.c:125: PASS: io_submit() with zero buf size failed with SUCCESS > io_submit01.c:125: PASS: io_submit() with zero nr failed with SUCCESS > > Summary: > passed 9 > failed 0 > skipped 0 > warnings 0 > > On 2/22/19 12:40 AM, Qian Cai wrote: >> This is only reproducible on linux-next (20190221), as v5.0-rc7 is fine. Running >> two LTP tests and then reboot will trigger this on ppc64le (CONFIG_IO_URING=n >> and CONFIG_SHUFFLE_PAGE_ALLOCATOR=y). >> >> # fgetxattr02 >> # io_submit01 >> # systemctl reboot >> >> There is a 32-bit (with all ones) overwritten of free slab objects (poisoned). >> >> [23424.121182] BUG aio_kiocb (Tainted: G B W L ): Poison overwritten >> [23424.121189] >> ----------------------------------------------------------------------------- >> [23424.121189] >> [23424.121197] INFO: 0x000000009f1f5145-0x00000000841e301b. First byte 0xff >> instead of 0x6b >> [23424.121205] INFO: Allocated in io_submit_one+0x9c/0xb20 age=0 cpu=7 pid=12174 >> [23424.121212] __slab_alloc+0x34/0x60 >> [23424.121217] kmem_cache_alloc+0x504/0x5c0 >> [23424.121221] io_submit_one+0x9c/0xb20 >> [23424.121224] sys_io_submit+0xe0/0x350 >> [23424.121227] system_call+0x5c/0x70 >> [23424.121231] INFO: Freed in aio_complete+0x31c/0x410 age=0 cpu=7 pid=12174 >> [23424.121234] kmem_cache_free+0x4bc/0x540 >> [23424.121237] aio_complete+0x31c/0x410 >> [23424.121240] blkdev_bio_end_io+0x238/0x3e0 >> [23424.121243] bio_endio.part.3+0x214/0x330 >> [23424.121247] brd_make_request+0x2d8/0x314 [brd] >> [23424.121250] generic_make_request+0x220/0x510 >> [23424.121254] submit_bio+0xc8/0x1f0 >> [23424.121256] blkdev_direct_IO+0x36c/0x610 >> [23424.121260] generic_file_read_iter+0xbc/0x230 >> [23424.121263] blkdev_read_iter+0x50/0x80 >> [23424.121266] aio_read+0x138/0x200 >> [23424.121269] io_submit_one+0x7c4/0xb20 >> [23424.121272] sys_io_submit+0xe0/0x350 >> [23424.121275] system_call+0x5c/0x70 >> [23424.121278] INFO: Slab 0x00000000841158ec objects=85 used=85 fp=0x >> (null) flags=0x13fffc000000200 >> [23424.121282] INFO: Object 0x000000007e677ed8 @offset=5504 fp=0x00000000e42bdf6f >> [23424.121282] >> [23424.121287] Redzone 000000005483b8fc: bb bb bb bb bb bb bb bb bb bb bb bb bb >> bb bb bb ................ >> [23424.121291] Redzone 00000000b842fe53: bb bb bb bb bb bb bb bb bb bb bb bb bb >> bb bb bb ................ >> [23424.121295] Redzone 00000000deb0d052: bb bb bb bb bb bb bb bb bb bb bb bb bb >> bb bb bb ................ >> [23424.121299] Redzone 0000000014045233: bb bb bb bb bb bb bb bb bb bb bb bb bb >> bb bb bb ................ >> [23424.121302] Redzone 00000000dd5d6c16: bb bb bb bb bb bb bb bb bb bb bb bb bb >> bb bb bb ................ >> [23424.121306] Redzone 00000000538b5478: bb bb bb bb bb bb bb bb bb bb bb bb bb >> bb bb bb ................ >> [23424.121310] Redzone 000000001f7fb704: bb bb bb bb bb bb bb bb bb bb bb bb bb >> bb bb bb ................ >> [23424.121314] Redzone 0000000000e0484d: bb bb bb bb bb bb bb bb bb bb bb bb bb >> bb bb bb ................ >> [23424.121318] Object 000000007e677ed8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b >> 6b 6b 6b kkkkkkkkkkkkkkkk >> [23424.121322] Object 00000000e207f30b: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b >> 6b 6b 6b kkkkkkkkkkkkkkkk >> [23424.121326] Object 00000000a7a45634: 6b 6b 6b 6b 6b 6b 6b 6b ff ff ff ff 6b >> 6b 6b 6b kkkkkkkk....kkkk >> [23424.121330] Object 00000000c85d951d: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b >> 6b 6b 6b kkkkkkkkkkkkkkkk >> [23424.121334] Object 000000003104522f: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b >> 6b 6b 6b kkkkkkkkkkkkkkkk >> [23424.121338] Object 00000000cfcdd820: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b >> 6b 6b 6b kkkkkkkkkkkkkkkk >> [23424.121342] Object 00000000dded4924: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b >> 6b 6b 6b kkkkkkkkkkkkkkkk >> [23424.121346] Object 00000000ff6687a4: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b >> 6b 6b 6b kkkkkkkkkkkkkkkk >> [23424.121350] Object 00000000df3d67f6: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b >> 6b 6b 6b kkkkkkkkkkkkkkkk >> [23424.121354] Object 00000000ddc188d1: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b >> 6b 6b 6b kkkkkkkkkkkkkkkk >> [23424.121358] Object 000000002cee751a: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b >> 6b 6b 6b kkkkkkkkkkkkkkkk >> [23424.121362] Object 00000000a994f007: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b >> 6b 6b a5 kkkkkkkkkkkkkkk. >> [23424.121366] Redzone 000000009f3d62e2: bb bb bb bb bb bb bb bb >> ........ >> [23424.121370] Padding 00000000e5ccead8: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a >> 5a 5a 5a ZZZZZZZZZZZZZZZZ >> [23424.121374] Padding 000000002b0c1778: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a >> 5a 5a 5a ZZZZZZZZZZZZZZZZ >> [23424.121378] Padding 00000000c67656c7: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a >> 5a 5a 5a ZZZZZZZZZZZZZZZZ >> [23424.121382] Padding 0000000078348c5a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a >> 5a 5a 5a ZZZZZZZZZZZZZZZZ >> [23424.121386] Padding 00000000f3297820: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a >> 5a 5a 5a ZZZZZZZZZZZZZZZZ >> [23424.121390] Padding 00000000e55789f4: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a >> 5a 5a 5a ZZZZZZZZZZZZZZZZ >> [23424.121394] Padding 00000000d0fbb94c: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a >> 5a 5a 5a ZZZZZZZZZZZZZZZZ >> [23424.121397] Padding 00000000bcb27a87: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a >> 5a 5a 5a ZZZZZZZZZZZZZZZZ >> [23424.121743] CPU: 7 PID: 12174 Comm: vgs Tainted: G B W L >> 5.0.0-rc7-next-20190221+ #7 >> [23424.121758] Call Trace: >> [23424.121762] [c0000004ce5bf7b0] [c0000000007deb8c] dump_stack+0xb0/0xf4 >> (unreliable) >> [23424.121770] [c0000004ce5bf7f0] [c00000000037d310] print_trailer+0x250/0x278 >> [23424.121775] [c0000004ce5bf880] [c00000000036d578] >> check_bytes_and_report+0x138/0x160 >> [23424.121779] [c0000004ce5bf920] [c00000000036fac8] check_object+0x348/0x3e0 >> [23424.121784] [c0000004ce5bf990] [c00000000036fd18] >> alloc_debug_processing+0x1b8/0x2c0 >> [23424.121788] [c0000004ce5bfa30] [c000000000372d14] ___slab_alloc+0xbb4/0xfa0 >> [23424.121792] [c0000004ce5bfb60] [c000000000373134] __slab_alloc+0x34/0x60 >> [23424.121802] [c0000004ce5bfb90] [c000000000373664] kmem_cache_alloc+0x504/0x5c0 >> [23424.121812] [c0000004ce5bfc20] [c000000000476a9c] io_submit_one+0x9c/0xb20 >> [23424.121824] [c0000004ce5bfd50] [c000000000477f10] sys_io_submit+0xe0/0x350 >> [23424.121832] [c0000004ce5bfe20] [c00000000000b000] system_call+0x5c/0x70 >> [23424.121836] FIX aio_kiocb: Restoring 0x000000009f1f5145-0x00000000841e301b=0x6b >> [23424.121836] >> [23424.121840] FIX aio_kiocb: Marking all objects used >> > ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: io_submit with slab free object overwritten 2019-02-22 21:42 ` Eric Sandeen @ 2019-02-22 21:48 ` Qian Cai 2019-02-22 21:58 ` Eric Sandeen 0 siblings, 1 reply; 5+ messages in thread From: Qian Cai @ 2019-02-22 21:48 UTC (permalink / raw) To: Eric Sandeen, hch Cc: axboe, viro, hare, bcrl, linux-aio, Linux-MM, jthumshirn, linux-fsdevel, Christoph Lameter On 2/22/19 4:42 PM, Eric Sandeen wrote: > On 2/22/19 3:07 PM, Qian Cai wrote: >> Reverted the commit 75374d062756 ("fs: add an iopoll method to struct >> file_operations") fixed the problem. Christoph mentioned that the field can be >> calculated by the offset (40 bytes). > > I'm a little confused, you can't revert just that patch, right, because others > in the iopoll series depend on it. Is the above commit really the culprit, or do > you mean you backed out the whole series? No, I can revert that single commit on the top of linux-next (next-20190222) just fine. ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: io_submit with slab free object overwritten 2019-02-22 21:48 ` Qian Cai @ 2019-02-22 21:58 ` Eric Sandeen 2019-02-22 22:06 ` Qian Cai 0 siblings, 1 reply; 5+ messages in thread From: Eric Sandeen @ 2019-02-22 21:58 UTC (permalink / raw) To: Qian Cai, hch Cc: axboe, viro, hare, bcrl, linux-aio, Linux-MM, jthumshirn, linux-fsdevel, Christoph Lameter On 2/22/19 3:48 PM, Qian Cai wrote: > > > On 2/22/19 4:42 PM, Eric Sandeen wrote: >> On 2/22/19 3:07 PM, Qian Cai wrote: >>> Reverted the commit 75374d062756 ("fs: add an iopoll method to struct >>> file_operations") fixed the problem. Christoph mentioned that the field can be >>> calculated by the offset (40 bytes). >> >> I'm a little confused, you can't revert just that patch, right, because others >> in the iopoll series depend on it. Is the above commit really the culprit, or do >> you mean you backed out the whole series? > > No, I can revert that single commit on the top of linux-next (next-20190222) > just fine. Sorry for being pedantic, but this commit is still in your tree? How can this build with just 75374d062756 reverted? (I'm confused about how simply changing the size of the 2 structures via 75374d062756 could cause memory corruption, so trying to really understand what got tested...) commit 06eca8c02eb3e171dc5721ddca4218d41b09b3aa Author: Christoph Hellwig <hch@lst.de> Date: Fri Nov 30 08:31:52 2018 -0700 block: wire up block device iopoll method Just call blk_poll on the iocb cookie, we can derive the block device from the inode trivially. Reviewed-by: Hannes Reinecke <hare@suse.com> Reviewed-by: Johannes Thumshirn <jthumshirn@suse.de> Signed-off-by: Christoph Hellwig <hch@lst.de> Signed-off-by: Jens Axboe <axboe@kernel.dk> diff --git a/fs/block_dev.c b/fs/block_dev.c index 7758ade..d1277a1 100644 --- a/fs/block_dev.c +++ b/fs/block_dev.c @@ -294,6 +294,14 @@ struct blkdev_dio { static struct bio_set blkdev_dio_pool; +static int blkdev_iopoll(struct kiocb *kiocb, bool wait) +{ + struct block_device *bdev = I_BDEV(kiocb->ki_filp->f_mapping->host); + struct request_queue *q = bdev_get_queue(bdev); + + return blk_poll(q, READ_ONCE(kiocb->ki_cookie), wait); +} + static void blkdev_bio_end_io(struct bio *bio) { struct blkdev_dio *dio = bio->bi_private; @@ -412,6 +420,7 @@ __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter, int nr_pages) bio->bi_opf |= REQ_HIPRI; qc = submit_bio(bio); + WRITE_ONCE(iocb->ki_cookie, qc); break; } @@ -2078,6 +2087,7 @@ const struct file_operations def_blk_fops = { .llseek = block_llseek, .read_iter = blkdev_read_iter, .write_iter = blkdev_write_iter, + .iopoll = blkdev_iopoll, .mmap = generic_file_mmap, .fsync = blkdev_fsync, .unlocked_ioctl = block_ioctl, ^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: io_submit with slab free object overwritten 2019-02-22 21:58 ` Eric Sandeen @ 2019-02-22 22:06 ` Qian Cai 0 siblings, 0 replies; 5+ messages in thread From: Qian Cai @ 2019-02-22 22:06 UTC (permalink / raw) To: Eric Sandeen, hch Cc: axboe, viro, hare, bcrl, linux-aio, Linux-MM, jthumshirn, linux-fsdevel, Christoph Lameter On 2/22/19 4:58 PM, Eric Sandeen wrote: > On 2/22/19 3:48 PM, Qian Cai wrote: >> >> >> On 2/22/19 4:42 PM, Eric Sandeen wrote: >>> On 2/22/19 3:07 PM, Qian Cai wrote: >>>> Reverted the commit 75374d062756 ("fs: add an iopoll method to struct >>>> file_operations") fixed the problem. Christoph mentioned that the field can be >>>> calculated by the offset (40 bytes). >>> >>> I'm a little confused, you can't revert just that patch, right, because others >>> in the iopoll series depend on it. Is the above commit really the culprit, or do >>> you mean you backed out the whole series? >> >> No, I can revert that single commit on the top of linux-next (next-20190222) >> just fine. > > Sorry for being pedantic, but this commit is still in your tree? How can this build > with just 75374d062756 reverted? > > (I'm confused about how simply changing the size of the 2 structures via > 75374d062756 could cause memory corruption, so trying to really understand > what got tested...) > > commit 06eca8c02eb3e171dc5721ddca4218d41b09b3aa > Author: Christoph Hellwig <hch@lst.de> > Date: Fri Nov 30 08:31:52 2018 -0700 > > block: wire up block device iopoll method > > Just call blk_poll on the iocb cookie, we can derive the block device > from the inode trivially. > > Reviewed-by: Hannes Reinecke <hare@suse.com> > Reviewed-by: Johannes Thumshirn <jthumshirn@suse.de> > Signed-off-by: Christoph Hellwig <hch@lst.de> > Signed-off-by: Jens Axboe <axboe@kernel.dk> > > diff --git a/fs/block_dev.c b/fs/block_dev.c > index 7758ade..d1277a1 100644 > --- a/fs/block_dev.c > +++ b/fs/block_dev.c > @@ -294,6 +294,14 @@ struct blkdev_dio { > > static struct bio_set blkdev_dio_pool; > > +static int blkdev_iopoll(struct kiocb *kiocb, bool wait) > +{ > + struct block_device *bdev = I_BDEV(kiocb->ki_filp->f_mapping->host); > + struct request_queue *q = bdev_get_queue(bdev); > + > + return blk_poll(q, READ_ONCE(kiocb->ki_cookie), wait); > +} > + > static void blkdev_bio_end_io(struct bio *bio) > { > struct blkdev_dio *dio = bio->bi_private; > @@ -412,6 +420,7 @@ __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter, int nr_pages) > bio->bi_opf |= REQ_HIPRI; > > qc = submit_bio(bio); > + WRITE_ONCE(iocb->ki_cookie, qc); > break; > } > > @@ -2078,6 +2087,7 @@ const struct file_operations def_blk_fops = { > .llseek = block_llseek, > .read_iter = blkdev_read_iter, > .write_iter = blkdev_write_iter, > + .iopoll = blkdev_iopoll, > .mmap = generic_file_mmap, > .fsync = blkdev_fsync, > .unlocked_ioctl = block_ioctl, > Sorry, I had a copy-and-paste error here while looking at the surrounding commits. I meant, Reverted 06eca8c02eb3 (block: wire up block device iopoll method) fixed the problem. ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2019-02-22 22:07 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <4a56fc9f-27f7-5cb5-feed-a4e33f05a5d1@lca.pw>
2019-02-22 21:07 ` io_submit with slab free object overwritten Qian Cai
2019-02-22 21:42 ` Eric Sandeen
2019-02-22 21:48 ` Qian Cai
2019-02-22 21:58 ` Eric Sandeen
2019-02-22 22:06 ` Qian Cai
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).