From: Eric Sandeen <sandeen@sandeen.net>
To: Qian Cai <cai@lca.pw>, hch@lst.de
Cc: axboe@kernel.dk, viro@zeniv.linux.org.uk, hare@suse.com,
bcrl@kvack.org, linux-aio@kvack.org,
Linux-MM <linux-mm@kvack.org>,
jthumshirn@suse.de, linux-fsdevel@vger.kernel.org,
Christoph Lameter <cl@linux.com>
Subject: Re: io_submit with slab free object overwritten
Date: Fri, 22 Feb 2019 15:42:39 -0600 [thread overview]
Message-ID: <0a28db73-7e52-9879-276c-adc6aaf05d4d@sandeen.net> (raw)
In-Reply-To: <64b860a3-7946-ca72-8669-18ad01a78c7c@lca.pw>
On 2/22/19 3:07 PM, Qian Cai wrote:
> Reverted the commit 75374d062756 ("fs: add an iopoll method to struct
> file_operations") fixed the problem. Christoph mentioned that the field can be
> calculated by the offset (40 bytes).
I'm a little confused, you can't revert just that patch, right, because others
in the iopoll series depend on it. Is the above commit really the culprit, or do
you mean you backed out the whole series?
thanks,
-Eric
> struct kmem_cache {
> struct kmem_cache_cpu __percpu *cpu_slab; (8 bytes)
> slab_flags_t flags; (4)
> unsigned long min_partial; (8)
> unsigned int size; (4)
> unsigned int object_size; (4)
> unsigned int offset; (4)
> unsigned int cpu_partial; (4)
> struct kmem_cache_order_objects oo; (4)
>
> /* Allocation and freeing of slabs */
> struct kmem_cache_order_objects max;
>
> So, it looks like "max" was overwritten after freed.
>
> # cat /opt/ltp/runtest/syscalls
> fgetxattr02 fgetxattr02
> io_submit01 io_submit01
>
> # /opt/ltp/runltp -f syscalls
>
> uname:
> Linux 5.0.0-rc7-next-20190222+ #11 SMP Fri Feb 22 14:57:10 EST 2019 ppc64le
> ppc64le ppc64le GNU/Linux
>
> /proc/cmdline
> BOOT_IMAGE=/vmlinuz-5.0.0-rc7-next-20190222+
> root=/dev/mapper/rhel_ibm--p8--01--lp5-root ro rd.lvm.lv=rhel_ibm-p8-01-lp5/root
> rd.lvm.lv=rhel_ibm-p8-01-lp5/swap crashkernel=768M numa_balancing=enable earlyprintk
>
> free reports:
> total used free shared buff/cache available
> Mem: 24305408 919552 23120832 12032 265024 22976896
> Swap: 8388544 0 8388544
>
> cpuinfo:
> Architecture: ppc64le
> Byte Order: Little Endian
> CPU(s): 16
> On-line CPU(s) list: 0-15
> Thread(s) per core: 8
> Core(s) per socket: 1
> Socket(s): 2
> NUMA node(s): 2
> Model: 2.1 (pvr 004b 0201)
> Model name: POWER8 (architected), altivec supported
> Hypervisor vendor: pHyp
> Virtualization type: para
> L1d cache: 64K
> L1i cache: 32K
> L2 cache: 512K
> L3 cache: 8192K
> NUMA node0 CPU(s):
> NUMA node1 CPU(s): 0-15
>
> Running tests.......
> <<<test_start>>>
> tag=fgetxattr02 stime=1550865820
> cmdline="fgetxattr02"
> contacts=""
> analysis=exit
> <<<test_output>>>
> tst_test.c:1096: INFO: Timeout per run is 0h 05m 00s
> fgetxattr02.c:174: PASS: fgetxattr(2) on testfile passed
> fgetxattr02.c:188: PASS: fgetxattr(2) on testfile got the right value
> fgetxattr02.c:201: PASS: fgetxattr(2) on testfile passed: SUCCESS
> fgetxattr02.c:174: PASS: fgetxattr(2) on testdir passed
> fgetxattr02.c:188: PASS: fgetxattr(2) on testdir got the right value
> fgetxattr02.c:201: PASS: fgetxattr(2) on testdir passed: SUCCESS
> fgetxattr02.c:174: PASS: fgetxattr(2) on symlink passed
> fgetxattr02.c:188: PASS: fgetxattr(2) on symlink got the right value
> fgetxattr02.c:201: PASS: fgetxattr(2) on symlink passed: SUCCESS
> fgetxattr02.c:201: PASS: fgetxattr(2) on fifo passed: ENODATA
> fgetxattr02.c:201: PASS: fgetxattr(2) on chr passed: ENODATA
> fgetxattr02.c:201: PASS: fgetxattr(2) on blk passed: ENODATA
> fgetxattr02.c:201: PASS: fgetxattr(2) on sock passed: ENODATA
>
> Summary:
> passed 13
> failed 0
> skipped 0
> warnings 0
> <<<execution_status>>>
> initiation_status="ok"
> duration=0 termination_type=exited termination_id=0 corefile=no
> cutime=0 cstime=1
> <<<test_end>>>
> <<<test_start>>>
> tag=io_submit01 stime=1550865820
> cmdline="io_submit01"
> contacts=""
> analysis=exit
> <<<test_output>>>
> incrementing stop
> tst_test.c:1096: INFO: Timeout per run is 0h 05m 00s
> io_submit01.c:125: PASS: io_submit() with invalid ctx failed with EINVAL
> io_submit01.c:125: PASS: io_submit() with invalid nr failed with EINVAL
> io_submit01.c:125: PASS: io_submit() with invalid iocbpp pointer failed with EFAULT
> io_submit01.c:125: PASS: io_submit() with NULL iocb pointers failed with EFAULT
> io_submit01.c:125: PASS: io_submit() with invalid fd failed with EBADF
> io_submit01.c:125: PASS: io_submit() with readonly fd for write failed with EBADF
> io_submit01.c:125: PASS: io_submit() with writeonly fd for read failed with EBADF
> io_submit01.c:125: PASS: io_submit() with zero buf size failed with SUCCESS
> io_submit01.c:125: PASS: io_submit() with zero nr failed with SUCCESS
>
> Summary:
> passed 9
> failed 0
> skipped 0
> warnings 0
>
> On 2/22/19 12:40 AM, Qian Cai wrote:
>> This is only reproducible on linux-next (20190221), as v5.0-rc7 is fine. Running
>> two LTP tests and then reboot will trigger this on ppc64le (CONFIG_IO_URING=n
>> and CONFIG_SHUFFLE_PAGE_ALLOCATOR=y).
>>
>> # fgetxattr02
>> # io_submit01
>> # systemctl reboot
>>
>> There is a 32-bit (with all ones) overwritten of free slab objects (poisoned).
>>
>> [23424.121182] BUG aio_kiocb (Tainted: G B W L ): Poison overwritten
>> [23424.121189]
>> -----------------------------------------------------------------------------
>> [23424.121189]
>> [23424.121197] INFO: 0x000000009f1f5145-0x00000000841e301b. First byte 0xff
>> instead of 0x6b
>> [23424.121205] INFO: Allocated in io_submit_one+0x9c/0xb20 age=0 cpu=7 pid=12174
>> [23424.121212] __slab_alloc+0x34/0x60
>> [23424.121217] kmem_cache_alloc+0x504/0x5c0
>> [23424.121221] io_submit_one+0x9c/0xb20
>> [23424.121224] sys_io_submit+0xe0/0x350
>> [23424.121227] system_call+0x5c/0x70
>> [23424.121231] INFO: Freed in aio_complete+0x31c/0x410 age=0 cpu=7 pid=12174
>> [23424.121234] kmem_cache_free+0x4bc/0x540
>> [23424.121237] aio_complete+0x31c/0x410
>> [23424.121240] blkdev_bio_end_io+0x238/0x3e0
>> [23424.121243] bio_endio.part.3+0x214/0x330
>> [23424.121247] brd_make_request+0x2d8/0x314 [brd]
>> [23424.121250] generic_make_request+0x220/0x510
>> [23424.121254] submit_bio+0xc8/0x1f0
>> [23424.121256] blkdev_direct_IO+0x36c/0x610
>> [23424.121260] generic_file_read_iter+0xbc/0x230
>> [23424.121263] blkdev_read_iter+0x50/0x80
>> [23424.121266] aio_read+0x138/0x200
>> [23424.121269] io_submit_one+0x7c4/0xb20
>> [23424.121272] sys_io_submit+0xe0/0x350
>> [23424.121275] system_call+0x5c/0x70
>> [23424.121278] INFO: Slab 0x00000000841158ec objects=85 used=85 fp=0x
>> (null) flags=0x13fffc000000200
>> [23424.121282] INFO: Object 0x000000007e677ed8 @offset=5504 fp=0x00000000e42bdf6f
>> [23424.121282]
>> [23424.121287] Redzone 000000005483b8fc: bb bb bb bb bb bb bb bb bb bb bb bb bb
>> bb bb bb ................
>> [23424.121291] Redzone 00000000b842fe53: bb bb bb bb bb bb bb bb bb bb bb bb bb
>> bb bb bb ................
>> [23424.121295] Redzone 00000000deb0d052: bb bb bb bb bb bb bb bb bb bb bb bb bb
>> bb bb bb ................
>> [23424.121299] Redzone 0000000014045233: bb bb bb bb bb bb bb bb bb bb bb bb bb
>> bb bb bb ................
>> [23424.121302] Redzone 00000000dd5d6c16: bb bb bb bb bb bb bb bb bb bb bb bb bb
>> bb bb bb ................
>> [23424.121306] Redzone 00000000538b5478: bb bb bb bb bb bb bb bb bb bb bb bb bb
>> bb bb bb ................
>> [23424.121310] Redzone 000000001f7fb704: bb bb bb bb bb bb bb bb bb bb bb bb bb
>> bb bb bb ................
>> [23424.121314] Redzone 0000000000e0484d: bb bb bb bb bb bb bb bb bb bb bb bb bb
>> bb bb bb ................
>> [23424.121318] Object 000000007e677ed8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
>> 6b 6b 6b kkkkkkkkkkkkkkkk
>> [23424.121322] Object 00000000e207f30b: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
>> 6b 6b 6b kkkkkkkkkkkkkkkk
>> [23424.121326] Object 00000000a7a45634: 6b 6b 6b 6b 6b 6b 6b 6b ff ff ff ff 6b
>> 6b 6b 6b kkkkkkkk....kkkk
>> [23424.121330] Object 00000000c85d951d: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
>> 6b 6b 6b kkkkkkkkkkkkkkkk
>> [23424.121334] Object 000000003104522f: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
>> 6b 6b 6b kkkkkkkkkkkkkkkk
>> [23424.121338] Object 00000000cfcdd820: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
>> 6b 6b 6b kkkkkkkkkkkkkkkk
>> [23424.121342] Object 00000000dded4924: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
>> 6b 6b 6b kkkkkkkkkkkkkkkk
>> [23424.121346] Object 00000000ff6687a4: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
>> 6b 6b 6b kkkkkkkkkkkkkkkk
>> [23424.121350] Object 00000000df3d67f6: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
>> 6b 6b 6b kkkkkkkkkkkkkkkk
>> [23424.121354] Object 00000000ddc188d1: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
>> 6b 6b 6b kkkkkkkkkkkkkkkk
>> [23424.121358] Object 000000002cee751a: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
>> 6b 6b 6b kkkkkkkkkkkkkkkk
>> [23424.121362] Object 00000000a994f007: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
>> 6b 6b a5 kkkkkkkkkkkkkkk.
>> [23424.121366] Redzone 000000009f3d62e2: bb bb bb bb bb bb bb bb
>> ........
>> [23424.121370] Padding 00000000e5ccead8: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [23424.121374] Padding 000000002b0c1778: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [23424.121378] Padding 00000000c67656c7: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [23424.121382] Padding 0000000078348c5a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [23424.121386] Padding 00000000f3297820: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [23424.121390] Padding 00000000e55789f4: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [23424.121394] Padding 00000000d0fbb94c: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [23424.121397] Padding 00000000bcb27a87: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [23424.121743] CPU: 7 PID: 12174 Comm: vgs Tainted: G B W L
>> 5.0.0-rc7-next-20190221+ #7
>> [23424.121758] Call Trace:
>> [23424.121762] [c0000004ce5bf7b0] [c0000000007deb8c] dump_stack+0xb0/0xf4
>> (unreliable)
>> [23424.121770] [c0000004ce5bf7f0] [c00000000037d310] print_trailer+0x250/0x278
>> [23424.121775] [c0000004ce5bf880] [c00000000036d578]
>> check_bytes_and_report+0x138/0x160
>> [23424.121779] [c0000004ce5bf920] [c00000000036fac8] check_object+0x348/0x3e0
>> [23424.121784] [c0000004ce5bf990] [c00000000036fd18]
>> alloc_debug_processing+0x1b8/0x2c0
>> [23424.121788] [c0000004ce5bfa30] [c000000000372d14] ___slab_alloc+0xbb4/0xfa0
>> [23424.121792] [c0000004ce5bfb60] [c000000000373134] __slab_alloc+0x34/0x60
>> [23424.121802] [c0000004ce5bfb90] [c000000000373664] kmem_cache_alloc+0x504/0x5c0
>> [23424.121812] [c0000004ce5bfc20] [c000000000476a9c] io_submit_one+0x9c/0xb20
>> [23424.121824] [c0000004ce5bfd50] [c000000000477f10] sys_io_submit+0xe0/0x350
>> [23424.121832] [c0000004ce5bfe20] [c00000000000b000] system_call+0x5c/0x70
>> [23424.121836] FIX aio_kiocb: Restoring 0x000000009f1f5145-0x00000000841e301b=0x6b
>> [23424.121836]
>> [23424.121840] FIX aio_kiocb: Marking all objects used
>>
>
next prev parent reply other threads:[~2019-02-22 21:42 UTC|newest]
Thread overview: 5+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <4a56fc9f-27f7-5cb5-feed-a4e33f05a5d1@lca.pw>
2019-02-22 21:07 ` io_submit with slab free object overwritten Qian Cai
2019-02-22 21:42 ` Eric Sandeen [this message]
2019-02-22 21:48 ` Qian Cai
2019-02-22 21:58 ` Eric Sandeen
2019-02-22 22:06 ` Qian Cai
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=0a28db73-7e52-9879-276c-adc6aaf05d4d@sandeen.net \
--to=sandeen@sandeen.net \
--cc=axboe@kernel.dk \
--cc=bcrl@kvack.org \
--cc=cai@lca.pw \
--cc=cl@linux.com \
--cc=hare@suse.com \
--cc=hch@lst.de \
--cc=jthumshirn@suse.de \
--cc=linux-aio@kvack.org \
--cc=linux-fsdevel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=viro@zeniv.linux.org.uk \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).