From: Qian Cai <cai@lca.pw>
To: hch@lst.de
Cc: axboe@kernel.dk, viro@zeniv.linux.org.uk, hare@suse.com,
bcrl@kvack.org, linux-aio@kvack.org,
Linux-MM <linux-mm@kvack.org>,
jthumshirn@suse.de, linux-fsdevel@vger.kernel.org,
Christoph Lameter <cl@linux.com>
Subject: Re: io_submit with slab free object overwritten
Date: Fri, 22 Feb 2019 16:07:32 -0500 [thread overview]
Message-ID: <64b860a3-7946-ca72-8669-18ad01a78c7c@lca.pw> (raw)
In-Reply-To: <4a56fc9f-27f7-5cb5-feed-a4e33f05a5d1@lca.pw>
Reverted the commit 75374d062756 ("fs: add an iopoll method to struct
file_operations") fixed the problem. Christoph mentioned that the field can be
calculated by the offset (40 bytes).
struct kmem_cache {
struct kmem_cache_cpu __percpu *cpu_slab; (8 bytes)
slab_flags_t flags; (4)
unsigned long min_partial; (8)
unsigned int size; (4)
unsigned int object_size; (4)
unsigned int offset; (4)
unsigned int cpu_partial; (4)
struct kmem_cache_order_objects oo; (4)
/* Allocation and freeing of slabs */
struct kmem_cache_order_objects max;
So, it looks like "max" was overwritten after freed.
# cat /opt/ltp/runtest/syscalls
fgetxattr02 fgetxattr02
io_submit01 io_submit01
# /opt/ltp/runltp -f syscalls
uname:
Linux 5.0.0-rc7-next-20190222+ #11 SMP Fri Feb 22 14:57:10 EST 2019 ppc64le
ppc64le ppc64le GNU/Linux
/proc/cmdline
BOOT_IMAGE=/vmlinuz-5.0.0-rc7-next-20190222+
root=/dev/mapper/rhel_ibm--p8--01--lp5-root ro rd.lvm.lv=rhel_ibm-p8-01-lp5/root
rd.lvm.lv=rhel_ibm-p8-01-lp5/swap crashkernel=768M numa_balancing=enable earlyprintk
free reports:
total used free shared buff/cache available
Mem: 24305408 919552 23120832 12032 265024 22976896
Swap: 8388544 0 8388544
cpuinfo:
Architecture: ppc64le
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Thread(s) per core: 8
Core(s) per socket: 1
Socket(s): 2
NUMA node(s): 2
Model: 2.1 (pvr 004b 0201)
Model name: POWER8 (architected), altivec supported
Hypervisor vendor: pHyp
Virtualization type: para
L1d cache: 64K
L1i cache: 32K
L2 cache: 512K
L3 cache: 8192K
NUMA node0 CPU(s):
NUMA node1 CPU(s): 0-15
Running tests.......
<<<test_start>>>
tag=fgetxattr02 stime=1550865820
cmdline="fgetxattr02"
contacts=""
analysis=exit
<<<test_output>>>
tst_test.c:1096: INFO: Timeout per run is 0h 05m 00s
fgetxattr02.c:174: PASS: fgetxattr(2) on testfile passed
fgetxattr02.c:188: PASS: fgetxattr(2) on testfile got the right value
fgetxattr02.c:201: PASS: fgetxattr(2) on testfile passed: SUCCESS
fgetxattr02.c:174: PASS: fgetxattr(2) on testdir passed
fgetxattr02.c:188: PASS: fgetxattr(2) on testdir got the right value
fgetxattr02.c:201: PASS: fgetxattr(2) on testdir passed: SUCCESS
fgetxattr02.c:174: PASS: fgetxattr(2) on symlink passed
fgetxattr02.c:188: PASS: fgetxattr(2) on symlink got the right value
fgetxattr02.c:201: PASS: fgetxattr(2) on symlink passed: SUCCESS
fgetxattr02.c:201: PASS: fgetxattr(2) on fifo passed: ENODATA
fgetxattr02.c:201: PASS: fgetxattr(2) on chr passed: ENODATA
fgetxattr02.c:201: PASS: fgetxattr(2) on blk passed: ENODATA
fgetxattr02.c:201: PASS: fgetxattr(2) on sock passed: ENODATA
Summary:
passed 13
failed 0
skipped 0
warnings 0
<<<execution_status>>>
initiation_status="ok"
duration=0 termination_type=exited termination_id=0 corefile=no
cutime=0 cstime=1
<<<test_end>>>
<<<test_start>>>
tag=io_submit01 stime=1550865820
cmdline="io_submit01"
contacts=""
analysis=exit
<<<test_output>>>
incrementing stop
tst_test.c:1096: INFO: Timeout per run is 0h 05m 00s
io_submit01.c:125: PASS: io_submit() with invalid ctx failed with EINVAL
io_submit01.c:125: PASS: io_submit() with invalid nr failed with EINVAL
io_submit01.c:125: PASS: io_submit() with invalid iocbpp pointer failed with EFAULT
io_submit01.c:125: PASS: io_submit() with NULL iocb pointers failed with EFAULT
io_submit01.c:125: PASS: io_submit() with invalid fd failed with EBADF
io_submit01.c:125: PASS: io_submit() with readonly fd for write failed with EBADF
io_submit01.c:125: PASS: io_submit() with writeonly fd for read failed with EBADF
io_submit01.c:125: PASS: io_submit() with zero buf size failed with SUCCESS
io_submit01.c:125: PASS: io_submit() with zero nr failed with SUCCESS
Summary:
passed 9
failed 0
skipped 0
warnings 0
On 2/22/19 12:40 AM, Qian Cai wrote:
> This is only reproducible on linux-next (20190221), as v5.0-rc7 is fine. Running
> two LTP tests and then reboot will trigger this on ppc64le (CONFIG_IO_URING=n
> and CONFIG_SHUFFLE_PAGE_ALLOCATOR=y).
>
> # fgetxattr02
> # io_submit01
> # systemctl reboot
>
> There is a 32-bit (with all ones) overwritten of free slab objects (poisoned).
>
> [23424.121182] BUG aio_kiocb (Tainted: G B W L ): Poison overwritten
> [23424.121189]
> -----------------------------------------------------------------------------
> [23424.121189]
> [23424.121197] INFO: 0x000000009f1f5145-0x00000000841e301b. First byte 0xff
> instead of 0x6b
> [23424.121205] INFO: Allocated in io_submit_one+0x9c/0xb20 age=0 cpu=7 pid=12174
> [23424.121212] __slab_alloc+0x34/0x60
> [23424.121217] kmem_cache_alloc+0x504/0x5c0
> [23424.121221] io_submit_one+0x9c/0xb20
> [23424.121224] sys_io_submit+0xe0/0x350
> [23424.121227] system_call+0x5c/0x70
> [23424.121231] INFO: Freed in aio_complete+0x31c/0x410 age=0 cpu=7 pid=12174
> [23424.121234] kmem_cache_free+0x4bc/0x540
> [23424.121237] aio_complete+0x31c/0x410
> [23424.121240] blkdev_bio_end_io+0x238/0x3e0
> [23424.121243] bio_endio.part.3+0x214/0x330
> [23424.121247] brd_make_request+0x2d8/0x314 [brd]
> [23424.121250] generic_make_request+0x220/0x510
> [23424.121254] submit_bio+0xc8/0x1f0
> [23424.121256] blkdev_direct_IO+0x36c/0x610
> [23424.121260] generic_file_read_iter+0xbc/0x230
> [23424.121263] blkdev_read_iter+0x50/0x80
> [23424.121266] aio_read+0x138/0x200
> [23424.121269] io_submit_one+0x7c4/0xb20
> [23424.121272] sys_io_submit+0xe0/0x350
> [23424.121275] system_call+0x5c/0x70
> [23424.121278] INFO: Slab 0x00000000841158ec objects=85 used=85 fp=0x
> (null) flags=0x13fffc000000200
> [23424.121282] INFO: Object 0x000000007e677ed8 @offset=5504 fp=0x00000000e42bdf6f
> [23424.121282]
> [23424.121287] Redzone 000000005483b8fc: bb bb bb bb bb bb bb bb bb bb bb bb bb
> bb bb bb ................
> [23424.121291] Redzone 00000000b842fe53: bb bb bb bb bb bb bb bb bb bb bb bb bb
> bb bb bb ................
> [23424.121295] Redzone 00000000deb0d052: bb bb bb bb bb bb bb bb bb bb bb bb bb
> bb bb bb ................
> [23424.121299] Redzone 0000000014045233: bb bb bb bb bb bb bb bb bb bb bb bb bb
> bb bb bb ................
> [23424.121302] Redzone 00000000dd5d6c16: bb bb bb bb bb bb bb bb bb bb bb bb bb
> bb bb bb ................
> [23424.121306] Redzone 00000000538b5478: bb bb bb bb bb bb bb bb bb bb bb bb bb
> bb bb bb ................
> [23424.121310] Redzone 000000001f7fb704: bb bb bb bb bb bb bb bb bb bb bb bb bb
> bb bb bb ................
> [23424.121314] Redzone 0000000000e0484d: bb bb bb bb bb bb bb bb bb bb bb bb bb
> bb bb bb ................
> [23424.121318] Object 000000007e677ed8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b 6b 6b kkkkkkkkkkkkkkkk
> [23424.121322] Object 00000000e207f30b: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b 6b 6b kkkkkkkkkkkkkkkk
> [23424.121326] Object 00000000a7a45634: 6b 6b 6b 6b 6b 6b 6b 6b ff ff ff ff 6b
> 6b 6b 6b kkkkkkkk....kkkk
> [23424.121330] Object 00000000c85d951d: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b 6b 6b kkkkkkkkkkkkkkkk
> [23424.121334] Object 000000003104522f: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b 6b 6b kkkkkkkkkkkkkkkk
> [23424.121338] Object 00000000cfcdd820: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b 6b 6b kkkkkkkkkkkkkkkk
> [23424.121342] Object 00000000dded4924: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b 6b 6b kkkkkkkkkkkkkkkk
> [23424.121346] Object 00000000ff6687a4: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b 6b 6b kkkkkkkkkkkkkkkk
> [23424.121350] Object 00000000df3d67f6: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b 6b 6b kkkkkkkkkkkkkkkk
> [23424.121354] Object 00000000ddc188d1: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b 6b 6b kkkkkkkkkkkkkkkk
> [23424.121358] Object 000000002cee751a: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b 6b 6b kkkkkkkkkkkkkkkk
> [23424.121362] Object 00000000a994f007: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> 6b 6b a5 kkkkkkkkkkkkkkk.
> [23424.121366] Redzone 000000009f3d62e2: bb bb bb bb bb bb bb bb
> ........
> [23424.121370] Padding 00000000e5ccead8: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a
> 5a 5a 5a ZZZZZZZZZZZZZZZZ
> [23424.121374] Padding 000000002b0c1778: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a
> 5a 5a 5a ZZZZZZZZZZZZZZZZ
> [23424.121378] Padding 00000000c67656c7: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a
> 5a 5a 5a ZZZZZZZZZZZZZZZZ
> [23424.121382] Padding 0000000078348c5a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a
> 5a 5a 5a ZZZZZZZZZZZZZZZZ
> [23424.121386] Padding 00000000f3297820: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a
> 5a 5a 5a ZZZZZZZZZZZZZZZZ
> [23424.121390] Padding 00000000e55789f4: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a
> 5a 5a 5a ZZZZZZZZZZZZZZZZ
> [23424.121394] Padding 00000000d0fbb94c: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a
> 5a 5a 5a ZZZZZZZZZZZZZZZZ
> [23424.121397] Padding 00000000bcb27a87: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a
> 5a 5a 5a ZZZZZZZZZZZZZZZZ
> [23424.121743] CPU: 7 PID: 12174 Comm: vgs Tainted: G B W L
> 5.0.0-rc7-next-20190221+ #7
> [23424.121758] Call Trace:
> [23424.121762] [c0000004ce5bf7b0] [c0000000007deb8c] dump_stack+0xb0/0xf4
> (unreliable)
> [23424.121770] [c0000004ce5bf7f0] [c00000000037d310] print_trailer+0x250/0x278
> [23424.121775] [c0000004ce5bf880] [c00000000036d578]
> check_bytes_and_report+0x138/0x160
> [23424.121779] [c0000004ce5bf920] [c00000000036fac8] check_object+0x348/0x3e0
> [23424.121784] [c0000004ce5bf990] [c00000000036fd18]
> alloc_debug_processing+0x1b8/0x2c0
> [23424.121788] [c0000004ce5bfa30] [c000000000372d14] ___slab_alloc+0xbb4/0xfa0
> [23424.121792] [c0000004ce5bfb60] [c000000000373134] __slab_alloc+0x34/0x60
> [23424.121802] [c0000004ce5bfb90] [c000000000373664] kmem_cache_alloc+0x504/0x5c0
> [23424.121812] [c0000004ce5bfc20] [c000000000476a9c] io_submit_one+0x9c/0xb20
> [23424.121824] [c0000004ce5bfd50] [c000000000477f10] sys_io_submit+0xe0/0x350
> [23424.121832] [c0000004ce5bfe20] [c00000000000b000] system_call+0x5c/0x70
> [23424.121836] FIX aio_kiocb: Restoring 0x000000009f1f5145-0x00000000841e301b=0x6b
> [23424.121836]
> [23424.121840] FIX aio_kiocb: Marking all objects used
>
next parent reply other threads:[~2019-02-22 21:07 UTC|newest]
Thread overview: 5+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <4a56fc9f-27f7-5cb5-feed-a4e33f05a5d1@lca.pw>
2019-02-22 21:07 ` Qian Cai [this message]
2019-02-22 21:42 ` io_submit with slab free object overwritten Eric Sandeen
2019-02-22 21:48 ` Qian Cai
2019-02-22 21:58 ` Eric Sandeen
2019-02-22 22:06 ` Qian Cai
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=64b860a3-7946-ca72-8669-18ad01a78c7c@lca.pw \
--to=cai@lca.pw \
--cc=axboe@kernel.dk \
--cc=bcrl@kvack.org \
--cc=cl@linux.com \
--cc=hare@suse.com \
--cc=hch@lst.de \
--cc=jthumshirn@suse.de \
--cc=linux-aio@kvack.org \
--cc=linux-fsdevel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=viro@zeniv.linux.org.uk \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).