From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.1 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8BB8FC43381 for ; Fri, 22 Feb 2019 21:07:38 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 3D161207E0 for ; Fri, 22 Feb 2019 21:07:38 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lca.pw header.i=@lca.pw header.b="Lom/xUgl" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726490AbfBVVHh (ORCPT ); Fri, 22 Feb 2019 16:07:37 -0500 Received: from mail-qt1-f196.google.com ([209.85.160.196]:35326 "EHLO mail-qt1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726152AbfBVVHh (ORCPT ); Fri, 22 Feb 2019 16:07:37 -0500 Received: by mail-qt1-f196.google.com with SMTP id p48so4174997qtk.2 for ; Fri, 22 Feb 2019 13:07:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lca.pw; s=google; h=from:subject:to:cc:references:message-id:date:user-agent :mime-version:in-reply-to:content-language:content-transfer-encoding; bh=LiaWbODHYOh+ZnfJmFKaQwDMnQck/cyCHyDsVxz0fQU=; b=Lom/xUglchz9raIHwUiu8kx89cD0sgH1BrJcRe4L1EquelC/ttgzV9nvi9/vCHtjiv 7cfdLCSNZmSpYeGKeuHV5XHHkE9LGYmR+IQhJnLXaSEpTRklw881ipy9+ZBb0MXUoZep LnvCUAVUcAYvcyh8c5WVCAQzDzlre5t+ppg1Pj2Ax/M3XvchvoIsQ3E0mrO2Yg/6TiS4 fONYJqEa+TxFm94Mri2hpxfqktETwqPbL7sI3GvlmMbbFbAa03N9v22G+Ku8NFMgySsI dL1C3Yh8QnYL2ssUtyuVRiUn0KPF7JGGlJeXteRs5uk1RQwF458/6d0iZZMTkD2qLpiI 1Trw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:subject:to:cc:references:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=LiaWbODHYOh+ZnfJmFKaQwDMnQck/cyCHyDsVxz0fQU=; b=AsJmA6/SlZNACul7kP4EVvSxJfx1rV/7+n+sr+5/wa2Pq+zr9hzgMwOOBhkRbaV5FX Vl6cm+Tt78d1kdXUTwv54Xm2Cjx0aEbRhwGZVYAis8K0ZhpeLNOGlJeVjZdCw7syQ8cj GQN+kO8WlF18jCl0j3h2h4QgPbfhotakSsikbcsg08lz6xecgNCoX+IkUmJG6XbF/s5B T4gStdvmWjQiVe40BZ3RWJjtUWgvYbs67/6OrIQKob+n6UPq0B94LM5JLEBw8fV9FoRm REJOviGikjcd3feUnEkc02XUbR7NntOwO1sBr9myVlStjjEojwSwx5QunHzM0FZyqEd6 lkPA== X-Gm-Message-State: AHQUAubFedkGENrYOq2BD1QxKlPyfyGGuY0EMGh4xI3gCrzUfAjkfxNr aO3cahr+6s42k1EYWs/ZLrdTn1YQ4IQ= X-Google-Smtp-Source: AHgI3IYdvCp4QTO0Ou3ofEAAvVi6FH5V7L2PMN14Zwz7vy6zxbQkY539/7GZpfj/ac8m4TanCcAnHg== X-Received: by 2002:a0c:891a:: with SMTP id 26mr4776956qvp.163.1550869655524; Fri, 22 Feb 2019 13:07:35 -0800 (PST) Received: from ovpn-120-150.rdu2.redhat.com (pool-71-184-117-43.bstnma.fios.verizon.net. [71.184.117.43]) by smtp.gmail.com with ESMTPSA id c9sm1920854qkj.61.2019.02.22.13.07.33 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 22 Feb 2019 13:07:34 -0800 (PST) From: Qian Cai Subject: Re: io_submit with slab free object overwritten To: hch@lst.de Cc: axboe@kernel.dk, viro@zeniv.linux.org.uk, hare@suse.com, bcrl@kvack.org, linux-aio@kvack.org, Linux-MM , jthumshirn@suse.de, linux-fsdevel@vger.kernel.org, Christoph Lameter References: <4a56fc9f-27f7-5cb5-feed-a4e33f05a5d1@lca.pw> Message-ID: <64b860a3-7946-ca72-8669-18ad01a78c7c@lca.pw> Date: Fri, 22 Feb 2019 16:07:32 -0500 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:60.0) Gecko/20100101 Thunderbird/60.3.3 MIME-Version: 1.0 In-Reply-To: <4a56fc9f-27f7-5cb5-feed-a4e33f05a5d1@lca.pw> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit Sender: linux-fsdevel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Reverted the commit 75374d062756 ("fs: add an iopoll method to struct file_operations") fixed the problem. Christoph mentioned that the field can be calculated by the offset (40 bytes). struct kmem_cache { struct kmem_cache_cpu __percpu *cpu_slab; (8 bytes) slab_flags_t flags; (4) unsigned long min_partial; (8) unsigned int size; (4) unsigned int object_size; (4) unsigned int offset; (4) unsigned int cpu_partial; (4) struct kmem_cache_order_objects oo; (4) /* Allocation and freeing of slabs */ struct kmem_cache_order_objects max; So, it looks like "max" was overwritten after freed. # cat /opt/ltp/runtest/syscalls fgetxattr02 fgetxattr02 io_submit01 io_submit01 # /opt/ltp/runltp -f syscalls uname: Linux 5.0.0-rc7-next-20190222+ #11 SMP Fri Feb 22 14:57:10 EST 2019 ppc64le ppc64le ppc64le GNU/Linux /proc/cmdline BOOT_IMAGE=/vmlinuz-5.0.0-rc7-next-20190222+ root=/dev/mapper/rhel_ibm--p8--01--lp5-root ro rd.lvm.lv=rhel_ibm-p8-01-lp5/root rd.lvm.lv=rhel_ibm-p8-01-lp5/swap crashkernel=768M numa_balancing=enable earlyprintk free reports: total used free shared buff/cache available Mem: 24305408 919552 23120832 12032 265024 22976896 Swap: 8388544 0 8388544 cpuinfo: Architecture: ppc64le Byte Order: Little Endian CPU(s): 16 On-line CPU(s) list: 0-15 Thread(s) per core: 8 Core(s) per socket: 1 Socket(s): 2 NUMA node(s): 2 Model: 2.1 (pvr 004b 0201) Model name: POWER8 (architected), altivec supported Hypervisor vendor: pHyp Virtualization type: para L1d cache: 64K L1i cache: 32K L2 cache: 512K L3 cache: 8192K NUMA node0 CPU(s): NUMA node1 CPU(s): 0-15 Running tests....... <<>> tag=fgetxattr02 stime=1550865820 cmdline="fgetxattr02" contacts="" analysis=exit <<>> tst_test.c:1096: INFO: Timeout per run is 0h 05m 00s fgetxattr02.c:174: PASS: fgetxattr(2) on testfile passed fgetxattr02.c:188: PASS: fgetxattr(2) on testfile got the right value fgetxattr02.c:201: PASS: fgetxattr(2) on testfile passed: SUCCESS fgetxattr02.c:174: PASS: fgetxattr(2) on testdir passed fgetxattr02.c:188: PASS: fgetxattr(2) on testdir got the right value fgetxattr02.c:201: PASS: fgetxattr(2) on testdir passed: SUCCESS fgetxattr02.c:174: PASS: fgetxattr(2) on symlink passed fgetxattr02.c:188: PASS: fgetxattr(2) on symlink got the right value fgetxattr02.c:201: PASS: fgetxattr(2) on symlink passed: SUCCESS fgetxattr02.c:201: PASS: fgetxattr(2) on fifo passed: ENODATA fgetxattr02.c:201: PASS: fgetxattr(2) on chr passed: ENODATA fgetxattr02.c:201: PASS: fgetxattr(2) on blk passed: ENODATA fgetxattr02.c:201: PASS: fgetxattr(2) on sock passed: ENODATA Summary: passed 13 failed 0 skipped 0 warnings 0 <<>> initiation_status="ok" duration=0 termination_type=exited termination_id=0 corefile=no cutime=0 cstime=1 <<>> <<>> tag=io_submit01 stime=1550865820 cmdline="io_submit01" contacts="" analysis=exit <<>> incrementing stop tst_test.c:1096: INFO: Timeout per run is 0h 05m 00s io_submit01.c:125: PASS: io_submit() with invalid ctx failed with EINVAL io_submit01.c:125: PASS: io_submit() with invalid nr failed with EINVAL io_submit01.c:125: PASS: io_submit() with invalid iocbpp pointer failed with EFAULT io_submit01.c:125: PASS: io_submit() with NULL iocb pointers failed with EFAULT io_submit01.c:125: PASS: io_submit() with invalid fd failed with EBADF io_submit01.c:125: PASS: io_submit() with readonly fd for write failed with EBADF io_submit01.c:125: PASS: io_submit() with writeonly fd for read failed with EBADF io_submit01.c:125: PASS: io_submit() with zero buf size failed with SUCCESS io_submit01.c:125: PASS: io_submit() with zero nr failed with SUCCESS Summary: passed 9 failed 0 skipped 0 warnings 0 On 2/22/19 12:40 AM, Qian Cai wrote: > This is only reproducible on linux-next (20190221), as v5.0-rc7 is fine. Running > two LTP tests and then reboot will trigger this on ppc64le (CONFIG_IO_URING=n > and CONFIG_SHUFFLE_PAGE_ALLOCATOR=y). > > # fgetxattr02 > # io_submit01 > # systemctl reboot > > There is a 32-bit (with all ones) overwritten of free slab objects (poisoned). > > [23424.121182] BUG aio_kiocb (Tainted: G B W L ): Poison overwritten > [23424.121189] > ----------------------------------------------------------------------------- > [23424.121189] > [23424.121197] INFO: 0x000000009f1f5145-0x00000000841e301b. First byte 0xff > instead of 0x6b > [23424.121205] INFO: Allocated in io_submit_one+0x9c/0xb20 age=0 cpu=7 pid=12174 > [23424.121212] __slab_alloc+0x34/0x60 > [23424.121217] kmem_cache_alloc+0x504/0x5c0 > [23424.121221] io_submit_one+0x9c/0xb20 > [23424.121224] sys_io_submit+0xe0/0x350 > [23424.121227] system_call+0x5c/0x70 > [23424.121231] INFO: Freed in aio_complete+0x31c/0x410 age=0 cpu=7 pid=12174 > [23424.121234] kmem_cache_free+0x4bc/0x540 > [23424.121237] aio_complete+0x31c/0x410 > [23424.121240] blkdev_bio_end_io+0x238/0x3e0 > [23424.121243] bio_endio.part.3+0x214/0x330 > [23424.121247] brd_make_request+0x2d8/0x314 [brd] > [23424.121250] generic_make_request+0x220/0x510 > [23424.121254] submit_bio+0xc8/0x1f0 > [23424.121256] blkdev_direct_IO+0x36c/0x610 > [23424.121260] generic_file_read_iter+0xbc/0x230 > [23424.121263] blkdev_read_iter+0x50/0x80 > [23424.121266] aio_read+0x138/0x200 > [23424.121269] io_submit_one+0x7c4/0xb20 > [23424.121272] sys_io_submit+0xe0/0x350 > [23424.121275] system_call+0x5c/0x70 > [23424.121278] INFO: Slab 0x00000000841158ec objects=85 used=85 fp=0x > (null) flags=0x13fffc000000200 > [23424.121282] INFO: Object 0x000000007e677ed8 @offset=5504 fp=0x00000000e42bdf6f > [23424.121282] > [23424.121287] Redzone 000000005483b8fc: bb bb bb bb bb bb bb bb bb bb bb bb bb > bb bb bb ................ > [23424.121291] Redzone 00000000b842fe53: bb bb bb bb bb bb bb bb bb bb bb bb bb > bb bb bb ................ > [23424.121295] Redzone 00000000deb0d052: bb bb bb bb bb bb bb bb bb bb bb bb bb > bb bb bb ................ > [23424.121299] Redzone 0000000014045233: bb bb bb bb bb bb bb bb bb bb bb bb bb > bb bb bb ................ > [23424.121302] Redzone 00000000dd5d6c16: bb bb bb bb bb bb bb bb bb bb bb bb bb > bb bb bb ................ > [23424.121306] Redzone 00000000538b5478: bb bb bb bb bb bb bb bb bb bb bb bb bb > bb bb bb ................ > [23424.121310] Redzone 000000001f7fb704: bb bb bb bb bb bb bb bb bb bb bb bb bb > bb bb bb ................ > [23424.121314] Redzone 0000000000e0484d: bb bb bb bb bb bb bb bb bb bb bb bb bb > bb bb bb ................ > [23424.121318] Object 000000007e677ed8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b > 6b 6b 6b kkkkkkkkkkkkkkkk > [23424.121322] Object 00000000e207f30b: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b > 6b 6b 6b kkkkkkkkkkkkkkkk > [23424.121326] Object 00000000a7a45634: 6b 6b 6b 6b 6b 6b 6b 6b ff ff ff ff 6b > 6b 6b 6b kkkkkkkk....kkkk > [23424.121330] Object 00000000c85d951d: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b > 6b 6b 6b kkkkkkkkkkkkkkkk > [23424.121334] Object 000000003104522f: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b > 6b 6b 6b kkkkkkkkkkkkkkkk > [23424.121338] Object 00000000cfcdd820: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b > 6b 6b 6b kkkkkkkkkkkkkkkk > [23424.121342] Object 00000000dded4924: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b > 6b 6b 6b kkkkkkkkkkkkkkkk > [23424.121346] Object 00000000ff6687a4: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b > 6b 6b 6b kkkkkkkkkkkkkkkk > [23424.121350] Object 00000000df3d67f6: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b > 6b 6b 6b kkkkkkkkkkkkkkkk > [23424.121354] Object 00000000ddc188d1: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b > 6b 6b 6b kkkkkkkkkkkkkkkk > [23424.121358] Object 000000002cee751a: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b > 6b 6b 6b kkkkkkkkkkkkkkkk > [23424.121362] Object 00000000a994f007: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b > 6b 6b a5 kkkkkkkkkkkkkkk. > [23424.121366] Redzone 000000009f3d62e2: bb bb bb bb bb bb bb bb > ........ > [23424.121370] Padding 00000000e5ccead8: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a > 5a 5a 5a ZZZZZZZZZZZZZZZZ > [23424.121374] Padding 000000002b0c1778: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a > 5a 5a 5a ZZZZZZZZZZZZZZZZ > [23424.121378] Padding 00000000c67656c7: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a > 5a 5a 5a ZZZZZZZZZZZZZZZZ > [23424.121382] Padding 0000000078348c5a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a > 5a 5a 5a ZZZZZZZZZZZZZZZZ > [23424.121386] Padding 00000000f3297820: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a > 5a 5a 5a ZZZZZZZZZZZZZZZZ > [23424.121390] Padding 00000000e55789f4: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a > 5a 5a 5a ZZZZZZZZZZZZZZZZ > [23424.121394] Padding 00000000d0fbb94c: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a > 5a 5a 5a ZZZZZZZZZZZZZZZZ > [23424.121397] Padding 00000000bcb27a87: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a > 5a 5a 5a ZZZZZZZZZZZZZZZZ > [23424.121743] CPU: 7 PID: 12174 Comm: vgs Tainted: G B W L > 5.0.0-rc7-next-20190221+ #7 > [23424.121758] Call Trace: > [23424.121762] [c0000004ce5bf7b0] [c0000000007deb8c] dump_stack+0xb0/0xf4 > (unreliable) > [23424.121770] [c0000004ce5bf7f0] [c00000000037d310] print_trailer+0x250/0x278 > [23424.121775] [c0000004ce5bf880] [c00000000036d578] > check_bytes_and_report+0x138/0x160 > [23424.121779] [c0000004ce5bf920] [c00000000036fac8] check_object+0x348/0x3e0 > [23424.121784] [c0000004ce5bf990] [c00000000036fd18] > alloc_debug_processing+0x1b8/0x2c0 > [23424.121788] [c0000004ce5bfa30] [c000000000372d14] ___slab_alloc+0xbb4/0xfa0 > [23424.121792] [c0000004ce5bfb60] [c000000000373134] __slab_alloc+0x34/0x60 > [23424.121802] [c0000004ce5bfb90] [c000000000373664] kmem_cache_alloc+0x504/0x5c0 > [23424.121812] [c0000004ce5bfc20] [c000000000476a9c] io_submit_one+0x9c/0xb20 > [23424.121824] [c0000004ce5bfd50] [c000000000477f10] sys_io_submit+0xe0/0x350 > [23424.121832] [c0000004ce5bfe20] [c00000000000b000] system_call+0x5c/0x70 > [23424.121836] FIX aio_kiocb: Restoring 0x000000009f1f5145-0x00000000841e301b=0x6b > [23424.121836] > [23424.121840] FIX aio_kiocb: Marking all objects used >