From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.5 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_PASS,URIBL_BLOCKED,USER_AGENT_MUTT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7B8BFC43381 for ; Thu, 21 Feb 2019 10:17:09 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 4A3812148D for ; Thu, 21 Feb 2019 10:17:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727696AbfBUKRC (ORCPT ); Thu, 21 Feb 2019 05:17:02 -0500 Received: from mx1.redhat.com ([209.132.183.28]:52140 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725943AbfBUKRB (ORCPT ); Thu, 21 Feb 2019 05:17:01 -0500 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 21746300BBA8; Thu, 21 Feb 2019 10:17:00 +0000 (UTC) Received: from ming.t460p (ovpn-8-27.pek2.redhat.com [10.72.8.27]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 980B35C277; Thu, 21 Feb 2019 10:16:24 +0000 (UTC) Date: Thu, 21 Feb 2019 18:16:19 +0800 From: Ming Lei To: Marek Szyprowski Cc: Jens Axboe , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Theodore Ts'o , Omar Sandoval , Sagi Grimberg , Dave Chinner , Kent Overstreet , Mike Snitzer , dm-devel@redhat.com, Alexander Viro , linux-fsdevel@vger.kernel.org, linux-raid@vger.kernel.org, David Sterba , linux-btrfs@vger.kernel.org, "Darrick J . Wong" , linux-xfs@vger.kernel.org, Gao Xiang , Christoph Hellwig , linux-ext4@vger.kernel.org, Coly Li , linux-bcache@vger.kernel.org, Boaz Harrosh , Bob Peterson , cluster-devel@redhat.com, Ulf Hansson , "linux-mmc@vger.kernel.org" , 'Linux Samsung SOC' , Krzysztof Kozlowski , Adrian Hunter , Bartlomiej Zolnierkiewicz Subject: Re: [PATCH V15 14/18] block: enable multipage bvecs Message-ID: <20190221101618.GB12448@ming.t460p> References: <20190215111324.30129-1-ming.lei@redhat.com> <20190215111324.30129-15-ming.lei@redhat.com> <6c9ae4de-c56f-a2b3-2542-da7d8b95601d@samsung.com> <20190221095733.GA12448@ming.t460p> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: User-Agent: Mutt/1.9.1 (2017-09-22) X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.41]); Thu, 21 Feb 2019 10:17:01 +0000 (UTC) Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org On Thu, Feb 21, 2019 at 11:08:19AM +0100, Marek Szyprowski wrote: > Hi Ming, > > On 2019-02-21 10:57, Ming Lei wrote: > > On Thu, Feb 21, 2019 at 09:42:59AM +0100, Marek Szyprowski wrote: > >> On 2019-02-15 12:13, Ming Lei wrote: > >>> This patch pulls the trigger for multi-page bvecs. > >>> > >>> Reviewed-by: Omar Sandoval > >>> Signed-off-by: Ming Lei > >> Since Linux next-20190218 I've observed problems with block layer on one > >> of my test devices (Odroid U3 with EXT4 rootfs on SD card). Bisecting > >> this issue led me to this change. This is also the first linux-next > >> release with this change merged. The issue is fully reproducible and can > >> be observed in the following kernel log: > >> > >> sdhci: Secure Digital Host Controller Interface driver > >> sdhci: Copyright(c) Pierre Ossman > >> s3c-sdhci 12530000.sdhci: clock source 2: mmc_busclk.2 (100000000 Hz) > >> s3c-sdhci 12530000.sdhci: Got CD GPIO > >> mmc0: SDHCI controller on samsung-hsmmc [12530000.sdhci] using ADMA > >> mmc0: new high speed SDHC card at address aaaa > >> mmcblk0: mmc0:aaaa SL16G 14.8 GiB > >> > >> ... > >> > >> EXT4-fs (mmcblk0p2): INFO: recovery required on readonly filesystem > >> EXT4-fs (mmcblk0p2): write access will be enabled during recovery > >> EXT4-fs (mmcblk0p2): recovery complete > >> EXT4-fs (mmcblk0p2): mounted filesystem with ordered data mode. Opts: (null) > >> VFS: Mounted root (ext4 filesystem) readonly on device 179:2. > >> devtmpfs: mounted > >> Freeing unused kernel memory: 1024K > >> hub 1-3:1.0: USB hub found > >> Run /sbin/init as init process > >> hub 1-3:1.0: 3 ports detected > >> *** stack smashing detected ***: terminated > >> Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000004 > >> CPU: 1 PID: 1 Comm: init Not tainted 5.0.0-rc6-next-20190218 #1546 > >> Hardware name: SAMSUNG EXYNOS (Flattened Device Tree) > >> [] (unwind_backtrace) from [] (show_stack+0x10/0x14) > >> [] (show_stack) from [] (dump_stack+0x90/0xc8) > >> [] (dump_stack) from [] (panic+0xfc/0x304) > >> [] (panic) from [] (do_exit+0xabc/0xc6c) > >> [] (do_exit) from [] (do_group_exit+0x3c/0xbc) > >> [] (do_group_exit) from [] (get_signal+0x130/0xbf4) > >> [] (get_signal) from [] (do_work_pending+0x130/0x618) > >> [] (do_work_pending) from [] > >> (slow_work_pending+0xc/0x20) > >> Exception stack(0xe88c3fb0 to 0xe88c3ff8) > >> 3fa0:                                     00000000 bea7787c 00000005 > >> b6e8d0b8 > >> 3fc0: bea77a18 b6f92010 b6e8d0b8 00000001 b6e8d0c8 00000001 b6e8c000 > >> bea77b60 > >> 3fe0: 00000020 bea77998 ffffffff b6d52368 60000050 ffffffff > >> CPU3: stopping > >> > >> I would like to help debugging and fixing this issue, but I don't really > >> have idea where to start. Here are some more detailed information about > >> my test system: > >> > >> 1. Board: ARM 32bit Samsung Exynos4412-based Odroid U3 (device tree > >> source: arch/arm/boot/dts/exynos4412-odroidu3.dts) > >> > >> 2. Block device: MMC/SDHCI/SDHCI-S3C with SD card > >> (drivers/mmc/host/sdhci-s3c.c driver, sdhci_2 device node in the device > >> tree) > >> > >> 3. Rootfs: Ext4 > >> > >> 4. Kernel config: arch/arm/configs/exynos_defconfig > >> > >> I can gather more logs if needed, just let me which kernel option to > >> enable. Reverting this commit on top of next-20190218 as well as current > >> linux-next (tested with next-20190221) fixes this issue and makes the > >> system bootable again. > > Could you test the patch in following link and see if it can make a difference? > > > > https://marc.info/?l=linux-aio&m=155070355614541&w=2 > > I've tested that patch, but it doesn't make any difference on the test > system. In the log I see no warning added by it. I guess it might be related with memory corruption, could you enable the following debug options and post the dmesg log? CONFIG_DEBUG_STACKOVERFLOW=y CONFIG_KASAN=y Thanks, Ming