From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-11.4 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1EA74C433E7 for ; Mon, 19 Oct 2020 02:33:47 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 8C82322269 for ; Mon, 19 Oct 2020 02:33:46 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="hV63/hxW" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8C82322269 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id C9FF96B005D; Sun, 18 Oct 2020 22:33:45 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C28276B0062; Sun, 18 Oct 2020 22:33:45 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AEEA86B0068; Sun, 18 Oct 2020 22:33:45 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 7A4B76B005D for ; Sun, 18 Oct 2020 22:33:45 -0400 (EDT) Received: from smtpin25.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 0305B824CDA4 for ; Mon, 19 Oct 2020 02:33:45 +0000 (UTC) X-FDA: 77387104410.25.net29_5011bd127233 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin25.hostedemail.com (Postfix) with ESMTP id D00601804E3A0 for ; Mon, 19 Oct 2020 02:33:44 +0000 (UTC) X-HE-Tag: net29_5011bd127233 X-Filterd-Recvd-Size: 7693 Received: from mail-wm1-f67.google.com (mail-wm1-f67.google.com [209.85.128.67]) by imf16.hostedemail.com (Postfix) with ESMTP for ; Mon, 19 Oct 2020 02:33:44 +0000 (UTC) Received: by mail-wm1-f67.google.com with SMTP id q5so11153504wmq.0 for ; Sun, 18 Oct 2020 19:33:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=VBN/6CP/+ltPqdamjKu98WNiNi7S6ZEKhupkah4Br1c=; b=hV63/hxW3sBUBXLnuV54MuJa2DdDe7YEv/dYeKTGNfQvDOv3Ui+Ggz7T7AAyMX2isX VdYbmBeSqXJqfT32pfF+YbRk4kino2tGi9nWzLmzqAEg7OSzMEFy/W4nqugmkBsU5nqx 07yGPNKxCy7WEch9pi0GKaOxelBIwUcW1xElYftBxEPtvfDOO8n36H1Fd980GZacGDEk VvT6+p+eo7oajZPaU5gfmDQ9boYl51YTWsN6CaC/kFLQ6Vkmj4JsnaKv20/xt40GtUlX ljvwOOOvDAzxehh1nSg8I9qavUIRGO8LwACuk3F7nrZCsFFiL6ELNehxggDAOacp4+xS 1t6g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=VBN/6CP/+ltPqdamjKu98WNiNi7S6ZEKhupkah4Br1c=; b=bz2XLHacYt7YkcfVYUL3+F8Nnjyh+7pma8RfJOXD+nc9++aimpEiDxGl4nZse76yCv SHtLQTtqkcml3BVXERZ1hQgBuWUoJaRThKmMMXQXBoCRKg9kEj/DDsduKMx4DrajZMXn av5Sb8+XnB+/hlTbAjdg+fvjLp8Qi9H+b2U+AUBwZjO5NlNhUbcOgWxLO5epQ4ZOmR1l bO0T42D8GvvOm4gmOI+xGv9G97OKXgLB4LQG0cCCFGYtT7XTLzSki2ZGs0i7QLSyMCEj 5jJL+4S6PMspR4ifCVzkJ/7WyJ9vqziQxOXaoOENuSiK4aXLtLd7NCsjILyFjeLhjxEe hHlA== X-Gm-Message-State: AOAM533Vpmc1ZrYdiEK42uF381wu1O0yW1Exe2Ie3ScrCmI+XnoulI5f vZNtAsg8/hoqW3S9rUQvfZpum8qgy0oZ1ViSsL3vfA== X-Google-Smtp-Source: ABdhPJzB5DOvyDDSp7xrHu51FkKWPOlUT1KhYRd62+NKzH8xN0RiVPg6E7qSaODPdlUXlyjb34W2tfyApr6feUN4rkY= X-Received: by 2002:a1c:e256:: with SMTP id z83mr14534843wmg.37.1603074822730; Sun, 18 Oct 2020 19:33:42 -0700 (PDT) MIME-Version: 1.0 References: <20200527201119.1692513-1-bigeasy@linutronix.de> <20200527201119.1692513-8-bigeasy@linutronix.de> <20201019015252.GA61728@google.com> In-Reply-To: <20201019015252.GA61728@google.com> From: Hugh Dickins Date: Sun, 18 Oct 2020 19:33:15 -0700 Message-ID: Subject: Re: [PATCH v3 7/7] zram: Use local lock to protect per-CPU data To: Yu Zhao Cc: Sebastian Andrzej Siewior , linux-kernel , Peter Zijlstra , Ingo Molnar , Steven Rostedt , Will Deacon , Thomas Gleixner , "Paul E . McKenney" , Linus Torvalds , Matthew Wilcox , Mike Galbraith , Minchan Kim , Nitin Gupta , Sergey Senozhatsky , linux-mm Content-Type: text/plain; charset="UTF-8" X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Sun, Oct 18, 2020 at 6:53 PM Yu Zhao wrote: > > On Wed, May 27, 2020 at 10:11:19PM +0200, Sebastian Andrzej Siewior wrote: > > From: Mike Galbraith > > > > The zcomp driver uses per-CPU compression. The per-CPU data pointer is > > acquired with get_cpu_ptr() which implicitly disables preemption. > > It allocates memory inside the preempt disabled region which conflicts > > with the PREEMPT_RT semantics. > > > > Replace the implicit preemption control with an explicit local lock. > > This allows RT kernels to substitute it with a real per CPU lock, which > > serializes the access but keeps the code section preemptible. On non RT > > kernels this maps to preempt_disable() as before, i.e. no functional > > change. > > Hi, > > This change seems to have introduced a potential deadlock. Can you > please take a look? Probably needs Peter's fix https://lore.kernel.org/lkml/20201016124009.GQ2611@hirez.programming.kicks-ass.net/ > > Thank you. > > [ 40.030778] ====================================================== > [ 40.037706] WARNING: possible circular locking dependency detected > [ 40.044637] 5.9.0-74216-g5c9472ed6825 #1 Tainted: G W > [ 40.051759] ------------------------------------------------------ > [ 40.058685] swapon/586 is trying to acquire lock: > [ 40.063950] ffffe8ffffc0ee60 (&zstrm->lock){+.+.}-{2:2}, at: local_lock_acquire+0x5/0x70 [zram] > [ 40.073739] > [ 40.073739] but task is already holding lock: > [ 40.080277] ffff888101a1f438 (&zspage->lock){.+.+}-{2:2}, at: zs_map_object+0x73/0x28d > [ 40.089182] > [ 40.089182] which lock already depends on the new lock. > [ 40.089182] > [ 40.098344] > [ 40.098344] the existing dependency chain (in reverse order) is: > [ 40.106715] > [ 40.106715] -> #1 (&zspage->lock){.+.+}-{2:2}: > [ 40.113386] lock_acquire+0x1cd/0x2c3 > [ 40.118083] _raw_read_lock+0x44/0x78 > [ 40.122781] zs_map_object+0x73/0x28d > [ 40.127479] zram_bvec_rw+0x42e/0x75d [zram] > [ 40.132855] zram_submit_bio+0x1fc/0x2d7 [zram] > [ 40.138526] submit_bio_noacct+0x11b/0x372 > [ 40.143709] submit_bio+0xfd/0x1b5 > [ 40.148113] __block_write_full_page+0x302/0x56f > [ 40.153877] __writepage+0x1e/0x74 > [ 40.158281] write_cache_pages+0x404/0x59a > [ 40.163461] generic_writepages+0x53/0x82 > [ 40.168545] do_writepages+0x33/0x74 > [ 40.173145] __filemap_fdatawrite_range+0x91/0xac > [ 40.179005] file_write_and_wait_range+0x39/0x87 > [ 40.184769] blkdev_fsync+0x19/0x3e > [ 40.189272] do_fsync+0x39/0x5c > [ 40.193384] __x64_sys_fsync+0x13/0x17 > [ 40.198178] do_syscall_64+0x37/0x45 > [ 40.202776] entry_SYSCALL_64_after_hwframe+0x44/0xa9 > [ 40.209022] > [ 40.209022] -> #0 (&zstrm->lock){+.+.}-{2:2}: > [ 40.215589] validate_chain+0x1966/0x21a8 > [ 40.220673] __lock_acquire+0x941/0xbba > [ 40.225552] lock_acquire+0x1cd/0x2c3 > [ 40.230250] local_lock_acquire+0x21/0x70 [zram] > [ 40.236015] zcomp_stream_get+0x33/0x4d [zram] > [ 40.241585] zram_bvec_rw+0x476/0x75d [zram] > [ 40.246963] zram_rw_page+0xd8/0x17c [zram] > [ 40.252240] bdev_read_page+0x7a/0x9d > [ 40.256933] do_mpage_readpage+0x6b2/0x860 > [ 40.262101] mpage_readahead+0x136/0x245 > [ 40.267089] read_pages+0x60/0x1f9 > [ 40.271492] page_cache_ra_unbounded+0x211/0x27b > [ 40.277251] generic_file_buffered_read+0x188/0xd4d > [ 40.283296] new_sync_read+0x10c/0x143 > [ 40.288088] vfs_read+0xf4/0x1a5 > [ 40.292285] ksys_read+0x73/0xd3 > [ 40.296483] do_syscall_64+0x37/0x45 > [ 40.301072] entry_SYSCALL_64_after_hwframe+0x44/0xa9 > [ 40.307319] > [ 40.307319] other info that might help us debug this: > [ 40.307319] > [ 40.316285] Possible unsafe locking scenario: > [ 40.316285] > [ 40.322907] CPU0 CPU1 > [ 40.327972] ---- ---- > [ 40.333041] lock(&zspage->lock); > [ 40.336874] lock(&zstrm->lock); > [ 40.343424] lock(&zspage->lock); > [ 40.350071] lock(&zstrm->lock); > [ 40.353803] > [ 40.353803] *** DEADLOCK *** >