From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5CEFEC433DB for ; Mon, 1 Mar 2021 18:39:40 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 37234601FC for ; Mon, 1 Mar 2021 18:39:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240249AbhCASiV (ORCPT ); Mon, 1 Mar 2021 13:38:21 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39140 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233299AbhCAQkQ (ORCPT ); Mon, 1 Mar 2021 11:40:16 -0500 Received: from mail-lf1-x134.google.com (mail-lf1-x134.google.com [IPv6:2a00:1450:4864:20::134]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DBD13C06178B for ; Mon, 1 Mar 2021 08:39:35 -0800 (PST) Received: by mail-lf1-x134.google.com with SMTP id z11so26473014lfb.9 for ; Mon, 01 Mar 2021 08:39:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=8RorKlKUIwAeZ2vu8I46zt7CBWlrUNEpLpF3jdorkm8=; b=Hx5rm6bgWnW2rzcg2ftQXDvzUq9+b9iZL+48q+8+FVQIhe6LJSSulGtfBsHTS3gkns 4oDOY3TvQLXS9UMnfUpKBGeLsmOQoJojGPdx8k3ndakA4m87/EgWxdg5pSOAdnG+g9Bn mzK0T+tRjpSvPlBF/8obIi7KyX0lPLpsKMm6yi+wP4Gx9klCmizayy1TpFqUR4wFLqwi heoKUIkdL09+MasjGb9IM0eI0alY97t9dxVgdKvPxZ9j9ZupH53PSBcQ/g2E9ixxeVTM rEB7rFK05UxoTMMiy78IXLK2TmBvobe9M3FA6wc3TKZg3JyjAUm3L6WIW3u2PniUcELi uVTQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=8RorKlKUIwAeZ2vu8I46zt7CBWlrUNEpLpF3jdorkm8=; b=uLPe3BfcnH6zOWaABXmFRAti9r4Tmanx7tWU3oz3QO0Arm5szuvi9YeP7tGcyhGN6I vMtdELpyBzCTc9gPdt3Tr1QxgE9GX94I09kSOgOkDmlYX2RmB6EvZqiK6sMMkMGzKsko Jl+As1/MfmA/6CVSvFmSU8sy+GPAbO+9Yes9YcBt0GehRIzkDFcd1k4o0BWLT6qWecE5 i3etJegnl4Z7NOoXgnt3r6vhW2965u+azvDvzXiKCR93i+T6p4ksARr3dSeyad+PUncH a5vhFbDUpj3TTiLKXp0AHKxLGfvPAtBsp1V51ybI9DxMrfjn9T8j6WyKapnV1tvUnwfy CCOA== X-Gm-Message-State: AOAM533DaA0+pU1QlqbFaX0v2gnKdgX0Cw6lqxkDf0xJl6s5A9RHHrSO /8OaRGm75v2p6jWz6NLB4mfuy+QD2nOmyo4HVnyODA== X-Google-Smtp-Source: ABdhPJx0KbzhJIVoM4/GcjChCBVWTV/vBF4C/e1Kdb2FPa+us+EDkJoa0wsgIaAANgRdocbZkBVKQCHsKYU/dtN/IeA= X-Received: by 2002:a05:6512:942:: with SMTP id u2mr10175495lft.117.1614616774183; Mon, 01 Mar 2021 08:39:34 -0800 (PST) MIME-Version: 1.0 References: <000000000000f1c03b05bc43aadc@google.com> <7b7c4f41-b72e-840f-278a-320b9d97f887@oracle.com> In-Reply-To: From: Shakeel Butt Date: Mon, 1 Mar 2021 08:39:22 -0800 Message-ID: Subject: Re: possible deadlock in sk_clone_lock To: Michal Hocko Cc: Mike Kravetz , syzbot , Andrew Morton , LKML , Linux MM , syzkaller-bugs , Eric Dumazet , Mina Almasry Content-Type: text/plain; charset="UTF-8" Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Mon, Mar 1, 2021 at 7:57 AM Michal Hocko wrote: > > On Mon 01-03-21 07:10:11, Shakeel Butt wrote: > > On Mon, Mar 1, 2021 at 4:12 AM Michal Hocko wrote: > > > > > > On Fri 26-02-21 16:00:30, Shakeel Butt wrote: > > > > On Fri, Feb 26, 2021 at 3:14 PM Mike Kravetz wrote: > > > > > > > > > > Cc: Michal > > > > > > > > > > On 2/26/21 2:44 PM, Shakeel Butt wrote: > > > > > > On Fri, Feb 26, 2021 at 2:09 PM syzbot > > > > > > wrote: > > > > > > > > > > >> other info that might help us debug this: > > > > > >> > > > > > >> Possible interrupt unsafe locking scenario: > > > > > >> > > > > > >> CPU0 CPU1 > > > > > >> ---- ---- > > > > > >> lock(hugetlb_lock); > > > > > >> local_irq_disable(); > > > > > >> lock(slock-AF_INET); > > > > > >> lock(hugetlb_lock); > > > > > >> > > > > > >> lock(slock-AF_INET); > > > > > >> > > > > > >> *** DEADLOCK *** > > > > > > > > > > > > This has been reproduced on 4.19 stable kernel as well [1] and there > > > > > > is a reproducer as well. > > > > > > > > > > > > It seems like sendmsg(MSG_ZEROCOPY) from a buffer backed by hugetlb. I > > > > > > wonder if we just need to make hugetlb_lock softirq-safe. > > > > > > > > > > > > [1] https://syzkaller.appspot.com/bug?extid=6383ce4b0b8ec575ad93 > > > > > > > > > > Thanks Shakeel, > > > > > > > > > > Commit c77c0a8ac4c5 ("mm/hugetlb: defer freeing of huge pages if in non-task > > > > > context") attempted to address this issue. It uses a work queue to > > > > > acquire hugetlb_lock if the caller is !in_task(). > > > > > > > > > > In another recent thread, there was the suggestion to change the > > > > > !in_task to in_atomic. > > > > > > > > > > I need to do some research on the subtle differences between in_task, > > > > > in_atomic, etc. TBH, I 'thought' !in_task would prevent the issue > > > > > reported here. But, that obviously is not the case. > > > > > > > > I think the freeing is happening in the process context in this report > > > > but it is creating the lock chain from softirq-safe slock to > > > > irq-unsafe hugetlb_lock. So, two solutions I can think of are: (1) > > > > always defer the freeing of hugetlb pages to a work queue or (2) make > > > > hugetlb_lock softirq-safe. > > > > > > There is __do_softirq so this should be in the soft IRQ context no? > > > Is this really reproducible with kernels which have c77c0a8ac4c5 > > > applied? > > > > Yes this is softirq context and syzbot has reproduced this on > > linux-next 20210224. > > Then how come this can ever be a problem? in_task() should exclude soft > irq context unless I am mistaken. > If I take the following example of syzbot's deadlock scenario then CPU1 is the one freeing the hugetlb pages. It is in the process context but has disabled softirqs (see __tcp_close()). CPU0 CPU1 ---- ---- lock(hugetlb_lock); local_irq_disable(); lock(slock-AF_INET); lock(hugetlb_lock); lock(slock-AF_INET); So, this deadlock scenario is very much possible. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C0723C433E0 for ; Mon, 1 Mar 2021 16:39:52 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 4865664FFE for ; Mon, 1 Mar 2021 16:39:52 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4865664FFE Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 948AE8D0087; Mon, 1 Mar 2021 11:39:51 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 8D1C28D0063; Mon, 1 Mar 2021 11:39:51 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 798F78D0087; Mon, 1 Mar 2021 11:39:51 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0099.hostedemail.com [216.40.44.99]) by kanga.kvack.org (Postfix) with ESMTP id 5E8318D0063 for ; Mon, 1 Mar 2021 11:39:51 -0500 (EST) Received: from smtpin20.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 1B78E8151 for ; Mon, 1 Mar 2021 16:39:51 +0000 (UTC) X-FDA: 77871866982.20.DE435D6 Received: from mail-lf1-f41.google.com (mail-lf1-f41.google.com [209.85.167.41]) by imf07.hostedemail.com (Postfix) with ESMTP id 03A8BA0049CE for ; Mon, 1 Mar 2021 16:39:45 +0000 (UTC) Received: by mail-lf1-f41.google.com with SMTP id b1so15743481lfb.7 for ; Mon, 01 Mar 2021 08:39:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=8RorKlKUIwAeZ2vu8I46zt7CBWlrUNEpLpF3jdorkm8=; b=Hx5rm6bgWnW2rzcg2ftQXDvzUq9+b9iZL+48q+8+FVQIhe6LJSSulGtfBsHTS3gkns 4oDOY3TvQLXS9UMnfUpKBGeLsmOQoJojGPdx8k3ndakA4m87/EgWxdg5pSOAdnG+g9Bn mzK0T+tRjpSvPlBF/8obIi7KyX0lPLpsKMm6yi+wP4Gx9klCmizayy1TpFqUR4wFLqwi heoKUIkdL09+MasjGb9IM0eI0alY97t9dxVgdKvPxZ9j9ZupH53PSBcQ/g2E9ixxeVTM rEB7rFK05UxoTMMiy78IXLK2TmBvobe9M3FA6wc3TKZg3JyjAUm3L6WIW3u2PniUcELi uVTQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=8RorKlKUIwAeZ2vu8I46zt7CBWlrUNEpLpF3jdorkm8=; b=XqF+0ycK8+5cg/iAq2JJyrtyTrxfeB9ChazkoaxAgN+wpe1VeUMmtyKXLiC0t0XxK9 YkGimcWyYadYAGEfcszHzT+SpMD0PSixG/MHePHYpZ80FsY/yJ+h6LCAass0aBRpRakd hTTZUXshX0Xul/xZb2f5c+KJeOUb0xiHzREot1DecI1J27p6ReUjXscmEERCabYitg9E KtTz8cEWqgsy8Kr6ppnGyiEWXk5/MR0TzE+cjGkmw+WKguK+A3Nk97oIme0Eegi8fNIp h0YbUqRNsz6YyEyST6GxsCI/05riVF508HgEIC80sD320SFW1EVRWCyyzbkai+FeujxO Kf8g== X-Gm-Message-State: AOAM531sxsAqbEuZI2mZqGDjiPVxZYUyBfako8q8fJbOFIaWG1Rynw7q YPlx+5QYcGDY/PTJ9PmhzEOua9mRLK6vmfOqnY5cWg== X-Google-Smtp-Source: ABdhPJx0KbzhJIVoM4/GcjChCBVWTV/vBF4C/e1Kdb2FPa+us+EDkJoa0wsgIaAANgRdocbZkBVKQCHsKYU/dtN/IeA= X-Received: by 2002:a05:6512:942:: with SMTP id u2mr10175495lft.117.1614616774183; Mon, 01 Mar 2021 08:39:34 -0800 (PST) MIME-Version: 1.0 References: <000000000000f1c03b05bc43aadc@google.com> <7b7c4f41-b72e-840f-278a-320b9d97f887@oracle.com> In-Reply-To: From: Shakeel Butt Date: Mon, 1 Mar 2021 08:39:22 -0800 Message-ID: Subject: Re: possible deadlock in sk_clone_lock To: Michal Hocko Cc: Mike Kravetz , syzbot , Andrew Morton , LKML , Linux MM , syzkaller-bugs , Eric Dumazet , Mina Almasry Content-Type: text/plain; charset="UTF-8" X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 03A8BA0049CE X-Stat-Signature: gf65uzq1iguxmpezbgrd3psotxeg593f Received-SPF: none (google.com>: No applicable sender policy available) receiver=imf07; identity=mailfrom; envelope-from=""; helo=mail-lf1-f41.google.com; client-ip=209.85.167.41 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1614616785-438318 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Mon, Mar 1, 2021 at 7:57 AM Michal Hocko wrote: > > On Mon 01-03-21 07:10:11, Shakeel Butt wrote: > > On Mon, Mar 1, 2021 at 4:12 AM Michal Hocko wrote: > > > > > > On Fri 26-02-21 16:00:30, Shakeel Butt wrote: > > > > On Fri, Feb 26, 2021 at 3:14 PM Mike Kravetz wrote: > > > > > > > > > > Cc: Michal > > > > > > > > > > On 2/26/21 2:44 PM, Shakeel Butt wrote: > > > > > > On Fri, Feb 26, 2021 at 2:09 PM syzbot > > > > > > wrote: > > > > > > > > > > >> other info that might help us debug this: > > > > > >> > > > > > >> Possible interrupt unsafe locking scenario: > > > > > >> > > > > > >> CPU0 CPU1 > > > > > >> ---- ---- > > > > > >> lock(hugetlb_lock); > > > > > >> local_irq_disable(); > > > > > >> lock(slock-AF_INET); > > > > > >> lock(hugetlb_lock); > > > > > >> > > > > > >> lock(slock-AF_INET); > > > > > >> > > > > > >> *** DEADLOCK *** > > > > > > > > > > > > This has been reproduced on 4.19 stable kernel as well [1] and there > > > > > > is a reproducer as well. > > > > > > > > > > > > It seems like sendmsg(MSG_ZEROCOPY) from a buffer backed by hugetlb. I > > > > > > wonder if we just need to make hugetlb_lock softirq-safe. > > > > > > > > > > > > [1] https://syzkaller.appspot.com/bug?extid=6383ce4b0b8ec575ad93 > > > > > > > > > > Thanks Shakeel, > > > > > > > > > > Commit c77c0a8ac4c5 ("mm/hugetlb: defer freeing of huge pages if in non-task > > > > > context") attempted to address this issue. It uses a work queue to > > > > > acquire hugetlb_lock if the caller is !in_task(). > > > > > > > > > > In another recent thread, there was the suggestion to change the > > > > > !in_task to in_atomic. > > > > > > > > > > I need to do some research on the subtle differences between in_task, > > > > > in_atomic, etc. TBH, I 'thought' !in_task would prevent the issue > > > > > reported here. But, that obviously is not the case. > > > > > > > > I think the freeing is happening in the process context in this report > > > > but it is creating the lock chain from softirq-safe slock to > > > > irq-unsafe hugetlb_lock. So, two solutions I can think of are: (1) > > > > always defer the freeing of hugetlb pages to a work queue or (2) make > > > > hugetlb_lock softirq-safe. > > > > > > There is __do_softirq so this should be in the soft IRQ context no? > > > Is this really reproducible with kernels which have c77c0a8ac4c5 > > > applied? > > > > Yes this is softirq context and syzbot has reproduced this on > > linux-next 20210224. > > Then how come this can ever be a problem? in_task() should exclude soft > irq context unless I am mistaken. > If I take the following example of syzbot's deadlock scenario then CPU1 is the one freeing the hugetlb pages. It is in the process context but has disabled softirqs (see __tcp_close()). CPU0 CPU1 ---- ---- lock(hugetlb_lock); local_irq_disable(); lock(slock-AF_INET); lock(hugetlb_lock); lock(slock-AF_INET); So, this deadlock scenario is very much possible.