From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.0 required=3.0 tests=BAYES_00,INCLUDES_PATCH, MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id EB747C433DF for ; Tue, 21 Jul 2020 06:33:33 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id AF14020B1F for ; Tue, 21 Jul 2020 06:33:33 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org AF14020B1F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 10D3A6B0007; Tue, 21 Jul 2020 02:33:33 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0BE806B0008; Tue, 21 Jul 2020 02:33:33 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F15EC8D0001; Tue, 21 Jul 2020 02:33:32 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0177.hostedemail.com [216.40.44.177]) by kanga.kvack.org (Postfix) with ESMTP id DB9726B0007 for ; Tue, 21 Jul 2020 02:33:32 -0400 (EDT) Received: from smtpin06.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 8B93418034A60 for ; Tue, 21 Jul 2020 06:33:32 +0000 (UTC) X-FDA: 77061116664.06.grape61_5008ddc26f2b Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin06.hostedemail.com (Postfix) with ESMTP id 5C0AA1003E0FE for ; Tue, 21 Jul 2020 06:33:32 +0000 (UTC) X-HE-Tag: grape61_5008ddc26f2b X-Filterd-Recvd-Size: 4925 Received: from mail-wr1-f65.google.com (mail-wr1-f65.google.com [209.85.221.65]) by imf26.hostedemail.com (Postfix) with ESMTP for ; Tue, 21 Jul 2020 06:33:31 +0000 (UTC) Received: by mail-wr1-f65.google.com with SMTP id f2so19950502wrp.7 for ; Mon, 20 Jul 2020 23:33:31 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=tV1LRzNgPSEgDwGe9LGBIejRICI799y9RAnDI8eKpbg=; b=Gx3QUGnEGJNfj/dxxDaLZISxvlKiygxaI3Oo2lmeEH19UvVar1G8+8cRElfqfWtZMC 5hDr2bEXQdvyCR6dLYhU5nu4rknX5HtfGYSMQgGcuUBC7s/AoMZh9BFeaTxsOc3P/Dw3 hJCtFW6QpJh7FmEDnSVSitVncXxxxiArTjNu+fFGRWRg9485lwwwWLaWyQm/aftgwk8L z+u7kG5afBvhmz0OxXBotAeTo62u5kZqSZFnxVvE3Htl0snzpLGZ34mEwxO+xBOO0k4f RwzXpf6jSQCOGaTJvQ9PgtMNTEsoW9MGT7vxKvtyC3s5S1NWN8vQvO73XB5Vxdqw7gPX t5ow== X-Gm-Message-State: AOAM532TPSXKRCNP10aJ4AHRTGOXyp93+bVKIUGOnGwrn6SDwORTkwFE HUsPc2dwn3U4JfhOAf7JgVV/jepu X-Google-Smtp-Source: ABdhPJxiB/dYqgFor+GO+S7YCh7ZoYChxfsvo91Dw+1AOToJGHEczK4ZX3skesAUKaqixM9i4q+sjQ== X-Received: by 2002:a5d:6702:: with SMTP id o2mr462986wru.364.1595313210451; Mon, 20 Jul 2020 23:33:30 -0700 (PDT) Received: from tiehlicka.suse.cz (ip-37-188-169-187.eurotel.cz. [37.188.169.187]) by smtp.gmail.com with ESMTPSA id q7sm36617204wra.56.2020.07.20.23.33.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 20 Jul 2020 23:33:29 -0700 (PDT) From: Michal Hocko To: Cc: LKML , Andrew Morton , Linus Torvalds , Tim Chen , Michal Hocko Subject: [RFC PATCH] mm: silence soft lockups from unlock_page Date: Tue, 21 Jul 2020 08:32:58 +0200 Message-Id: <20200721063258.17140-1-mhocko@kernel.org> X-Mailer: git-send-email 2.27.0 MIME-Version: 1.0 X-Rspamd-Queue-Id: 5C0AA1003E0FE X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam03 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Michal Hocko We have seen a bug report with huge number of soft lockups during the system boot on !PREEMPT kernel NMI watchdog: BUG: soft lockup - CPU#1291 stuck for 22s! [systemd-udevd:4= 3283] [...] NIP [c00000000094e66c] _raw_spin_lock_irqsave+0xac/0x100 LR [c00000000094e654] _raw_spin_lock_irqsave+0x94/0x100 Call Trace: [c00002293d883b30] [c0000000012cdee8] __pmd_index_size+0x0/0x8 (unreliabl= e) [c00002293d883b70] [c0000000002b7490] wake_up_page_bit+0xc0/0x150 [c00002293d883bf0] [c00000000030ceb8] do_fault+0x448/0x870 [c00002293d883c40] [c000000000310080] __handle_mm_fault+0x880/0x16b0 [c00002293d883d10] [c000000000311018] handle_mm_fault+0x168/0x250 [c00002293d883d50] [c00000000006c488] do_page_fault+0x568/0x8d0 [c00002293d883e30] [c00000000000a534] handle_page_fault+0x18/0x38 on a large ppc machine. The very likely cause is a suboptimal configuration when systed-udev spawns way too many workders to bring the system up. The lockup is in page_unlock in do_read_fault and I suspect that this is yet another effect of a very long waitqueue chain which has been addresses by 11a19c7b099f ("sched/wait: Introduce wakeup boomark in wake_up_page_bit") previously. The commit primarily aimed at hard lockup prevention but it doesn't really help !PREEMPT case which still has to process all the work without any rescheduling point. This is however not really trivial because page_unlock is called from many contexts many of which are likely called from an atomic context. Introducing page_unlock_sleepable is certainly an option but it seems like a hard to maintain option which doesn't really fix the underlying problem as the same might happen from other unlock_page callers. This patch doesn't address the underlying problem but it reduces the visible effect. Tell the soft lockup to shut up when retrying batches on queued waiters. This will also allow systems configured to panic on warning to proceed with the boot. Signed-off-by: Michal Hocko --- mm/filemap.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/mm/filemap.c b/mm/filemap.c index 385759c4ce4b..74681c40a6e5 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -41,6 +41,7 @@ #include #include #include +#include #include "internal.h" =20 #define CREATE_TRACE_POINTS @@ -1055,6 +1056,7 @@ static void wake_up_page_bit(struct page *page, int= bit_nr) */ spin_unlock_irqrestore(&q->lock, flags); cpu_relax(); + touch_softlockup_watchdog(); spin_lock_irqsave(&q->lock, flags); __wake_up_locked_key_bookmark(q, TASK_NORMAL, &key, &bookmark); } --=20 2.27.0