From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.9 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1, USER_IN_DEF_DKIM_WL autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2972BC433E1 for ; Sun, 26 Jul 2020 20:30:53 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id B40F0206D8 for ; Sun, 26 Jul 2020 20:30:52 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="F0Y/7PYe" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B40F0206D8 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 130206B0002; Sun, 26 Jul 2020 16:30:52 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0E29F6B0003; Sun, 26 Jul 2020 16:30:52 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F13446B0005; Sun, 26 Jul 2020 16:30:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0094.hostedemail.com [216.40.44.94]) by kanga.kvack.org (Postfix) with ESMTP id D733E6B0002 for ; Sun, 26 Jul 2020 16:30:51 -0400 (EDT) Received: from smtpin14.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 6B5B21EE6 for ; Sun, 26 Jul 2020 20:30:51 +0000 (UTC) X-FDA: 77081370702.14.town33_0f0256026f5b Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin14.hostedemail.com (Postfix) with ESMTP id 3FEDA18229835 for ; Sun, 26 Jul 2020 20:30:51 +0000 (UTC) X-HE-Tag: town33_0f0256026f5b X-Filterd-Recvd-Size: 7795 Received: from mail-qk1-f193.google.com (mail-qk1-f193.google.com [209.85.222.193]) by imf36.hostedemail.com (Postfix) with ESMTP for ; Sun, 26 Jul 2020 20:30:50 +0000 (UTC) Received: by mail-qk1-f193.google.com with SMTP id g26so13558186qka.3 for ; Sun, 26 Jul 2020 13:30:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:from:to:cc:subject:in-reply-to:message-id:references :user-agent:mime-version; bh=eJ8N3m9hHBs2DJD7AGqPgD7eK30TwK2EzEcmU7vvZK0=; b=F0Y/7PYeOQN/imu6lHcT+ttXXOTbRaMB1OK3HA/5JWKeD6itG1Hf7Yr0bEtbxGz+4a hBNuZYLSFlQOr3i+ct6+B9bwbAPeWPeNyRNys4WiX4ubFK/A3ULA/2DLgWCfYpbCWh6M w27ps7vjILtS+jXWpt8g2PTvl8dHkHr2lNRnPCOm9aUdXN/qViAfSiecm4Ny/k6CHxip CKuR2EmgRyYACEs3QR8NGD53y7LhqaY6x0hilcBgheHsBUxoQwGxqA4iH7fEyxg6rsgF ai2JTaQG3nZs3v1DIzspKwPqgVR6+yJ/7w6/1XX0eAMXTzoPoV8NzwdVCKMDdqZGsQAy Y1WQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:in-reply-to:message-id :references:user-agent:mime-version; bh=eJ8N3m9hHBs2DJD7AGqPgD7eK30TwK2EzEcmU7vvZK0=; b=t6jdburXM35m/3q/PTvbzX9t3n2uWUMhJfSSLUQXS0TM3qOTVTxUGr3C5rGoELXwsI 7m0hVFelRXhqWhR6lbqcLtT6M32uI3Bej4RgpxdA60m9F4MAlCg7GFjTO849O9/mj//C Ff2Rj9g0ZbmxwfvJFMW6lgfxL84npzw44kpRWoWSkggE2YQ7/hr1NGK1aIfMughnYAVV Xqk9ZZnnVkK+1OrbBS0s8kuwNShnHZ+W9VarbdDEhCCBLxC9C7aMaephdwm9tJIooF0O wGMWr8aLrPpLDUaJ/nzjqEoaqVurbg9UDgP8F2P284T3j0/qCU/aXpFwpwDpv1ao4QEu 9JzA== X-Gm-Message-State: AOAM533x4BU4b5sHfiykWfJSkcoypy90HnEJqHBtH5CG29aOBOoYYXqx f0CSjIwfG7eijcnAVdkG2te/xQ== X-Google-Smtp-Source: ABdhPJyizh6pIuMdA1aNv4N31IYvAMG4NiMFI2wCeELIs+WEQHTandpa6FxJf9lqHa8pIvHW0CJo9A== X-Received: by 2002:a37:62d4:: with SMTP id w203mr19681461qkb.463.1595795449848; Sun, 26 Jul 2020 13:30:49 -0700 (PDT) Received: from eggly.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id w8sm16821218qka.52.2020.07.26.13.30.47 (version=TLS1 cipher=ECDHE-ECDSA-AES128-SHA bits=128/128); Sun, 26 Jul 2020 13:30:48 -0700 (PDT) Date: Sun, 26 Jul 2020 13:30:32 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@eggly.anvils To: Linus Torvalds cc: Hugh Dickins , Oleg Nesterov , Michal Hocko , Linux-MM , LKML , Andrew Morton , Tim Chen , Michal Hocko Subject: Re: [RFC PATCH] mm: silence soft lockups from unlock_page In-Reply-To: Message-ID: References: <20200723124749.GA7428@redhat.com> <20200724152424.GC17209@redhat.com> <20200725101445.GB3870@redhat.com> User-Agent: Alpine 2.11 (LSU 23 2013-08-11) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-Rspamd-Queue-Id: 3FEDA18229835 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam02 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Sat, 25 Jul 2020, Hugh Dickins wrote: > On Sat, 25 Jul 2020, Hugh Dickins wrote: > > On Sat, 25 Jul 2020, Linus Torvalds wrote: > > > On Sat, Jul 25, 2020 at 3:14 AM Oleg Nesterov wrote: > > > > > > > > Heh. I too thought about this. And just in case, your patch looks correct > > > > to me. But I can't really comment this behavioural change. Perhaps it > > > > should come in a separate patch? > > > > > > We could do that. At the same time, I think both parts change how the > > > waitqueue works that it might as well just be one "fix page_bit_wait > > > waitqueue usage". > > > > > > But let's wait to see what Hugh's numbers say. > > > > Oh no, no no: sorry for getting your hopes up there, I won't come up > > with any numbers more significant than "0 out of 10" machines crashed. > > I know it would be *really* useful if I could come up with performance > > comparisons, or steer someone else to do so: but I'm sorry, cannot. > > > > Currently it's actually 1 out of 10 machines crashed, for the same > > driverland issue seen last time, maybe it's a bad machine; and another > > 1 out of the 10 machines went AWOL for unknown reasons, but probably > > something outside the kernel got confused by the stress. No reason > > to suspect your changes at all (but some unanalyzed "failure"s, of > > dubious significance, accumulating like last time). > > > > I'm optimistic: nothing has happened to warn us off your changes. > > Less optimistic now, I'm afraid. > > The machine I said had (twice) crashed coincidentally in driverland > (some USB completion thing): that machine I set running a comparison > kernel without your changes this morning, while the others still > running with your changes; and it has now passed the point where it > twice crashed before (the most troublesome test), without crashing. > > Surprising: maybe still just coincidence, but I must look closer at > the crashes. > > The others have now completed, and one other crashed in that > troublesome test, but sadly without yielding any crash info. > > I've just set comparison runs going on them all, to judge whether > to take the "failure"s seriously; and I'll look more closely at them. The comparison runs have not yet completed (except for the one started early), but they have all got past the most interesting tests, and it's clear that they do not have the "failure"s seen with your patches. >From that I can only conclude that your patches make a difference. I've deduced nothing useful from the logs, will have to leave that to others here with more experience of them. But my assumption now is that you have successfully removed one bottleneck, so the tests get somewhat further and now stick in the next bottleneck, whatever that may be. Which shows up as "failure", where the unlock_page() wake_up_page_bit() bottleneck had allowed the tests to proceed in a more serially sedate way. The xhci handle_cmd_completion list_del bugs (on an older version of the driver): weird, nothing to do with page wakeups, I'll just have to assume that it's some driver bug exposed by the greater stress allowed down, and let driver people investigate (if it still manifests) when we take in your improvements. One nice thing from the comparison runs without your patches: watchdog panic did crash one of those with exactly the unlock_page() wake_up_page_bit() softlockup symptom we've been fighting, that did not appear with your patches. So although the sample size is much too small to justify a conclusion, it does tend towards confirming your changes. Thank you for your work on this! And I'm sure you'd have preferred some hard data back, rather than a diary of my mood swings, but... we do what we can. Hugh