From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.1 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1, USER_IN_DEF_DKIM_WL autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 495AFC433DF for ; Thu, 6 Aug 2020 11:38:25 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id D3D4C22CF6 for ; Thu, 6 Aug 2020 11:38:24 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="oyj/A4qt" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D3D4C22CF6 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id C6B8A6B0002; Thu, 6 Aug 2020 01:21:35 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BF4CA6B0005; Thu, 6 Aug 2020 01:21:35 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id ABC876B0006; Thu, 6 Aug 2020 01:21:35 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0247.hostedemail.com [216.40.44.247]) by kanga.kvack.org (Postfix) with ESMTP id 93A556B0002 for ; Thu, 6 Aug 2020 01:21:35 -0400 (EDT) Received: from smtpin30.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 3E7283624 for ; Thu, 6 Aug 2020 05:21:35 +0000 (UTC) X-FDA: 77118996150.30.truck97_380052d26fb5 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin30.hostedemail.com (Postfix) with ESMTP id 1798B180B3C83 for ; Thu, 6 Aug 2020 05:21:35 +0000 (UTC) X-HE-Tag: truck97_380052d26fb5 X-Filterd-Recvd-Size: 7984 Received: from mail-oi1-f195.google.com (mail-oi1-f195.google.com [209.85.167.195]) by imf33.hostedemail.com (Postfix) with ESMTP for ; Thu, 6 Aug 2020 05:21:34 +0000 (UTC) Received: by mail-oi1-f195.google.com with SMTP id e6so21585486oii.4 for ; Wed, 05 Aug 2020 22:21:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:from:to:cc:subject:in-reply-to:message-id:references :user-agent:mime-version; bh=NawncW6kcN7J/8qHcLn2vH5R5YQS2/laJgJm6p/2YCk=; b=oyj/A4qtZQFmUNx3d8X6GCw9WEuOghnm1XwugQyROixks1XkjMb3pWc838vhBkvDR+ dQ4776yWmKpUGQKAR/vPQWeCIcnXjSdbqeaMRKbakwt6AEf94LgWKI1IyCg82bM38xKV V7/eANJ7VR9Lb8Y5Tlv7QYbXRG4p6FEPFNaZHUKDKGasYuXfp+t501S28LB9KgmB45Gf 5c6BuP0WDKhsJEYi2P5UIwlvfsrzmob+gsJwOZgqgGCL36oOb1v64lzzvLvvboVs3Xst FeIIw7QihwkcTJNwXuQh7N/mRKJXi9E20nesrqi05PydTk8ZyPueKMlil21ChOC9pbD8 YmIA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:in-reply-to:message-id :references:user-agent:mime-version; bh=NawncW6kcN7J/8qHcLn2vH5R5YQS2/laJgJm6p/2YCk=; b=rReiQ46ht1DOr/GWU2ac8UtfWBk8Bzb0SgtaYhTHySeNkU4DIo+NHzz1im7lBJTYP7 FaJj/zlkk4Nw2IsLSNOWbuqY5VUikBjDxKeCPUr4o9SNx6t2Xfu01WO0rkKPqWELkYoa atjbeqTP6lXn/eA/eH71/MNUafAcP2QfCVqfFzuIDTFQdqUq184eIHjc/itw3AXsIA4B ZhyGvwHeaqGRaArfJwtOH3x+LYP3JiM3DYjU0xnprcRStnZ6rUKrNRjCU5i2PU+kfljz oXfBXOIurQFqylANsW1oiD90T+cxbxsp3r8stmjC/Lane7rlsUpDrPiE+fhgueu/fPeD pw1Q== X-Gm-Message-State: AOAM532SAYnYOb4Pq7sNywWXtw8CxokYEl/Cy2SVxo+DZ5VKmRgkHj7s SthkX6mtidhI1J8ec+Tz7XhmlQ== X-Google-Smtp-Source: ABdhPJwP0SUs4y6xBmSM+VlR4LZd/gb+f0cyUOs/WRnv2k3YOa6BUmM28Erezs7jF6n/63Hheq1okQ== X-Received: by 2002:aca:5585:: with SMTP id j127mr5299089oib.120.1596691293575; Wed, 05 Aug 2020 22:21:33 -0700 (PDT) Received: from eggly.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id t83sm945595oot.22.2020.08.05.22.21.31 (version=TLS1 cipher=ECDHE-ECDSA-AES128-SHA bits=128/128); Wed, 05 Aug 2020 22:21:32 -0700 (PDT) Date: Wed, 5 Aug 2020 22:21:18 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@eggly.anvils To: Linus Torvalds cc: Hugh Dickins , Oleg Nesterov , Michal Hocko , Linux-MM , LKML , Andrew Morton , Tim Chen , Michal Hocko , Greg KH Subject: Re: [RFC PATCH] mm: silence soft lockups from unlock_page In-Reply-To: Message-ID: References: <20200723124749.GA7428@redhat.com> <20200724152424.GC17209@redhat.com> <20200725101445.GB3870@redhat.com> User-Agent: Alpine 2.11 (LSU 23 2013-08-11) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-Rspamd-Queue-Id: 1798B180B3C83 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam05 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Nice to see the +130.0% this morning. I got back on to this on Monday, here's some follow-up. On Sun, 26 Jul 2020, Hugh Dickins wrote: > > The comparison runs have not yet completed (except for the one started > early), but they have all got past the most interesting tests, and it's > clear that they do not have the "failure"s seen with your patches. > > From that I can only conclude that your patches make a difference. > > I've deduced nothing useful from the logs, will have to leave that > to others here with more experience of them. But my assumption now > is that you have successfully removed one bottleneck, so the tests > get somewhat further and now stick in the next bottleneck, whatever > that may be. Which shows up as "failure", where the unlock_page() > wake_up_page_bit() bottleneck had allowed the tests to proceed in > a more serially sedate way. Yes, that's still how it appears to me. The test failures, all of them, came from fork() returning ENOSPC, which originated from alloc_pid()'s idr_alloc_cyclic(). I did try doubling our already large pid_max, that did not work out well, there are probably good reasons for it to be where it is and I was wrong to dabble with it. I also tried an rcu_barrier() and retry when getting -ENOSPC there, thinking maybe RCU was not freeing them up fast enough, but that didn't help either. I think (but didn't quite make the effort to double-check with an independent count) it was simply running out of pids: that your change speeds up the forking enough, that exiting could not quite keep up (SIGCHLD is SIG_IGNed); whereas before your change, the unlock_page() in do_wp_page(), on a PageAnon stack page, slowed the forking down enough when heavily contended. (I think we could improve the checks there, to avoid taking page lock in more cases; but I don't know if that would help any real-life workload - I see now that Michal's case is do_read_fault() not do_wp_page().) And FWIW a further speedup there is the opposite of what these tests are wanting: for the moment I've enabled a delay to get them passing as before. Something I was interested to realize in looking at this: trylock_page() on a contended lock is now much less likely to jump the queue and succeed than before, since your lock holder hands off the page lock to the next holder: much smaller window than waiting for the next to wake to take it. Nothing wrong with that, but effect might be seen somewhere. > > The xhci handle_cmd_completion list_del bugs (on an older version > of the driver): weird, nothing to do with page wakeups, I'll just > have to assume that it's some driver bug exposed by the greater > stress allowed down, and let driver people investigate (if it > still manifests) when we take in your improvements. Complete red herring. I'll give Greg more info in response to his mail, and there may be an xhci bug in there; but when I looked back, found I'd come across the same bug back in October, and find that occasionally it's been seen in our fleet. Yes, it's odd that your change coincided with it becoming more common on that machine (which I've since replaced by another), yes it's funny that it's in __list_del_entry_valid(), which is exactly where I got crashes on pages with your initial patch; but it's just a distraction. > > One nice thing from the comparison runs without your patches: > watchdog panic did crash one of those with exactly the unlock_page() > wake_up_page_bit() softlockup symptom we've been fighting, that did > not appear with your patches. So although the sample size is much > too small to justify a conclusion, it does tend towards confirming > your changes. > > Thank you for your work on this! And I'm sure you'd have preferred > some hard data back, rather than a diary of my mood swings, but... > we do what we can. > > Hugh