From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BEE88C433FE for ; Thu, 7 Apr 2022 00:27:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231822AbiDGA3r (ORCPT ); Wed, 6 Apr 2022 20:29:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33778 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231575AbiDGA3q (ORCPT ); Wed, 6 Apr 2022 20:29:46 -0400 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.220.29]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 99BDF4E398; Wed, 6 Apr 2022 17:27:48 -0700 (PDT) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 20D3E1F859; Thu, 7 Apr 2022 00:27:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1649291267; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=v3pW6Wkg4mFZYigeP74v+8mIq5ePJzzu1u51lT1zjcQ=; b=VcMcFLYuO6Zbk96saqYasfXMTJw2nTwyy3FRhy/SdxZ6rv7AaG3ZwM9FD9IKYyQA4BW6R1 or1TFcb8PfVgLjbmmjHEM91HJW17cLDav/LfsG20ttzeqRDWQHzdB/CJSow1SuLX1zP5FT OKzE0cnLsVdn12NWUp1ol5ovF7NOG9Y= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1649291267; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=v3pW6Wkg4mFZYigeP74v+8mIq5ePJzzu1u51lT1zjcQ=; b=Y/nzWcidoiH4397dxiunxniX3T/ko8ndFmzYp6hgZrJ+7GqyU06Lz804wmSFWVGvC9fB/C vIsa3/MSbt7i2EDg== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 02A1C13A9C; Thu, 7 Apr 2022 00:27:44 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id 8xmTKwAwTmK0bwAAMHmgww (envelope-from ); Thu, 07 Apr 2022 00:27:44 +0000 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit MIME-Version: 1.0 From: "NeilBrown" To: "Dave Chinner" , Trond Myklebust Cc: "J. Bruce Fields" , "Chuck Lever" , linux-nfs@vger.kernel.org, linux-fsdevel@vger.kernel.org Subject: Re: sporadic hangs on generic/186 In-reply-to: <20220407001453.GE1609613@dread.disaster.area> References: <20220406195424.GA1242@fieldses.org>, <20220407001453.GE1609613@dread.disaster.area> Date: Thu, 07 Apr 2022 10:27:41 +1000 Message-id: <164929126156.10985.11316778982526844125@noble.neil.brown.name> Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org On Thu, 07 Apr 2022, Dave Chinner wrote: > On Wed, Apr 06, 2022 at 03:54:24PM -0400, J. Bruce Fields wrote: > > In the last couple days I've started getting hangs on xfstests > > generic/186 on upstream. I also notice the test completes after 10+ > > hours (usually it takes about 5 minutes). Sometimes this is accompanied > > by "nfs: RPC call returned error 12" on the client. > > #define ENOMEM 12 /* Out of memory */ > > So either the client or the server is running out of memory > somewhere? Probably the client. There are a bunch of changes recently which add __GFP_NORETRY to memory allocations from PF_WQ_WORKERs because that can result in deadlocks when swapping over NFS. This means that kmalloc request that previously never failed (because GFP_KERNEL never fails for kernel threads I think) can now fail. This has tickled one bug that I know of. There are likely to be more. The RPC code should simply retry these allocations after a short delay. HZ/4 is the number that is used in a couple of places. Possibly there are more places that need to handle -ENOMEM with rpc_delay(). NeilBrown