From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 68D7CC32772 for ; Wed, 17 Aug 2022 11:57:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239086AbiHQL5d (ORCPT ); Wed, 17 Aug 2022 07:57:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51248 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238700AbiHQL5b (ORCPT ); Wed, 17 Aug 2022 07:57:31 -0400 Received: from wnew4-smtp.messagingengine.com (wnew4-smtp.messagingengine.com [64.147.123.18]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D247782D3B; Wed, 17 Aug 2022 04:57:30 -0700 (PDT) Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailnew.west.internal (Postfix) with ESMTP id 830E72B059F6; Wed, 17 Aug 2022 07:57:29 -0400 (EDT) Received: from imap50 ([10.202.2.100]) by compute3.internal (MEProxy); Wed, 17 Aug 2022 07:57:30 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= colorremedies.com; h=cc:cc:content-transfer-encoding :content-type:date:date:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to; s=fm1; t=1660737449; x=1660741049; bh=nMczVAGpca 7H++XwV1RmIcObkzLoHhiXbmBHdwyghyU=; b=WLdp1RQQHgQVxczuf7pW15tSAX 2fY2owzazhdQ4Vag4SdC9OvQD8X0ecFf+Jh6e8dyXMnXQuMPp/BdL+jyeMnuOE4H OPnuYzkxkoycBiFkTuCPpA1UXkQ9QdD/Dox/EZ+pXrpaTck4gM7ixADCvMqyuSMd Z0DxqNifJ4mCdf+mow2fZTea6WzwqGYe3tYwdI7gu8iPV7JJYqmIVBPsO0keMjWQ /0Y7t79djaGZTlH9vwhfsT8EAcZhY4/yQ1MUG8wwd2ENASQ0Rge0IG/taXYjwcaP FB69dJMdB3vd398ro1EN64h4y1T4rPL2Q/l+OONmI1Z/JnP8AXYl/OSWIBfA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:sender:subject:subject:to:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t=1660737449; x= 1660741049; bh=nMczVAGpca7H++XwV1RmIcObkzLoHhiXbmBHdwyghyU=; b=R iAymQvL3bRE/7i5N7wzVWFFiIJBHAFb9Iu1SgWK8qjROI9K64jqeK8ZKnhYQZyYc E9FctukwQssAcs4io5G+Jl26u3y3uOV/DrRmHiP1+rdSQI0sxSiofx0G12aApgls 1iWyAfjYhXJkKLtg8UIIwYDshHkyGJLPnedA+BfpDtKhBtDTNcHjWf3ece6tSOla +V3kY2eVhXAceF72W+Xkpwaw7Y79WUh2ec7tTlXK22PQ6lJpNwipV1GifC/D20yD 5N07Zu0GJ0jZU9Gm2Md5ntVrhFtHR2nRVeNZeYrMEcoHpGyqADP4Vwav26oCD2/h eNREEVANYr+vdC9Fj1fLg== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvfedrvdehiedggeeiucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepofgfggfkjghffffhvfevufgtgfesthhqredtreerjeenucfhrhhomhepfdev hhhrihhsucfouhhrphhhhidfuceolhhishhtshestgholhhorhhrvghmvgguihgvshdrtg homheqnecuggftrfgrthhtvghrnhepveeghfeiveehvdeukedvtedtfefhuddtleekffek keefheeivdehgfffleeileehnecuffhomhgrihhnpehkvghrnhgvlhdrohhrghenucevlh hushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehlihhsthhssegt ohhlohhrrhgvmhgvughivghsrdgtohhm X-ME-Proxy: Feedback-ID: i06494636:Fastmail Received: by mailuser.nyi.internal (Postfix, from userid 501) id 761291700082; Wed, 17 Aug 2022 07:57:28 -0400 (EDT) X-Mailer: MessagingEngine.com Webmail Interface User-Agent: Cyrus-JMAP/3.7.0-alpha0-841-g7899e99a45-fm-20220811.002-g7899e99a Mime-Version: 1.0 Message-Id: <8b361f8e-cc4f-466c-90f0-031a43436af2@www.fastmail.com> In-Reply-To: <7c830487-95a6-b008-920b-8bc4a318f10a@applied-asynchrony.com> References: <2220d403-e443-4e60-b7c3-d149e402c13e@www.fastmail.com> <61e5ccda-a527-4fea-9850-91095ffa91c4@www.fastmail.com> <4995baed-c561-421d-ba3e-3a75d6a738a3@www.fastmail.com> <2b8a38fa-f15f-45e8-8caa-61c5f8cd52de@www.fastmail.com> <7c830487-95a6-b008-920b-8bc4a318f10a@applied-asynchrony.com> Date: Wed, 17 Aug 2022 07:57:07 -0400 From: "Chris Murphy" To: =?UTF-8?Q?Holger_Hoffst=C3=A4tte?= , "Nikolay Borisov" , "Jens Axboe" , "Jan Kara" , "Paolo Valente" Cc: Linux-RAID , linux-block , linux-kernel , "Josef Bacik" Subject: Re: stalling IO regression since linux 5.12, through 5.18 Content-Type: text/plain;charset=utf-8 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wed, Aug 17, 2022, at 5:52 AM, Holger Hoffst=C3=A4tte wrote: > On 2022-08-16 17:34, Chris Murphy wrote: >>=20 >> On Tue, Aug 16, 2022, at 11:25 AM, Nikolay Borisov wrote: >>> How about changing the scheduler either mq-deadline or noop, just >>> to see if this is also reproducible with a different scheduler. I >>> guess noop would imply the blk cgroup controller is going to be >>> disabled >>=20 >> I already reported on that: always happens with bfq within an hour or >> less. Doesn't happen with mq-deadline for ~25+ hours. Does happen >> with bfq with the above patches removed. Does happen with >> cgroup.disabled=3Dio set. >>=20 >> Sounds to me like it's something bfq depends on and is somehow >> becoming perturbed in a way that mq-deadline does not, and has >> changed between 5.11 and 5.12. I have no idea what's under bfq that >> matches this description. > > Chris, just a shot in the dark but can you try the patch from > > https://lore.kernel.org/linux-block/20220803121504.212071-1-yukuai1@hu= aweicloud.com/ > > on top of something more recent than 5.12? Ideally 5.19 where it appli= es > cleanly. The problem doesn't reliably reproduce on 5.19. A patch for 5.12..5.18 w= ould be much more testable. --=20 Chris Murphy