From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.8 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 43F86C43381 for ; Mon, 25 Mar 2019 19:58:00 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 16B2C20879 for ; Mon, 25 Mar 2019 19:58:00 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="ZrGkwVUP" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729626AbfCYT57 (ORCPT ); Mon, 25 Mar 2019 15:57:59 -0400 Received: from mail-yw1-f66.google.com ([209.85.161.66]:36180 "EHLO mail-yw1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729475AbfCYT56 (ORCPT ); Mon, 25 Mar 2019 15:57:58 -0400 Received: by mail-yw1-f66.google.com with SMTP id u197so8032030ywf.3; Mon, 25 Mar 2019 12:57:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=KDQD0fdqhJXjKjlRxHs2X4+uo6UUNuD7klav1qzmcsY=; b=ZrGkwVUPlS9Rdt1sY5ZR16ZPOdzA+n9LSpjyxxnkcS9UwSkBvMJ5KV+wzInusy15I2 +HgnzHg1Y41Dl4wyE7/iddttS45lu9G3okUalAj83xdVlycBvlYvO18uEmAb8dg1lQlR odF6OJ565A9rkNErY+G4QADktZnP2IRXnuJrOF0XEWWYYFdAhGT02mp0LD2t1UNMu1j5 UFBGB+u9x7czky3k1E/YseyWCI10S+y2mlEzuv+RgCLeI9pgNB5yD3ULU0epdc1t7fOb omLMhc1vmc+w0HiYjOOJsP8ekG/5TsRXFFjZ+iYYWUGS5sXj8u+EsRdM546afGe2X4TU pVlQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=KDQD0fdqhJXjKjlRxHs2X4+uo6UUNuD7klav1qzmcsY=; b=DxoBGWCceiDEivh4QswJ2Zra/6OB4L/V8XgSva3Djj33xLPdv99KktGEgHLzaOmUnV /v9q4ImsLu72i4m0ItVonBd9xjFrz0OoxhSBRU98AvAz8Ur5Z/DAhRlMyIz/VUYYRWWy up7/6Ucdm7BxlG9qgVz3ICOptJGuWekEDiJV/y2H99cEx8bbMdsKOVbmaavhQDsSTPLC 7gA7DgpscivHX7iKDMWqz9MNBjWWGKBwKIc90P7641SgaoQcrTarVGameaCwrC193VME L1aNg+hH7jnZicPcqxUbSRBX8XMSuJTDx/A/bcemBGWdoHelaBTwZWIBykMnrjL1BcG7 8kJQ== X-Gm-Message-State: APjAAAX2fS1mt/AGRo+r3elKruy12VWonDuWZch90XmWiwTBZSV+Myvr OUsnw9ckJTsdGxKbC0wHI8GD0FEhtNVZPn1gAY4= X-Google-Smtp-Source: APXvYqzm1VvX1p+uTWfqG4gBpd+X2YFlIXIhe2VyFCBq83jvRvqBcXWB1z4DHxqVCgnCr3RP90/zeUkOaD4yuExh1zk= X-Received: by 2002:a25:9c89:: with SMTP id y9mr22003198ybo.320.1553543877592; Mon, 25 Mar 2019 12:57:57 -0700 (PDT) MIME-Version: 1.0 References: <20190325001044.GA23020@dastard> <20190325154731.GT1183@magnolia> <20190325164129.GH10344@bombadil.infradead.org> <20190325182239.GI10344@bombadil.infradead.org> <20190325194021.GJ10344@bombadil.infradead.org> In-Reply-To: <20190325194021.GJ10344@bombadil.infradead.org> From: Amir Goldstein Date: Mon, 25 Mar 2019 21:57:46 +0200 Message-ID: Subject: Re: [QUESTION] Long read latencies on mixed rw buffered IO To: Matthew Wilcox Cc: "Darrick J. Wong" , Dave Chinner , linux-xfs , Christoph Hellwig , linux-fsdevel Content-Type: text/plain; charset="UTF-8" Sender: linux-fsdevel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org On Mon, Mar 25, 2019 at 9:40 PM Matthew Wilcox wrote: > > On Mon, Mar 25, 2019 at 09:18:51PM +0200, Amir Goldstein wrote: > > On Mon, Mar 25, 2019 at 8:22 PM Matthew Wilcox wrote: > > > On Mon, Mar 25, 2019 at 07:30:39PM +0200, Amir Goldstein wrote: > > > > On Mon, Mar 25, 2019 at 6:41 PM Matthew Wilcox wrote: > > > > > I think it is a bug that we only wake readers at the front of the queue; > > > > > I think we would get better performance if we wake all readers. ie here: > > > > So I have no access to the test machine of former tests right now, > > but when running the same filebench randomrw workload > > (8 writers, 8 readers) on VM with 2 CPUs and SSD drive, results > > are not looking good for this patch: > > > > --- v5.1-rc1 / xfs --- > > rand-write1 852404ops 14202ops/s 110.9mb/s 0.6ms/op > > [0.01ms - 553.45ms] > > rand-read1 26117ops 435ops/s 3.4mb/s 18.4ms/op > > [0.04ms - 632.29ms] > > 61.088: IO Summary: 878521 ops 14636.774 ops/s 435/14202 rd/wr > > 114.3mb/s 1.1ms/op > > --- v5.1-rc1 / xfs + patch v2 below --- rand-write1 852487ops 14175ops/s 110.7mb/s 0.6ms/op [0.01ms - 755.24ms] rand-read1 23194ops 386ops/s 3.0mb/s 20.7ms/op [0.03ms - 755.25ms] 61.187: IO Summary: 875681 ops 14560.980 ops/s 386/14175 rd/wr 113.8mb/s 1.1ms/op Not as bad as v1. Only a little bit worse than master... The whole deal with the read/write balance and on SSD, I imagine the balance really changes. That's why I was skeptical about one-size-fits all read/write balance. Keeping an open mind. Please throw more patches at me. I will also test them on machine with spindles tomorrow. Thanks, Amir. > > --- v5.1-rc1 / xfs + patch above --- > > rand-write1 1117998ops 18621ops/s 145.5mb/s 0.4ms/op > > [0.01ms - 788.19ms] > > rand-read1 7089ops 118ops/s 0.9mb/s 67.4ms/op > > [0.03ms - 792.67ms] > > 61.091: IO Summary: 1125087 ops 18738.961 ops/s 118/18621 rd/wr > > 146.4mb/s 0.8ms/op > > > > --- v5.1-rc1 / xfs + remove XFS_IOLOCK_SHARED from > > xfs_file_buffered_aio_read --- > > rand-write1 1025826ops 17091ops/s 133.5mb/s 0.5ms/op > > [0.01ms - 909.20ms] > > rand-read1 115162ops 1919ops/s 15.0mb/s 4.2ms/op > > [0.00ms - 157.46ms] > > 61.084: IO Summary: 1140988 ops 19009.369 ops/s 1919/17091 rd/wr > > 148.5mb/s 0.8ms/op > > > > --- v5.1-rc1 / ext4 --- > > rand-write1 867926ops 14459ops/s 113.0mb/s 0.6ms/op > > [0.01ms - 886.89ms] > > rand-read1 121893ops 2031ops/s 15.9mb/s 3.9ms/op > > [0.00ms - 149.24ms] > > 61.102: IO Summary: 989819 ops 16489.132 ops/s 2031/14459 rd/wr > > 128.8mb/s 1.0ms/op > > > > So rw_semaphore fix is not in the ballpark, not even looking in the > > right direction... > > > > Any other ideas to try? > > Sure! Maybe the problem is walking the list over and over. So add new > readers to the front of the list if the head of the list is a reader; > otherwise add them to the tail of the list. > > (this won't have quite the same effect as the previous patch because > new readers coming in while the head of the list is a writer will still > get jumbled with new writers, but it should be better than we have now, > assuming the problem is that readers are being delayed behind writers). > > diff --git a/kernel/locking/rwsem-xadd.c b/kernel/locking/rwsem-xadd.c > index fbe96341beee..56dbbaea90ee 100644 > --- a/kernel/locking/rwsem-xadd.c > +++ b/kernel/locking/rwsem-xadd.c > @@ -250,8 +250,15 @@ __rwsem_down_read_failed_common(struct rw_semaphore *sem, int state) > return sem; > } > adjustment += RWSEM_WAITING_BIAS; > + list_add_tail(&waiter.list, &sem->wait_list); > + } else { > + struct rwsem_waiter *first = list_first_entry(&sem->wait_list, > + typeof(*first), list); > + if (first->type == RWSEM_WAITING_FOR_READ) > + list_add(&waiter.list, &sem->wait_list); > + else > + list_add_tail(&waiter.list, &sem->wait_list); > } > - list_add_tail(&waiter.list, &sem->wait_list); > > /* we're now waiting on the lock, but no longer actively locking */ > count = atomic_long_add_return(adjustment, &sem->count);