From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-23.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9109AC63777 for ; Thu, 26 Nov 2020 04:52:43 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id E56DF2145D for ; Thu, 26 Nov 2020 04:52:42 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="D9/hcrRH" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E56DF2145D Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 09F986B006E; Wed, 25 Nov 2020 23:52:42 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 0513F6B0070; Wed, 25 Nov 2020 23:52:42 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E81916B0071; Wed, 25 Nov 2020 23:52:41 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0173.hostedemail.com [216.40.44.173]) by kanga.kvack.org (Postfix) with ESMTP id CE65E6B006E for ; Wed, 25 Nov 2020 23:52:41 -0500 (EST) Received: from smtpin06.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 8BC39180AD80F for ; Thu, 26 Nov 2020 04:52:41 +0000 (UTC) X-FDA: 77525348922.06.shoes38_0c161362737c Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin06.hostedemail.com (Postfix) with ESMTP id 6CECB100470F3 for ; Thu, 26 Nov 2020 04:52:41 +0000 (UTC) X-HE-Tag: shoes38_0c161362737c X-Filterd-Recvd-Size: 6840 Received: from mail-il1-f194.google.com (mail-il1-f194.google.com [209.85.166.194]) by imf10.hostedemail.com (Postfix) with ESMTP for ; Thu, 26 Nov 2020 04:52:40 +0000 (UTC) Received: by mail-il1-f194.google.com with SMTP id q1so681561ilt.6 for ; Wed, 25 Nov 2020 20:52:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=sDklPCBSDngDVVB9iqv91v571+XOJMImwRyy6FkmCZQ=; b=D9/hcrRHg0RFirj0OCtux5afxgDTvkT1TmebOXEd2NCmanzryQQK+1+Ki6AChd3hJO GZMjYWX7MONr8UtDTt8KRIIympnbZsxGj9pBE9I23+WUgkZthRdoxD5Zszlp0NC6QMHM TM/yObf4BkB1SE/2Ki2M9d4pZNY6e2p1YSfHlOFQoCc1UljJEq2bVFq+MeHxwIzxg9uK t42VYloNAasJdJeVYVFDc3BrCZw1AmZg8Croz8+FgLeQ44tdT21T+zs11UTMOjuBl8Yj iMhbHN8gqDN2T+4QBQtJCfiLhZVgBdet66N7uGiv9UuoyrDqr+L5b+nDgwL5BrmdEQuJ XTOA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=sDklPCBSDngDVVB9iqv91v571+XOJMImwRyy6FkmCZQ=; b=RNUpkxzwt1XwSk6NijAhzajWnh54xqf+tPa6g+dZ94DaOgWoHQgHPSgJzlXBra13F/ KCI3j7BQtPS6u1UxB0qDlsQJNK5zNOOrIriIciCLKRJjw/Ry2qPZVd/EKo+3a/iBv0Na KSlHQiOGt3MxoUU25gFyNmklFWrsmv22Ti2k7XQCIhq3TJQw5SzDOKv9+ALFfRR4B+rd bzQa9SL75paFRq4rxk6EzgAv25TflKUTUS0NzgMnnZxJB32mej1c6noyZDsoq+CRjWT4 oJqzSfcRM2+GauPJLHRwSdypBEkLleBzPnasjz3yIckUsqQ1P7zynsV5yVh4TtJrZMtV 5sDQ== X-Gm-Message-State: AOAM533XXRui+hRadeM006url9675k7tka22PmGgjgdXcUOOudmpIpyN i+prJQjmgOTRWwH+ZHcpQ9xRBA== X-Google-Smtp-Source: ABdhPJz2/EsdZl2ZpBPptMIR825pX3h04q4itLK7ZTujfRUxp4kbqoE148hcGCw3lr68vCoPT5kVaA== X-Received: by 2002:a05:6e02:e44:: with SMTP id l4mr1140572ilk.208.1606366360337; Wed, 25 Nov 2020 20:52:40 -0800 (PST) Received: from google.com ([2620:15c:183:200:7220:84ff:fe09:2d90]) by smtp.gmail.com with ESMTPSA id i8sm2479613ilj.1.2020.11.25.20.52.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 Nov 2020 20:52:39 -0800 (PST) Date: Wed, 25 Nov 2020 21:52:34 -0700 From: Yu Zhao To: Alex Shi Cc: Konstantin Khlebnikov , Andrew Morton , Hugh Dickins , Michal Hocko , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: Re: [PATCH next] mm/swap.c: reduce lock contention in lru_cache_add Message-ID: <20201126045234.GA1014081@google.com> References: <1605860847-47445-1-git-send-email-alex.shi@linux.alibaba.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1605860847-47445-1-git-send-email-alex.shi@linux.alibaba.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Fri, Nov 20, 2020 at 04:27:27PM +0800, Alex Shi wrote: > The current relock logical will change lru_lock when found a new > lruvec, so if 2 memcgs are reading file or alloc page at same time, > they could hold the lru_lock alternately, and wait for each other for > fairness attribute of ticket spin lock. > > This patch will sort that all lru_locks and only hold them once in > above scenario. That could reduce fairness waiting for lock reget. > Than, vm-scalability/case-lru-file-readtwice could get ~5% performance > gain on my 2P*20core*HT machine. > > Suggested-by: Konstantin Khlebnikov > Signed-off-by: Alex Shi > Cc: Konstantin Khlebnikov > Cc: Andrew Morton > Cc: Hugh Dickins > Cc: Yu Zhao > Cc: Michal Hocko > Cc: linux-mm@kvack.org > Cc: linux-kernel@vger.kernel.org > --- > mm/swap.c | 57 +++++++++++++++++++++++++++++++++++++++++++++++-------- > 1 file changed, 49 insertions(+), 8 deletions(-) > > diff --git a/mm/swap.c b/mm/swap.c > index 490553f3f9ef..c787b38bf9c0 100644 > --- a/mm/swap.c > +++ b/mm/swap.c > @@ -1009,24 +1009,65 @@ static void __pagevec_lru_add_fn(struct page *page, struct lruvec *lruvec) > trace_mm_lru_insertion(page, lru); > } > > +struct lruvecs { > + struct list_head lists[PAGEVEC_SIZE]; > + struct lruvec *vecs[PAGEVEC_SIZE]; > +}; > + > +/* Sort pvec pages on their lruvec */ > +int sort_page_lruvec(struct lruvecs *lruvecs, struct pagevec *pvec) > +{ > + int i, j, nr_lruvec; > + struct page *page; > + struct lruvec *lruvec = NULL; > + > + lruvecs->vecs[0] = NULL; > + for (i = nr_lruvec = 0; i < pagevec_count(pvec); i++) { > + page = pvec->pages[i]; > + lruvec = mem_cgroup_page_lruvec(page, page_pgdat(page)); > + > + /* Try to find a same lruvec */ > + for (j = 0; j <= nr_lruvec; j++) > + if (lruvec == lruvecs->vecs[j]) > + break; > + > + /* A new lruvec */ > + if (j > nr_lruvec) { > + INIT_LIST_HEAD(&lruvecs->lists[nr_lruvec]); > + lruvecs->vecs[nr_lruvec] = lruvec; > + j = nr_lruvec++; > + lruvecs->vecs[nr_lruvec] = 0; > + } > + > + list_add_tail(&page->lru, &lruvecs->lists[j]); > + } > + > + return nr_lruvec; > +} > + > /* > * Add the passed pages to the LRU, then drop the caller's refcount > * on them. Reinitialises the caller's pagevec. > */ > void __pagevec_lru_add(struct pagevec *pvec) > { > - int i; > - struct lruvec *lruvec = NULL; > + int i, nr_lruvec; > unsigned long flags = 0; > + struct page *page; > + struct lruvecs lruvecs; > > - for (i = 0; i < pagevec_count(pvec); i++) { > - struct page *page = pvec->pages[i]; > + nr_lruvec = sort_page_lruvec(&lruvecs, pvec); Simply looping pvec multiple times (15 at most) for different lruvecs would be better because 1) it requires no extra data structures and therefore has better cache locality (theoretically faster) 2) it only loops once when !CONFIG_MEMCG and !CONFIG_NUMA and therefore has no impact on Android and Chrome OS. > - lruvec = relock_page_lruvec_irqsave(page, lruvec, &flags); > - __pagevec_lru_add_fn(page, lruvec); > + for (i = 0; i < nr_lruvec; i++) { > + spin_lock_irqsave(&lruvecs.vecs[i]->lru_lock, flags); > + while (!list_empty(&lruvecs.lists[i])) { > + page = lru_to_page(&lruvecs.lists[i]); > + list_del(&page->lru); > + __pagevec_lru_add_fn(page, lruvecs.vecs[i]); > + } > + spin_unlock_irqrestore(&lruvecs.vecs[i]->lru_lock, flags); > } > - if (lruvec) > - unlock_page_lruvec_irqrestore(lruvec, flags); > + > release_pages(pvec->pages, pvec->nr); > pagevec_reinit(pvec); > } > -- > 2.29.GIT >