From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.6 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4A01EC35242 for ; Tue, 11 Feb 2020 22:47:17 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id E88C9206ED for ; Tue, 11 Feb 2020 22:47:16 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="gdqkBY5t" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E88C9206ED Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 935756B0357; Tue, 11 Feb 2020 17:47:16 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 8C0256B0359; Tue, 11 Feb 2020 17:47:16 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7AE6C6B035A; Tue, 11 Feb 2020 17:47:16 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0221.hostedemail.com [216.40.44.221]) by kanga.kvack.org (Postfix) with ESMTP id 59C876B0357 for ; Tue, 11 Feb 2020 17:47:16 -0500 (EST) Received: from smtpin12.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 1071545A0 for ; Tue, 11 Feb 2020 22:47:16 +0000 (UTC) X-FDA: 76479333672.12.sand35_22477fba96663 X-HE-Tag: sand35_22477fba96663 X-Filterd-Recvd-Size: 6176 Received: from mail-wr1-f65.google.com (mail-wr1-f65.google.com [209.85.221.65]) by imf48.hostedemail.com (Postfix) with ESMTP for ; Tue, 11 Feb 2020 22:47:15 +0000 (UTC) Received: by mail-wr1-f65.google.com with SMTP id m16so14558672wrx.11 for ; Tue, 11 Feb 2020 14:47:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=q6OeJXPwWEnUOz8Lo8TsCrdbcxtEJkAXopv2X8QuM5w=; b=gdqkBY5tGbiMNDJ4wqiCepJ8YXg/3EiJi0JOAI2tx4qlfJJlzhFoNdbuz/sNNNq3PG U+L7mlN0oUB4muF4Ai4keJpGp39n7UWVFwPR9BlHlyEi2bp0ksiGnU1tyLkwIN4OKfhF JqT5Na4647BS1x9rsipCv2TqUJGtNfMP3TugVRM2uxqFB/UK48tJMEB/tonKFZFhOj5/ HzPRXPhMd/Y5dcKzbxzveT2Y87nhgn9QVjse7rlPTWPddpBk3O81Es39GAYCXFxkN2tS lS3ppIhVptxaXYTP7Qp91OJ3PVwD5fNcZFkmEUexSbxOFbzTp0SKWWgCGqRVpUmqwBxB N2ug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:from:to:cc:date:message-id:in-reply-to :references:user-agent:mime-version:content-transfer-encoding; bh=q6OeJXPwWEnUOz8Lo8TsCrdbcxtEJkAXopv2X8QuM5w=; b=hyi1S9UOgfidazirZ1ZIBTAccCuxNAv2AifA62XW/BYWpuQUHKO5WMVh45xQlFnsCQ PU1UKzCKQ7sXnX38p4OBG3z6EVdDKL0h1FIz47p/1JYhPMJ3iyJzeEih9q1FV+it3zL4 uO/AZeoO7zxJCdAnJ1DUGDhujxeqe7Wu09oHvF4qFh06ZI3Zt2caUjhUy3po0hPF/Fwb 8tBmUkg5pQiIUuawh4vKMD4wBIvlfMjg7xoOYKYYxuLt927WGYhBrjDVaySYypV7j7VJ V8TH7uO1ErkEGUshusHf59f7SYPYiYeLBtE3pMeEM3AWV3mbMNYEGiWmCOkCQo3nzntW OMdg== X-Gm-Message-State: APjAAAWsHkDPb2qJSyJMK74ohgrgXBFCwpYdoEpBcEWkvFe/y7sKNnqM gRJBoTe/gjBM1vEvRtdw3T4= X-Google-Smtp-Source: APXvYqxlgc69mpm5mOtrTz4wpHiodKuG0hVB/KKC8SEPcsJZBDWdjBxrGa3+eksMaEXqt5GHMRf/vQ== X-Received: by 2002:a5d:5345:: with SMTP id t5mr11932990wrv.0.1581461234292; Tue, 11 Feb 2020 14:47:14 -0800 (PST) Received: from localhost.localdomain ([2001:470:b:9c3:9e5c:8eff:fe4f:f2d0]) by smtp.gmail.com with ESMTPSA id 18sm5760932wmf.1.2020.02.11.14.47.10 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 11 Feb 2020 14:47:13 -0800 (PST) Subject: [PATCH v17 7/9] mm/page_reporting: Rotate reported pages to the tail of the list From: Alexander Duyck To: kvm@vger.kernel.org, david@redhat.com, mst@redhat.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, mgorman@techsingularity.net Cc: yang.zhang.wz@gmail.com, pagupta@redhat.com, konrad.wilk@oracle.com, nitesh@redhat.com, riel@surriel.com, willy@infradead.org, lcapitulino@redhat.com, dave.hansen@intel.com, wei.w.wang@intel.com, aarcange@redhat.com, pbonzini@redhat.com, dan.j.williams@intel.com, mhocko@kernel.org, alexander.h.duyck@linux.intel.com, vbabka@suse.cz, osalvador@suse.de Date: Tue, 11 Feb 2020 14:47:08 -0800 Message-ID: <20200211224708.29318.16862.stgit@localhost.localdomain> In-Reply-To: <20200211224416.29318.44077.stgit@localhost.localdomain> References: <20200211224416.29318.44077.stgit@localhost.localdomain> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Alexander Duyck Rather than walking over the same pages again and again to get to the pages that have yet to be reported we can save ourselves a significant amount of time by simply rotating the list so that when we have a full list of reported pages the head of the list is pointing to the next non-reported page. Doing this should save us some significant time when processing each free list. This doesn't gain us much in the standard case as all of the non-reported pages should be near the top of the list already. However in the case of page shuffling this results in a noticeable improvement. Below are the will-it-scale page_fault1 w/ THP numbers for 16 tasks with and without this patch. Without: tasks processes processes_idle threads threads_idle 16 8093776.25 0.17 5393242.00 38.20 With: tasks processes processes_idle threads threads_idle 16 8283274.75 0.17 5594261.00 38.15 Signed-off-by: Alexander Duyck --- mm/page_reporting.c | 30 ++++++++++++++++++++++-------- 1 file changed, 22 insertions(+), 8 deletions(-) diff --git a/mm/page_reporting.c b/mm/page_reporting.c index 1047c6872d4f..6885e74c2367 100644 --- a/mm/page_reporting.c +++ b/mm/page_reporting.c @@ -131,17 +131,27 @@ void __page_reporting_notify(void) if (PageReported(page)) continue; - /* Attempt to pull page from list */ - if (!__isolate_free_page(page, order)) - break; + /* Attempt to pull page from list and place in scatterlist */ + if (*offset) { + if (!__isolate_free_page(page, order)) { + next = page; + break; + } - /* Add page to scatter list */ - --(*offset); - sg_set_page(&sgl[*offset], page, page_len, 0); + /* Add page to scatter list */ + --(*offset); + sg_set_page(&sgl[*offset], page, page_len, 0); - /* If scatterlist isn't full grab more pages */ - if (*offset) continue; + } + + /* + * Make the first non-processed page in the free list + * the new head of the free list before we release the + * zone lock. + */ + if (&page->lru != list && !list_is_first(&page->lru, list)) + list_rotate_to_front(&page->lru, list); /* release lock before waiting on report processing */ spin_unlock_irq(&zone->lock); @@ -169,6 +179,10 @@ void __page_reporting_notify(void) break; } + /* Rotate any leftover pages to the head of the freelist */ + if (&next->lru != list && !list_is_first(&next->lru, list)) + list_rotate_to_front(&next->lru, list); + spin_unlock_irq(&zone->lock); return err;