From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D3365C433C1 for ; Thu, 25 Mar 2021 11:43:23 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 49D2261A0A for ; Thu, 25 Mar 2021 11:43:23 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 49D2261A0A Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=techsingularity.net Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id D03166B0078; Thu, 25 Mar 2021 07:43:22 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CB0CE6B007B; Thu, 25 Mar 2021 07:43:22 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B2B446B007D; Thu, 25 Mar 2021 07:43:22 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0175.hostedemail.com [216.40.44.175]) by kanga.kvack.org (Postfix) with ESMTP id 950026B0078 for ; Thu, 25 Mar 2021 07:43:22 -0400 (EDT) Received: from smtpin19.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 603291DAD7 for ; Thu, 25 Mar 2021 11:43:22 +0000 (UTC) X-FDA: 77958211044.19.71C8CBD Received: from outbound-smtp33.blacknight.com (outbound-smtp33.blacknight.com [81.17.249.66]) by imf25.hostedemail.com (Postfix) with ESMTP id 8EF3C6000109 for ; Thu, 25 Mar 2021 11:43:20 +0000 (UTC) Received: from mail.blacknight.com (pemlinmail01.blacknight.ie [81.17.254.10]) by outbound-smtp33.blacknight.com (Postfix) with ESMTPS id 96F6EBAA40 for ; Thu, 25 Mar 2021 11:43:20 +0000 (GMT) Received: (qmail 17095 invoked from network); 25 Mar 2021 11:43:20 -0000 Received: from unknown (HELO stampy.112glenside.lan) (mgorman@techsingularity.net@[84.203.22.4]) by 81.17.254.9 with ESMTPA; 25 Mar 2021 11:43:20 -0000 From: Mel Gorman To: Andrew Morton Cc: Chuck Lever , Jesper Dangaard Brouer , Christoph Hellwig , Alexander Duyck , Vlastimil Babka , Matthew Wilcox , Ilias Apalodimas , LKML , Linux-Net , Linux-MM , Linux-NFS , Mel Gorman Subject: [PATCH 4/9] mm/page_alloc: optimize code layout for __alloc_pages_bulk Date: Thu, 25 Mar 2021 11:42:23 +0000 Message-Id: <20210325114228.27719-5-mgorman@techsingularity.net> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210325114228.27719-1-mgorman@techsingularity.net> References: <20210325114228.27719-1-mgorman@techsingularity.net> MIME-Version: 1.0 X-Stat-Signature: 1o5wndi3tadfw9pkk4w1th8i8ub6ekac X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 8EF3C6000109 Received-SPF: none (techsingularity.net>: No applicable sender policy available) receiver=imf25; identity=mailfrom; envelope-from=""; helo=outbound-smtp33.blacknight.com; client-ip=81.17.249.66 X-HE-DKIM-Result: none/none X-HE-Tag: 1616672600-408878 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Jesper Dangaard Brouer Looking at perf-report and ASM-code for __alloc_pages_bulk() it is clear that the code activated is suboptimal. The compiler guesses wrong and places unlikely code at the beginning. Due to the use of WARN_ON_ONCE() macro the UD2 asm instruction is added to the code, which confuse the I-cache prefetcher in the CPU. [mgorman: Minor changes and rebasing] Signed-off-by: Jesper Dangaard Brouer Signed-off-by: Mel Gorman --- mm/page_alloc.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index be1e33a4df39..1ec18121268b 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -5001,7 +5001,7 @@ int __alloc_pages_bulk(gfp_t gfp, int preferred_nid= , unsigned int alloc_flags; int nr_populated =3D 0; =20 - if (WARN_ON_ONCE(nr_pages <=3D 0)) + if (unlikely(nr_pages <=3D 0)) return 0; =20 /* @@ -5048,7 +5048,7 @@ int __alloc_pages_bulk(gfp_t gfp, int preferred_nid= , * If there are no allowed local zones that meets the watermarks then * try to allocate a single page and reclaim if necessary. */ - if (!zone) + if (unlikely(!zone)) goto failed; =20 /* Attempt the batch allocation */ @@ -5066,7 +5066,7 @@ int __alloc_pages_bulk(gfp_t gfp, int preferred_nid= , =20 page =3D __rmqueue_pcplist(zone, ac.migratetype, alloc_flags, pcp, pcp_list); - if (!page) { + if (unlikely(!page)) { /* Try and get at least one page */ if (!nr_populated) goto failed_irq; --=20 2.26.2