From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.0 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 51727C49ED9 for ; Tue, 10 Sep 2019 22:14:50 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 262CC2171F for ; Tue, 10 Sep 2019 22:14:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726411AbfIJWOt (ORCPT ); Tue, 10 Sep 2019 18:14:49 -0400 Received: from mga04.intel.com ([192.55.52.120]:57998 "EHLO mga04.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725856AbfIJWOt (ORCPT ); Tue, 10 Sep 2019 18:14:49 -0400 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga104.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 10 Sep 2019 15:14:48 -0700 X-IronPort-AV: E=Sophos;i="5.64,490,1559545200"; d="scan'208";a="178822353" Received: from ahduyck-desk1.jf.intel.com ([10.7.198.76]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 10 Sep 2019 15:14:48 -0700 Message-ID: Subject: Re: [PATCH v9 1/8] mm: Add per-cpu logic to page shuffling From: Alexander Duyck To: Michal Hocko Cc: David Hildenbrand , Alexander Duyck , virtio-dev@lists.oasis-open.org, kvm@vger.kernel.org, mst@redhat.com, catalin.marinas@arm.com, dave.hansen@intel.com, linux-kernel@vger.kernel.org, willy@infradead.org, linux-mm@kvack.org, akpm@linux-foundation.org, will@kernel.org, linux-arm-kernel@lists.infradead.org, osalvador@suse.de, yang.zhang.wz@gmail.com, pagupta@redhat.com, konrad.wilk@oracle.com, nitesh@redhat.com, riel@surriel.com, lcapitulino@redhat.com, wei.w.wang@intel.com, aarcange@redhat.com, ying.huang@intel.com, pbonzini@redhat.com, dan.j.williams@intel.com, fengguang.wu@intel.com, kirill.shutemov@linux.intel.com Date: Tue, 10 Sep 2019 15:14:47 -0700 In-Reply-To: <20190910121130.GU2063@dhcp22.suse.cz> References: <20190907172225.10910.34302.stgit@localhost.localdomain> <20190907172512.10910.74435.stgit@localhost.localdomain> <0df2e5d0-af92-04b4-aa7d-891387874039@redhat.com> <0ca58fea280b51b83e7b42e2087128789bc9448d.camel@linux.intel.com> <20190910121130.GU2063@dhcp22.suse.cz> Content-Type: text/plain; charset="UTF-8" User-Agent: Evolution 3.30.5 (3.30.5-1.fc29) MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, 2019-09-10 at 14:11 +0200, Michal Hocko wrote: > On Mon 09-09-19 08:11:36, Alexander Duyck wrote: > > On Mon, 2019-09-09 at 10:14 +0200, David Hildenbrand wrote: > > > On 07.09.19 19:25, Alexander Duyck wrote: > > > > From: Alexander Duyck > > > > > > > > Change the logic used to generate randomness in the suffle path so that we > > > > can avoid cache line bouncing. The previous logic was sharing the offset > > > > and entropy word between all CPUs. As such this can result in cache line > > > > bouncing and will ultimately hurt performance when enabled. > > > > > > So, usually we perform such changes if there is real evidence. Do you > > > have any such performance numbers to back your claims? > > > > I'll have to go rerun the test to get the exact numbers. The reason this > > came up is that my original test was spanning NUMA nodes and that made > > this more expensive as a result since the memory was both not local to the > > CPU and was being updated by multiple sockets. > > What was the pattern of page freeing in your testing? I am wondering > because order 0 pages should be prevailing and those usually go via pcp > lists so they do not get shuffled unless the batch is full IIRC. So I am pretty sure my previous data was faulty. One side effect of the page reporting is that it was evicting pages out of the guest and when the pages were faulted back in they were coming from local page pools. This was throwing off my early numbers and making tests look better than they should have for the reported case. I had this patch previously merged with another one so I wasn't testing it on its own, it was instead a part of a bigger set. Now that I have tried testing it on its own I can see that it has no significant impact on performance. With that being the case I will probably just drop it. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.0 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 51329C5AE59 for ; Tue, 10 Sep 2019 22:14:51 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 1CEE82171F for ; Tue, 10 Sep 2019 22:14:51 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1CEE82171F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.intel.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id AA7EB6B0007; Tue, 10 Sep 2019 18:14:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A30BB6B0008; Tue, 10 Sep 2019 18:14:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8F7756B000A; Tue, 10 Sep 2019 18:14:50 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0018.hostedemail.com [216.40.44.18]) by kanga.kvack.org (Postfix) with ESMTP id 670F36B0007 for ; Tue, 10 Sep 2019 18:14:50 -0400 (EDT) Received: from smtpin01.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with SMTP id F3EAC8243765 for ; Tue, 10 Sep 2019 22:14:49 +0000 (UTC) X-FDA: 75920416740.01.floor38_6453a9bcc0063 X-HE-Tag: floor38_6453a9bcc0063 X-Filterd-Recvd-Size: 4033 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by imf19.hostedemail.com (Postfix) with ESMTP for ; Tue, 10 Sep 2019 22:14:49 +0000 (UTC) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga101.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 10 Sep 2019 15:14:48 -0700 X-IronPort-AV: E=Sophos;i="5.64,490,1559545200"; d="scan'208";a="178822353" Received: from ahduyck-desk1.jf.intel.com ([10.7.198.76]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 10 Sep 2019 15:14:48 -0700 Message-ID: Subject: Re: [PATCH v9 1/8] mm: Add per-cpu logic to page shuffling From: Alexander Duyck To: Michal Hocko Cc: David Hildenbrand , Alexander Duyck , virtio-dev@lists.oasis-open.org, kvm@vger.kernel.org, mst@redhat.com, catalin.marinas@arm.com, dave.hansen@intel.com, linux-kernel@vger.kernel.org, willy@infradead.org, linux-mm@kvack.org, akpm@linux-foundation.org, will@kernel.org, linux-arm-kernel@lists.infradead.org, osalvador@suse.de, yang.zhang.wz@gmail.com, pagupta@redhat.com, konrad.wilk@oracle.com, nitesh@redhat.com, riel@surriel.com, lcapitulino@redhat.com, wei.w.wang@intel.com, aarcange@redhat.com, ying.huang@intel.com, pbonzini@redhat.com, dan.j.williams@intel.com, fengguang.wu@intel.com, kirill.shutemov@linux.intel.com Date: Tue, 10 Sep 2019 15:14:47 -0700 In-Reply-To: <20190910121130.GU2063@dhcp22.suse.cz> References: <20190907172225.10910.34302.stgit@localhost.localdomain> <20190907172512.10910.74435.stgit@localhost.localdomain> <0df2e5d0-af92-04b4-aa7d-891387874039@redhat.com> <0ca58fea280b51b83e7b42e2087128789bc9448d.camel@linux.intel.com> <20190910121130.GU2063@dhcp22.suse.cz> Content-Type: text/plain; charset="UTF-8" User-Agent: Evolution 3.30.5 (3.30.5-1.fc29) MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Tue, 2019-09-10 at 14:11 +0200, Michal Hocko wrote: > On Mon 09-09-19 08:11:36, Alexander Duyck wrote: > > On Mon, 2019-09-09 at 10:14 +0200, David Hildenbrand wrote: > > > On 07.09.19 19:25, Alexander Duyck wrote: > > > > From: Alexander Duyck > > > > > > > > Change the logic used to generate randomness in the suffle path so that we > > > > can avoid cache line bouncing. The previous logic was sharing the offset > > > > and entropy word between all CPUs. As such this can result in cache line > > > > bouncing and will ultimately hurt performance when enabled. > > > > > > So, usually we perform such changes if there is real evidence. Do you > > > have any such performance numbers to back your claims? > > > > I'll have to go rerun the test to get the exact numbers. The reason this > > came up is that my original test was spanning NUMA nodes and that made > > this more expensive as a result since the memory was both not local to the > > CPU and was being updated by multiple sockets. > > What was the pattern of page freeing in your testing? I am wondering > because order 0 pages should be prevailing and those usually go via pcp > lists so they do not get shuffled unless the batch is full IIRC. So I am pretty sure my previous data was faulty. One side effect of the page reporting is that it was evicting pages out of the guest and when the pages were faulted back in they were coming from local page pools. This was throwing off my early numbers and making tests look better than they should have for the reported case. I had this patch previously merged with another one so I wasn't testing it on its own, it was instead a part of a bigger set. Now that I have tried testing it on its own I can see that it has no significant impact on performance. With that being the case I will probably just drop it. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.0 required=3.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 097FEC49ED9 for ; Tue, 10 Sep 2019 22:14:57 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id C7D17216F4 for ; Tue, 10 Sep 2019 22:14:56 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="itrOV+WG" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C7D17216F4 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.intel.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+infradead-linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Date:To:From:Subject:Message-ID:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=8VKrKK6z4pd1HkRZP4QGwiOajM5QKYPZ/7kHQe43S/U=; b=itrOV+WGznlVES HW1UVF1POjP2roP1grJSsGtilQeg+7AZXqpAI1RwvOGfe1ytKE/25FApq6ejn2gs9MX0gcPIijm6a clziPmpf68o2cNKckCf4oVU/bBO4NukWhyaO8IBHFGJ3Qxvd51hOdJpLONnxoLPt7oVwOie813fCT dkJXU0i4OzUdnFJxkB5qxncP8gwp6V15Wb9R+OTUSHpZQaZI8Ei/Lf9uy9ol+XtWQBN02inCL6I0d X+AuSvhDm1JR/F6EijSkC4aVnoV8IdH+09zLb/bw7jKVEve0Onm5JQTxtbM/JsRqeEomMiLuw5rMG GhUf9kG1JIbPn57lYnWQ==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92 #3 (Red Hat Linux)) id 1i7oPT-0001V4-Mt; Tue, 10 Sep 2019 22:14:51 +0000 Received: from mga12.intel.com ([192.55.52.136]) by bombadil.infradead.org with esmtps (Exim 4.92 #3 (Red Hat Linux)) id 1i7oPQ-0001Uj-TB for linux-arm-kernel@lists.infradead.org; Tue, 10 Sep 2019 22:14:50 +0000 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga106.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 10 Sep 2019 15:14:48 -0700 X-IronPort-AV: E=Sophos;i="5.64,490,1559545200"; d="scan'208";a="178822353" Received: from ahduyck-desk1.jf.intel.com ([10.7.198.76]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 10 Sep 2019 15:14:48 -0700 Message-ID: Subject: Re: [PATCH v9 1/8] mm: Add per-cpu logic to page shuffling From: Alexander Duyck To: Michal Hocko Date: Tue, 10 Sep 2019 15:14:47 -0700 In-Reply-To: <20190910121130.GU2063@dhcp22.suse.cz> References: <20190907172225.10910.34302.stgit@localhost.localdomain> <20190907172512.10910.74435.stgit@localhost.localdomain> <0df2e5d0-af92-04b4-aa7d-891387874039@redhat.com> <0ca58fea280b51b83e7b42e2087128789bc9448d.camel@linux.intel.com> <20190910121130.GU2063@dhcp22.suse.cz> User-Agent: Evolution 3.30.5 (3.30.5-1.fc29) MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20190910_151448_956992_F099CC3B X-CRM114-Status: GOOD ( 17.54 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yang.zhang.wz@gmail.com, pagupta@redhat.com, kvm@vger.kernel.org, mst@redhat.com, catalin.marinas@arm.com, Alexander Duyck , lcapitulino@redhat.com, linux-mm@kvack.org, will@kernel.org, aarcange@redhat.com, virtio-dev@lists.oasis-open.org, David Hildenbrand , willy@infradead.org, wei.w.wang@intel.com, ying.huang@intel.com, riel@surriel.com, dan.j.williams@intel.com, linux-arm-kernel@lists.infradead.org, osalvador@suse.de, nitesh@redhat.com, konrad.wilk@oracle.com, dave.hansen@intel.com, linux-kernel@vger.kernel.org, pbonzini@redhat.com, akpm@linux-foundation.org, fengguang.wu@intel.com, kirill.shutemov@linux.intel.com Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+infradead-linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Tue, 2019-09-10 at 14:11 +0200, Michal Hocko wrote: > On Mon 09-09-19 08:11:36, Alexander Duyck wrote: > > On Mon, 2019-09-09 at 10:14 +0200, David Hildenbrand wrote: > > > On 07.09.19 19:25, Alexander Duyck wrote: > > > > From: Alexander Duyck > > > > > > > > Change the logic used to generate randomness in the suffle path so that we > > > > can avoid cache line bouncing. The previous logic was sharing the offset > > > > and entropy word between all CPUs. As such this can result in cache line > > > > bouncing and will ultimately hurt performance when enabled. > > > > > > So, usually we perform such changes if there is real evidence. Do you > > > have any such performance numbers to back your claims? > > > > I'll have to go rerun the test to get the exact numbers. The reason this > > came up is that my original test was spanning NUMA nodes and that made > > this more expensive as a result since the memory was both not local to the > > CPU and was being updated by multiple sockets. > > What was the pattern of page freeing in your testing? I am wondering > because order 0 pages should be prevailing and those usually go via pcp > lists so they do not get shuffled unless the batch is full IIRC. So I am pretty sure my previous data was faulty. One side effect of the page reporting is that it was evicting pages out of the guest and when the pages were faulted back in they were coming from local page pools. This was throwing off my early numbers and making tests look better than they should have for the reported case. I had this patch previously merged with another one so I wasn't testing it on its own, it was instead a part of a bigger set. Now that I have tried testing it on its own I can see that it has no significant impact on performance. With that being the case I will probably just drop it. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: virtio-dev-return-6110-cohuck=redhat.com@lists.oasis-open.org Sender: List-Post: List-Help: List-Unsubscribe: List-Subscribe: Received: from lists.oasis-open.org (oasis-open.org [10.110.1.242]) by lists.oasis-open.org (Postfix) with ESMTP id 47C20985C1E for ; Tue, 10 Sep 2019 22:14:50 +0000 (UTC) Message-ID: From: Alexander Duyck Date: Tue, 10 Sep 2019 15:14:47 -0700 In-Reply-To: <20190910121130.GU2063@dhcp22.suse.cz> References: <20190907172225.10910.34302.stgit@localhost.localdomain> <20190907172512.10910.74435.stgit@localhost.localdomain> <0df2e5d0-af92-04b4-aa7d-891387874039@redhat.com> <0ca58fea280b51b83e7b42e2087128789bc9448d.camel@linux.intel.com> <20190910121130.GU2063@dhcp22.suse.cz> Content-Type: text/plain; charset="UTF-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: [virtio-dev] Re: [PATCH v9 1/8] mm: Add per-cpu logic to page shuffling To: Michal Hocko Cc: David Hildenbrand , Alexander Duyck , virtio-dev@lists.oasis-open.org, kvm@vger.kernel.org, mst@redhat.com, catalin.marinas@arm.com, dave.hansen@intel.com, linux-kernel@vger.kernel.org, willy@infradead.org, linux-mm@kvack.org, akpm@linux-foundation.org, will@kernel.org, linux-arm-kernel@lists.infradead.org, osalvador@suse.de, yang.zhang.wz@gmail.com, pagupta@redhat.com, konrad.wilk@oracle.com, nitesh@redhat.com, riel@surriel.com, lcapitulino@redhat.com, wei.w.wang@intel.com, aarcange@redhat.com, ying.huang@intel.com, pbonzini@redhat.com, dan.j.williams@intel.com, fengguang.wu@intel.com, kirill.shutemov@linux.intel.com List-ID: On Tue, 2019-09-10 at 14:11 +0200, Michal Hocko wrote: > On Mon 09-09-19 08:11:36, Alexander Duyck wrote: > > On Mon, 2019-09-09 at 10:14 +0200, David Hildenbrand wrote: > > > On 07.09.19 19:25, Alexander Duyck wrote: > > > > From: Alexander Duyck > > > > > > > > Change the logic used to generate randomness in the suffle path so that we > > > > can avoid cache line bouncing. The previous logic was sharing the offset > > > > and entropy word between all CPUs. As such this can result in cache line > > > > bouncing and will ultimately hurt performance when enabled. > > > > > > So, usually we perform such changes if there is real evidence. Do you > > > have any such performance numbers to back your claims? > > > > I'll have to go rerun the test to get the exact numbers. The reason this > > came up is that my original test was spanning NUMA nodes and that made > > this more expensive as a result since the memory was both not local to the > > CPU and was being updated by multiple sockets. > > What was the pattern of page freeing in your testing? I am wondering > because order 0 pages should be prevailing and those usually go via pcp > lists so they do not get shuffled unless the batch is full IIRC. So I am pretty sure my previous data was faulty. One side effect of the page reporting is that it was evicting pages out of the guest and when the pages were faulted back in they were coming from local page pools. This was throwing off my early numbers and making tests look better than they should have for the reported case. I had this patch previously merged with another one so I wasn't testing it on its own, it was instead a part of a bigger set. Now that I have tried testing it on its own I can see that it has no significant impact on performance. With that being the case I will probably just drop it. --------------------------------------------------------------------- To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org