From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.3 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F0F6FECE58C for ; Wed, 9 Oct 2019 16:25:15 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 9FC9E218AC for ; Wed, 9 Oct 2019 16:25:15 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9FC9E218AC Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 500678E0005; Wed, 9 Oct 2019 12:25:15 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4D6958E0003; Wed, 9 Oct 2019 12:25:15 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 39E928E0007; Wed, 9 Oct 2019 12:25:15 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0126.hostedemail.com [216.40.44.126]) by kanga.kvack.org (Postfix) with ESMTP id 18A848E0003 for ; Wed, 9 Oct 2019 12:25:15 -0400 (EDT) Received: from smtpin27.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with SMTP id AACDF824CA3D for ; Wed, 9 Oct 2019 16:25:14 +0000 (UTC) X-FDA: 76024770948.27.bun30_2dae69353b149 X-HE-Tag: bun30_2dae69353b149 X-Filterd-Recvd-Size: 10351 Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by imf17.hostedemail.com (Postfix) with ESMTP for ; Wed, 9 Oct 2019 16:25:13 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 769C5806A75; Wed, 9 Oct 2019 16:25:12 +0000 (UTC) Received: from [10.18.17.163] (dhcp-17-163.bos.redhat.com [10.18.17.163]) by smtp.corp.redhat.com (Postfix) with ESMTPS id A3BBD60166; Wed, 9 Oct 2019 16:25:01 +0000 (UTC) Subject: Re: [PATCH v11 0/6] mm / virtio: Provide support for unused page reporting To: Alexander Duyck Cc: Alexander Duyck , LKML , linux-mm , David Hildenbrand , virtio-dev@lists.oasis-open.org, kvm list , "Michael S. Tsirkin" , Dave Hansen , Matthew Wilcox , Michal Hocko , Andrew Morton , Mel Gorman , Vlastimil Babka , Oscar Salvador , Yang Zhang , Pankaj Gupta , Konrad Rzeszutek Wilk , Rik van Riel , lcapitulino@redhat.com, "Wang, Wei W" , Andrea Arcangeli , Paolo Bonzini , Dan Williams References: <20191001152441.27008.99285.stgit@localhost.localdomain> <7233498c-2f64-d661-4981-707b59c78fd5@redhat.com> <1ea1a4e11617291062db81f65745b9c95fd0bb30.camel@linux.intel.com> <8bd303a6-6e50-b2dc-19ab-4c3f176c4b02@redhat.com> <0a16b11e-ec3b-7196-5b7f-e7395876cf28@redhat.com> <7fc13837-546c-9c4a-1456-753df199e171@redhat.com> <5b6e0b6df46c03bfac906313071ac0362d43c432.camel@linux.intel.com> From: Nitesh Narayan Lal Openpgp: preference=signencrypt Autocrypt: addr=nitesh@redhat.com; prefer-encrypt=mutual; keydata= mQINBFl4pQoBEADT/nXR2JOfsCjDgYmE2qonSGjkM1g8S6p9UWD+bf7YEAYYYzZsLtbilFTe z4nL4AV6VJmC7dBIlTi3Mj2eymD/2dkKP6UXlliWkq67feVg1KG+4UIp89lFW7v5Y8Muw3Fm uQbFvxyhN8n3tmhRe+ScWsndSBDxYOZgkbCSIfNPdZrHcnOLfA7xMJZeRCjqUpwhIjxQdFA7 n0s0KZ2cHIsemtBM8b2WXSQG9CjqAJHVkDhrBWKThDRF7k80oiJdEQlTEiVhaEDURXq+2XmG jpCnvRQDb28EJSsQlNEAzwzHMeplddfB0vCg9fRk/kOBMDBtGsTvNT9OYUZD+7jaf0gvBvBB lbKmmMMX7uJB+ejY7bnw6ePNrVPErWyfHzR5WYrIFUtgoR3LigKnw5apzc7UIV9G8uiIcZEn C+QJCK43jgnkPcSmwVPztcrkbC84g1K5v2Dxh9amXKLBA1/i+CAY8JWMTepsFohIFMXNLj+B RJoOcR4HGYXZ6CAJa3Glu3mCmYqHTOKwezJTAvmsCLd3W7WxOGF8BbBjVaPjcZfavOvkin0u DaFvhAmrzN6lL0msY17JCZo046z8oAqkyvEflFbC0S1R/POzehKrzQ1RFRD3/YzzlhmIowkM BpTqNBeHEzQAlIhQuyu1ugmQtfsYYq6FPmWMRfFPes/4JUU/PQARAQABtCVOaXRlc2ggTmFy YXlhbiBMYWwgPG5pbGFsQHJlZGhhdC5jb20+iQI9BBMBCAAnBQJZeKUKAhsjBQkJZgGABQsJ CAcCBhUICQoLAgQWAgMBAh4BAheAAAoJEKOGQNwGMqM56lEP/A2KMs/pu0URcVk/kqVwcBhU SnvB8DP3lDWDnmVrAkFEOnPX7GTbactQ41wF/xwjwmEmTzLrMRZpkqz2y9mV0hWHjqoXbOCS 6RwK3ri5e2ThIPoGxFLt6TrMHgCRwm8YuOSJ97o+uohCTN8pmQ86KMUrDNwMqRkeTRW9wWIQ EdDqW44VwelnyPwcmWHBNNb1Kd8j3xKlHtnS45vc6WuoKxYRBTQOwI/5uFpDZtZ1a5kq9Ak/ MOPDDZpd84rqd+IvgMw5z4a5QlkvOTpScD21G3gjmtTEtyfahltyDK/5i8IaQC3YiXJCrqxE r7/4JMZeOYiKpE9iZMtS90t4wBgbVTqAGH1nE/ifZVAUcCtycD0f3egX9CHe45Ad4fsF3edQ ESa5tZAogiA4Hc/yQpnnf43a3aQ67XPOJXxS0Qptzu4vfF9h7kTKYWSrVesOU3QKYbjEAf95 NewF9FhAlYqYrwIwnuAZ8TdXVDYt7Z3z506//sf6zoRwYIDA8RDqFGRuPMXUsoUnf/KKPrtR ceLcSUP/JCNiYbf1/QtW8S6Ca/4qJFXQHp0knqJPGmwuFHsarSdpvZQ9qpxD3FnuPyo64S2N Dfq8TAeifNp2pAmPY2PAHQ3nOmKgMG8Gn5QiORvMUGzSz8Lo31LW58NdBKbh6bci5+t/HE0H pnyVf5xhNC/FuQINBFl4pQoBEACr+MgxWHUP76oNNYjRiNDhaIVtnPRqxiZ9v4H5FPxJy9UD Bqr54rifr1E+K+yYNPt/Po43vVL2cAyfyI/LVLlhiY4yH6T1n+Di/hSkkviCaf13gczuvgz4 KVYLwojU8+naJUsiCJw01MjO3pg9GQ+47HgsnRjCdNmmHiUQqksMIfd8k3reO9SUNlEmDDNB XuSzkHjE5y/R/6p8uXaVpiKPfHoULjNRWaFc3d2JGmxJpBdpYnajoz61m7XJlgwl/B5Ql/6B dHGaX3VHxOZsfRfugwYF9CkrPbyO5PK7yJ5vaiWre7aQ9bmCtXAomvF1q3/qRwZp77k6i9R3 tWfXjZDOQokw0u6d6DYJ0Vkfcwheg2i/Mf/epQl7Pf846G3PgSnyVK6cRwerBl5a68w7xqVU 4KgAh0DePjtDcbcXsKRT9D63cfyfrNE+ea4i0SVik6+N4nAj1HbzWHTk2KIxTsJXypibOKFX 2VykltxutR1sUfZBYMkfU4PogE7NjVEU7KtuCOSAkYzIWrZNEQrxYkxHLJsWruhSYNRsqVBy KvY6JAsq/i5yhVd5JKKU8wIOgSwC9P6mXYRgwPyfg15GZpnw+Fpey4bCDkT5fMOaCcS+vSU1 UaFmC4Ogzpe2BW2DOaPU5Ik99zUFNn6cRmOOXArrryjFlLT5oSOe4IposgWzdwARAQABiQIl BBgBCAAPBQJZeKUKAhsMBQkJZgGAAAoJEKOGQNwGMqM5ELoP/jj9d9gF1Al4+9bngUlYohYu 0sxyZo9IZ7Yb7cHuJzOMqfgoP4tydP4QCuyd9Q2OHHL5AL4VFNb8SvqAxxYSPuDJTI3JZwI7 d8JTPKwpulMSUaJE8ZH9n8A/+sdC3CAD4QafVBcCcbFe1jifHmQRdDrvHV9Es14QVAOTZhnJ vweENyHEIxkpLsyUUDuVypIo6y/Cws+EBCWt27BJi9GH/EOTB0wb+2ghCs/i3h8a+bi+bS7L FCCm/AxIqxRurh2UySn0P/2+2eZvneJ1/uTgfxnjeSlwQJ1BWzMAdAHQO1/lnbyZgEZEtUZJ x9d9ASekTtJjBMKJXAw7GbB2dAA/QmbA+Q+Xuamzm/1imigz6L6sOt2n/X/SSc33w8RJUyor SvAIoG/zU2Y76pKTgbpQqMDmkmNYFMLcAukpvC4ki3Sf086TdMgkjqtnpTkEElMSFJC8npXv 3QnGGOIfFug/qs8z03DLPBz9VYS26jiiN7QIJVpeeEdN/LKnaz5LO+h5kNAyj44qdF2T2AiF HxnZnxO5JNP5uISQH3FjxxGxJkdJ8jKzZV7aT37sC+Rp0o3KNc+GXTR+GSVq87Xfuhx0LRST NK9ZhT0+qkiN7npFLtNtbzwqaqceq3XhafmCiw8xrtzCnlB/C4SiBr/93Ip4kihXJ0EuHSLn VujM7c/b4pps Organization: Red Hat Inc, Message-ID: <9bd52b8e-fa9e-a5ad-de39-660684757cdb@redhat.com> Date: Wed, 9 Oct 2019 12:25:00 -0400 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.8.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Content-Language: en-US X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.6.2 (mx1.redhat.com [10.5.110.67]); Wed, 09 Oct 2019 16:25:12 +0000 (UTC) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 10/7/19 1:20 PM, Alexander Duyck wrote: > On Mon, Oct 7, 2019 at 10:07 AM Nitesh Narayan Lal = wrote: >> >> On 10/7/19 12:27 PM, Alexander Duyck wrote: >>> On Mon, 2019-10-07 at 12:19 -0400, Nitesh Narayan Lal wrote: >>>> On 10/7/19 11:33 AM, Alexander Duyck wrote: >>>>> On Mon, 2019-10-07 at 08:29 -0400, Nitesh Narayan Lal wrote: >>>>>> On 10/2/19 10:25 AM, Alexander Duyck wrote: > > >>>>>> page_reporting.c change: >>>>>> @@ -101,8 +101,12 @@ static void scan_zone_bitmap(struct page_repo= rting_config >>>>>> *phconf, >>>>>> /* Process only if the page is still online */ >>>>>> page =3D pfn_to_online_page((setbit << PAGE_REPORT= ING_MIN_ORDER) + >>>>>> zone->base_pfn); >>>>>> - if (!page) >>>>>> + if (!page || !PageBuddy(page)) { >>>>>> + clear_bit(setbit, zone->bitmap); >>>>>> + atomic_dec(&zone->free_pages); >>>>>> continue; >>>>>> + } >>>>>> >>>>> I suspect the zone->free_pages is going to be expensive for you to = deal >>>>> with. It is a global atomic value and is going to have the cachelin= e >>>>> bouncing that it is contained in. As a result thinks like setting t= he >>>>> bitmap with be more expensive as every tome a CPU increments free_p= ages it >>>>> will likely have to take the cache line containing the bitmap point= er as >>>>> well. >>>> I see I will have to explore this more. I am wondering if there is a= way to >>>> measure this If its effect is not visible in will-it-scale/page_faul= t1. If >>>> there is a noticeable amount of degradation, I will have to address = this. >>> If nothing else you might look at seeing if you can split up the >>> structures so that the bitmap and nr_bits is in a different region >>> somewhere since those are read-mostly values. >> ok, I will try to understand the issue and your suggestion. >> Thank you for bringing this up. >> >>> Also you are now updating the bitmap and free_pages both inside and >>> outside of the zone lock so that will likely have some impact. >> So as per your previous suggestion, I have made the bitmap structure >> object as a rcu protected pointer. So we are safe from that side. >> The other downside which I can think of is a race where one page >> trying to increment free_pages and other trying to decrements it. >> However, being an atomic variable that should not be a problem. >> Did I miss anything? > I'm not so much worried about a race as the cache line bouncing > effect. Basically your notifier combined within this hinting thread > will likely result in more time spent by the thread that holds the > lock since it will be trying to access the bitmap to set the bit and > the free_pages to report the bit, but at the same time you will have > this thread clearing bits and decrementing the free_pages values. > > One thing you could consider in your worker thread would be to do > reallocate and replace the bitmap every time you plan to walk it. By > doing that you would avoid the cacheline bouncing on the bitmap since > you would only have to read it, and you would no longer have another > thread dirtying it. You could essentially reset the free_pages at the > same time you replace the bitmap. It would need to all happen with the > zone lock held though when you swap it out. If I am not mistaken then from what you are suggesting, I will have to ho= ld the zone lock for the entire duration of swap & scan which would be costl= y if the bitmap is large, isn't? Also, we might end up missing free pages that= are getting freed while we are scanning. As far as free_pages count is concerned, I am thinking if I should replace it with zone->free_area[REPORTING_ORDER].nr_free which is already= there (I still need to explore this in a bit more depth). > > - Alex --=20 Thanks Nitesh From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.3 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DC5FFC4360C for ; Sun, 13 Oct 2019 03:00:45 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 7E6732089C for ; Sun, 13 Oct 2019 03:00:45 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7E6732089C Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 2D6F66B0003; Sat, 12 Oct 2019 23:00:45 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2AD196B0005; Sat, 12 Oct 2019 23:00:45 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0FEB98E0001; Sat, 12 Oct 2019 23:00:45 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0044.hostedemail.com [216.40.44.44]) by kanga.kvack.org (Postfix) with ESMTP id D8DF66B0003 for ; Sat, 12 Oct 2019 23:00:44 -0400 (EDT) Received: from smtpin24.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with SMTP id 6D377180ACF61 for ; Sun, 13 Oct 2019 03:00:44 +0000 (UTC) X-FDA: 76037258808.24.frame57_5da47a3190407 X-HE-Tag: frame57_5da47a3190407 X-Filterd-Recvd-Size: 19877 Received: from listssympa-test.colorado.edu (listssympa-test.colorado.edu [128.138.129.156]) by imf16.hostedemail.com (Postfix) with ESMTP for ; Sun, 13 Oct 2019 03:00:43 +0000 (UTC) Received: from listssympa-test.colorado.edu (localhost [127.0.0.1]) by listssympa-test.colorado.edu (8.15.2/8.15.2/MJC-8.0/sympa) with ESMTPS id x9D30XHj006618 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Sat, 12 Oct 2019 21:00:34 -0600 Received: (from root@localhost) by listssympa-test.colorado.edu (8.15.2/8.15.2/MJC-8.0/submit) id x9D30Vsw006541; Sat, 12 Oct 2019 21:00:31 -0600 Received: from MWHPR03MB2815.namprd03.prod.outlook.com (2603:10b6:a03:74::26) by BYAPR03MB4376.namprd03.prod.outlook.com with HTTPS via BYAPR05CA0049.NAMPRD05.PROD.OUTLOOK.COM; Wed, 9 Oct 2019 17:59:06 +0000 Received: from BN3PR03CA0109.namprd03.prod.outlook.com (2603:10b6:400:4::27) by MWHPR03MB2815.namprd03.prod.outlook.com (2603:10b6:300:11e::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2327.24; Wed, 9 Oct 2019 17:35:35 +0000 Received: from SN1NAM01FT041.eop-nam01.prod.protection.outlook.com (2a01:111:f400:7e40::203) by BN3PR03CA0109.outlook.office365.com (2603:10b6:400:4::27) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.2347.16 via Frontend Transport; Wed, 9 Oct 2019 17:35:35 +0000 Received: from ipmx4.colorado.edu (128.138.67.75) by SN1NAM01FT041.mail.protection.outlook.com (10.152.65.86) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.2347.16 via Frontend Transport; Wed, 9 Oct 2019 17:35:35 +0000 Received: from ipmx2.colorado.edu ([128.138.128.232]) by mx.colorado.edu with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Oct 2019 11:01:08 -0600 Received: from vger.kernel.org ([209.132.180.67]) by mx.colorado.edu with ESMTP; 09 Oct 2019 10:25:14 -0600 Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731743AbfJIQZN convert rfc822-to-8bit (ORCPT ); Wed, 9 Oct 2019 12:25:13 -0400 Received: from mx1.redhat.com ([209.132.183.28]:38138 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730503AbfJIQZN (ORCPT ); Wed, 9 Oct 2019 12:25:13 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 769C5806A75; Wed, 9 Oct 2019 16:25:12 +0000 (UTC) Received: from [10.18.17.163] (dhcp-17-163.bos.redhat.com [10.18.17.163]) by smtp.corp.redhat.com (Postfix) with ESMTPS id A3BBD60166; Wed, 9 Oct 2019 16:25:01 +0000 (UTC) Authentication-Results: spf=none (sender IP is 128.138.67.75) smtp.mailfrom=vger.kernel.org; o365.colorado.edu; dkim=none (message not signed) header.d=none;o365.colorado.edu; dmarc=fail action=none header.from=redhat.com; Received-SPF: None (protection.outlook.com: vger.kernel.org does not designate permitted sender hosts) Authentication-Results-Original: mx.colorado.edu; dkim=none (message not signed) header.i=none IronPort-SDR: gn+6dVGTDga5YkOsOSkRgd4jMY63eo+2xhjP4t97+7hOQOS0vH7XJG0xVG5BuVHVqNhSLADVEt pkdiRibuEtDe8/fWSC9AJRwWdo/QnWQbM= IronPort-SDR: iq4J3W4KbvzAn9CVDES7xo9TFwoJ/NGa1RQuAl5lz+2Kw1T4iRwXD6ad9+34IqN/w0CMogCYZx tZAR4sZCCpEBTMvNo/mvy+zLlehjY7tXQ= IronPort-PHdr: =?us-ascii?q?9a23=3AtUuk5R+gDkuTx/9uRHGN80YQeigqvan1NQcJ65?= =?us-ascii?q?0hzohDabmn44+8ZR7a9bNmi1vOR4zX7LRJh/eF+6zjWGlV55GHvThCdZFXTB?= =?us-ascii?q?YKhI0QmBBoG8+KD0D3bbbqYiU2Ed4EVQpj+He2PA=3D=3D?= IronPort-PHdr: =?us-ascii?q?9a23=3A9xXdBBN6avwAErkMWXsl6mtXPHoOpqn0MwgJ65?= =?us-ascii?q?Eul7NJdOGZ8o//OFDEu6w/iU+PVoLf7OhNh+2Qvqz9CiQM4peE5XYFdpEEFx?= =?us-ascii?q?oIkt4fkAFoBsmZQVb6I/jnY21yHMlLWFJ/uXDuN09TFcs=3D?= X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: =?us-ascii?q?A0EFAACKFJ5dbeiAioBlGgEBAQEBAQE?= =?us-ascii?q?BAQMBAQEBEQEBAQICAQEBAYFoBAEBAQELAYIaBYFAMSqEI4kChVqCD4NulT0?= =?us-ascii?q?UgV8RAQEBAQEBAQEBCBgVAgEBAYcRIzUIDgIDCQEBAQMBAQECAQUCAQECAhA?= =?us-ascii?q?LDQkGK4VAhiEBBQECIAQMRgYJAQEKGAICJgICA1QGDQYCAQEBgx2CeLBefzO?= =?us-ascii?q?IfIFIgQwoAYwNEgaBf4E4gms+hC2DJYJeBIxhglOeD4Iskl+CNAYbgjqLQRI?= =?us-ascii?q?niwyjHoRtgVQBgg8zGggbFYMnUBAUgVsXjiIdJgExgQgcj3cBAQ?= X-IronPort-Anti-Spam-Result: =?us-ascii?q?A0EOAACpDJ5dh0O0hNFlGgEBAQEBAQE?= =?us-ascii?q?BAQMBAQEBEQEBAQICAQEBAYFoBAEBAQELAYIagUMBMSqEI4kChVmCDxSDWpU?= =?us-ascii?q?9FIFfEQEBAQEBAQEBAQgYFAECAQEBhxEjNQgOAgMJAQEBAwEBAQIBBQIBAQI?= =?us-ascii?q?CEAEBAQoLCQgphUCGIQECAwECIAQMRgYJAQEKGAICJgICA1QGDQYCAQEBgx2?= =?us-ascii?q?CC7BtfzOIfIFIgQwoAYwNEgaBf4E4gms+hC2DJYJYBIxhglOeD4Iskl+CNAY?= =?us-ascii?q?bgjqLQRIniwyjHoRtgVQBgg8zGggbFYMnUBAUgVsXjiIdJQEBMYEGAQEck0E?= =?us-ascii?q?BAQ?= X-IPAS-Result: =?us-ascii?q?A0EFAACKFJ5dbeiAioBlGgEBAQEBAQEBAQMBAQEBEQEBA?= =?us-ascii?q?QICAQEBAYFoBAEBAQELAYIaBYFAMSqEI4kChVqCD4NulT0UgV8RAQEBAQEBA?= =?us-ascii?q?QEBCBgVAgEBAYcRIzUIDgIDCQEBAQMBAQECAQUCAQECAhALDQkGK4VAhiEBB?= =?us-ascii?q?QECIAQMRgYJAQEKGAICJgICA1QGDQYCAQEBgx2CeLBefzOIfIFIgQwoAYwNE?= =?us-ascii?q?gaBf4E4gms+hC2DJYJeBIxhglOeD4Iskl+CNAYbgjqLQRIniwyjHoRtgVQBg?= =?us-ascii?q?g8zGggbFYMnUBAUgVsXjiIdJgExgQgcj3cBAQ?= X-IPAS-Result: =?us-ascii?q?A0EOAACpDJ5dh0O0hNFlGgEBAQEBAQEBAQMBAQEBEQEBA?= =?us-ascii?q?QICAQEBAYFoBAEBAQELAYIagUMBMSqEI4kChVmCDxSDWpU9FIFfEQEBAQEBA?= =?us-ascii?q?QEBAQgYFAECAQEBhxEjNQgOAgMJAQEBAwEBAQIBBQIBAQICEAEBAQoLCQgph?= =?us-ascii?q?UCGIQECAwECIAQMRgYJAQEKGAICJgICA1QGDQYCAQEBgx2CC7BtfzOIfIFIg?= =?us-ascii?q?QwoAYwNEgaBf4E4gms+hC2DJYJYBIxhglOeD4Iskl+CNAYbgjqLQRIniwyjH?= =?us-ascii?q?oRtgVQBgg8zGggbFYMnUBAUgVsXjiIdJQEBMYEGAQEck0EBAQ?= X-IronPort-AV: E=Sophos;i="5.67,277,1566885600"; d="scan'208";a="369263420" X-IronPort-AV: E=Sophos;i="5.67,276,1566885600"; d="scan'208";a="414131265" X-IronPort-Outbreak-Status: No, level 0, Unknown - Unknown X-IronPort-Outbreak-Status: No, level 0, Unknown - Unknown X-Original-Recipients: migi9492@g.colorado.edu Subject: Re: [PATCH v11 0/6] mm / virtio: Provide support for unused page reporting To: "Alexander Duyck" Cc: "Alexander Duyck" , LKML , linux-mm , "David Hildenbrand" , "virtio-dev@lists.oasis-open.org" , "kvm list" , "Michael S. Tsirkin" , "Dave Hansen" , "Matthew Wilcox" , "Michal Hocko" , "Andrew Morton" , "Mel Gorman" , "Vlastimil Babka" , "Oscar Salvador" , "Yang Zhang" , "Pankaj Gupta" , "Konrad Rzeszutek Wilk" , "Rik van Riel" , "lcapitulino@redhat.com" , "Wang, Wei W" , "Andrea Arcangeli" , "Paolo Bonzini" , "Dan Williams" References: <20191001152441.27008.99285.stgit@localhost.localdomain> <7233498c-2f64-d661-4981-707b59c78fd5@redhat.com> <1ea1a4e11617291062db81f65745b9c95fd0bb30.camel@linux.intel.com> <8bd303a6-6e50-b2dc-19ab-4c3f176c4b02@redhat.com> <0a16b11e-ec3b-7196-5b7f-e7395876cf28@redhat.com> <7fc13837-546c-9c4a-1456-753df199e171@redhat.com> <5b6e0b6df46c03bfac906313071ac0362d43c432.camel@linux.intel.com> From: "Nitesh Narayan Lal" Openpgp: preference=signencrypt Autocrypt: addr=nitesh@redhat.com; prefer-encrypt=mutual; keydata= mQINBFl4pQoBEADT/nXR2JOfsCjDgYmE2qonSGjkM1g8S6p9UWD+bf7YEAYYYzZsLtbilFTe z4nL4AV6VJmC7dBIlTi3Mj2eymD/2dkKP6UXlliWkq67feVg1KG+4UIp89lFW7v5Y8Muw3Fm uQbFvxyhN8n3tmhRe+ScWsndSBDxYOZgkbCSIfNPdZrHcnOLfA7xMJZeRCjqUpwhIjxQdFA7 n0s0KZ2cHIsemtBM8b2WXSQG9CjqAJHVkDhrBWKThDRF7k80oiJdEQlTEiVhaEDURXq+2XmG jpCnvRQDb28EJSsQlNEAzwzHMeplddfB0vCg9fRk/kOBMDBtGsTvNT9OYUZD+7jaf0gvBvBB lbKmmMMX7uJB+ejY7bnw6ePNrVPErWyfHzR5WYrIFUtgoR3LigKnw5apzc7UIV9G8uiIcZEn C+QJCK43jgnkPcSmwVPztcrkbC84g1K5v2Dxh9amXKLBA1/i+CAY8JWMTepsFohIFMXNLj+B RJoOcR4HGYXZ6CAJa3Glu3mCmYqHTOKwezJTAvmsCLd3W7WxOGF8BbBjVaPjcZfavOvkin0u DaFvhAmrzN6lL0msY17JCZo046z8oAqkyvEflFbC0S1R/POzehKrzQ1RFRD3/YzzlhmIowkM BpTqNBeHEzQAlIhQuyu1ugmQtfsYYq6FPmWMRfFPes/4JUU/PQARAQABtCVOaXRlc2ggTmFy YXlhbiBMYWwgPG5pbGFsQHJlZGhhdC5jb20+iQI9BBMBCAAnBQJZeKUKAhsjBQkJZgGABQsJ CAcCBhUICQoLAgQWAgMBAh4BAheAAAoJEKOGQNwGMqM56lEP/A2KMs/pu0URcVk/kqVwcBhU SnvB8DP3lDWDnmVrAkFEOnPX7GTbactQ41wF/xwjwmEmTzLrMRZpkqz2y9mV0hWHjqoXbOCS 6RwK3ri5e2ThIPoGxFLt6TrMHgCRwm8YuOSJ97o+uohCTN8pmQ86KMUrDNwMqRkeTRW9wWIQ EdDqW44VwelnyPwcmWHBNNb1Kd8j3xKlHtnS45vc6WuoKxYRBTQOwI/5uFpDZtZ1a5kq9Ak/ MOPDDZpd84rqd+IvgMw5z4a5QlkvOTpScD21G3gjmtTEtyfahltyDK/5i8IaQC3YiXJCrqxE r7/4JMZeOYiKpE9iZMtS90t4wBgbVTqAGH1nE/ifZVAUcCtycD0f3egX9CHe45Ad4fsF3edQ ESa5tZAogiA4Hc/yQpnnf43a3aQ67XPOJXxS0Qptzu4vfF9h7kTKYWSrVesOU3QKYbjEAf95 NewF9FhAlYqYrwIwnuAZ8TdXVDYt7Z3z506//sf6zoRwYIDA8RDqFGRuPMXUsoUnf/KKPrtR ceLcSUP/JCNiYbf1/QtW8S6Ca/4qJFXQHp0knqJPGmwuFHsarSdpvZQ9qpxD3FnuPyo64S2N Dfq8TAeifNp2pAmPY2PAHQ3nOmKgMG8Gn5QiORvMUGzSz8Lo31LW58NdBKbh6bci5+t/HE0H pnyVf5xhNC/FuQINBFl4pQoBEACr+MgxWHUP76oNNYjRiNDhaIVtnPRqxiZ9v4H5FPxJy9UD Bqr54rifr1E+K+yYNPt/Po43vVL2cAyfyI/LVLlhiY4yH6T1n+Di/hSkkviCaf13gczuvgz4 KVYLwojU8+naJUsiCJw01MjO3pg9GQ+47HgsnRjCdNmmHiUQqksMIfd8k3reO9SUNlEmDDNB XuSzkHjE5y/R/6p8uXaVpiKPfHoULjNRWaFc3d2JGmxJpBdpYnajoz61m7XJlgwl/B5Ql/6B dHGaX3VHxOZsfRfugwYF9CkrPbyO5PK7yJ5vaiWre7aQ9bmCtXAomvF1q3/qRwZp77k6i9R3 tWfXjZDOQokw0u6d6DYJ0Vkfcwheg2i/Mf/epQl7Pf846G3PgSnyVK6cRwerBl5a68w7xqVU 4KgAh0DePjtDcbcXsKRT9D63cfyfrNE+ea4i0SVik6+N4nAj1HbzWHTk2KIxTsJXypibOKFX 2VykltxutR1sUfZBYMkfU4PogE7NjVEU7KtuCOSAkYzIWrZNEQrxYkxHLJsWruhSYNRsqVBy KvY6JAsq/i5yhVd5JKKU8wIOgSwC9P6mXYRgwPyfg15GZpnw+Fpey4bCDkT5fMOaCcS+vSU1 UaFmC4Ogzpe2BW2DOaPU5Ik99zUFNn6cRmOOXArrryjFlLT5oSOe4IposgWzdwARAQABiQIl BBgBCAAPBQJZeKUKAhsMBQkJZgGAAAoJEKOGQNwGMqM5ELoP/jj9d9gF1Al4+9bngUlYohYu 0sxyZo9IZ7Yb7cHuJzOMqfgoP4tydP4QCuyd9Q2OHHL5AL4VFNb8SvqAxxYSPuDJTI3JZwI7 d8JTPKwpulMSUaJE8ZH9n8A/+sdC3CAD4QafVBcCcbFe1jifHmQRdDrvHV9Es14QVAOTZhnJ vweENyHEIxkpLsyUUDuVypIo6y/Cws+EBCWt27BJi9GH/EOTB0wb+2ghCs/i3h8a+bi+bS7L FCCm/AxIqxRurh2UySn0P/2+2eZvneJ1/uTgfxnjeSlwQJ1BWzMAdAHQO1/lnbyZgEZEtUZJ x9d9ASekTtJjBMKJXAw7GbB2dAA/QmbA+Q+Xuamzm/1imigz6L6sOt2n/X/SSc33w8RJUyor SvAIoG/zU2Y76pKTgbpQqMDmkmNYFMLcAukpvC4ki3Sf086TdMgkjqtnpTkEElMSFJC8npXv 3QnGGOIfFug/qs8z03DLPBz9VYS26jiiN7QIJVpeeEdN/LKnaz5LO+h5kNAyj44qdF2T2AiF HxnZnxO5JNP5uISQH3FjxxGxJkdJ8jKzZV7aT37sC+Rp0o3KNc+GXTR+GSVq87Xfuhx0LRST NK9ZhT0+qkiN7npFLtNtbzwqaqceq3XhafmCiw8xrtzCnlB/C4SiBr/93Ip4kihXJ0EuHSLn VujM7c/b4pps Organization: Red Hat Inc, Message-ID: <9bd52b8e-fa9e-a5ad-de39-660684757cdb@redhat.com> Date: Wed, 9 Oct 2019 12:25:00 -0400 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.8.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Content-Language: en-US X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.6.2 (mx1.redhat.com [10.5.110.67]); Wed, 09 Oct 2019 16:25:12 +0000 (UTC) List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-MS-Exchange-Organization-ExpirationStartTime: 09 Oct 2019 17:35:35.3362 (UTC) X-MS-Exchange-Organization-ExpirationStartTimeReason: OriginalSubmit X-MS-Exchange-Organization-ExpirationInterval: 1:00:00:00.0000000 X-MS-Exchange-Organization-ExpirationIntervalReason: OriginalSubmit X-MS-Exchange-Organization-Network-Message-Id: 2008687f-ebd0-429e-5b2c-08d74cdf1796 X-EOPAttributedMessage: 0 X-MS-Exchange-Organization-MessageDirectionality: Originating X-Forefront-Antispam-Report: CIP:128.138.67.75;IPV:CAL;CTRY:US;EFV:NLI;SFV:SKN;SFS:;DIR:INB;SFP:;SCL:-1;SRVR:MWHPR03MB2815;H:ipmx4.colorado.edu;FPR:;SPF:None;LANG:en;;SKIP:1; X-MS-Exchange-Organization-AuthSource: SN1NAM01FT041.eop-nam01.prod.protection.outlook.com X-MS-Exchange-Organization-AuthAs: Anonymous X-OriginatorOrg: colorado.edu X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 2008687f-ebd0-429e-5b2c-08d74cdf1796 X-MS-TrafficTypeDiagnostic: MWHPR03MB2815:|MWHPR03MB2815: X-MS-Exchange-Organization-SCL: -1 X-MS-Oob-TLC-OOBClassifiers: OLM:10000; X-Microsoft-Antispam: BCL:0; X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Oct 2019 17:35:35.0990 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 2008687f-ebd0-429e-5b2c-08d74cdf1796 X-MS-Exchange-CrossTenant-Id: 3ded8b1b-070d-4629-82e4-c0b019f46057 X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3ded8b1b-070d-4629-82e4-c0b019f46057;Ip=[128.138.67.75];Helo=[ipmx4.colorado.edu] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR03MB2815 X-MS-Exchange-Transport-EndToEndLatency: 00:23:32.4069755 X-MS-Exchange-Processed-By-BccFoldering: 15.20.2347.014 X-Microsoft-Antispam-Mailbox-Delivery: ucf:0;jmr:0;ex:0;auth:0;dest:I;ENG:(750127)(520002050)(944506383)(944626516); X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?6bw8eFEgV31cF7DA8lnVbg+TRV2aPe7+6B28zgFdxW5eTjr6z3bMDlx5AFLD?= =?us-ascii?Q?m9Pa2gi0zLLOjpxLSREFL8/5tK22rYbArS+0Y6eqmhyKssz+GiQxgzqM97eD?= =?us-ascii?Q?Z/ycdgQzF7F1C/QWA0q6kB3HJACP+24m1pjQHEzxGh3OT6ZIsOdtO8KqZYY6?= =?us-ascii?Q?Bz6lErm69woHvbDPkwhnO85VkngYx1Gcd9TAhvvr7lLj/8N64fGsnDG22GT+?= =?us-ascii?Q?owou5FT24rNj9j/p15cYWLll7rRXoCAptg+Y3CeE0CXkY2UoIiQNAzTWcH4d?= =?us-ascii?Q?O02eHWegCJ4u7W9wuJk83cr1rnXQhR2jm3Vs67hjDkXpbG5CCdKNSKoPazkq?= =?us-ascii?Q?LTvtXDcwid4tKZuAj4ot57Bqe1urOW5JzXc3bF1202vQh+X5bfgDa6ea7XHI?= =?us-ascii?Q?fK4kE8uG0hf4vaURXMbvvCtKuewzIx2rbl5cSwsHHareNmbLXOr0RFZinQ1R?= =?us-ascii?Q?QhAzHf0xrEtyF5XKwbd1ooPtfwwue/49P9ZXNyoaeMT3HOfnm88uhMtL8BKw?= =?us-ascii?Q?5g5nZgi1ksgXBAMp/WMDatYglm5AddRo1P+YPsbDXalmXZcJY346cd/lQpyD?= =?us-ascii?Q?UnOlKXqD5gcJyPa+kGGA/cJgowtR6elN0c667dputAUETppW9eGSDcxpQtmM?= =?us-ascii?Q?s8g8rREH+IoIUqYfjE6FO8Rn8kY58m7Wavw6UoiSCtFqxtc2cBtDrbebzUQ+?= =?us-ascii?Q?w31/LeB9nCnpSTaZ6FRL45uviIktxYeHGijkTp9wCJSpR1+UuSS99RfnlDBv?= =?us-ascii?Q?ovnEpb2pIlhQ1dV9kadb+P7XhvPJbHIVp8vfd9svdYIJUySWPoY9MNSf3dym?= =?us-ascii?Q?NUZeF6teMVZcD9rdDQNFIdOhV/wjNxoT9FcnOTHXBCv3wt5Ai8JhHsigtUKV?= =?us-ascii?Q?aUA26Hve0tXEMe4GDV9KNclde5F5HAxBfxZA/B3SssQtmFXUiUmU6wRnH6p4?= =?us-ascii?Q?+iJX4LSN/Efy0hJBMPYsnHr8jvU4UmhydwYZ6SUnlgknJCjc/A/plO8I11IA?= =?us-ascii?Q?QTXIAHPfKLJvb63nr9I9IITmSPQpnLxQ7vKtyEQKweauwG5UoYnlCycy9iyw?= =?us-ascii?Q?jGOKQuhvuA9oXK6tX0UPdVdNGv8OXc7Ll1BsZ7u8ezECpL4rOaI=3D?= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Message-ID: <20191009162500.A6wP7q2q0y-jfLIAu0me6Z5xy3AC77gbgPqlmSTJXU8@z> On 10/7/19 1:20 PM, Alexander Duyck wrote: > On Mon, Oct 7, 2019 at 10:07 AM Nitesh Narayan Lal = wrote: >> >> On 10/7/19 12:27 PM, Alexander Duyck wrote: >>> On Mon, 2019-10-07 at 12:19 -0400, Nitesh Narayan Lal wrote: >>>> On 10/7/19 11:33 AM, Alexander Duyck wrote: >>>>> On Mon, 2019-10-07 at 08:29 -0400, Nitesh Narayan Lal wrote: >>>>>> On 10/2/19 10:25 AM, Alexander Duyck wrote: > > >>>>>> page_reporting=2Ec change: >>>>>> @@ -101,8 +101,12 @@ static void scan_zone_bitmap(struct page_report= ing_config >>>>>> *phconf, >>>>>> /* Process only if the page is still online */ >>>>>> page =3D pfn_to_online_page((setbit << PAGE_REPORTIN= G_MIN_ORDER) + >>>>>> zone->base_pfn); >>>>>> - if (!page) >>>>>> + if (!page || !PageBuddy(page)) { >>>>>> + clear_bit(setbit, zone->bitmap); >>>>>> + atomic_dec(&zone->free_pages); >>>>>> continue; >>>>>> + } >>>>>> >>>>> I suspect the zone->free_pages is going to be expensive for you to de= al >>>>> with=2E It is a global atomic value and is going to have the cachelin= e >>>>> bouncing that it is contained in=2E As a result thinks like setting t= he >>>>> bitmap with be more expensive as every tome a CPU increments free_pag= es it >>>>> will likely have to take the cache line containing the bitmap pointer= as >>>>> well=2E >>>> I see I will have to explore this more=2E I am wondering if there is a= way to >>>> measure this If its effect is not visible in will-it-scale/page_fault1= =2E If >>>> there is a noticeable amount of degradation, I will have to address th= is=2E >>> If nothing else you might look at seeing if you can split up the >>> structures so that the bitmap and nr_bits is in a different region >>> somewhere since those are read-mostly values=2E >> ok, I will try to understand the issue and your suggestion=2E >> Thank you for bringing this up=2E >> >>> Also you are now updating the bitmap and free_pages both inside and >>> outside of the zone lock so that will likely have some impact=2E >> So as per your previous suggestion, I have made the bitmap structure >> object as a rcu protected pointer=2E So we are safe from that side=2E >> The other downside which I can think of is a race where one page >> trying to increment free_pages and other trying to decrements it=2E >> However, being an atomic variable that should not be a problem=2E >> Did I miss anything? > I'm not so much worried about a race as the cache line bouncing > effect=2E Basically your notifier combined within this hinting thread > will likely result in more time spent by the thread that holds the > lock since it will be trying to access the bitmap to set the bit and > the free_pages to report the bit, but at the same time you will have > this thread clearing bits and decrementing the free_pages values=2E > > One thing you could consider in your worker thread would be to do > reallocate and replace the bitmap every time you plan to walk it=2E By > doing that you would avoid the cacheline bouncing on the bitmap since > you would only have to read it, and you would no longer have another > thread dirtying it=2E You could essentially reset the free_pages at the > same time you replace the bitmap=2E It would need to all happen with the > zone lock held though when you swap it out=2E If I am not mistaken then from what you are suggesting, I will have to hold= the zone lock for the entire duration of swap & scan which would be costly = if the bitmap is large, isn't? Also, we might end up missing free pages that a= re getting freed while we are scanning=2E As far as free_pages count is concerned, I am thinking if I should replace it with zone->free_area[REPORTING_ORDER]=2Enr_free which is already= there (I still need to explore this in a bit more depth)=2E > > - Alex --=20 Thanks Nitesh