From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.3 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 351F7C3F2D0 for ; Fri, 28 Feb 2020 09:50:05 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id D86B52469F for ; Fri, 28 Feb 2020 09:50:04 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D86B52469F Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=suse.de Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 368F26B0005; Fri, 28 Feb 2020 04:50:04 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 318556B0006; Fri, 28 Feb 2020 04:50:04 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 22F1C6B0007; Fri, 28 Feb 2020 04:50:04 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0098.hostedemail.com [216.40.44.98]) by kanga.kvack.org (Postfix) with ESMTP id 089226B0005 for ; Fri, 28 Feb 2020 04:50:04 -0500 (EST) Received: from smtpin08.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id B39DB180AD801 for ; Fri, 28 Feb 2020 09:50:03 +0000 (UTC) X-FDA: 76539064686.08.drum67_3aadefeb8515f X-HE-Tag: drum67_3aadefeb8515f X-Filterd-Recvd-Size: 4986 Received: from mx2.suse.de (mx2.suse.de [195.135.220.15]) by imf37.hostedemail.com (Postfix) with ESMTP for ; Fri, 28 Feb 2020 09:50:03 +0000 (UTC) X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay2.suse.de (unknown [195.135.220.254]) by mx2.suse.de (Postfix) with ESMTP id 093F6B2D4; Fri, 28 Feb 2020 09:50:01 +0000 (UTC) Date: Fri, 28 Feb 2020 09:49:54 +0000 From: Mel Gorman To: "Huang, Ying" Cc: David Hildenbrand , Matthew Wilcox , Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Vlastimil Babka , Zi Yan , Michal Hocko , Peter Zijlstra , Dave Hansen , Minchan Kim , Johannes Weiner , Hugh Dickins , Alexander Duyck Subject: Re: [RFC 0/3] mm: Discard lazily freed pages when migrating Message-ID: <20200228094954.GB3772@suse.de> References: <20200228033819.3857058-1-ying.huang@intel.com> <20200228034248.GE29971@bombadil.infradead.org> <87a7538977.fsf@yhuang-dev.intel.com> <871rqf850z.fsf@yhuang-dev.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-15 Content-Disposition: inline In-Reply-To: <871rqf850z.fsf@yhuang-dev.intel.com> User-Agent: Mutt/1.10.1 (2018-07-13) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Fri, Feb 28, 2020 at 04:55:40PM +0800, Huang, Ying wrote: > > E.g., free page reporting in QEMU wants to use MADV_FREE. The guest will > > report currently free pages to the hypervisor, which will MADV_FREE the > > reported memory. As long as there is no memory pressure, there is no > > need to actually free the pages. Once the guest reuses such a page, it > > could happen that there is still the old page and pulling in in a fresh > > (zeroed) page can be avoided. > > > > AFAIKs, after your change, we would get more pages discarded from our > > guest, resulting in more fresh (zeroed) pages having to be pulled in > > when a guest touches a reported free page again. But OTOH, page > > migration is speed up (avoiding to migrate these pages). > > Let's look at this problem in another perspective. To migrate the > MADV_FREE pages of the QEMU process from the node A to the node B, we > need to free the original pages in the node A, and (maybe) allocate the > same number of pages in the node B. So the question becomes > > - we may need to allocate some pages in the node B > - these pages may be accessed by the application or not > - we should allocate all these pages in advance or allocate them lazily > when they are accessed. > > We thought the common philosophy in Linux kernel is to allocate lazily. > I also think there needs to be an example of a real application that benefits from this new behaviour. Consider the possible sources of page migration 1. NUMA balancing -- The application has to read/write the data for this to trigger. In the case of write, MADV_FREE is cancelled and it's mostly likely going to be a write unless it's an application bug. 2. sys_movepages -- the application has explictly stated the data is in use on a particular node yet any MADV_FREE page gets discarded 3. Compaction -- there may be no memory pressure at all but the MADV_FREE memory is discarded prematurely In the first case, the data is explicitly in use, most likely due to a write in which case it's inappropriate to discard. Discarding and reallocating a zero'd page is not expected. In second case, the data is likely in use or else why would the system call be used? In the third case the timing of when MADV_FREE pages disappear is arbitrary as it can happen without any actual memory pressure. This may or may not be problematic but it leads to unpredictable latencies for applications that use MADV_FREE for a quick malloc/free implementation. Before, as long as there is no pressure, the reuse of a MADV_FREE incurs just a small penalty but now with compaction it does not matter if the system avoids memory pressure because they may still incur a fault to allocate and zero a new page. There is a hypothetical fourth case which I only mention because of your email address. If persistent memory is ever used for tiered memory then MADV_FREE pages that migrate from dram to pmem gets discarded instead of migrated. When it's reused, it gets reallocated from dram regardless of whether that region is hot or not. This may lead to an odd scenario whereby applications occupy dram prematurely due to a single reference of a MADV_FREE page. It's all subtle enough that we really should have an example application in mind that benefits so we can weigh the benefits against the potential risks. -- Mel Gorman SUSE Labs