From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.5 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, HTML_MESSAGE,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F2352C432BE for ; Thu, 26 Aug 2021 15:04:38 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 4EDAC60F44 for ; Thu, 26 Aug 2021 15:04:38 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 4EDAC60F44 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=nutanix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id D646C900002; Thu, 26 Aug 2021 11:04:37 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C7C248D0001; Thu, 26 Aug 2021 11:04:37 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AA762900002; Thu, 26 Aug 2021 11:04:37 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0126.hostedemail.com [216.40.44.126]) by kanga.kvack.org (Postfix) with ESMTP id 864278D0001 for ; Thu, 26 Aug 2021 11:04:37 -0400 (EDT) Received: from smtpin33.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 26D9F8249980 for ; Thu, 26 Aug 2021 15:04:37 +0000 (UTC) X-FDA: 78517553394.33.A33509E Received: from mx0a-002c1b01.pphosted.com (mx0a-002c1b01.pphosted.com [148.163.151.68]) by imf11.hostedemail.com (Postfix) with ESMTP id 24F1FF00020E for ; Thu, 26 Aug 2021 15:04:35 +0000 (UTC) Received: from pps.filterd (m0127840.ppops.net [127.0.0.1]) by mx0a-002c1b01.pphosted.com (8.16.1.2/8.16.0.43) with SMTP id 17QEkYe7003561; Thu, 26 Aug 2021 08:02:18 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nutanix.com; h=from : to : cc : subject : date : message-id : references : in-reply-to : content-type : mime-version; s=proofpoint20171006; bh=QR/cXU7e3VUQwTXkvYtPyKzXlLI1PA30ijobjqmehwg=; b=hZZ7iTqJdZq08ewXTKrTmVt4ggYZoQ4Z/EeWy+VEOzZ307Yhtw3nKjiwgiHgJPwxxSMd Ksr/HRmST6kbVuQ66WyF13DXunwAseqYcxKgxnw4p8xpFb5q7nLSub+4TjiZYzFm5GD7 yCyLIsAKmt71xhIDs4wxOksbRLl4j7aKoWH0q22ncOGRZKwXNMCrKaDaWFozipTwxqZJ wzXdajrz3IxaJ+khT7IaFKrXZKPWmoQAxidh0xLKY3h/fEWkAGm3C9ka7DrGzFEWH1Md kkvamNVn4oouVNXfF0y8DwSj+fF9dBlfox6wOcRr6TVEoXb9/6R9Ku4bW4H5xXYQvzzd 9Q== Received: from nam12-dm6-obe.outbound.protection.outlook.com (mail-dm6nam12lp2172.outbound.protection.outlook.com [104.47.59.172]) by mx0a-002c1b01.pphosted.com with ESMTP id 3ap6ct8v2v-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 26 Aug 2021 08:02:16 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Aia7lu/WT2oNBbJyxObZ9GNvqFQrbswuP7RLprC10Mz2csqgGnlFljntus/bgh6BSMEgHpJk3ElgrRbqXpO2HX4zy3EiRhme9mqYyIRygNbgTf6j3h58Hx+u6AYS9oG04UKOAkig6sLS3PfZPldVEYdjKh5bokTyTtRSvSiuruXpi/KF5I5sgOvTyx7vK313wJzXwYKPVXCU5i4CIkDWeBTSoPEtgcYkbESbccDKO7314u2F0kzjicR6GNrhCZIgDPCEc4A9xENuwuHvQ1P+/FH7g5UkYPXDj8rj5ODqH07IlSflyI+v9C5qxmWwu3yG9RuUyTI69B0W0eE3kLsZpA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=QR/cXU7e3VUQwTXkvYtPyKzXlLI1PA30ijobjqmehwg=; b=IO7C9UoSUjxRPGhBOwznV14s/GuKXuVtKXsMjRahHQpIjDsgWCTITwkDaV+lAvhQcv2gdt1exjiI6Hq3VblyF7ee6Hv1IXsukTPnYAzc43AQ+qO/YQzXZ2DihsiH7uExvgUePKKXAE3gBFrfeobbRmC8OeMRM7Mjrouy8HujqCVac5GZlEjNOO9HWpZ+c+ZeRgYwIINPyfsK5pTC0o7alEJhPXguSlp5MK22sUCOtlbAVhy5acWY5I+cnFrLKBpsapMyXqWRlPSHo6GQFxV+CZoHyaKV25+fzwR4O+VT40KakPjl6tSN5s/EFzsHcTyWpcDRVPSS5kDEbSQJZzeV0Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nutanix.com; dmarc=pass action=none header.from=nutanix.com; dkim=pass header.d=nutanix.com; arc=none Received: from DM6PR02MB5578.namprd02.prod.outlook.com (2603:10b6:5:79::13) by DM6PR02MB4811.namprd02.prod.outlook.com (2603:10b6:5:18::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4436.22; Thu, 26 Aug 2021 15:01:45 +0000 Received: from DM6PR02MB5578.namprd02.prod.outlook.com ([fe80::5116:80a5:77d9:fcc4]) by DM6PR02MB5578.namprd02.prod.outlook.com ([fe80::5116:80a5:77d9:fcc4%7]) with mapi id 15.20.4457.019; Thu, 26 Aug 2021 15:01:45 +0000 From: Tiberiu Georgescu To: Peter Xu CC: David Hildenbrand , "linux-kernel@vger.kernel.org" , "linux-mm@kvack.org" , Alistair Popple , Ivan Teterevkov , Mike Rapoport , Hugh Dickins , Matthew Wilcox , Andrea Arcangeli , "Kirill A . Shutemov" , Andrew Morton , Mike Kravetz , "Carl Waldspurger [C]" , Florian Schmidt , Jonathan Davies , Chris Riches Subject: Re: [PATCH RFC 0/4] mm: Enable PM_SWAP for shmem with PTE_MARKER Thread-Topic: [PATCH RFC 0/4] mm: Enable PM_SWAP for shmem with PTE_MARKER Thread-Index: AQHXizvfxbSDphJsCEy6t9AqGpL5qqt3d0AAgACHggCAABs6gIAAGzmAgADJJACAAJ7TgIAABe+AgAFamQCAACqDAIABiCIAgAAnsYCAB37yAIAAFjcAgAGS4oA= Date: Thu, 26 Aug 2021 15:01:45 +0000 Message-ID: <40A87343-266E-469D-BAE5-667A2E1DB238@nutanix.com> References: <7F645772-1212-4F0D-88AF-2569D5BBC2CD@nutanix.com> <6ab58270-c487-2a56-b522-ea5100edb13c@redhat.com> <0A4C4E37-88C9-4490-9D8B-6990D805F447@nutanix.com> <5766d353-6ff8-fdfa-f8f9-764e8de9b5aa@redhat.com> <53EBB3A1-06C0-418A-B20E-5BFD5212D7C3@nutanix.com> In-Reply-To: Accept-Language: en-GB, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-mailer: Apple Mail (2.3654.120.0.1.13) x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: bb92e0d1-440c-48ef-28c6-08d968a26bb6 x-ms-traffictypediagnostic: DM6PR02MB4811: x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-proofpoint-crosstenant: true x-ms-oob-tlc-oobclassifiers: OLM:10000; x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: /kwDagx36EhAHWd38zwt4Nb3j+R+8CtXnBjWW2moHXUh3l531Reix2HHB6YxtWlUpJb18JIU2KgwEI/xuUb95coEDWBzjJP4bFAPM3iULb+68MzdNG3RgmRINwysySZNXfsLgOSMcyvZ40WCLvy1/s6yeDrnHaYQPKAusYyk2Oq4tSRSRhGXLo8Zp35/GHSr+173/ljavLgFbRP5x1eeSzp+onlCmDR9dK3YotVnwSH4W57vEP4ToJqwFRN4GK44FNTqNmIGv4wSF+hwtTs6WLPshSVUN/KGg3tidvjqKiUzV/3ytW9r8Z8KQcsqJr5UZIaTl2Xq8CJ4AKkRnjRo/omfQp4Xx2Mkxhsc2ifAIWtXbavJPTYBPUDctcxNMlvKehMi+uR4ikWXP97dhUP6elM8YQpjHgufgD1EQuhzuem66RJM3ksQY89pIpj2uR9ptMOCibyRvbW88TjyB6t/kgwvTn0uQUsQoUlp6kZ53/BmPLxkPw0eyxdWtwUh/f9zgowf59PCi7KRSEymTTpS/Ivt1dtldAKNQh5hVVp/KuJbSRQIbfEbxG72p82K8U8eeInVhBZxLUSONeiOFTYYqnB5BMviUYLM38oKv0IcG31oGyw7XEtBaQA29uv/1c4daZzNbY7a/rmMtMCjKEV7Zl7+9QoJLk93Zv1SV1NILBhLiZ6nePXDkhnizbfu/kMEKuQWuQ/xPM2GPogp07T1UYF736FQls4cCK1PrUsIa9p7Qadpylrn1m9/wPIZScjj x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR02MB5578.namprd02.prod.outlook.com;PTR:;CAT:NONE;SFS:(346002)(376002)(366004)(39860400002)(396003)(136003)(38100700002)(53546011)(54906003)(86362001)(6512007)(316002)(91956017)(478600001)(6916009)(7416002)(71200400001)(6506007)(33656002)(4326008)(76116006)(83380400001)(44832011)(5660300002)(107886003)(66476007)(66556008)(36756003)(8676002)(66946007)(66446008)(64756008)(186003)(122000001)(2616005)(2906002)(8936002)(38070700005)(6486002)(45980500001)(559001)(579004);DIR:OUT;SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?JcLXlPHA9pSV4WWW8Y8ym9UoPXTwyy5tTORVxFDfH0u8D80S/y/ybnkMaMPB?= =?us-ascii?Q?YRhl44PkYKh+QMnGtOo1xvzmcMnj2Qyy0M6XFl6EOdSujg4CB4XFraUEBNi0?= =?us-ascii?Q?ty7F5L43G2CwgCkdfAtf58TMBxCtBIZTH0OPpaDZxBX7hBfMFX+sp5VTbxWv?= =?us-ascii?Q?tDjAWrsLH+DWqLtYaNf/twml2O8cogEVFJLscPrSF90pwWZKXgExsuLT/RP7?= =?us-ascii?Q?T8SqVLx73RKkiY1yYdSXi5Q0tXHEYEsDDr+b9diQ+dgsnJABfhGIj1A39A99?= =?us-ascii?Q?quAPUJlChZU6S8efASF/OWQaOAhyKJJEQUWtQWPk4Bc9amgDM1apx2FYkbaE?= =?us-ascii?Q?zpY1flKHiuL99qdusTJOOWAqYcDfXPzZMLHIoS3jXaGjNg+cWuqO9ddJJfaz?= =?us-ascii?Q?NZWudAeJVEHQJPJl7eKRriPZkWbXEVgQJADgW4cy+S4DSwC0oBexxiDGINtE?= =?us-ascii?Q?F/ZUsjaJyZtlRhKbweLLO9BbLd+LAonejY0c7akhzuPnT4c+xMLxrPvlJ5zH?= =?us-ascii?Q?Td665aQVj+gwy2kTbJwiRpJFwltQka+B2shEhwrCBP8HdmbbxEhoe+RZxB2N?= =?us-ascii?Q?9CIkcYxjUo2G3uGfo8lwDiI5sEWviHKiunTW/4EyHbJop2nKrr6Y+VkjM8YG?= =?us-ascii?Q?3dkarqWiu3BJfknMPMzHcXTcQutIpnk39lHFXmRZEMSCJ4dBOdw/RkdKok3n?= =?us-ascii?Q?hwtY/AgnP4CpGuKhoYULIJcaj07SGn0+6jT8hd4MtIUIUX+svLswcvL+fa5w?= =?us-ascii?Q?vAOQKT3Ost6UqJdWqlSwdrlArwor4gEGInO0cSDNOHKs576OE1m+pZZF+9JO?= =?us-ascii?Q?ioZcXE3tEYAA81Qn1q3IDt4Hi0er3pPTF4McoDED48dBNQBqvCoyEC/2F3zE?= =?us-ascii?Q?vMJYtD2p1WzdPLBLTstPnF5WS45vhRqEq98NN+MCON9NaZfkkeQsUau0VtTP?= =?us-ascii?Q?Ug71dfkuSBo3Cx2z0QtNgQU8OXxQW2AoTSP0pcJzwfZrW7NjmPfG6xr8Ybgn?= =?us-ascii?Q?33SYj5RQoYJsBU2AYd5feUBI4SGRnLN5+vUls3sZNUJG3PYqX2jPjetqxTpn?= =?us-ascii?Q?S8kYJoPXCfQxJleKapAt+JnF0EC8zKdfIfLR+Plt+g8Voh9RSTAVuPo2/vRq?= =?us-ascii?Q?szY8K7vsKYH0Un0/9k1kjfbT8xs9jehMal9+cilSf9h9R9C1gmwyNrDkkDG+?= =?us-ascii?Q?u+ERShXx6M8X6cW8uTSJlydCAvhF6VtgDV8XYO85xL3o4rFRqAxNIgOIAmp3?= =?us-ascii?Q?sI3K3N8gAWToLVOdi230M48L8YH6I+RtkiYGTwJpYbkRnjxd6rbRQqbi9+1p?= =?us-ascii?Q?H4gv98qKSmJNl+nvLJY8zlFnSXxz2M7VQ2+22Qp8hxJrN2OFqMVf8amW2jDK?= =?us-ascii?Q?fbDxbeXn08SMdn1qANRWwsSM3RS4?= Content-Type: multipart/alternative; boundary="_000_40A87343266E469DBAE5667A2E1DB238nutanixcom_" MIME-Version: 1.0 X-OriginatorOrg: nutanix.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: DM6PR02MB5578.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: bb92e0d1-440c-48ef-28c6-08d968a26bb6 X-MS-Exchange-CrossTenant-originalarrivaltime: 26 Aug 2021 15:01:45.0226 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: bb047546-786f-4de1-bd75-24e5b6f79043 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: M46mN0SVKP4ziLY15wICw/9v3zPO7ioI9b/PI9N/Hpvg1AWMKRvNPuXW4CyWTxCQqRl4n6pTe7SqEUu83Ara0e1ZI2Pv7tazgULM9/ibiXA= X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR02MB4811 X-Proofpoint-ORIG-GUID: u7usqlYG9qXfbuns_2YQCYO04q5M2ANf X-Proofpoint-GUID: u7usqlYG9qXfbuns_2YQCYO04q5M2ANf X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.182.1,Aquarius:18.0.790,Hydra:6.0.391,FMLib:17.0.607.475 definitions=2021-08-26_04,2021-08-26_02,2020-04-07_01 X-Proofpoint-Spam-Reason: safe Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=nutanix.com header.s=proofpoint20171006 header.b=hZZ7iTqJ; dmarc=pass (policy=none) header.from=nutanix.com; spf=none (imf11.hostedemail.com: domain of tiberiu.georgescu@nutanix.com has no SPF policy when checking 148.163.151.68) smtp.mailfrom=tiberiu.georgescu@nutanix.com X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 24F1FF00020E X-Stat-Signature: yj8754pf5wcwkrk8d35acyf7bpj9ix5z X-HE-Tag: 1629990275-633388 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: --_000_40A87343266E469DBAE5667A2E1DB238nutanixcom_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable On 25 Aug 2021, at 15:59, Peter Xu > wrote: On Wed, Aug 25, 2021 at 01:40:13PM +0000, Tiberiu Georgescu wrote: Hello Peter, sorry for the late reply, Hi, Tiberiu, No worries on that. On Fri, Aug 20, 2021 at 04:49:58PM +0000, Tiberiu Georgescu wrote: Firstly, I am worried lseek with the SEEK_HOLE flag would page in pages fro= m swap, so using it would be a direct factor on its own output. If people are= working on Live Migration, this would not be ideal. I am not 100% sure this is how = lseek works, so please feel free to contradict me, but I think it would swap in s= ome of the pages that it seeks through, if not all, to figure out when to stop.= Unless it leverages the page cache somehow, or an internal bitmap. It shouldn't. Man page is clear on that: SEEK_DATA Adjust the file offset to the next location in the file greater than or equal to offset containing data. If offset points to data, then the file offset is set to offset. Ok, I got to test it out and you are right. lseek does not swap in pages. T= hat is great news. Again, I think your requirement is different from CRIU, so I think mincore(= ) is the right thing for you. Secondly, mincore() could return some "false positives" for this particular= use case. That is because it returns flag=3D1 for pages which are still in the = swap cache, so the output becomes ambiguous. I don't think so; mincore() should return flag=3D0 if it's either in swap c= ache or even got dropped from it. I think its name/doc also shows that in the f= act that "as long as it's not in RAM, the flag is cleared". That's why I think that should indeed be what you're looking for, if swp entry can be ignored. More below on that. By saying there are "false positives", I do not mean that the mincore() wou= ld not work as expected, only that its definition is a little more subtle than= that. And that it does not suit our needs entirely by itself. I tested mincore() compared to the pagemap, and I discovered that there are more flags set to 1 (not necessarily contiguous) compared to the pages page= map was reporting as present. By also looking through the code, I could only co= nclude that pages in the swap cache were considered "still in RAM", so were set to= 1 as well. When looking into what the swap cache does, it makes sense. Please see mincore_page(): /* * When tmpfs swaps out a page from a file, any process mapping that * file will not get a swp_entry_t in its pte, but rather it is like * any other file mapping (ie. marked !present and faulted in with * tmpfs's .fault). So swapped out tmpfs mappings are tested here. */ page =3D find_get_incore_page(mapping, index); if (page) { present =3D PageUptodate(page); put_page(page); } I think the "testing" means when swapped out, the page will be NULL. If it'= s just the pte got zapped, the page will be returned. The call stack should = look like: find_get_incore_page -> find_get_page -> pagecache_get_page(fgp_flags=3D= =3D0). I think the fgp_flags guaranteed it, with no FGP_ENTRY. Did you test mincore() without my patch (as my current patch will indeed ca= use more 1's returned than it should)? My guess is there's something else that made your test read more 1's with mincore() than pagemap, but I have no sol= id idea on that. I made sure to avoid any of our patches while testing mincore and counted t= he flags to make sure. There are more set than they are present. I will look i= nto the code again and test on a non-rc kernel version, just to be safe. I still think it makes sense for mincore to consider pages in the swap cach= e to be "in RAM". I start seeing how useful it can be to differentiate betwee= n present pages and in-swap-cache pages. We could use mincore() and pagemap to find the pages in the swap cache. In short, mincore() is not enough because it does not differentiate between present pages and swap-cache entries, as both are in RAM, but the latter is also in swap. It can be used with other tools to get more specific infor= mation though, so it is useful. Note that my series is as you mentioned missing the changes to support mincore() (otherwise I'll know the existance of it!). It'll be trivial to = add that, but let's see whether mincore() will satisfy your need. We are currently trying to make use of all tools that we have learned of so= far during our discussions (lseek, map_files, even mincore) to get the informat= ion that we need about swap pages. In theory, for many of our use cases, a combination of 2 or 3 should be enough. It is a little more convoluted than a simple pagemap call, but it can be mo= re versatile (using lseek to skip multiple unallocated pages). As to whether t= he swap bit (and more) should be eventually added on the pagemap, maybe this topic makes more sense to continue on the Documentation thread. [...] It is possible for the swap device to be network attached and shared, so mu= ltiple hosts would need to understand its content. Then it is no longer internal t= o one kernel only. By being swap-aware, we can skip swapped-out pages during migration (to pre= vent IO and potential thrashing), and transfer those pages in another way t= hat is zero-copy. That sounds reasonable, but I'm not aware of any user-API that exposes swap entries to userspace, or is there one? Good question. AFAIK, the swap device can be retrieved by using the swap ty= pe, which is part of the swap entry. During our discussions, I was always assum= ing that, if the pagemap entry kept track of the swap offset, it would keep tra= ck of the swap type and, conversely, the swap device as well. Sorry if I haven't made= this assumption clear until now. So we were relying on the pagemap to expose swap entry information. Seeing = it works for private pages, we thought it made sense to have worked on shared = pages as well. I.e., how do you know which swap device is which? How do you guarantee the kernel swp entry information won't change along with time? I don't think we can guarantee it unless we halt the guest. Yes, halting the guest helps, though then performance start to matter becau= se all time consumed in either pagemap or mincore() will be counted in as down= time of the VM live migration, and it's not "live" at all during this period. That's true. That is why this "halting" is supposed to only happen once, an= d its duration needs to be minimised by pre-copy. Live migration AFAIK is int= ended to have a single imperceptible pause at some point in order to converge mor= e quickly and cleanly. Without this halt, the whole migration procedure could= run indefinitely, or for days instead of hours/minutes. I'm not sure how it was done with private mappings before, because I though= t that's a pre-requisite knowledge to decide whether we should migrate a page= or not, but I might have missed something. We can stop vm, sample, start vm, = but it could become hiccups in the guest too, or otherwise contribute to downti= me when src/dst vm switches. I see your point. Safe to say the vm should never be stopped during pre-cop= y. See my comment above. But QEMU does most migration work in pre-copy using a best-effort approach anyway. So, having a way to retrieve temporary, but accurate information about swap entries (i.e. post-patch pagemap) should be enough to guarantee a smoother migration process. It is intended to be repeated, unless there is no change between iterations. The kernel will allocate swap device index which will be assigned as swp_ty= pe, right? If there're multiple swap devices, how do you know which swp_entry = is located on which device? I wanted to look for that info in "swapon -s" but= I didn't. Or maybe that solution only works if there's only one swap device? Besides, I also still have a question on the accuracy. If there's no formal= way for userspace to interact with the kernel, I'm wondering how to guarantee t= he page will be kept swapped out, and data located on the swap device will alw= ays be the latest? Because IMHO the kernel can swap in pages as wish, even if = it's not accessed from the userspace. After all, all these operations should be transparent to userspace. One example in my mind is we do have page fault-around enabled for shmem, s= o that even if the VM is stopped, its pages can be faulted in if (unluckily, = in this case, though) some page near the swapped out page got faulted in - it could be some qemu malloc()ed region that (again, unluckily) was allocated = to have a virtual address very close to the mmap()ed guest memories. I am not sure whether it's a real problem, e.g., even if some page swapped = in during guest halted for some reason, if no one is writting to that page, at least the data on the page will still be identical to the one located on th= e swap device. However I think that still sounds too tricky, and maybe fragi= le. Great example, and good point. Thank you for raising that up. This looks li= ke a very real problem that we need to take into consideration in our prototypes= . -- Kind regards, Tibi --_000_40A87343266E469DBAE5667A2E1DB238nutanixcom_ Content-Type: text/html; charset="us-ascii" Content-ID: Content-Transfer-Encoding: quoted-printable
On 25 Aug 2021, at 15:59, Peter Xu <peterx@redhat.com> wrote:

On Wed, Aug 25, 2021 at 01:40:13PM +0000, Tiberiu Georgescu wrote:
Hello Peter, sorry for the late reply,

Hi, Tiberiu,

No worries on that.


On Fri, Aug 20, 2021 at 04:49:58PM +00= 00, Tiberiu Georgescu wrote:
Firstly, I am worried lseek with the S= EEK_HOLE flag would page in pages from
swap, so using it would be a direct factor on its own output. If people are= working
on Live Migration, this would not be ideal. I am not 100% sure this is how = lseek
works, so please feel free to contradict me, but I think it would swap in s= ome
of the pages that it seeks through, if not all, to figure out when to stop.= Unless it
leverages the page cache somehow, or an internal bitmap.

It shouldn't.  Man page is clear on that:

     SEEK_DATA
            Adj= ust the file offset to the next location in the file greater
            tha= n or equal to offset containing data.  If offset points to
            dat= a, then the file offset is set to offset.

Ok, I got to test it out and you are right. lseek does not swap in pages. T= hat is
great news.

Again, I think your requirement is different from CRIU, so I think mincore(= ) is
the right thing for you.


Secondly, mincore() could return some "false positives" for this = particular use
case. That is because it returns flag=3D1 for pages which are still in the = swap
cache, so the output becomes ambiguous.

I don't think so; mincore() should return flag=3D0 if it's either in swap c= ache
or even got dropped from it.  I think its name/doc also shows that in = the fact
that "as long as it's not in RAM, the flag is cleared".  Tha= t's why I think
that should indeed be what you're looking for, if swp entry can be ignored.=
More below on that.

By saying there are "false positives", I do not mean that the min= core() would
not work as expected, only that its definition is a little more subtle than= that. And
that it does not suit our needs entirely by itself.

I tested mincore() compared to the pagemap, and I discovered that there are=
more flags set to 1 (not necessarily contiguous) compared to the pages page= map 
was reporting as present. By also looking through the code, I could only co= nclude
that pages in the swap cache were considered "still in RAM", so w= ere set to 1 as
well. When looking into what the swap cache does, it makes sense.

Please see mincore_page():

/*=
 * When tmpfs swaps out a page from a file, any process mapping that  * file will not get a swp_entry_t in its pte, but rather it is like  * any other file mapping (ie. marked !present and faulted in with
 * tmpfs's .fault). So swapped out tmpfs mappings are tested here.
 */
page =3D find_get_incore_page(mapping, index);
if (page) {
present =3D PageUptodate(page);
put_page(page); }<= br style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 1= 8px; font-style: normal; font-variant-caps: normal; font-weight: normal; le= tter-spacing: normal; text-align: start; text-indent: 0px; text-transform: = none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0p= x; text-decoration: none;" class=3D"">
I think the "testing" means when swapped out, the page will be NUL= L. If it's
just the pte got zapped, the page will be returned.  The call stack should= look
like:

 find_get_incore_page -> find_get_page -> pagecache_get_page(fgp_flags=3D=3D0).

I think the fgp_flags guaranteed it, with no FGP_ENTRY.

Did you test mincore() without my patch (as my current patch will indeed cause=
more 1's returned than it should)?  My guess is there's something else tha= t
made your test read more 1's with mincore() than pagemap, but I have no solid
idea on that.

I made sure to avoid any of our patches while testing mincore and counted t= he
flags to make sure. There are more set than they are present. I will l= ook into the 
code again and test on a non-rc kernel version, just to be safe.

I still think it makes sense for mincore to consider pages in the swap= cache
to be "in RAM". I start seeing how useful it can be to diffe= rentiate between
present pages and in-swap-cache pages.


We could use mincore() and pagemap to find the pages in the swap cache.

In short, mincore() is not enough because it does not differentiate between=
present pages and swap-cache entries, as both are in RAM, but the latter is also in swap. It can be used with other tools to get more specific infor= mation
though, so it is useful.

Note that my series is as you mentioned missing the changes to support
mincore() (otherwise I'll know the existance of it!).  It'll be trivia= l to add
that, but let's see whether mincore() will satisfy your need.

We are currently trying to make use of all tools that we have learned of so= far
during our discussions (lseek, map_files, even mincore) to get the informat= ion
that we need about swap pages. In theory, for many of our use cases, a
combination of 2 or 3 should be enough.

It is a little more convoluted than a simple pagemap call, but it can be mo= re
versatile (using lseek to skip multiple unallocated pages). As to whether t= he swap
bit (and more) should be eventually added on the pagemap, maybe this topic<= br class=3D""> makes more sense to continue on the Documentation thread.

[...]

It is possible for the swap device to = be network attached and shared, so multiple
hosts would need to understand its content. Then it is no longer internal t= o one
kernel only.

By being swap-aware, we can skip swapped-out pages during migration (to pre= vent IO and potential thrashing), and transfer those pages in another way t= hat
is zero-copy.

That sounds reasonable, but I'm not aware of any user-API that exposes swap=
entries to userspace, or is there one?

Good question. AFAIK, the swap device can be retrieved by using the swap ty= pe,
which is part of the swap entry. During our discussions, I was always assum= ing
that, if the pagemap entry kept track of the swap offset, it would keep tra= ck of the
swap type and, conversely, the swap device as well. Sorry if I haven't made= this
assumption clear until now.

So we were relying on the pagemap to expose swap entry information. Seeing = it
works for private pages, we thought it made sense to have worked on shared = pages as well.

I.e., how do you know which swap device is which?  How do you guarante= e the
kernel swp entry information won't change along with time?

I don't think we can guarantee it unless we halt the guest.

Yes, halting the guest helps, though then performance start to matter because
all time consumed in either pagemap or mincore() will be counted in as downtim= e
of the VM live migration, and it's not "live" at all during this pe= riod.

That's true. That is why this "halting" is supposed to only happe= n once, and
its duration needs to be minimised by pre-copy. Live migration AFAIK i= s intended
to have a single imperceptible pause at some point in order to converg= e more
quickly and cleanly. Without this halt, the whole migration procedure = could run
indefinitely, or for days instead of hours/minutes.

I'm not sure how it was done with private mappings before, because I thought
that's a pre-requisite knowledge to decide whether we should migrate a page or
not, but I might have missed something.  We can stop vm, sample, start vm,= but
it could become hiccups in the guest too, or otherwise contribute to downtime=
when src/dst vm switches.

I see your point. Safe to say the vm should never be stopped during pre-cop= y.
See my comment above.

But QEMU does most
migration work in pre-copy using a best-effort approach anyway.

So, having a way to retrieve temporary, but accurate information about swap=
entries (i.e. post-patch pagemap) should be enough to guarantee a smoother<= br class=3D""> migration process. It is intended to be repeated, unless there is no change=
between iterations.

The kernel will allocate swap device index which will be assigned as swp_type,=
right?  If there're multiple swap devices, how do you know which swp_entry i= s
located on which device?  I wanted to look for that info in "swapon -s&q= uot; but I
didn't.  Or maybe that solution only works if there's only one swap device?

Besides, I also still have a question on the accuracy. If there's no formal way
for userspace to interact with the kernel, I'm wondering how to guarantee the<= /span>
page will be kept swapped out, and data located on the swap device will always<= /span>
be the latest?  Because IMHO the kernel can swap in pages as wish, even = if it's
not accessed from the userspace.  After all, all these operations should = be
transparent to userspace.

One example in my mind is we do have page fault-around enabled for shmem, so
that even if the VM is stopped, its pages can be faulted in if (unluckily, in
this case, though) some page near the swapped out page got faulted in - it
could be some qemu malloc()ed region that (again, unluckily) was allocated to
have a virtual address very close to the mmap()ed guest memories.

I am not sure whether it's a real problem, e.g., even if some page swapped i= n
during guest halted for some reason, if no one is writting to that page, at
least the data on the page will still be identical to the one located on the
swap device.  However I think that still sounds too tricky, and maybe frag= ile.

Great example, and good point. Thank you for raising that up. This looks li= ke a
very real problem that we need to take into consideration in our proto= types.

--
Kind regards,

Tibi

--_000_40A87343266E469DBAE5667A2E1DB238nutanixcom_--