From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2C450C7619A for ; Wed, 12 Apr 2023 16:34:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A7F9B6B0074; Wed, 12 Apr 2023 12:34:48 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A2FC9900003; Wed, 12 Apr 2023 12:34:48 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8F67E900002; Wed, 12 Apr 2023 12:34:48 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 81CD56B0074 for ; Wed, 12 Apr 2023 12:34:48 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 4E52180196 for ; Wed, 12 Apr 2023 16:34:48 +0000 (UTC) X-FDA: 80673287856.26.DD049D8 Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2051.outbound.protection.outlook.com [40.107.94.51]) by imf14.hostedemail.com (Postfix) with ESMTP id 79487100002 for ; Wed, 12 Apr 2023 16:34:45 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=memverge.com header.s=selector2 header.b=aTetL5u3; spf=none (imf14.hostedemail.com: domain of gregory.price@memverge.com has no SPF policy when checking 40.107.94.51) smtp.mailfrom=gregory.price@memverge.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=none) header.from=memverge.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1681317285; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=7hIj0nVhsMBQYcGr4RfUDhyMJkcxdJpv7I5gJkldxkQ=; b=qYNW+jzOGw56CTc88K3QBbujs95fuOv4B6QyDZW8BaQ/94k+hm6DFavzd6RfSliXIzYPaM qz8ySUj8srRUFXgOD93PsA1e3nJbMpmMQu6CES9YFjB90uipBhgCMev5RMkKrcSRosQYyG NaSVSAe5mVli/1Ae7k8ZKNIGeOLwHIs= ARC-Authentication-Results: i=2; imf14.hostedemail.com; dkim=pass header.d=memverge.com header.s=selector2 header.b=aTetL5u3; spf=none (imf14.hostedemail.com: domain of gregory.price@memverge.com has no SPF policy when checking 40.107.94.51) smtp.mailfrom=gregory.price@memverge.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=none) header.from=memverge.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1681317285; a=rsa-sha256; cv=pass; b=JYx9l4uIagVAQVIBvznxZpzJTD6On2f7Xkt0ASLRbaYl+Fb9RuX/3Z5Do27+2adIkdZgcQ Iy4EH4lRxzdQXKGYCW69aEX9qX3wOU8uzXhX9zjxb+Nvt2TDJ5LarqKUltGoPG8lGNLgNY BeEgs1/kHDYKzidN8fmaKMmIUDyuI0E= ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=eG+HjrNyc4UfkIjW+VJ+TdD60PMI3ZKu2mxvBdxsQI9D2SaJMjD7oViQjd9oPXP1A4dnq9Y8nWEkND/BZBxJD2LKt8VvcRD2zM8FPHYGxFOsFdpZAkmSpmQ/WCEUUcgOIVjFa83sStFmeMBAz5lXnLYrrObSYTtZGv6IpmxoYQkW3bJ1q/o/LyXySAGT38MBLrRB1osk7lrM3+NPyjVzwPHPa9yzBI3S3EpQkXqSY9wiqODQ4hzBTgbfolX4qABhVUFEW15UUdPZXHTJv7K9ZukdmYHiP9ZZnsFLijk+6yLyo/+ULffxJVmxxpfxz2IdYL1RZXfX3fNoLiBZnYBnEQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=7hIj0nVhsMBQYcGr4RfUDhyMJkcxdJpv7I5gJkldxkQ=; b=ZurDFZ+V+b2QCP1GejIYXPPUHqYzdFeiWWhluAocEtvfKNdd4Uem7TJPvDoLXiFWEHUXdXGerX8yxoS7W9mX1s+5hCnk2JZXAMrTCuEj+2S7JAbV0PsfKFX4E4SK1z8MTfyH3n6yEnfc/+mxRY9ajJMJ+6gSbOFh/xcsbTNDNIEEiAsviyUn07niQcFjik38MQ243sZVGjiesTtfL+IMQMqlzNg5UEUUZyeSIPiSig35R8L5rPvJPsrrVy2Ik5o+m2s+3mXJgxAqt9ij7kJhwHkPeOA6GRPZKlb0rnEE6L1y3PB1XQtKKe/OQrE99O932UJbXtqiBM5t6Vf5yigXAA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=memverge.com; dmarc=pass action=none header.from=memverge.com; dkim=pass header.d=memverge.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=memverge.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=7hIj0nVhsMBQYcGr4RfUDhyMJkcxdJpv7I5gJkldxkQ=; b=aTetL5u3asytfz/xuoAytiFf2AROTYOnpV5skkM96zL2b+qGrn4UNyrAN4LNErneaWaPbsbITCqCEO5CJn0wgOG+u7aUu+xdtQlVK7Mw/LeHcduRGbRzjShWd8pmawPRheKCyo8P2DxX60E1nktNyU9PBXwFXu/ualu8lKMewvM= Received: from SJ0PR17MB5512.namprd17.prod.outlook.com (2603:10b6:a03:394::19) by PH7PR17MB6108.namprd17.prod.outlook.com (2603:10b6:510:1f4::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6156.28; Wed, 12 Apr 2023 16:34:41 +0000 Received: from SJ0PR17MB5512.namprd17.prod.outlook.com ([fe80::7b97:62c3:4602:b47a]) by SJ0PR17MB5512.namprd17.prod.outlook.com ([fe80::7b97:62c3:4602:b47a%6]) with mapi id 15.20.6277.036; Wed, 12 Apr 2023 16:34:41 +0000 Date: Wed, 12 Apr 2023 12:34:37 -0400 From: Gregory Price To: David Hildenbrand Cc: "Huang, Ying" , Dragan Stancevic , lsf-pc@lists.linux-foundation.org, nil-migration@lists.linux.dev, linux-cxl@vger.kernel.org, linux-mm@kvack.org Subject: Re: [LSF/MM/BPF TOPIC] BoF VM =?utf-8?Q?li?= =?utf-8?Q?ve_migration_over_CXL_memory=E2=80=8B?= Message-ID: References: <5d1156eb-02ae-a6cc-54bb-db3df3ca0597@stancevic.com> <87v8i22abl.fsf@yhuang6-desk2.ccr.corp.intel.com> <87bkjtzu7e.fsf@yhuang6-desk2.ccr.corp.intel.com> Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-ClientProxiedBy: SJ0PR05CA0086.namprd05.prod.outlook.com (2603:10b6:a03:332::31) To SJ0PR17MB5512.namprd17.prod.outlook.com (2603:10b6:a03:394::19) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SJ0PR17MB5512:EE_|PH7PR17MB6108:EE_ X-MS-Office365-Filtering-Correlation-Id: d1847cb5-9f47-4678-6c06-08db3b73d0b0 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:SJ0PR17MB5512.namprd17.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230028)(366004)(136003)(346002)(396003)(376002)(39850400004)(451199021)(38100700002)(2906002)(8936002)(44832011)(5660300002)(36756003)(86362001)(83380400001)(316002)(478600001)(54906003)(2616005)(6666004)(6486002)(6512007)(186003)(26005)(6506007)(66946007)(6916009)(4326008)(41300700001)(66476007)(66556008);DIR:OUT;SFP:1101; X-OriginatorOrg: memverge.com X-MS-Exchange-CrossTenant-Network-Message-Id: d1847cb5-9f47-4678-6c06-08db3b73d0b0 X-MS-Exchange-CrossTenant-AuthSource: SJ0PR17MB5512.namprd17.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Apr 2023 16:34:41.4277 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 5c90cb59-37e7-4c81-9c07-00473d5fb682 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: RJHDooXr8Rux98MNwajAGGRtRcLCZs7YqyiaprFxJp3oYabYTpDcOYYxnEaC0SimqhPi0r8XJOwj9gEx/WSESXhksN4deqOHUIG5tP3kXhU= X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR17MB6108 X-Stat-Signature: r3d79ggpm34scaugwm9dbh647i3dp69c X-Rspam-User: X-Rspamd-Queue-Id: 79487100002 X-Rspamd-Server: rspam06 X-HE-Tag: 1681317285-854058 X-HE-Meta: U2FsdGVkX1/gMEgzZ/MMrzClRkqFa86IcD1Ze+SwASXzB9zEHNMFueeKf/Wz58grvUZY06gwBLP5RPaUr0m3dO5SLXERFHYVKlAQsSYFGnHHl6w+NSQrF4JfybBVGiuP/AbI1zy+iG7YD5fVM2FV+G6M2TYHEfKRJRRdBs8quen9WF4ViYqv0rRtQnkUdi7zA+YGpM4Y3PrWsgCZ/IjTrk1yAbPi4vUwvFCeYmrHc23kR9fE/0xQNVWHwsndKAy3D/NU0sCjlmCmWb0bBVHvbHPHYrgrEYR2qP522PsVcXQGxMopluzf2ZFVRllhLJodsxlncrgmVqMuB0ev0xi3eZvY3/+e6Wgr9IYobq57iND38KiWqxhZLa9xkFRBVG9kMmm6xQdzb1mWhuPxwnc8evVqlBnzi1p+WCW7wE93mpOYyz8Um9ozIaMzxRMku6B2vaNrp5fzatQSd1vW7kNRW4qI1ts+YTTRxrE+GDsL/8O1MPyhtNm0CpdZqFzgJiW215jhJRS/eOceCQ63SND3gAbIKY2uLs3Igu9DPzlF4jVCVrsio0Cv4K5fFktvIsL2GBAU7/qxRaPO0K5h6pcVetOQZTUD222v2XIgj6qSIwrGYr7sYATPkjqE6WThagAqHKGq/ZFE/wXUFFUKEGjKm87+U3yWpitO9qTwdIpVtD+N0Ab+c/IEKFVTCzYW3GK8uhABi7bY0HDztXHNhsPrixnHCgJ19LeGXFwDgLpyrj+UYdyEsFWF4qgjdxkPF+e5KP2ecpfJ2RDNGsryx9BoCWE3PSpYdWMHOlG/rUVyBRBvxLP+akYvFxDBXxS4XZ6nQXW1NER42/r8BgvbPEduwEzUisp+HN+4XS6+GJvvJv6rxZPWHmioUcuEsgxomq8XssI4IwYpUNAQwmltEDy5kjAvITGfTxg8AshKng7g97V9/7YRhQhKA3g4uy4M1rnHChGMFKsoQcjTYZkenpi r4G6oQaN ctl5bUXpWS98ULx0GsT5C5HezktAb94HxHf6JqAYFdlYtZ5ZOtgXAwfzwUOZEkA3aEimLP0zBwwqU1yFfuMe7yIjfgLKrU5zkjrw1GX57FGvFftGke2EI0JuulKYztbtXLGoO2639xJ0kWXZMRRVYG0G7WW4nhPryZHnPpMRFcCCnJJaDC7DHDeD/UUZBsu3emOBBhnS0Idp/G+mPap3dlhewThTQIeVkupuqs1YjaYb5FE8qK83wRlw2TQ0QVCsw/xgZwSSQhUoClNo= X-Bogosity: Ham, tests=bogofilter, spamicity=0.010001, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, Apr 12, 2023 at 05:50:55PM +0200, David Hildenbrand wrote: > > long-term: possibly forever, controlled by user space. In practice, anything > longer than ~10 seconds ( best guess :) ). There can be long-term pinnings > that are of very short duration, we just don't know what user space is up to > and when it will decide to unpin. > > Assume user space requests to trigger read/write of a user space page to a > file: the page is pinned, DMA is started, once DMA completes the page is > unpinned. Short-term. User space does not control how long the page remains > pinned. > > In contrast: > > Example #1: mapping VM guest memory into an IOMMU using vfio for PCI > passthrough requires pinning the pages. Until user space decides to unmap > the pages from the IOMMU, the pages will remain pinned. -> long-term > > Example #2: mapping a user space address range into an IOMMU to repeatedly > perform RDMA using that address range requires pinning the pages. Until user > space decides to unregister that range, the pages remain pinned. -> > long-term > > Example #3: registering a user space address range with io_uring as a fixed > buffer, such that io_uring OPS can avoid the page table walks by simply > using the pinned pages that were looked up once. As long as the fixed buffer > remains registered, the pages stay pinned. -> long-term > > -- > Thanks, > > David / dhildenb > That pretty much precludes live migration from using CXL as a transport mechanism, since live migration would be a user-initiated process, you would need what amounts to an atomic move between hosts to ensure pages are not left pinned. The more i'm reading the more i'm somewhat convinced CXL memory should not allow pinning at all. I suppose you could implement a new RDMA feature where the remote host's CXL memory is temporarily mapped, data is migrated, and then that area is unmapped. Basically the exact same RDMA mechanism, but using memory instead of network. This would make the operation a kernel-controlled if pin/unpin is required. Lots to talk about. ~Gregory