From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 53811C352A1 for ; Tue, 6 Dec 2022 06:26:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233797AbiLFG0Q (ORCPT ); Tue, 6 Dec 2022 01:26:16 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56366 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229936AbiLFG0K (ORCPT ); Tue, 6 Dec 2022 01:26:10 -0500 Received: from verein.lst.de (verein.lst.de [213.95.11.211]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 52AAC275D2; Mon, 5 Dec 2022 22:26:09 -0800 (PST) Received: by verein.lst.de (Postfix, from userid 2407) id AC88E68B05; Tue, 6 Dec 2022 07:26:04 +0100 (CET) Date: Tue, 6 Dec 2022 07:26:04 +0100 From: Christoph Hellwig To: Lei Rao Cc: kbusch@kernel.org, axboe@fb.com, kch@nvidia.com, hch@lst.de, sagi@grimberg.me, alex.williamson@redhat.com, cohuck@redhat.com, jgg@ziepe.ca, yishaih@nvidia.com, shameerali.kolothum.thodi@huawei.com, kevin.tian@intel.com, mjrosato@linux.ibm.com, linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, kvm@vger.kernel.org, eddie.dong@intel.com, yadong.li@intel.com, yi.l.liu@intel.com, Konrad.wilk@oracle.com, stephen@eideticom.com, hang.yuan@intel.com Subject: Re: [RFC PATCH 5/5] nvme-vfio: Add a document for the NVMe device Message-ID: <20221206062604.GB6595@lst.de> References: <20221206055816.292304-1-lei.rao@intel.com> <20221206055816.292304-6-lei.rao@intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20221206055816.292304-6-lei.rao@intel.com> User-Agent: Mutt/1.5.17 (2007-11-01) Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, Dec 06, 2022 at 01:58:16PM +0800, Lei Rao wrote: > The documentation describes the details of the NVMe hardware > extension to support VFIO live migration. This is not a NVMe hardware extension, this is some really strange and half-assed intel-specific extension to nvme, which like any other vendors specific non-standard extensions to nvme we refused to support in Linux. There is a TPAR for live migration building blocks under discussion in the NVMe technical working group. It will still require mediatation of access to the admin queue to deal with the huge amout of state nvme has that needs to be migrated (and which doesn't seem to be covered at all here). In Linux the equivalent would be to implement a mdev driver that allows passing through the I/O qeues to a guest, but it might be a better idea to handle the device model emulation entirely in Qemu (or other userspace device models) and just find a way to expose enough of the I/O queues to userspace. The current TPAR seems to be very complicated for that, as in many cases we'd only need a way to tіe certain namespaces to certain I/O queues and not waste a lot of resources on the rest of the controller.