From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 95A75C352A1 for ; Tue, 6 Dec 2022 14:49:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235087AbiLFOtx (ORCPT ); Tue, 6 Dec 2022 09:49:53 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46826 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234364AbiLFOtp (ORCPT ); Tue, 6 Dec 2022 09:49:45 -0500 Received: from mail-pf1-x434.google.com (mail-pf1-x434.google.com [IPv6:2607:f8b0:4864:20::434]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CC8AD2AE39 for ; Tue, 6 Dec 2022 06:48:24 -0800 (PST) Received: by mail-pf1-x434.google.com with SMTP id g1so6681049pfk.2 for ; Tue, 06 Dec 2022 06:48:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=E3b5bdWA85OzMOTjNeR+xYtDEAbTCZli0u3K40D9kx4=; b=OFUOd4PNQR8bRxCK6qegIBSYa3MK8J/1a32WRukCTxAY5fTbE+omxk8YDjbYTugGMt YeXjCnUiKvlBk7edXLLva1LG+EYEPJj21wfIj1MmX7CR4SOWyOYUEzYSPs05BYdfhIq5 z1IaXMgmuZsbEAXA7r01df8vhuih3NfsWItIHWKZi0n6rXD8IocqnRosa8UqEyN/8mVL i6iXH2dMdNFy5yDeIFaIvU+1FfeV0PfX9RbViXvdq9awFDH4LYyYXr/zLukIzNQ2rY4g WSup0IALJEblLWRnTR/3BL3KE6NdkyZhPvl0lshlmlVISJYUUF2mgQiKaTs/T2Pmpe+Y t8FQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=E3b5bdWA85OzMOTjNeR+xYtDEAbTCZli0u3K40D9kx4=; b=froUQfs2iW1h4TNEk3ualfjd6KU2IIBJjWg7tQaNJ1/zVwc7ux2zBxOEaI0RAdeD8s /UnoEbGNIK5ngSz2abe77bareiBTpBnlQEBPfn9iN5a7T4Rnl3niSDYcBnQ4DPiVLn/C YDCziy24B2lU9ghTxq0e2KgGyVLXn36hMS7uYnv9G1XAvTJdf2R/eizoyef0wLrfk7cl eYkeeT0qT20oJoCWbgkIBzvULMzLLmMNy881iPky2HR89Ty267nWRuZ0xy2Srn+3Kd/E 87T0gQxRSskIkv1v5P3OZZSYNKZewGhMYvvhCM8+7KCZhVcQVL7RyKS4TV/xf8T3honz iHVQ== X-Gm-Message-State: ANoB5pkZjSVLO3Xgc7YaABDkGE/irnXXdulMWuLmF8iB5k1Cnf6+mmyg RPICRkOYWsJ/rRKKGF5eH6g4JA== X-Google-Smtp-Source: AA0mqf6wZNYJR+nxEDntmIwoB1WTx5lMm+/0X87R2SrkgFXKtucbmHuGQYhQGln4hWgCmve4UUXs+g== X-Received: by 2002:a63:105c:0:b0:46f:f72c:cdae with SMTP id 28-20020a63105c000000b0046ff72ccdaemr62687659pgq.237.1670338104328; Tue, 06 Dec 2022 06:48:24 -0800 (PST) Received: from ziepe.ca ([206.223.160.26]) by smtp.gmail.com with ESMTPSA id u2-20020a170902e5c200b00186c5e8a8d7sm12785747plf.171.2022.12.06.06.48.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Dec 2022 06:48:23 -0800 (PST) Received: from jgg by wakko with local (Exim 4.95) (envelope-from ) id 1p2ZF8-004cPA-AK; Tue, 06 Dec 2022 10:48:22 -0400 Date: Tue, 6 Dec 2022 10:48:22 -0400 From: Jason Gunthorpe To: Christoph Hellwig Cc: Lei Rao , kbusch@kernel.org, axboe@fb.com, kch@nvidia.com, sagi@grimberg.me, alex.williamson@redhat.com, cohuck@redhat.com, yishaih@nvidia.com, shameerali.kolothum.thodi@huawei.com, kevin.tian@intel.com, mjrosato@linux.ibm.com, linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, kvm@vger.kernel.org, eddie.dong@intel.com, yadong.li@intel.com, yi.l.liu@intel.com, Konrad.wilk@oracle.com, stephen@eideticom.com, hang.yuan@intel.com Subject: Re: [RFC PATCH 5/5] nvme-vfio: Add a document for the NVMe device Message-ID: References: <20221206055816.292304-1-lei.rao@intel.com> <20221206055816.292304-6-lei.rao@intel.com> <20221206062604.GB6595@lst.de> <20221206130901.GB24358@lst.de> <20221206140002.GB27689@lst.de> <20221206143126.GB30297@lst.de> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20221206143126.GB30297@lst.de> Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, Dec 06, 2022 at 03:31:26PM +0100, Christoph Hellwig wrote: > On Tue, Dec 06, 2022 at 10:20:26AM -0400, Jason Gunthorpe wrote: > > In the VFIO restore model there is no "live OS" on resume. The > > load/resume cycle is as destructive as reset to the vfio device. > > Of course there may be and OS. As soon as the VF is live Linux > will by default bind to it. And that's the big problem here, > the VF should not actually exist or at least not be usable when > such a restore happens - or to say it in NVMe terms, the Secondary > Controller better be in offline state when state is loaded into it. Sadly in Linux we don't have a SRIOV VF lifecycle model that is any use. What we do have is a transfer of control from the normal OS driver (eg nvme) to the VFIO driver. Also, remember, that VFIO only does live migration between VFIO devices. We cannot use live migration and end up with a situation where the normal nvme driver is controlling the VF. The VFIO load model is explicitly destructive. We replace the current VF with the loading VF. Both the VFIO variant driver and the VFIO userspace issuing the load have to be aware of this and understand that the whole device will change. >From an implementation perspective, I would expect the nvme varient driver to either place the nvme device in the correct state during load, or refuse to execute load if it is in the wrong state. To be compatible with what qemu is doing the "right state" should be entered by completing function level reset of the VF. The Linux/qemu parts are still being finalized, so if you see something that could be changed to better match nvme it would be a great time to understand that. Jason