Re: [PATCH v3 8/8] vfio/pci: Add the support for PCI D3cold state

From: Jason Gunthorpe <jgg@nvidia.com>
To: Alex Williamson <alex.williamson@redhat.com>
Cc: Abhishek Sahu <abhsahu@nvidia.com>,
	Cornelia Huck <cohuck@redhat.com>,
	Yishai Hadas <yishaih@nvidia.com>,
	Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>,
	Kevin Tian <kevin.tian@intel.com>,
	"Rafael J . Wysocki" <rafael@kernel.org>,
	Max Gurtovoy <mgurtovoy@nvidia.com>,
	Bjorn Helgaas <bhelgaas@google.com>,
	linux-kernel@vger.kernel.org, kvm@vger.kernel.org,
	linux-pm@vger.kernel.org, linux-pci@vger.kernel.org
Subject: Re: [PATCH v3 8/8] vfio/pci: Add the support for PCI D3cold state
Date: Wed, 1 Jun 2022 20:17:20 -0300	[thread overview]
Message-ID: <20220601231720.GT1343366@nvidia.com> (raw)
In-Reply-To: <20220601121547.03ebbf64.alex.williamson@redhat.com>

On Wed, Jun 01, 2022 at 12:15:47PM -0600, Alex Williamson wrote:
> On Wed, 1 Jun 2022 14:30:54 -0300
> Jason Gunthorpe <jgg@nvidia.com> wrote:
> 
> > On Wed, Jun 01, 2022 at 10:21:51AM -0600, Alex Williamson wrote:
> > 
> > > Some ioctls clearly cannot occur while the device is in low power, such
> > > as resets and interrupt control, but even less obvious things like
> > > getting region info require device access.  Migration also provides a
> > > channel to device access.    
> > 
> > I wonder what power management means in a case like that.
> > 
> > For the migration drivers they all rely on a PF driver that is not
> > VFIO, so it should be impossible for power management to cause the PF
> > to stop working.
> > 
> > I would expect any sane design of power management for a VF to not
> > cause any harm to the migration driver..
> 
> Is there even a significant benefit or use case for power management
> for VFs?  The existing D3hot support should be ok, but I imagine to
> support D3cold, all the VFs and the PF would need to move to low power.
> It might be safe to simply exclude VFs from providing this feature for
> now.

I know of no use case, I think it would be a good idea to exclude VFs.

> Yes, but that's also penalizing devices that require no special
> support, for the few that do.  I'm not opposed to some sort of
> vfio-pci-nvidia-gpu variant driver to provide that device specific
> support, but I'd think the device table for such a driver might just be
> added to the exclusion list for power management support in vfio-pci.
> vfio-pci-core would need some way for drivers to opt-out/in for power
> management. 

If you think it can be done generically with a small exclusion list
then that probably makes sense.

Jason