Re: [PATCH 1/3] PCI: Add a flag to notify PCI drivers about powerdown during suspend

From: Bjorn Helgaas <helgaas@kernel.org>
To: Manivannan Sadhasivam <manivannan.sadhasivam@linaro.org>
Cc: Bjorn Helgaas <bhelgaas@google.com>,
	Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>,
	Keith Busch <kbusch@kernel.org>, Christoph Hellwig <hch@lst.de>,
	linux-nvme@lists.infradead.org, linux-pci@vger.kernel.org,
	linux-arm-msm@vger.kernel.org, linux-kernel@vger.kernel.org,
	Stanimir Varbanov <svarbanov@mm-sol.com>,
	Bjorn Andersson <bjorn.andersson@linaro.org>,
	Jens Axboe <axboe@fb.com>,
	Veerabhadrarao Badiganti <quic_vbadigan@quicinc.com>,
	quic_krichai@quicinc.com, Nitin Rawat <quic_nitirawa@quicinc.com>,
	Vidya Sagar <vidyas@nvidia.com>, Sagi Grimberg <sagi@grimberg.me>
Subject: Re: [PATCH 1/3] PCI: Add a flag to notify PCI drivers about powerdown during suspend
Date: Tue, 17 May 2022 12:24:23 -0500	[thread overview]
Message-ID: <20220517172423.GA1083672@bhelgaas> (raw)
In-Reply-To: <20220517150908.GA4528@thinkpad>

On Tue, May 17, 2022 at 08:39:08PM +0530, Manivannan Sadhasivam wrote:
> On Mon, May 16, 2022 at 03:18:17PM -0500, Bjorn Helgaas wrote:
> > On Fri, May 13, 2022 at 04:30:25PM +0530, Manivannan Sadhasivam wrote:
> > > On some systems like Chromebooks based on Qcom chipsets, the OS may
> > > powerdown all PCIe devices during system suspend for aggressive
> > > powersaving. In that case, the PCI host controller drivers need to notify
> > > the PCI device drivers that the power will be taken off during system
> > > suspend so that the drivers can prepare the devices accordingly.
> > 
> > "The OS may powerdown all PCIe devices ..." makes it sound like this
> > is an OS policy decision.  Where exactly (what function) is that?
> > 
> > Or if it's not an OS policy decision, but rather some property of the
> > hardware, say that specifically.
> 
> On SC7280, it is the Resource Power Manager(RPMh) that's powering
> the devices down by cutting off the PCIe voltage domain. But the
> SC7280 RC driver itself may put the PCIe devices into D3cold state
> during system suspend.
> https://lore.kernel.org/lkml/CAE-0n53ho2DX2rqQMvvKAuDCfsWW62TceTaNPzv5Mn_NQ-U6dA@mail.gmail.com/T/
> 
> So to cover both cases (one is a hardware independent of SoC and
> another one is the device driver), and to be generic, I've used the
> term "OS" after looking at the previous flags.

This sort of device-specific behavior definitely needs a pointer to an
example.  Otherwise it seems like it could be generic PCIe behavior
that should be documented in the PCIe base spec.

> > > One prime example is the PCI NVMe driver. This flag can be used by the
> > > driver to shutdown the NVMe device during suspend and recover it during
> > > resume.

Apparently nvme is broken, or at least sub-optimal, without this flag.
What other drivers will be similarly affected?

> > > Signed-off-by: Manivannan Sadhasivam <manivannan.sadhasivam@linaro.org>
> > > ---
> > >  include/linux/pci.h | 1 +
> > >  1 file changed, 1 insertion(+)
> > > 
> > > diff --git a/include/linux/pci.h b/include/linux/pci.h
> > > index 60adf42460ab..069caf1fe88d 100644
> > > --- a/include/linux/pci.h
> > > +++ b/include/linux/pci.h
> > > @@ -578,6 +578,7 @@ struct pci_host_bridge {
> > >  	unsigned int	preserve_config:1;	/* Preserve FW resource setup */
> > >  	unsigned int	size_windows:1;		/* Enable root bus sizing */
> > >  	unsigned int	msi_domain:1;		/* Bridge wants MSI domain */
> > > +	unsigned int	suspend_poweroff:1;	/* OS may poweroff devices during system suspend */
> > >  
> > >  	/* Resource alignment requirements */
> > >  	resource_size_t (*align_resource)(struct pci_dev *dev,