Re: [PATCH 1/3] PCI: Add a flag to notify PCI drivers about powerdown during suspend

From: Manivannan Sadhasivam <manivannan.sadhasivam@linaro.org>
To: Bjorn Helgaas <helgaas@kernel.org>
Cc: Bjorn Helgaas <bhelgaas@google.com>,
	Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>,
	Keith Busch <kbusch@kernel.org>, Christoph Hellwig <hch@lst.de>,
	linux-nvme@lists.infradead.org, linux-pci@vger.kernel.org,
	linux-arm-msm@vger.kernel.org, linux-kernel@vger.kernel.org,
	Stanimir Varbanov <svarbanov@mm-sol.com>,
	Bjorn Andersson <bjorn.andersson@linaro.org>,
	Jens Axboe <axboe@fb.com>,
	Veerabhadrarao Badiganti <quic_vbadigan@quicinc.com>,
	quic_krichai@quicinc.com, Nitin Rawat <quic_nitirawa@quicinc.com>,
	Vidya Sagar <vidyas@nvidia.com>, Sagi Grimberg <sagi@grimberg.me>
Subject: Re: [PATCH 1/3] PCI: Add a flag to notify PCI drivers about powerdown during suspend
Date: Wed, 18 May 2022 09:29:15 +0530	[thread overview]
Message-ID: <20220518035915.GB4791@thinkpad> (raw)
In-Reply-To: <20220517172423.GA1083672@bhelgaas>

On Tue, May 17, 2022 at 12:24:23PM -0500, Bjorn Helgaas wrote:
> On Tue, May 17, 2022 at 08:39:08PM +0530, Manivannan Sadhasivam wrote:
> > On Mon, May 16, 2022 at 03:18:17PM -0500, Bjorn Helgaas wrote:
> > > On Fri, May 13, 2022 at 04:30:25PM +0530, Manivannan Sadhasivam wrote:
> > > > On some systems like Chromebooks based on Qcom chipsets, the OS may
> > > > powerdown all PCIe devices during system suspend for aggressive
> > > > powersaving. In that case, the PCI host controller drivers need to notify
> > > > the PCI device drivers that the power will be taken off during system
> > > > suspend so that the drivers can prepare the devices accordingly.
> > > 
> > > "The OS may powerdown all PCIe devices ..." makes it sound like this
> > > is an OS policy decision.  Where exactly (what function) is that?
> > > 
> > > Or if it's not an OS policy decision, but rather some property of the
> > > hardware, say that specifically.
> > 
> > On SC7280, it is the Resource Power Manager(RPMh) that's powering
> > the devices down by cutting off the PCIe voltage domain. But the
> > SC7280 RC driver itself may put the PCIe devices into D3cold state
> > during system suspend.
> > https://lore.kernel.org/lkml/CAE-0n53ho2DX2rqQMvvKAuDCfsWW62TceTaNPzv5Mn_NQ-U6dA@mail.gmail.com/T/
> > 
> > So to cover both cases (one is a hardware independent of SoC and
> > another one is the device driver), and to be generic, I've used the
> > term "OS" after looking at the previous flags.
> 
> This sort of device-specific behavior definitely needs a pointer to an
> example.  Otherwise it seems like it could be generic PCIe behavior
> that should be documented in the PCIe base spec.
> 

Okay.

> > > > One prime example is the PCI NVMe driver. This flag can be used by the
> > > > driver to shutdown the NVMe device during suspend and recover it during
> > > > resume.
> 
> Apparently nvme is broken, or at least sub-optimal, without this flag.

Yes, broken on SC7280 or any other SoCs that turn off power.

> What other drivers will be similarly affected?
> 

I don't have a list but the drivers that don't expect the device to be turned
off or reset during suspend may experience this issue. Right now, we have only
identified the issue with NVMe because that's what used on Chromebooks.

But in the coming days, we may need to fix some of the drivers also.

Thanks,
Mani

> > > > Signed-off-by: Manivannan Sadhasivam <manivannan.sadhasivam@linaro.org>
> > > > ---
> > > >  include/linux/pci.h | 1 +
> > > >  1 file changed, 1 insertion(+)
> > > > 
> > > > diff --git a/include/linux/pci.h b/include/linux/pci.h
> > > > index 60adf42460ab..069caf1fe88d 100644
> > > > --- a/include/linux/pci.h
> > > > +++ b/include/linux/pci.h
> > > > @@ -578,6 +578,7 @@ struct pci_host_bridge {
> > > >  	unsigned int	preserve_config:1;	/* Preserve FW resource setup */
> > > >  	unsigned int	size_windows:1;		/* Enable root bus sizing */
> > > >  	unsigned int	msi_domain:1;		/* Bridge wants MSI domain */
> > > > +	unsigned int	suspend_poweroff:1;	/* OS may poweroff devices during system suspend */
> > > >  
> > > >  	/* Resource alignment requirements */
> > > >  	resource_size_t (*align_resource)(struct pci_dev *dev,

-- 
மணிவண்ணன் சதாசிவம்