From: "Rafael J. Wysocki" <rjw@rjwysocki.net>
To: Bjorn Helgaas <helgaas@kernel.org>
Cc: Dexuan Cui <decui@microsoft.com>,
"Rafael J . Wysocki" <rafael.j.wysocki@intel.com>,
Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>,
Michael Kelley <mikelley@microsoft.com>,
Sasha Levin <Alexander.Levin@microsoft.com>,
Haiyang Zhang <haiyangz@microsoft.com>,
KY Srinivasan <kys@microsoft.com>,
Stephen Hemminger <sthemmin@microsoft.com>,
olaf@aepfle.de, apw@canonical.com, jasowang@redhat.com,
vkuznets@redhat.com, marcelo.cerri@canonical.com,
jackm@mellanox.com, linux-pci@vger.kernel.org,
linux-hyperv@vger.kernel.org, linux-kernel@vger.kernel.org,
driverdev-devel@linuxdriverproject.org,
Bjorn Helgaas <bhelgaas@google.com>,
Linux PM <linux-pm@vger.kernel.org>
Subject: Re: [PATCH 1/7] PCI/PM: Always return devices to D0 when thawing
Date: Tue, 15 Oct 2019 19:15:46 +0200 [thread overview]
Message-ID: <15877028.U7HUqG5fmY@kreacher> (raw)
In-Reply-To: <20191014230016.240912-2-helgaas@kernel.org>
On Tuesday, October 15, 2019 1:00:10 AM CEST Bjorn Helgaas wrote:
> From: Dexuan Cui <decui@microsoft.com>
>
> pci_pm_thaw_noirq() is supposed to return the device to D0 and restore its
> configuration registers, but previously it only did that for devices whose
> drivers implemented the new power management ops.
>
> Hibernation, e.g., via "echo disk > /sys/power/state", involves freezing
> devices, creating a hibernation image, thawing devices, writing the image,
> and powering off. The fact that thawing did not return devices with legacy
> power management to D0 caused errors, e.g., in this path:
>
> pci_pm_thaw_noirq
> if (pci_has_legacy_pm_support(pci_dev)) # true for Mellanox VF driver
> return pci_legacy_resume_early(dev) # ... legacy PM skips the rest
> pci_set_power_state(pci_dev, PCI_D0)
> pci_restore_state(pci_dev)
> pci_pm_thaw
> if (pci_has_legacy_pm_support(pci_dev))
> pci_legacy_resume
> drv->resume
> mlx4_resume
> ...
> pci_enable_msix_range
> ...
> if (dev->current_state != PCI_D0) # <---
> return -EINVAL;
>
> which caused these warnings:
>
> mlx4_core a6d1:00:02.0: INTx is not supported in multi-function mode, aborting
> PM: dpm_run_callback(): pci_pm_thaw+0x0/0xd7 returns -95
> PM: Device a6d1:00:02.0 failed to thaw: error -95
>
> Return devices to D0 and restore config registers for all devices, not just
> those whose drivers support new power management.
>
> [bhelgaas: also call pci_restore_state() before pci_legacy_resume_early(),
> update comment, add stable tag, commit log]
> Link: https://lore.kernel.org/r/KU1P153MB016637CAEAD346F0AA8E3801BFAD0@KU1P153MB0166.APCP153.PROD.OUTLOOK.COM
> Signed-off-by: Dexuan Cui <decui@microsoft.com>
> Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
> Cc: stable@vger.kernel.org # v4.13+
No issues found, so
Reviewed-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
> ---
> drivers/pci/pci-driver.c | 17 +++++++++++------
> 1 file changed, 11 insertions(+), 6 deletions(-)
>
> diff --git a/drivers/pci/pci-driver.c b/drivers/pci/pci-driver.c
> index a8124e47bf6e..d4ac8ce8c1f9 100644
> --- a/drivers/pci/pci-driver.c
> +++ b/drivers/pci/pci-driver.c
> @@ -1076,17 +1076,22 @@ static int pci_pm_thaw_noirq(struct device *dev)
> return error;
> }
>
> - if (pci_has_legacy_pm_support(pci_dev))
> - return pci_legacy_resume_early(dev);
> -
> /*
> - * pci_restore_state() requires the device to be in D0 (because of MSI
> - * restoration among other things), so force it into D0 in case the
> - * driver's "freeze" callbacks put it into a low-power state directly.
> + * Both the legacy ->resume_early() and the new pm->thaw_noirq()
> + * callbacks assume the device has been returned to D0 and its
> + * config state has been restored.
> + *
> + * In addition, pci_restore_state() restores MSI-X state in MMIO
> + * space, which requires the device to be in D0, so return it to D0
> + * in case the driver's "freeze" callbacks put it into a low-power
> + * state.
> */
> pci_set_power_state(pci_dev, PCI_D0);
> pci_restore_state(pci_dev);
>
> + if (pci_has_legacy_pm_support(pci_dev))
> + return pci_legacy_resume_early(dev);
> +
> if (drv && drv->pm && drv->pm->thaw_noirq)
> error = drv->pm->thaw_noirq(dev);
>
>
next prev parent reply other threads:[~2019-10-15 17:15 UTC|newest]
Thread overview: 18+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-10-14 23:00 [PATCH v3 0/7] PCI: PM: Move to D0 before calling pci_legacy_resume_early() Bjorn Helgaas
2019-10-14 23:00 ` [PATCH 1/7] PCI/PM: Always return devices to D0 when thawing Bjorn Helgaas
2019-10-15 17:15 ` Rafael J. Wysocki [this message]
2019-10-14 23:00 ` [PATCH 2/7] PCI/PM: Correct pci_pm_thaw_noirq() documentation Bjorn Helgaas
2019-10-15 17:17 ` Rafael J. Wysocki
2019-10-14 23:00 ` [PATCH 3/7] PCI/PM: Clear PCIe PME Status even for legacy power management Bjorn Helgaas
2019-10-15 17:20 ` Rafael J. Wysocki
2019-10-14 23:00 ` [PATCH 4/7] PCI/PM: Run resume fixups before disabling wakeup events Bjorn Helgaas
2019-10-15 17:21 ` Rafael J. Wysocki
2019-10-14 23:00 ` [PATCH 5/7] PCI/PM: Make power management op coding style consistent Bjorn Helgaas
2019-10-15 17:22 ` Rafael J. Wysocki
2019-10-16 13:50 ` Dan Carpenter
2019-10-14 23:00 ` [PATCH 6/7] PCI/PM: Wrap long lines in documentation Bjorn Helgaas
2019-10-15 17:23 ` Rafael J. Wysocki
2019-10-14 23:00 ` [PATCH 7/7] PCI/MSI: Move power state check out of pci_msi_supported() Bjorn Helgaas
2019-10-15 17:24 ` Rafael J. Wysocki
2019-10-15 18:24 ` [PATCH v3 0/7] PCI: PM: Move to D0 before calling pci_legacy_resume_early() Dexuan Cui
2019-10-15 18:42 ` Bjorn Helgaas
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=15877028.U7HUqG5fmY@kreacher \
--to=rjw@rjwysocki.net \
--cc=Alexander.Levin@microsoft.com \
--cc=apw@canonical.com \
--cc=bhelgaas@google.com \
--cc=decui@microsoft.com \
--cc=driverdev-devel@linuxdriverproject.org \
--cc=haiyangz@microsoft.com \
--cc=helgaas@kernel.org \
--cc=jackm@mellanox.com \
--cc=jasowang@redhat.com \
--cc=kys@microsoft.com \
--cc=linux-hyperv@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-pci@vger.kernel.org \
--cc=linux-pm@vger.kernel.org \
--cc=lorenzo.pieralisi@arm.com \
--cc=marcelo.cerri@canonical.com \
--cc=mikelley@microsoft.com \
--cc=olaf@aepfle.de \
--cc=rafael.j.wysocki@intel.com \
--cc=sthemmin@microsoft.com \
--cc=vkuznets@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).