archive mirror
 help / color / mirror / Atom feed
From: Abhishek Sahu <>
To: Alex Williamson <>,
	Cornelia Huck <>,
	Yishai Hadas <>,
	Jason Gunthorpe <>,
	Shameer Kolothum <>,
	Kevin Tian <>,
	"Rafael J . Wysocki" <>
Cc: Max Gurtovoy <>,
	Bjorn Helgaas <>,
	<>, <>,
	<>, <>,
	Abhishek Sahu <>
Subject: [PATCH v4 2/4] vfio/pci: Change the PF power state to D0 before enabling VFs
Date: Tue, 17 May 2022 15:32:17 +0530	[thread overview]
Message-ID: <> (raw)
In-Reply-To: <>

According to [PCIe v5 9.6.2] for PF Device Power Management States

 "The PF's power management state (D-state) has global impact on its
  associated VFs. If a VF does not implement the Power Management
  Capability, then it behaves as if it is in an equivalent
  power state of its associated PF.

  If a VF implements the Power Management Capability, the Device behavior
  is undefined if the PF is placed in a lower power state than the VF.
  Software should avoid this situation by placing all VFs in lower power
  state before lowering their associated PF's power state."

From the vfio driver side, user can enable SR-IOV when the PF is in D3hot
state. If VF does not implement the Power Management Capability, then
the VF will be actually in D3hot state and then the VF BAR access will
fail. If VF implements the Power Management Capability, then VF will
assume that its current power state is D0 when the PF is D3hot and
in this case, the behavior is undefined.

To support PF power management, we need to create power management
dependency between PF and its VF's. The runtime power management support
may help with this where power management dependencies are supported
through device links. But till we have such support in place, we can
disallow the PF to go into low power state, if PF has VF enabled.
There can be a case, where user first enables the VF's and then
disables the VF's. If there is no user of PF, then the PF can put into
D3hot state again. But with this patch, the PF will still be in D0
state after disabling VF's since detecting this case inside
vfio_pci_core_sriov_configure() requires access to
struct vfio_device::open_count along with its locks. But the subsequent
patches related to runtime PM will handle this case since runtime PM
maintains its own usage count.

Also, vfio_pci_core_sriov_configure() can be called at any time
(with and without vfio pci device user), so the power state change
needs to be protected with the required locks.

Signed-off-by: Abhishek Sahu <>
 drivers/vfio/pci/vfio_pci_core.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c
index b9f222ca48cf..4fe9a4efc751 100644
--- a/drivers/vfio/pci/vfio_pci_core.c
+++ b/drivers/vfio/pci/vfio_pci_core.c
@@ -217,6 +217,10 @@ int vfio_pci_set_power_state(struct vfio_pci_core_device *vdev, pci_power_t stat
 	bool needs_restore = false, needs_save = false;
 	int ret;
+	/* Prevent changing power state for PFs with VFs enabled */
+	if (pci_num_vf(pdev) && state > PCI_D0)
+		return -EBUSY;
 	if (vdev->needs_pm_restore) {
 		if (pdev->current_state < PCI_D3hot && state >= PCI_D3hot) {
@@ -1960,6 +1964,13 @@ int vfio_pci_core_sriov_configure(struct vfio_pci_core_device *vdev,
 		list_add_tail(&vdev->sriov_pfs_item, &vfio_pci_sriov_pfs);
+		/*
+		 * The PF power state should always be higher than the VF power
+		 * state. If PF is in the low power state, then change the
+		 * power state to D0 first before enabling SR-IOV.
+		 */
+		vfio_pci_lock_and_set_power_state(vdev, PCI_D0);
 		ret = pci_enable_sriov(pdev, nr_virtfn);
 		if (ret)
 			goto out_del;

  parent reply	other threads:[~2022-05-17 10:03 UTC|newest]

Thread overview: 12+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2022-05-17 10:02 [PATCH v4 0/4] vfio/pci: power management changes Abhishek Sahu
2022-05-17 10:02 ` [PATCH v4 1/4] vfio/pci: Invalidate mmaps and block the access in D3hot power state Abhishek Sahu
2022-05-17 23:30   ` kernel test robot
2022-05-17 10:02 ` Abhishek Sahu [this message]
2022-05-17 18:27   ` [PATCH v4 2/4] vfio/pci: Change the PF power state to D0 before enabling VFs Alex Williamson
2022-05-18  9:56     ` Abhishek Sahu
2022-05-17 10:02 ` [PATCH v4 3/4] vfio/pci: Virtualize PME related registers bits and initialize to zero Abhishek Sahu
2022-05-17 10:02 ` [PATCH v4 4/4] vfio/pci: Move the unused device into low power state with runtime PM Abhishek Sahu
2022-05-17 20:02   ` Alex Williamson
2022-05-18 10:06     ` Abhishek Sahu
2022-05-17 20:42   ` Alex Williamson
2022-05-17 20:55     ` Alex Williamson

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).