[Intel-wired-lan] [RFC v4 0/6] Introduce dynamic allocation/freeing of MSI-X vectors

[Intel-wired-lan] [RFC v4 0/6] Introduce dynamic allocation/freeing of MSI-X vectors

[Megha Dey]

Currently, MSI-X vector enabling and allocation for a PCIe device is static
i.e. a device driver gets only one chance to enable a specific number of
MSI-X vectors, usually during device probe. Also, in many cases, drivers
usually reserve more than required number of vectors anticipating their
use, which unnecessarily blocks resources that could have been made available
to other devices. Lastly, there is no way for drivers to reserve more vectors,
if the MSI-x has already been enabled for that device.

Hence, a dynamic MSI-X kernel infrastructure can benefit drivers by deferring
MSI-X allocation to post probe phase, where actual demand information is
available.

This patchset enables the dynamic allocation/de-allocation of MSI-X vectors
by introducing 2 new APIs: pci_alloc_irq_vectors_dyn() and
pci_free_irq_vectors_grp().

Changes V3->V4
1. Initialised first_desc for MSI as well.

Changes V2->V3
1. Added a separate patch to include documentation
2. Added a lookup function pci_irq_vector_group based on group id
3. Added a 'one_shot' field to struct device
4. Ensure msix table base gets mapped only once
5. Removed redundant checks
6. Use pointer for msi related data to reduce memory footprint of struct device
7. Initialization of idrs used internally by MSI code done by the core code
   instead of drivers/callers..

Changes V1->V2
1. Added relavant documentation
2. Avoid code duplication by ensuring pci_alloc_irq_vectors_affinity calls
   pci_alloc_irq_vectors_affinity_dyn function
3. Add the pci_free_irq_vectors_grp API

Megha Dey (6):
  PCI/MSI: New structures/macros for dynamic MSI-X allocation
  PCI/MSI: Dynamic allocation of MSI-X vectors by group
  x86: Introduce the dynamic teardown function
  PCI/MSI: Introduce new structure to manage MSI-x entries
  PCI/MSI: Free MSI-X resources by group
  Documentation: PCI/MSI: Document dynamic MSI-X infrastructure

 Documentation/PCI/MSI-HOWTO.txt |  36 +++++
 arch/x86/include/asm/x86_init.h |   1 +
 arch/x86/kernel/x86_init.c      |   6 +
 drivers/pci/msi.c               | 350 +++++++++++++++++++++++++++++++++++++---
 drivers/pci/probe.c             |  20 +++
 include/linux/device.h          |  14 ++
 include/linux/msi.h             |  11 ++
 include/linux/pci.h             |  56 +++++++
 kernel/irq/msi.c                |  37 ++++-
 9 files changed, 503 insertions(+), 28 deletions(-)

-- 
2.7.4

[Intel-wired-lan] [RFC v4 1/6] PCI/MSI: New structures/macros for dynamic MSI-X allocation

[Megha Dey]

This is a preparatory patch to introduce the dynamic allocation of
MSI-X vectors. In this patch, we add new structure members, introduce
relevant structures and macros which will be consumed by the API which
will dynamically allocate the MSI-X vectors.

Signed-off-by: Megha Dey <megha.dey@linux.intel.com>
---
 include/linux/device.h | 14 ++++++++++++++
 include/linux/msi.h    |  7 +++++++
 include/linux/pci.h    |  2 ++
 3 files changed, 23 insertions(+)

diff --git a/include/linux/device.h b/include/linux/device.h
index 6cb4640..3e9b138 100644
--- a/include/linux/device.h
+++ b/include/linux/device.h
@@ -887,6 +887,16 @@ struct dev_links_info {
 };
 
 /**
+ * struct dev_idr - Device group and entry ID allocator.
+ * @grp_idr: group ID allocator
+ * @entry_idr: entry ID allocator
+ */
+struct dev_idr {
+	struct idr *grp_idr;
+	struct idr *entry_idr;
+};
+
+/**
  * struct device - The basic device structure
  * @parent:	The device's "parent" device, the device to which it is attached.
  * 		In most cases, a parent device is some sort of bus or host
@@ -1051,6 +1061,10 @@ struct device {
     defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU_ALL)
 	bool			dma_coherent:1;
 #endif
+	/* For dynamic MSI-X allocation */
+	struct msi_desc		*first_desc;	/* 1st MSI-X desc of the latest allocated group */
+	struct dev_idr		*msix_dev_idr;	/* entry/group allocator */
+	bool			one_shot;	/* dynamic or oneshot MSI-X allocation */
 };
 
 static inline struct device *kobj_to_dev(struct kobject *kobj)
diff --git a/include/linux/msi.h b/include/linux/msi.h
index 784fb52..2092abf 100644
--- a/include/linux/msi.h
+++ b/include/linux/msi.h
@@ -118,11 +118,18 @@ struct msi_desc {
 	list_for_each_entry((desc), dev_to_msi_list((dev)), list)
 #define for_each_msi_entry_safe(desc, tmp, dev)	\
 	list_for_each_entry_safe((desc), (tmp), dev_to_msi_list((dev)), list)
+/* Iterate through MSI entries from a given entry */
+#define for_each_msi_entry_from(desc, dev)                             \
+	desc = (*dev).first_desc;                                       \
+	list_for_each_entry_from((desc), dev_to_msi_list((dev)), list)  \
 
 #ifdef CONFIG_PCI_MSI
 #define first_pci_msi_entry(pdev)	first_msi_entry(&(pdev)->dev)
 #define for_each_pci_msi_entry(desc, pdev)	\
 	for_each_msi_entry((desc), &(pdev)->dev)
+/* Iterate through PCI-MSI entries from a given entry */
+#define for_each_pci_msi_entry_from(desc, pdev)        \
+	for_each_msi_entry_from((desc), &(pdev)->dev)
 
 struct pci_dev *msi_desc_to_pci_dev(struct msi_desc *desc);
 void *msi_desc_to_pci_sysdata(struct msi_desc *desc);
diff --git a/include/linux/pci.h b/include/linux/pci.h
index 65f1d8c..177305f 100644
--- a/include/linux/pci.h
+++ b/include/linux/pci.h
@@ -459,6 +459,8 @@ struct pci_dev {
 	char		*driver_override; /* Driver name to force a match */
 
 	unsigned long	priv_flags;	/* Private flags for the PCI driver */
+	unsigned int	num_msix;	/* Number of MSI-X vectors supported */
+	void __iomem	*base;		/* Base address of MSI-X table */
 };
 
 static inline struct pci_dev *pci_physfn(struct pci_dev *dev)
-- 
2.7.4

[Intel-wired-lan] [RFC v4 2/6] PCI/MSI: Dynamic allocation of MSI-X vectors by group

[Megha Dey]

Currently, MSI-X vector enabling and allocation for a PCIe device is
static i.e. a device driver gets only one chance to enable a specific
number of MSI-X vectors, usually during device probe. Also, in many
cases, drivers usually reserve more than required number of vectors
anticipating their use, which unnecessarily blocks resources that
could have been made available to other devices. Lastly, there is no
way for drivers to reserve more vectors, if the MSI-x has already been
enabled for that device.

Hence, a dynamic MSI-X kernel infrastructure can benefit drivers by
deferring MSI-X allocation to post probe phase, where actual demand
information is available.

This patch enables the dynamic allocation of MSI-X vectors even after
MSI-X is enabled for a PCIe device by introducing a new API:
pci_alloc_irq_vectors_dyn().

This API can be called multiple times by the driver. The MSI-X vectors
allocated each time this API is called are associated with a group ID.

In order to obtain the Linux IRQ number associated with each vector in
a group, a new api, pci_irq_vector_group() has been introduced.

Signed-off-by: Megha Dey <megha.dey@linux.intel.com>
---
 drivers/pci/msi.c   | 186 +++++++++++++++++++++++++++++++++++++++++++++-------
 drivers/pci/probe.c |  19 ++++++
 include/linux/pci.h |  36 ++++++++++
 kernel/irq/msi.c    |   8 +--
 4 files changed, 222 insertions(+), 27 deletions(-)

diff --git a/drivers/pci/msi.c b/drivers/pci/msi.c
index 4c0b478..a0cf3d3 100644
--- a/drivers/pci/msi.c
+++ b/drivers/pci/msi.c
@@ -102,7 +102,7 @@ int __weak arch_setup_msi_irqs(struct pci_dev *dev, int nvec, int type)
 	if (type == PCI_CAP_ID_MSI && nvec > 1)
 		return 1;
 
-	for_each_pci_msi_entry(entry, dev) {
+	for_each_pci_msi_entry_from(entry, dev) {
 		ret = arch_setup_msi_irq(dev, entry);
 		if (ret < 0)
 			return ret;
@@ -468,7 +468,7 @@ static int populate_msi_sysfs(struct pci_dev *pdev)
 	int i;
 
 	/* Determine how many msi entries we have */
-	for_each_pci_msi_entry(entry, pdev)
+	for_each_pci_msi_entry_from(entry, pdev)
 		num_msi += entry->nvec_used;
 	if (!num_msi)
 		return 0;
@@ -477,7 +477,7 @@ static int populate_msi_sysfs(struct pci_dev *pdev)
 	msi_attrs = kcalloc(num_msi + 1, sizeof(void *), GFP_KERNEL);
 	if (!msi_attrs)
 		return -ENOMEM;
-	for_each_pci_msi_entry(entry, pdev) {
+	for_each_pci_msi_entry_from(entry, pdev) {
 		for (i = 0; i < entry->nvec_used; i++) {
 			msi_dev_attr = kzalloc(sizeof(*msi_dev_attr), GFP_KERNEL);
 			if (!msi_dev_attr)
@@ -506,7 +506,11 @@ static int populate_msi_sysfs(struct pci_dev *pdev)
 		goto error_irq_group;
 	msi_irq_groups[0] = msi_irq_group;
 
-	ret = sysfs_create_groups(&pdev->dev.kobj, msi_irq_groups);
+	if (!pdev->msix_enabled)
+		ret = sysfs_create_group(&pdev->dev.kobj, msi_irq_group);
+	else
+		ret = sysfs_merge_group(&pdev->dev.kobj, msi_irq_group);
+
 	if (ret)
 		goto error_irq_groups;
 	pdev->msi_irq_groups = msi_irq_groups;
@@ -574,7 +578,7 @@ static int msi_verify_entries(struct pci_dev *dev)
 {
 	struct msi_desc *entry;
 
-	for_each_pci_msi_entry(entry, dev) {
+	for_each_pci_msi_entry_from(entry, dev) {
 		if (!dev->no_64bit_msi || !entry->msg.address_hi)
 			continue;
 		pci_err(dev, "Device has broken 64-bit MSI but arch"
@@ -615,6 +619,9 @@ static int msi_capability_init(struct pci_dev *dev, int nvec,
 
 	list_add_tail(&entry->list, dev_to_msi_list(&dev->dev));
 
+	 dev->dev.first_desc = list_last_entry
+                        (dev_to_msi_list(&dev->dev), struct msi_desc, list);
+
 	/* Configure MSI capability structure */
 	ret = pci_msi_setup_msi_irqs(dev, nvec, PCI_CAP_ID_MSI);
 	if (ret) {
@@ -700,6 +707,17 @@ static int msix_setup_entries(struct pci_dev *dev, void __iomem *base,
 		entry->mask_base		= base;
 
 		list_add_tail(&entry->list, dev_to_msi_list(&dev->dev));
+
+		/*
+		 * Save the pointer to the first msi_desc entry of every
+		 * MSI-X group. This pointer is used by other functions
+		 * as the starting point to iterate through each of the
+		 * entries in that particular group.
+		 */
+		if (!i)
+			dev->dev.first_desc = list_last_entry
+			(dev_to_msi_list(&dev->dev), struct msi_desc, list);
+
 		if (masks)
 			curmsk++;
 	}
@@ -715,7 +733,7 @@ static void msix_program_entries(struct pci_dev *dev,
 	struct msi_desc *entry;
 	int i = 0;
 
-	for_each_pci_msi_entry(entry, dev) {
+	for_each_pci_msi_entry_from(entry, dev) {
 		if (entries)
 			entries[i++].vector = entry->irq;
 		entry->masked = readl(pci_msix_desc_addr(entry) +
@@ -740,18 +758,20 @@ static int msix_capability_init(struct pci_dev *dev, struct msix_entry *entries,
 {
 	int ret;
 	u16 control;
-	void __iomem *base;
 
 	/* Ensure MSI-X is disabled while it is set up */
 	pci_msix_clear_and_set_ctrl(dev, PCI_MSIX_FLAGS_ENABLE, 0);
 
 	pci_read_config_word(dev, dev->msix_cap + PCI_MSIX_FLAGS, &control);
+
 	/* Request & Map MSI-X table region */
-	base = msix_map_region(dev, msix_table_size(control));
-	if (!base)
-		return -ENOMEM;
+	if (!dev->msix_enabled) {
+		dev->base = msix_map_region(dev, msix_table_size(control));
+		if (!dev->base)
+			return -ENOMEM;
+	}
 
-	ret = msix_setup_entries(dev, base, entries, nvec, affd);
+	ret = msix_setup_entries(dev, dev->base, entries, nvec, affd);
 	if (ret)
 		return ret;
 
@@ -784,6 +804,7 @@ static int msix_capability_init(struct pci_dev *dev, struct msix_entry *entries,
 	pci_msix_clear_and_set_ctrl(dev, PCI_MSIX_FLAGS_MASKALL, 0);
 
 	pcibios_free_irq(dev);
+
 	return 0;
 
 out_avail:
@@ -795,7 +816,7 @@ static int msix_capability_init(struct pci_dev *dev, struct msix_entry *entries,
 		struct msi_desc *entry;
 		int avail = 0;
 
-		for_each_pci_msi_entry(entry, dev) {
+		for_each_pci_msi_entry_from(entry, dev) {
 			if (entry->irq != 0)
 				avail++;
 		}
@@ -932,7 +953,8 @@ int pci_msix_vec_count(struct pci_dev *dev)
 EXPORT_SYMBOL(pci_msix_vec_count);
 
 static int __pci_enable_msix(struct pci_dev *dev, struct msix_entry *entries,
-			     int nvec, const struct irq_affinity *affd)
+			     int nvec, const struct irq_affinity *affd,
+			     bool one_shot)
 {
 	int nr_entries;
 	int i, j;
@@ -1086,7 +1108,8 @@ EXPORT_SYMBOL(pci_enable_msi);
 
 static int __pci_enable_msix_range(struct pci_dev *dev,
 				   struct msix_entry *entries, int minvec,
-				   int maxvec, const struct irq_affinity *affd)
+				   int maxvec, const struct irq_affinity *affd,
+				   bool one_shot)
 {
 	int rc, nvec = maxvec;
 
@@ -1100,7 +1123,7 @@ static int __pci_enable_msix_range(struct pci_dev *dev,
 	if (affd && affd->nr_sets && minvec != maxvec)
 		return -EINVAL;
 
-	if (WARN_ON_ONCE(dev->msix_enabled))
+	if (one_shot && WARN_ON_ONCE(dev->msix_enabled))
 		return -EINVAL;
 
 	for (;;) {
@@ -1110,7 +1133,7 @@ static int __pci_enable_msix_range(struct pci_dev *dev,
 				return -ENOSPC;
 		}
 
-		rc = __pci_enable_msix(dev, entries, nvec, affd);
+		rc = __pci_enable_msix(dev, entries, nvec, affd, one_shot);
 		if (rc == 0)
 			return nvec;
 
@@ -1141,7 +1164,8 @@ static int __pci_enable_msix_range(struct pci_dev *dev,
 int pci_enable_msix_range(struct pci_dev *dev, struct msix_entry *entries,
 		int minvec, int maxvec)
 {
-	return __pci_enable_msix_range(dev, entries, minvec, maxvec, NULL);
+	return __pci_enable_msix_range(dev, entries, minvec, maxvec, NULL,
+									false);
 }
 EXPORT_SYMBOL(pci_enable_msix_range);
 
@@ -1167,9 +1191,45 @@ int pci_alloc_irq_vectors_affinity(struct pci_dev *dev, unsigned int min_vecs,
 				   unsigned int max_vecs, unsigned int flags,
 				   const struct irq_affinity *affd)
 {
+	int *group = NULL;
+
+	dev->dev.one_shot = true;
+
+	return pci_alloc_irq_vectors_affinity_dyn(dev, min_vecs, max_vecs,
+					flags, NULL, group, dev->dev.one_shot);
+}
+EXPORT_SYMBOL(pci_alloc_irq_vectors_affinity);
+
+/**
+ * pci_alloc_irq_vectors_affinity_dyn - allocate multiple IRQs for a device
+ * dynamically. Can be called multiple times.
+ * @dev:		PCI device to operate on
+ * @min_vecs:		minimum number of vectors required (must be >= 1)
+ * @max_vecs:		maximum (desired) number of vectors
+ * @flags:		flags or quirks for the allocation
+ * @affd:		optional description of the affinity requirements
+ * @group_id:		group ID assigned to vectors allocated
+ *
+ * Allocate up to @max_vecs interrupt vectors for @dev, using MSI-X. Return
+ * the number of vectors allocated (which might be smaller than @max_vecs)
+ * if successful, or a negative error code on error. If less than @min_vecs
+ * interrupt vectors are available for @dev the function will fail with -ENOSPC.
+ * Assign a unique group ID to the set of vectors being allocated.
+ *
+ * To get the Linux IRQ number used for a vector that can be passed to
+ * request_irq() use the pci_irq_vector() helper.
+ */
+int pci_alloc_irq_vectors_affinity_dyn(struct pci_dev *dev,
+				      unsigned int min_vecs,
+				      unsigned int max_vecs,
+				      unsigned int flags,
+				      const struct irq_affinity *affd,
+				      int *group_id, bool one_shot)
+{
 	static const struct irq_affinity msi_default_affd;
-	int msix_vecs = -ENOSPC;
+	int msix_vecs = -ENOSPC, i, *group = NULL;
 	int msi_vecs = -ENOSPC;
+	struct msix_entry *entries = NULL;
 
 	if (flags & PCI_IRQ_AFFINITY) {
 		if (!affd)
@@ -1180,15 +1240,53 @@ int pci_alloc_irq_vectors_affinity(struct pci_dev *dev, unsigned int min_vecs,
 	}
 
 	if (flags & PCI_IRQ_MSIX) {
-		msix_vecs = __pci_enable_msix_range(dev, NULL, min_vecs,
-						    max_vecs, affd);
+		if (!one_shot) {
+			entries = kcalloc(max_vecs, sizeof(struct msix_entry),
+								GFP_KERNEL);
+			if (entries == NULL)
+				return -ENOMEM;
+
+			group = kcalloc(max_vecs, sizeof(int), GFP_KERNEL);
+			if (group == NULL)
+				return -ENOMEM;
+
+			if (!dev->msix_enabled)
+				dev->num_msix = pci_msix_vec_count(dev);
+
+			/* Assign a unique group ID */
+			*group = idr_alloc(dev->dev.msix_dev_idr->grp_idr, NULL,
+						0, dev->num_msix, GFP_KERNEL);
+			if (*group < 0) {
+				if (*group == -ENOSPC)
+					pci_err(dev, "No free group IDs\n");
+				return *group;
+			}
+			*group_id = *group;
+
+			for (i = 0; i < max_vecs; i++) {
+				/* tag every entry with a group ID */
+				entries[i].entry = idr_alloc(
+					dev->dev.msix_dev_idr->entry_idr,
+					group, 0, dev->num_msix, GFP_KERNEL);
+				if (entries[i].entry < 0) {
+					if (entries[i].entry == -ENOSPC)
+						pci_err(dev, "No free IDs\n");
+					return entries[i].entry;
+				}
+			}
+		}
+
+		msix_vecs = __pci_enable_msix_range(dev, entries, min_vecs, max_vecs,
+								affd, one_shot);
+
+		kfree(entries);
+
 		if (msix_vecs > 0)
 			return msix_vecs;
 	}
 
 	if (flags & PCI_IRQ_MSI) {
-		msi_vecs = __pci_enable_msi_range(dev, min_vecs, max_vecs,
-						  affd);
+		msi_vecs = __pci_enable_msi_range(dev, min_vecs, max_vecs, affd);
 		if (msi_vecs > 0)
 			return msi_vecs;
 	}
@@ -1201,11 +1299,14 @@ int pci_alloc_irq_vectors_affinity(struct pci_dev *dev, unsigned int min_vecs,
 		}
 	}
 
+	if (flags & (PCI_IRQ_MSI | PCI_IRQ_LEGACY))
+		return -EINVAL;
+
 	if (msix_vecs == -ENOSPC)
 		return -ENOSPC;
 	return msi_vecs;
 }
-EXPORT_SYMBOL(pci_alloc_irq_vectors_affinity);
+EXPORT_SYMBOL(pci_alloc_irq_vectors_affinity_dyn);
 
 /**
  * pci_free_irq_vectors - free previously allocated IRQs for a device
@@ -1255,6 +1356,45 @@ int pci_irq_vector(struct pci_dev *dev, unsigned int nr)
 EXPORT_SYMBOL(pci_irq_vector);
 
 /**
+ * pci_irq_vector_group - return the IRQ number of a device vector associated
+ * with a group
+ * @dev: PCI device to operate on
+ * @nr: device-relative interrupt vector index (0-based).
+ * @group: group from which IRQ number should be returned
+ */
+int pci_irq_vector_group(struct pci_dev *dev, unsigned int nr,
+						unsigned int group_id)
+{
+	if (dev->msix_enabled) {
+		struct msi_desc *entry;
+		int *group, i = 0, grp_present = 0;
+
+		for_each_pci_msi_entry(entry, dev) {
+			group = idr_find(dev->dev.msix_dev_idr->entry_idr,
+						entry->msi_attrib.entry_nr);
+			if (*group == group_id) {
+				grp_present = 1;
+				if (i == nr)
+					return entry->irq;
+				i++;
+			}
+		}
+
+		if (!grp_present) {
+			pci_err(dev, "Group %d not present\n", group_id);
+			return -EINVAL;
+		}
+
+		pci_err(dev, "Interrupt vector index %d does not exist in "
+						"group %d\n", nr, group_id);
+	}
+
+	pci_err(dev, "MSI-X not enabled\n");
+	return -EINVAL;
+}
+EXPORT_SYMBOL(pci_irq_vector_group);
+
+/**
  * pci_irq_get_affinity - return the affinity of a particular msi vector
  * @dev:	PCI device to operate on
  * @nr:		device-relative interrupt vector index (0-based).
diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
index 257b9f6..dd4a6ef 100644
--- a/drivers/pci/probe.c
+++ b/drivers/pci/probe.c
@@ -2188,6 +2188,25 @@ struct pci_dev *pci_alloc_dev(struct pci_bus *bus)
 	if (!dev)
 		return NULL;
 
+	/* For dynamic MSI-x */
+	dev->dev.msix_dev_idr = kzalloc(sizeof(struct dev_idr), GFP_KERNEL);
+	if (!dev->dev.msix_dev_idr)
+		return NULL;
+
+	dev->dev.msix_dev_idr->grp_idr = kzalloc(sizeof(struct idr),
+								GFP_KERNEL);
+	if (!dev->dev.msix_dev_idr->grp_idr)
+		return NULL;
+
+	dev->dev.msix_dev_idr->entry_idr = kzalloc(sizeof(struct idr),
+								GFP_KERNEL);
+	if (!dev->dev.msix_dev_idr->entry_idr)
+		return NULL;
+
+	/* Initialise the IDR structures */
+	idr_init(dev->dev.msix_dev_idr->grp_idr);
+	idr_init(dev->dev.msix_dev_idr->entry_idr);
+
 	INIT_LIST_HEAD(&dev->bus_list);
 	dev->dev.type = &pci_dev_type;
 	dev->bus = pci_bus_get(bus);
diff --git a/include/linux/pci.h b/include/linux/pci.h
index 177305f..28eab4a 100644
--- a/include/linux/pci.h
+++ b/include/linux/pci.h
@@ -1396,9 +1396,16 @@ static inline int pci_enable_msix_exact(struct pci_dev *dev,
 int pci_alloc_irq_vectors_affinity(struct pci_dev *dev, unsigned int min_vecs,
 				   unsigned int max_vecs, unsigned int flags,
 				   const struct irq_affinity *affd);
+int pci_alloc_irq_vectors_affinity_dyn(struct pci_dev *dev,
+				   unsigned int min_vecs, unsigned int max_vecs,
+				   unsigned int flags,
+				   const struct irq_affinity *affd,
+				   int *group_id, bool one_shot);
 
 void pci_free_irq_vectors(struct pci_dev *dev);
 int pci_irq_vector(struct pci_dev *dev, unsigned int nr);
+int pci_irq_vector_group(struct pci_dev *dev, unsigned int nr,
+						unsigned int group_id);
 const struct cpumask *pci_irq_get_affinity(struct pci_dev *pdev, int vec);
 int pci_irq_get_node(struct pci_dev *pdev, int vec);
 
@@ -1428,6 +1435,17 @@ pci_alloc_irq_vectors_affinity(struct pci_dev *dev, unsigned int min_vecs,
 	return -ENOSPC;
 }
 
+static inline int
+pci_alloc_irq_vectors_affinity_dyn(struct pci_dev *dev, unsigned int min_vecs,
+				   unsigned int max_vecs, unsigned int flags,
+				   const struct irq_affinity *aff_desc,
+				   int *group_id, bool one_shot)
+{
+	if ((flags & PCI_IRQ_LEGACY) && min_vecs == 1 && dev->irq)
+		return 1;
+	return -ENOSPC;
+}
+
 static inline void pci_free_irq_vectors(struct pci_dev *dev)
 {
 }
@@ -1438,6 +1456,15 @@ static inline int pci_irq_vector(struct pci_dev *dev, unsigned int nr)
 		return -EINVAL;
 	return dev->irq;
 }
+
+static inline int pci_irq_vector_group(struct pci_dev *dev, unsigned int nr,
+							unsigned int group)
+{
+	if (WARN_ON_ONCE(nr > 0))
+		return -EINVAL;
+	return dev->irq;
+}
+
 static inline const struct cpumask *pci_irq_get_affinity(struct pci_dev *pdev,
 		int vec)
 {
@@ -1458,6 +1485,15 @@ pci_alloc_irq_vectors(struct pci_dev *dev, unsigned int min_vecs,
 					      NULL);
 }
 
+static inline int
+pci_alloc_irq_vectors_dyn(struct pci_dev *dev, unsigned int min_vecs,
+			  unsigned int max_vecs, unsigned int flags,
+			  int *group_id)
+{
+	return pci_alloc_irq_vectors_affinity_dyn(dev, min_vecs, max_vecs,
+					  flags, NULL, group_id, false);
+}
+
 /**
  * pci_irqd_intx_xlate() - Translate PCI INTx value to an IRQ domain hwirq
  * @d: the INTx IRQ domain
diff --git a/kernel/irq/msi.c b/kernel/irq/msi.c
index ad26fbc..5cfa931 100644
--- a/kernel/irq/msi.c
+++ b/kernel/irq/msi.c
@@ -411,7 +411,7 @@ int msi_domain_alloc_irqs(struct irq_domain *domain, struct device *dev,
 	if (ret)
 		return ret;
 
-	for_each_msi_entry(desc, dev) {
+	for_each_msi_entry_from(desc, dev) {
 		ops->set_desc(&arg, desc);
 
 		virq = __irq_domain_alloc_irqs(domain, -1, desc->nvec_used,
@@ -437,7 +437,7 @@ int msi_domain_alloc_irqs(struct irq_domain *domain, struct device *dev,
 
 	can_reserve = msi_check_reservation_mode(domain, info, dev);
 
-	for_each_msi_entry(desc, dev) {
+	for_each_msi_entry_from(desc, dev) {
 		virq = desc->irq;
 		if (desc->nvec_used == 1)
 			dev_dbg(dev, "irq %d for MSI\n", virq);
@@ -465,7 +465,7 @@ int msi_domain_alloc_irqs(struct irq_domain *domain, struct device *dev,
 	 * so request_irq() will assign the final vector.
 	 */
 	if (can_reserve) {
-		for_each_msi_entry(desc, dev) {
+		for_each_msi_entry_from(desc, dev) {
 			irq_data = irq_domain_get_irq_data(domain, desc->irq);
 			irqd_clr_activated(irq_data);
 		}
@@ -473,7 +473,7 @@ int msi_domain_alloc_irqs(struct irq_domain *domain, struct device *dev,
 	return 0;
 
 cleanup:
-	for_each_msi_entry(desc, dev) {
+	for_each_msi_entry_from(desc, dev) {
 		struct irq_data *irqd;
 
 		if (desc->irq == virq)
-- 
2.7.4

[Intel-wired-lan] [RFC v4 3/6] x86: Introduce the dynamic teardown function

[Megha Dey]

This is a preparatory patch to introduce disabling of MSI-X vectors
belonging to a particular group. In this patch, we introduce a x86
specific mechanism to teardown the IRQ vectors belonging to a
particular group.

Signed-off-by: Megha Dey <megha.dey@linux.intel.com>
---
 arch/x86/include/asm/x86_init.h |  1 +
 arch/x86/kernel/x86_init.c      |  6 ++++++
 drivers/pci/msi.c               | 20 ++++++++++++++++++++
 include/linux/msi.h             |  2 ++
 4 files changed, 29 insertions(+)

diff --git a/arch/x86/include/asm/x86_init.h b/arch/x86/include/asm/x86_init.h
index b85a7c5..50f26a0 100644
--- a/arch/x86/include/asm/x86_init.h
+++ b/arch/x86/include/asm/x86_init.h
@@ -283,6 +283,7 @@ struct pci_dev;
 struct x86_msi_ops {
 	int (*setup_msi_irqs)(struct pci_dev *dev, int nvec, int type);
 	void (*teardown_msi_irq)(unsigned int irq);
+	void (*teardown_msi_irqs_grp)(struct pci_dev *dev, int group_id);
 	void (*teardown_msi_irqs)(struct pci_dev *dev);
 	void (*restore_msi_irqs)(struct pci_dev *dev);
 };
diff --git a/arch/x86/kernel/x86_init.c b/arch/x86/kernel/x86_init.c
index 50a2b49..794e7d4 100644
--- a/arch/x86/kernel/x86_init.c
+++ b/arch/x86/kernel/x86_init.c
@@ -127,6 +127,7 @@ EXPORT_SYMBOL_GPL(x86_platform);
 struct x86_msi_ops x86_msi __ro_after_init = {
 	.setup_msi_irqs		= native_setup_msi_irqs,
 	.teardown_msi_irq	= native_teardown_msi_irq,
+	.teardown_msi_irqs_grp	= default_teardown_msi_irqs_grp,
 	.teardown_msi_irqs	= default_teardown_msi_irqs,
 	.restore_msi_irqs	= default_restore_msi_irqs,
 };
@@ -142,6 +143,11 @@ void arch_teardown_msi_irqs(struct pci_dev *dev)
 	x86_msi.teardown_msi_irqs(dev);
 }
 
+void arch_teardown_msi_irqs_grp(struct pci_dev *dev, int group_id)
+{
+	x86_msi.teardown_msi_irqs_grp(dev, group_id);
+}
+
 void arch_teardown_msi_irq(unsigned int irq)
 {
 	x86_msi.teardown_msi_irq(irq);
diff --git a/drivers/pci/msi.c b/drivers/pci/msi.c
index a0cf3d3..5e4f452 100644
--- a/drivers/pci/msi.c
+++ b/drivers/pci/msi.c
@@ -133,6 +133,26 @@ void __weak arch_teardown_msi_irqs(struct pci_dev *dev)
 	return default_teardown_msi_irqs(dev);
 }
 
+void default_teardown_msi_irqs_grp(struct pci_dev *dev, int group_id)
+{
+	int i, *group = NULL;
+	struct msi_desc *entry;
+
+	for_each_pci_msi_entry(entry, dev) {
+		group = idr_find(dev->dev.msix_dev_idr->entry_idr,
+						entry->msi_attrib.entry_nr);
+		if (*group == group_id && entry->irq) {
+			for (i = 0; i < entry->nvec_used; i++)
+				arch_teardown_msi_irq(entry->irq + i);
+		}
+	}
+}
+
+void __weak arch_teardown_msi_irqs_grp(struct pci_dev *dev, int group_id)
+{
+	return default_teardown_msi_irqs_grp(dev, group_id);
+}
+
 static void default_restore_msi_irq(struct pci_dev *dev, int irq)
 {
 	struct msi_desc *entry;
diff --git a/include/linux/msi.h b/include/linux/msi.h
index 2092abf..6be0377 100644
--- a/include/linux/msi.h
+++ b/include/linux/msi.h
@@ -182,9 +182,11 @@ int arch_setup_msi_irq(struct pci_dev *dev, struct msi_desc *desc);
 void arch_teardown_msi_irq(unsigned int irq);
 int arch_setup_msi_irqs(struct pci_dev *dev, int nvec, int type);
 void arch_teardown_msi_irqs(struct pci_dev *dev);
+void arch_teardown_msi_irqs_grp(struct pci_dev *dev, int group_id);
 void arch_restore_msi_irqs(struct pci_dev *dev);
 
 void default_teardown_msi_irqs(struct pci_dev *dev);
+void default_teardown_msi_irqs_grp(struct pci_dev *dev, int group_id);
 void default_restore_msi_irqs(struct pci_dev *dev);
 
 struct msi_controller {
-- 
2.7.4

[Intel-wired-lan] [RFC v4 4/6] PCI/MSI: Introduce new structure to manage MSI-x entries

[Megha Dey]

This is a preparatory patch to introduce disabling of MSI-X vectors
belonging to a particular group. In this patch, we introduce a new
structure msix_sysfs, which manages sysfs entries for dynamically
allocated MSI-X vectors belonging to a particular group.

Signed-off-by: Megha Dey <megha.dey@linux.intel.com>
---
 drivers/pci/msi.c   | 15 ++++++++++++++-
 drivers/pci/probe.c |  1 +
 include/linux/pci.h |  9 +++++++++
 3 files changed, 24 insertions(+), 1 deletion(-)

diff --git a/drivers/pci/msi.c b/drivers/pci/msi.c
index 5e4f452..e7bbd21 100644
--- a/drivers/pci/msi.c
+++ b/drivers/pci/msi.c
@@ -481,10 +481,11 @@ static int populate_msi_sysfs(struct pci_dev *pdev)
 	struct device_attribute *msi_dev_attr;
 	struct attribute_group *msi_irq_group;
 	const struct attribute_group **msi_irq_groups;
+	struct msix_sysfs *msix_sysfs_entry;
 	struct msi_desc *entry;
 	int ret = -ENOMEM;
 	int num_msi = 0;
-	int count = 0;
+	int count = 0, *group = NULL;
 	int i;
 
 	/* Determine how many msi entries we have */
@@ -511,6 +512,10 @@ static int populate_msi_sysfs(struct pci_dev *pdev)
 				goto error_attrs;
 			msi_dev_attr->attr.mode = S_IRUGO;
 			msi_dev_attr->show = msi_mode_show;
+			if (!i)
+				group = idr_find(
+					pdev->dev.msix_dev_idr->entry_idr,
+						entry->msi_attrib.entry_nr);
 			++count;
 		}
 	}
@@ -526,6 +531,14 @@ static int populate_msi_sysfs(struct pci_dev *pdev)
 		goto error_irq_group;
 	msi_irq_groups[0] = msi_irq_group;
 
+	msix_sysfs_entry = kzalloc(sizeof(*msix_sysfs_entry) * 2, GFP_KERNEL);
+	msix_sysfs_entry->msi_irq_group = msi_irq_group;
+	if (group)
+		msix_sysfs_entry->group_id = *group;
+	msix_sysfs_entry->vecs_in_grp = count;
+	INIT_LIST_HEAD(&msix_sysfs_entry->list);
+	list_add_tail(&msix_sysfs_entry->list, &pdev->msix_sysfs);
+
 	if (!pdev->msix_enabled)
 		ret = sysfs_create_group(&pdev->dev.kobj, msi_irq_group);
 	else
diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
index dd4a6ef..02f5d88 100644
--- a/drivers/pci/probe.c
+++ b/drivers/pci/probe.c
@@ -2208,6 +2208,7 @@ struct pci_dev *pci_alloc_dev(struct pci_bus *bus)
 	idr_init(dev->dev.msix_dev_idr->entry_idr);
 
 	INIT_LIST_HEAD(&dev->bus_list);
+	INIT_LIST_HEAD(&dev->msix_sysfs);
 	dev->dev.type = &pci_dev_type;
 	dev->bus = pci_bus_get(bus);
 
diff --git a/include/linux/pci.h b/include/linux/pci.h
index 28eab4a..2e2f4c9 100644
--- a/include/linux/pci.h
+++ b/include/linux/pci.h
@@ -461,6 +461,7 @@ struct pci_dev {
 	unsigned long	priv_flags;	/* Private flags for the PCI driver */
 	unsigned int	num_msix;	/* Number of MSI-X vectors supported */
 	void __iomem	*base;		/* Base address of MSI-X table */
+	struct list_head        msix_sysfs; /* Manage sysfs entries for each MSI-x group */
 };
 
 static inline struct pci_dev *pci_physfn(struct pci_dev *dev)
@@ -1375,6 +1376,14 @@ struct msix_entry {
 	u16	entry;	/* Driver uses to specify entry, OS writes */
 };
 
+/* Manage sysfs entries for dynamically allocated MSI-X vectors */
+struct msix_sysfs {
+	struct	attribute_group *msi_irq_group;
+	struct	list_head list;
+	int	group_id;
+	int	vecs_in_grp;
+};
+
 #ifdef CONFIG_PCI_MSI
 int pci_msi_vec_count(struct pci_dev *dev);
 void pci_disable_msi(struct pci_dev *dev);
-- 
2.7.4

[Intel-wired-lan] [RFC v4 5/6] PCI/MSI: Free MSI-X resources by group

[Megha Dey]

Currently, the pci_free_irq_vectors() frees all the allocated resources
associated with a PCIe device when the device is being shut down. With
the introduction of dynamic allocation of MSI-X vectors by group ID,
there should exist an API which can free the resources allocated only
to a particular group, which can be called even if the device is not
being shut down. The pci_free_irq_vectors_grp() function provides this
type of interface.

The existing pci_free_irq_vectors() can be called along side this API.

Signed-off-by: Megha Dey <megha.dey@linux.intel.com>
---
 drivers/pci/msi.c   | 129 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 include/linux/msi.h |   2 +
 include/linux/pci.h |   9 ++++
 kernel/irq/msi.c    |  29 ++++++++++++
 4 files changed, 169 insertions(+)

diff --git a/drivers/pci/msi.c b/drivers/pci/msi.c
index e7bbd21..a775c79 100644
--- a/drivers/pci/msi.c
+++ b/drivers/pci/msi.c
@@ -53,9 +53,23 @@ static void pci_msi_teardown_msi_irqs(struct pci_dev *dev)
 	else
 		arch_teardown_msi_irqs(dev);
 }
+
+static void pci_msi_teardown_msi_irqs_grp(struct pci_dev *dev, int group_id)
+{
+	struct irq_domain *domain;
+
+	domain = dev_get_msi_domain(&dev->dev);
+
+	if (domain && irq_domain_is_hierarchy(domain))
+		msi_domain_free_irqs_grp(domain, &dev->dev, group_id);
+	else
+		arch_teardown_msi_irqs_grp(dev, group_id);
+}
+
 #else
 #define pci_msi_setup_msi_irqs		arch_setup_msi_irqs
 #define pci_msi_teardown_msi_irqs	arch_teardown_msi_irqs
+#define pci_msi_teardown_msi_irqs_grp	default_teardown_msi_irqs_grp
 #endif
 
 /* Arch hooks */
@@ -400,6 +414,65 @@ static void free_msi_irqs(struct pci_dev *dev)
 	}
 }
 
+char msix_sysfs_grp[] = "msi_irqs";
+
+static int free_msi_irqs_grp(struct pci_dev *dev, int group_id)
+{
+	struct list_head *msi_list = dev_to_msi_list(&dev->dev);
+	struct msi_desc *entry, *tmp;
+	struct attribute **msi_attrs;
+	struct device_attribute *dev_attr;
+	int i, *group = NULL;
+	long vec;
+	struct msix_sysfs *msix_sysfs_entry, *tmp_msix;
+	struct list_head *pci_msix = &dev->msix_sysfs;
+	int num_vec = 0;
+
+	for_each_pci_msi_entry(entry, dev) {
+		group = idr_find(dev->dev.msix_dev_idr->entry_idr,
+					entry->msi_attrib.entry_nr);
+		if (*group == group_id && entry->irq)
+			for (i = 0; i < entry->nvec_used; i++)
+				BUG_ON(irq_has_action(entry->irq + i));
+	}
+
+	pci_msi_teardown_msi_irqs_grp(dev, group_id);
+
+	list_for_each_entry_safe(entry, tmp, msi_list, list) {
+		group = idr_find(dev->dev.msix_dev_idr->entry_idr,
+						entry->msi_attrib.entry_nr);
+		if (*group == group_id) {
+			idr_remove(dev->dev.msix_dev_idr->entry_idr,
+						entry->msi_attrib.entry_nr);
+			list_del(&entry->list);
+			free_msi_entry(entry);
+		}
+	}
+
+	list_for_each_entry_safe(msix_sysfs_entry, tmp_msix, pci_msix, list) {
+		if (msix_sysfs_entry->group_id == group_id) {
+			msi_attrs = msix_sysfs_entry->msi_irq_group->attrs;
+			for (i = 0; i < msix_sysfs_entry->vecs_in_grp; i++) {
+				if (!i)
+					num_vec = msix_sysfs_entry->vecs_in_grp;
+				dev_attr = container_of(msi_attrs[i],
+						struct device_attribute, attr);
+				sysfs_remove_file_from_group(&dev->dev.kobj,
+					&dev_attr->attr, msix_sysfs_grp);
+				if (kstrtol(dev_attr->attr.name, 10, &vec))
+					return -EINVAL;
+				kfree(dev_attr->attr.name);
+				kfree(dev_attr);
+			}
+			msix_sysfs_entry->msi_irq_group = NULL;
+			list_del(&msix_sysfs_entry->list);
+			idr_remove(dev->dev.msix_dev_idr->grp_idr, group_id);
+			kfree(msix_sysfs_entry);
+		}
+	}
+	return num_vec;
+}
+
 static void pci_intx_for_msi(struct pci_dev *dev, int enable)
 {
 	if (!(dev->dev_flags & PCI_DEV_FLAGS_MSI_INTX_DISABLE_BUG))
@@ -1055,6 +1128,47 @@ void pci_disable_msix(struct pci_dev *dev)
 }
 EXPORT_SYMBOL(pci_disable_msix);
 
+static void pci_msix_shutdown_grp(struct pci_dev *dev, int group_id)
+{
+	struct msi_desc *entry;
+	int grp_present = 0, *group = NULL;
+
+	if (pci_dev_is_disconnected(dev)) {
+		dev->msix_enabled = 0;
+		return;
+	}
+
+	/* Return the device with MSI-X masked as initial states */
+	for_each_pci_msi_entry(entry, dev) {
+		group = idr_find(dev->dev.msix_dev_idr->entry_idr,
+					entry->msi_attrib.entry_nr);
+		if (*group == group_id) {
+			/* Keep cached states to be restored */
+			__pci_msix_desc_mask_irq(entry, 1);
+			grp_present = 1;
+		}
+	}
+
+	if (!grp_present) {
+		pci_err(dev, "Group to be disabled not present\n");
+		return;
+	}
+}
+
+int pci_disable_msix_grp(struct pci_dev *dev, int group_id)
+{
+	int num_vecs;
+
+	if (!pci_msi_enable || !dev)
+		return -EINVAL;
+
+	pci_msix_shutdown_grp(dev, group_id);
+	num_vecs = free_msi_irqs_grp(dev, group_id);
+
+	return num_vecs;
+}
+EXPORT_SYMBOL(pci_disable_msix_grp);
+
 void pci_no_msi(void)
 {
 	pci_msi_enable = 0;
@@ -1355,6 +1469,21 @@ void pci_free_irq_vectors(struct pci_dev *dev)
 EXPORT_SYMBOL(pci_free_irq_vectors);
 
 /**
+ * pci_free_irq_vectors_grp - free previously allocated IRQs for a
+ * device associated with a group
+ * @dev:                PCI device to operate on
+ * @group:		group to be freed
+ *
+ * Undoes the allocations and enabling in pci_alloc_irq_vectors_dyn().
+ * Can be only called for MSIx vectors.
+ */
+int pci_free_irq_vectors_grp(struct pci_dev *dev, int group_id)
+{
+	return pci_disable_msix_grp(dev, group_id);
+}
+EXPORT_SYMBOL(pci_free_irq_vectors_grp);
+
+/**
  * pci_irq_vector - return Linux IRQ number of a device vector
  * @dev: PCI device to operate on
  * @nr: device-relative interrupt vector index (0-based).
diff --git a/include/linux/msi.h b/include/linux/msi.h
index 6be0377..fac011e 100644
--- a/include/linux/msi.h
+++ b/include/linux/msi.h
@@ -313,6 +313,8 @@ struct irq_domain *msi_create_irq_domain(struct fwnode_handle *fwnode,
 int msi_domain_alloc_irqs(struct irq_domain *domain, struct device *dev,
 			  int nvec);
 void msi_domain_free_irqs(struct irq_domain *domain, struct device *dev);
+void msi_domain_free_irqs_grp(struct irq_domain *domain, struct device *dev,
+								int group_id);
 struct msi_domain_info *msi_get_domain_info(struct irq_domain *domain);
 
 struct irq_domain *platform_msi_create_irq_domain(struct fwnode_handle *fwnode,
diff --git a/include/linux/pci.h b/include/linux/pci.h
index 2e2f4c9..25792dd 100644
--- a/include/linux/pci.h
+++ b/include/linux/pci.h
@@ -1389,6 +1389,7 @@ int pci_msi_vec_count(struct pci_dev *dev);
 void pci_disable_msi(struct pci_dev *dev);
 int pci_msix_vec_count(struct pci_dev *dev);
 void pci_disable_msix(struct pci_dev *dev);
+int pci_disable_msix_grp(struct pci_dev *dev, int group_id);
 void pci_restore_msi_state(struct pci_dev *dev);
 int pci_msi_enabled(void);
 int pci_enable_msi(struct pci_dev *dev);
@@ -1412,6 +1413,7 @@ int pci_alloc_irq_vectors_affinity_dyn(struct pci_dev *dev,
 				   int *group_id, bool one_shot);
 
 void pci_free_irq_vectors(struct pci_dev *dev);
+int pci_free_irq_vectors_grp(struct pci_dev *dev, int group_id);
 int pci_irq_vector(struct pci_dev *dev, unsigned int nr);
 int pci_irq_vector_group(struct pci_dev *dev, unsigned int nr,
 						unsigned int group_id);
@@ -1423,6 +1425,8 @@ static inline int pci_msi_vec_count(struct pci_dev *dev) { return -ENOSYS; }
 static inline void pci_disable_msi(struct pci_dev *dev) { }
 static inline int pci_msix_vec_count(struct pci_dev *dev) { return -ENOSYS; }
 static inline void pci_disable_msix(struct pci_dev *dev) { }
+static inline int pci_disable_msix_grp(struct pci_dev *dev, int group_id)
+							{ return -ENOSYS}
 static inline void pci_restore_msi_state(struct pci_dev *dev) { }
 static inline int pci_msi_enabled(void) { return 0; }
 static inline int pci_enable_msi(struct pci_dev *dev)
@@ -1459,6 +1463,11 @@ static inline void pci_free_irq_vectors(struct pci_dev *dev)
 {
 }
 
+static inline void pci_free_irq_vectors_grp(struct pci_dev *dev, int group_id)
+{
+	return 0;
+}
+
 static inline int pci_irq_vector(struct pci_dev *dev, unsigned int nr)
 {
 	if (WARN_ON_ONCE(nr > 0))
diff --git a/kernel/irq/msi.c b/kernel/irq/msi.c
index 5cfa931..07a120b 100644
--- a/kernel/irq/msi.c
+++ b/kernel/irq/msi.c
@@ -511,6 +511,35 @@ void msi_domain_free_irqs(struct irq_domain *domain, struct device *dev)
 }
 
 /**
+ * msi_domain_free_irqs_grp - Free interrupts belonging to a group from
+ * a MSI interrupt @domain associated to @dev
+ * @domain:	The domain to managing the interrupts
+ * @dev:	Pointer to device struct of the device for which the interrupt
+ *		should be freed
+ * @group_id:	The group ID to be freed
+ */
+void msi_domain_free_irqs_grp(struct irq_domain *domain, struct device *dev,
+								int group_id)
+{
+	struct msi_desc *desc;
+	int *group = NULL;
+
+	for_each_msi_entry(desc, dev) {
+		group = idr_find(dev->msix_dev_idr->entry_idr,
+						desc->msi_attrib.entry_nr);
+		/*
+		 * We might have failed to allocate an MSI early
+		 * enough that there is no IRQ associated to this
+		 * entry. If that's the case, don't do anything.
+		 */
+		if (*group == group_id) {
+			irq_domain_free_irqs(desc->irq, desc->nvec_used);
+			desc->irq = 0;
+		}
+	}
+}
+
+/**
  * msi_get_domain_info - Get the MSI interrupt domain info for @domain
  * @domain:	The interrupt domain to retrieve data from
  *
-- 
2.7.4

[Intel-wired-lan] [RFC v4 6/6] Documentation: PCI/MSI: Document dynamic MSI-X infrastructure

[Megha Dey]

Add Documentation for the newly introduced dynamic allocation
and deallocation of MSI-X vectors.

Signed-off-by: Megha Dey <megha.dey@linux.intel.com>
---
 Documentation/PCI/MSI-HOWTO.txt | 36 ++++++++++++++++++++++++++++++++++++
 1 file changed, 36 insertions(+)

diff --git a/Documentation/PCI/MSI-HOWTO.txt b/Documentation/PCI/MSI-HOWTO.txt
index 618e13d..28674bb 100644
--- a/Documentation/PCI/MSI-HOWTO.txt
+++ b/Documentation/PCI/MSI-HOWTO.txt
@@ -156,6 +156,42 @@ the driver can specify that only MSI or MSI-X is acceptable:
 	if (nvec < 0)
 		goto out_err;
 
+The pci_alloc_irq_vectors() API are one-shot methods to allocate MSI resources
+i.e. they cannot be called multiple times. In order to allocate MSI-X vectors
+post probe phase, multiple times, use the following API:
+
+  int pci_alloc_irq_vectors_dyn(struct pci_dev *dev, unsigned int min_vecs,
+                unsigned int max_vecs, unsigned int flags, int *group_id);
+
+This API allocates up to max_vecs interrupt vectors for a PCI device. It returns
+the number of vectors allocated or a negative error. If the device has a
+requirement for a minimum number of vectors the driver can pass a min_vecs
+argument set to this limit, and the PCI core will return -ENOSPC if it can't
+meet the minimum number of vectors. This API is only to be used for MSI-X vectors.
+
+A group ID pointer is passed which gets populated by this function. A unique
+group_id will associated with all the MSI-X vectors allocated each time this
+function is called:
+
+	int group_id;
+	nvec = pci_alloc_irq_vectors_dyn(pdev, minvecs, maxvecs,
+					flags | PCI_IRQ_MSIX, &group_id);
+	if (nvec < 0)
+		goto out_err;
+
+To get the Linux IRQ numbers passed to request_irq() and free_irq() and the
+vectors, use the following function:
+
+  int pci_irq_vec_grp(struct pci_dev *dev, unsigned int nr, unsigned int group_id);
+
+In order to free the MSI-X resources associated with a particular group, use
+the following function:
+
+  int pci_free_irq_vectors_grp(struct pci_dev *dev, int group_id);
+
+To delete the group allocated with the pci_alloc_irq_vectors_dyn() call:
+	nvec = pci_free_irq_vectors_grp(pdev, group_id);
+
 4.3 Legacy APIs
 
 The following old APIs to enable and disable MSI or MSI-X interrupts should
-- 
2.7.4