Resizeable PCI BAR support V5

Resizeable PCI BAR support V5

[Christian König]

Hi everyone,

This is the fith incarnation of this set of patches. It enables device
drivers to resize and most likely also relocate the PCI BAR of devices 
they manage to allow the CPU to access all of the device local memory at once.

This is very useful for GFX device drivers where the default PCI BAR is only
about 256MB in size for compatibility reasons, but the device easily have
multiple gigabyte of local memory.

Some changes since V4:
1. Rebased on 4.11.
2. added the rb from Andy Shevchenko to patches which look complete now.
3. Move releasing the BAR and reallocating it on error to the driver side.
4. Add amdgpu support for GMC V6 hardware generation as well.

Please review and/or comment,
Christian.

Resizeable PCI BAR support V5

[Christian König]

Hi everyone,

This is the fith incarnation of this set of patches. It enables device
drivers to resize and most likely also relocate the PCI BAR of devices 
they manage to allow the CPU to access all of the device local memory at once.

This is very useful for GFX device drivers where the default PCI BAR is only
about 256MB in size for compatibility reasons, but the device easily have
multiple gigabyte of local memory.

Some changes since V4:
1. Rebased on 4.11.
2. added the rb from Andy Shevchenko to patches which look complete now.
3. Move releasing the BAR and reallocating it on error to the driver side.
4. Add amdgpu support for GMC V6 hardware generation as well.

Please review and/or comment,
Christian.

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

[PATCH v5 1/6] PCI: add a define for the PCI resource type mask v2

[Christian König]

From: Christian König <christian.koenig@amd.com>

We use this mask multiple times in the bus setup.

v2: fix some style nit picks

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Andy Shevchenko <andy.shevchenko@gmail.com>
---
 drivers/pci/pci.h       |  4 ++++
 drivers/pci/setup-bus.c | 12 +++---------
 2 files changed, 7 insertions(+), 9 deletions(-)

diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h
index 8dd38e6..b3da553 100644
--- a/drivers/pci/pci.h
+++ b/drivers/pci/pci.h
@@ -3,6 +3,10 @@
 
 #define PCI_FIND_CAP_TTL	48
 
+#define PCI_RES_TYPE_MASK \
+	(IORESOURCE_IO | IORESOURCE_MEM | IORESOURCE_PREFETCH |\
+	 IORESOURCE_MEM_64)
+
 extern const unsigned char pcie_link_speed[];
 
 bool pcie_cap_has_lnkctl(const struct pci_dev *dev);
diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c
index cb389277..451a9c0 100644
--- a/drivers/pci/setup-bus.c
+++ b/drivers/pci/setup-bus.c
@@ -1523,8 +1523,6 @@ static void pci_bridge_release_resources(struct pci_bus *bus,
 {
 	struct pci_dev *dev = bus->self;
 	struct resource *r;
-	unsigned long type_mask = IORESOURCE_IO | IORESOURCE_MEM |
-				  IORESOURCE_PREFETCH | IORESOURCE_MEM_64;
 	unsigned old_flags = 0;
 	struct resource *b_res;
 	int idx = 1;
@@ -1567,7 +1565,7 @@ static void pci_bridge_release_resources(struct pci_bus *bus,
 	 */
 	release_child_resources(r);
 	if (!release_resource(r)) {
-		type = old_flags = r->flags & type_mask;
+		type = old_flags = r->flags & PCI_RES_TYPE_MASK;
 		dev_printk(KERN_DEBUG, &dev->dev, "resource %d %pR released\n",
 					PCI_BRIDGE_RESOURCES + idx, r);
 		/* keep the old size */
@@ -1758,8 +1756,6 @@ void pci_assign_unassigned_root_bus_resources(struct pci_bus *bus)
 	enum release_type rel_type = leaf_only;
 	LIST_HEAD(fail_head);
 	struct pci_dev_resource *fail_res;
-	unsigned long type_mask = IORESOURCE_IO | IORESOURCE_MEM |
-				  IORESOURCE_PREFETCH | IORESOURCE_MEM_64;
 	int pci_try_num = 1;
 	enum enable_type enable_local;
 
@@ -1818,7 +1814,7 @@ void pci_assign_unassigned_root_bus_resources(struct pci_bus *bus)
 	 */
 	list_for_each_entry(fail_res, &fail_head, list)
 		pci_bus_release_bridge_resources(fail_res->dev->bus,
-						 fail_res->flags & type_mask,
+						 fail_res->flags & PCI_RES_TYPE_MASK,
 						 rel_type);
 
 	/* restore size and flags */
@@ -1862,8 +1858,6 @@ void pci_assign_unassigned_bridge_resources(struct pci_dev *bridge)
 	LIST_HEAD(fail_head);
 	struct pci_dev_resource *fail_res;
 	int retval;
-	unsigned long type_mask = IORESOURCE_IO | IORESOURCE_MEM |
-				  IORESOURCE_PREFETCH | IORESOURCE_MEM_64;
 
 again:
 	__pci_bus_size_bridges(parent, &add_list);
@@ -1889,7 +1883,7 @@ void pci_assign_unassigned_bridge_resources(struct pci_dev *bridge)
 	 */
 	list_for_each_entry(fail_res, &fail_head, list)
 		pci_bus_release_bridge_resources(fail_res->dev->bus,
-						 fail_res->flags & type_mask,
+						 fail_res->flags & PCI_RES_TYPE_MASK,
 						 whole_subtree);
 
 	/* restore size and flags */
-- 
2.7.4

[PATCH v5 1/6] PCI: add a define for the PCI resource type mask v2

[Christian König]

From: Christian König <christian.koenig@amd.com>

We use this mask multiple times in the bus setup.

v2: fix some style nit picks

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Andy Shevchenko <andy.shevchenko@gmail.com>
---
 drivers/pci/pci.h       |  4 ++++
 drivers/pci/setup-bus.c | 12 +++---------
 2 files changed, 7 insertions(+), 9 deletions(-)

diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h
index 8dd38e6..b3da553 100644
--- a/drivers/pci/pci.h
+++ b/drivers/pci/pci.h
@@ -3,6 +3,10 @@
 
 #define PCI_FIND_CAP_TTL	48
 
+#define PCI_RES_TYPE_MASK \
+	(IORESOURCE_IO | IORESOURCE_MEM | IORESOURCE_PREFETCH |\
+	 IORESOURCE_MEM_64)
+
 extern const unsigned char pcie_link_speed[];
 
 bool pcie_cap_has_lnkctl(const struct pci_dev *dev);
diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c
index cb389277..451a9c0 100644
--- a/drivers/pci/setup-bus.c
+++ b/drivers/pci/setup-bus.c
@@ -1523,8 +1523,6 @@ static void pci_bridge_release_resources(struct pci_bus *bus,
 {
 	struct pci_dev *dev = bus->self;
 	struct resource *r;
-	unsigned long type_mask = IORESOURCE_IO | IORESOURCE_MEM |
-				  IORESOURCE_PREFETCH | IORESOURCE_MEM_64;
 	unsigned old_flags = 0;
 	struct resource *b_res;
 	int idx = 1;
@@ -1567,7 +1565,7 @@ static void pci_bridge_release_resources(struct pci_bus *bus,
 	 */
 	release_child_resources(r);
 	if (!release_resource(r)) {
-		type = old_flags = r->flags & type_mask;
+		type = old_flags = r->flags & PCI_RES_TYPE_MASK;
 		dev_printk(KERN_DEBUG, &dev->dev, "resource %d %pR released\n",
 					PCI_BRIDGE_RESOURCES + idx, r);
 		/* keep the old size */
@@ -1758,8 +1756,6 @@ void pci_assign_unassigned_root_bus_resources(struct pci_bus *bus)
 	enum release_type rel_type = leaf_only;
 	LIST_HEAD(fail_head);
 	struct pci_dev_resource *fail_res;
-	unsigned long type_mask = IORESOURCE_IO | IORESOURCE_MEM |
-				  IORESOURCE_PREFETCH | IORESOURCE_MEM_64;
 	int pci_try_num = 1;
 	enum enable_type enable_local;
 
@@ -1818,7 +1814,7 @@ void pci_assign_unassigned_root_bus_resources(struct pci_bus *bus)
 	 */
 	list_for_each_entry(fail_res, &fail_head, list)
 		pci_bus_release_bridge_resources(fail_res->dev->bus,
-						 fail_res->flags & type_mask,
+						 fail_res->flags & PCI_RES_TYPE_MASK,
 						 rel_type);
 
 	/* restore size and flags */
@@ -1862,8 +1858,6 @@ void pci_assign_unassigned_bridge_resources(struct pci_dev *bridge)
 	LIST_HEAD(fail_head);
 	struct pci_dev_resource *fail_res;
 	int retval;
-	unsigned long type_mask = IORESOURCE_IO | IORESOURCE_MEM |
-				  IORESOURCE_PREFETCH | IORESOURCE_MEM_64;
 
 again:
 	__pci_bus_size_bridges(parent, &add_list);
@@ -1889,7 +1883,7 @@ void pci_assign_unassigned_bridge_resources(struct pci_dev *bridge)
 	 */
 	list_for_each_entry(fail_res, &fail_head, list)
 		pci_bus_release_bridge_resources(fail_res->dev->bus,
-						 fail_res->flags & type_mask,
+						 fail_res->flags & PCI_RES_TYPE_MASK,
 						 whole_subtree);
 
 	/* restore size and flags */
-- 
2.7.4

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

[PATCH v5 2/6] PCI: add resizeable BAR infrastructure v5

[Christian König]

From: Christian König <christian.koenig@amd.com>

Just the defines and helper functions to read the possible sizes of a BAR and
update it's size.

See https://pcisig.com/sites/default/files/specification_documents/ECN_Resizable-BAR_24Apr2008.pdf
and PCIe r3.1, sec 7.22.

This is useful for hardware with large local storage (mostly GFX) which only
expose 256MB BARs initially to be compatible with 32bit systems.

v2: provide read helper as well
v3: improve function names, use unsigned values, add better comments.
v4: move definition, improve commit message, s/bar/BAR/
v5: split out helper to find ctrl reg pos, style fixes, comment fixes,
    add pci_rbar_size_to_bytes as well

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Andy Shevchenko <andy.shevchenko@gmail.com>
---
 drivers/pci/pci.c             | 104 ++++++++++++++++++++++++++++++++++++++++++
 drivers/pci/pci.h             |   8 ++++
 include/uapi/linux/pci_regs.h |  11 ++++-
 3 files changed, 121 insertions(+), 2 deletions(-)

diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
index 7904d02..d91ec39 100644
--- a/drivers/pci/pci.c
+++ b/drivers/pci/pci.c
@@ -2940,6 +2940,110 @@ bool pci_acs_path_enabled(struct pci_dev *start,
 }
 
 /**
+ * pci_rbar_find_pos - find position of resize ctrl reg for BAR
+ * @dev: PCI device
+ * @bar: BAR to find
+ *
+ * Helper to find the postion of the ctrl register for a BAR.
+ * Returns -ENOTSUPP of resizeable BARs are not supported at all.
+ * Returns -ENOENT if not ctrl register for the BAR could be found.
+ */
+static int pci_rbar_find_pos(struct pci_dev *pdev, int bar)
+{
+	unsigned int pos, nbars;
+	unsigned int i;
+	u32 ctrl;
+
+	pos = pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_REBAR);
+	if (!pos)
+		return -ENOTSUPP;
+
+	pci_read_config_dword(pdev, pos + PCI_REBAR_CTRL, &ctrl);
+	nbars = (ctrl & PCI_REBAR_CTRL_NBAR_MASK) >> PCI_REBAR_CTRL_NBAR_SHIFT;
+
+	for (i = 0; i < nbars; ++i, pos += 8) {
+		int bar_idx;
+
+		pci_read_config_dword(pdev, pos + PCI_REBAR_CTRL, &ctrl);
+		bar_idx = (ctrl & PCI_REBAR_CTRL_BAR_IDX_MASK) >>
+				PCI_REBAR_CTRL_BAR_IDX_SHIFT;
+		if (bar_idx == bar)
+			return pos;
+	}
+
+	return -ENOENT;
+}
+
+/**
+ * pci_rbar_get_possible_sizes - get possible sizes for BAR
+ * @dev: PCI device
+ * @bar: BAR to query
+ *
+ * Get the possible sizes of a resizeable BAR as bitmask defined in the spec
+ * (bit 0=1MB, bit 19=512GB). Returns 0 if BAR isn't resizeable.
+ */
+u32 pci_rbar_get_possible_sizes(struct pci_dev *pdev, int bar)
+{
+	u32 cap;
+	int pos;
+
+	pos = pci_rbar_find_pos(pdev, bar);
+	if (pos < 0)
+		return 0;
+
+	pci_read_config_dword(pdev, pos + PCI_REBAR_CAP, &cap);
+	return (cap & PCI_REBAR_CTRL_SIZES_MASK) >>
+		PCI_REBAR_CTRL_SIZES_SHIFT;
+}
+
+/**
+ * pci_rbar_get_current_size - get the current size of a BAR
+ * @dev: PCI device
+ * @bar: BAR to set size to
+ *
+ * Read the size of a BAR from the resizeable BAR config.
+ * Returns size if found or negative error code.
+ */
+int pci_rbar_get_current_size(struct pci_dev *pdev, int bar)
+{
+	u32 ctrl;
+	int pos;
+
+	pos = pci_rbar_find_pos(pdev, bar);
+	if (pos < 0)
+		return pos;
+
+	pci_read_config_dword(pdev, pos + PCI_REBAR_CTRL, &ctrl);
+	return (ctrl & PCI_REBAR_CTRL_BAR_SIZE_MASK) >>
+		PCI_REBAR_CTRL_BAR_SIZE_SHIFT;
+}
+
+/**
+ * pci_rbar_set_size - set a new size for a BAR
+ * @dev: PCI device
+ * @bar: BAR to set size to
+ * @size: new size as defined in the spec (0=1MB, 19=512GB)
+ *
+ * Set the new size of a BAR as defined in the spec.
+ * Returns zero if resizing was successful, error code otherwise.
+ */
+int pci_rbar_set_size(struct pci_dev *pdev, int bar, int size)
+{
+	u32 ctrl;
+	int pos;
+
+	pos = pci_rbar_find_pos(pdev, bar);
+	if (pos < 0)
+		return pos;
+
+	pci_read_config_dword(pdev, pos + PCI_REBAR_CTRL, &ctrl);
+	ctrl &= ~PCI_REBAR_CTRL_BAR_SIZE_MASK;
+	ctrl |= size << PCI_REBAR_CTRL_BAR_SIZE_SHIFT;
+	pci_write_config_dword(pdev, pos + PCI_REBAR_CTRL, ctrl);
+	return 0;
+}
+
+/**
  * pci_swizzle_interrupt_pin - swizzle INTx for device behind bridge
  * @dev: the PCI device
  * @pin: the INTx pin (1=INTA, 2=INTB, 3=INTC, 4=INTD)
diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h
index b3da553..23b75e8 100644
--- a/drivers/pci/pci.h
+++ b/drivers/pci/pci.h
@@ -357,4 +357,12 @@ int acpi_get_rc_resources(struct device *dev, const char *hid, u16 segment,
 			  struct resource *res);
 #endif
 
+u32 pci_rbar_get_possible_sizes(struct pci_dev *pdev, int bar);
+int pci_rbar_get_current_size(struct pci_dev *pdev, int bar);
+int pci_rbar_set_size(struct pci_dev *pdev, int bar, int size);
+static inline u64 pci_rbar_size_to_bytes(int size)
+{
+	return 1ULL << (size + 20);
+}
+
 #endif /* DRIVERS_PCI_H */
diff --git a/include/uapi/linux/pci_regs.h b/include/uapi/linux/pci_regs.h
index 634c9c4..b6bd6e5 100644
--- a/include/uapi/linux/pci_regs.h
+++ b/include/uapi/linux/pci_regs.h
@@ -941,9 +941,16 @@
 #define PCI_SATA_SIZEOF_LONG	16
 
 /* Resizable BARs */
+#define PCI_REBAR_CAP		4	/* capability register */
+#define  PCI_REBAR_CTRL_SIZES_MASK	(0xFFFFF << 4)	/* mask for sizes */
+#define  PCI_REBAR_CTRL_SIZES_SHIFT	4	/* shift for sizes */
 #define PCI_REBAR_CTRL		8	/* control register */
-#define  PCI_REBAR_CTRL_NBAR_MASK	(7 << 5)	/* mask for # bars */
-#define  PCI_REBAR_CTRL_NBAR_SHIFT	5	/* shift for # bars */
+#define  PCI_REBAR_CTRL_BAR_IDX_MASK	(7 << 0)	/* mask for BAR index */
+#define  PCI_REBAR_CTRL_BAR_IDX_SHIFT	0	/* shift for BAR index */
+#define  PCI_REBAR_CTRL_NBAR_MASK	(7 << 5)	/* mask for # BARs */
+#define  PCI_REBAR_CTRL_NBAR_SHIFT	5	/* shift for # BARs */
+#define  PCI_REBAR_CTRL_BAR_SIZE_MASK	(0x1F << 8)	/* mask for BAR size */
+#define  PCI_REBAR_CTRL_BAR_SIZE_SHIFT	8	/* shift for BAR size */
 
 /* Dynamic Power Allocation */
 #define PCI_DPA_CAP		4	/* capability register */
-- 
2.7.4

[PATCH v5 2/6] PCI: add resizeable BAR infrastructure v5

[Christian König]

From: Christian König <christian.koenig@amd.com>

Just the defines and helper functions to read the possible sizes of a BAR and
update it's size.

See https://pcisig.com/sites/default/files/specification_documents/ECN_Resizable-BAR_24Apr2008.pdf
and PCIe r3.1, sec 7.22.

This is useful for hardware with large local storage (mostly GFX) which only
expose 256MB BARs initially to be compatible with 32bit systems.

v2: provide read helper as well
v3: improve function names, use unsigned values, add better comments.
v4: move definition, improve commit message, s/bar/BAR/
v5: split out helper to find ctrl reg pos, style fixes, comment fixes,
    add pci_rbar_size_to_bytes as well

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Andy Shevchenko <andy.shevchenko@gmail.com>
---
 drivers/pci/pci.c             | 104 ++++++++++++++++++++++++++++++++++++++++++
 drivers/pci/pci.h             |   8 ++++
 include/uapi/linux/pci_regs.h |  11 ++++-
 3 files changed, 121 insertions(+), 2 deletions(-)

diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
index 7904d02..d91ec39 100644
--- a/drivers/pci/pci.c
+++ b/drivers/pci/pci.c
@@ -2940,6 +2940,110 @@ bool pci_acs_path_enabled(struct pci_dev *start,
 }
 
 /**
+ * pci_rbar_find_pos - find position of resize ctrl reg for BAR
+ * @dev: PCI device
+ * @bar: BAR to find
+ *
+ * Helper to find the postion of the ctrl register for a BAR.
+ * Returns -ENOTSUPP of resizeable BARs are not supported at all.
+ * Returns -ENOENT if not ctrl register for the BAR could be found.
+ */
+static int pci_rbar_find_pos(struct pci_dev *pdev, int bar)
+{
+	unsigned int pos, nbars;
+	unsigned int i;
+	u32 ctrl;
+
+	pos = pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_REBAR);
+	if (!pos)
+		return -ENOTSUPP;
+
+	pci_read_config_dword(pdev, pos + PCI_REBAR_CTRL, &ctrl);
+	nbars = (ctrl & PCI_REBAR_CTRL_NBAR_MASK) >> PCI_REBAR_CTRL_NBAR_SHIFT;
+
+	for (i = 0; i < nbars; ++i, pos += 8) {
+		int bar_idx;
+
+		pci_read_config_dword(pdev, pos + PCI_REBAR_CTRL, &ctrl);
+		bar_idx = (ctrl & PCI_REBAR_CTRL_BAR_IDX_MASK) >>
+				PCI_REBAR_CTRL_BAR_IDX_SHIFT;
+		if (bar_idx == bar)
+			return pos;
+	}
+
+	return -ENOENT;
+}
+
+/**
+ * pci_rbar_get_possible_sizes - get possible sizes for BAR
+ * @dev: PCI device
+ * @bar: BAR to query
+ *
+ * Get the possible sizes of a resizeable BAR as bitmask defined in the spec
+ * (bit 0=1MB, bit 19=512GB). Returns 0 if BAR isn't resizeable.
+ */
+u32 pci_rbar_get_possible_sizes(struct pci_dev *pdev, int bar)
+{
+	u32 cap;
+	int pos;
+
+	pos = pci_rbar_find_pos(pdev, bar);
+	if (pos < 0)
+		return 0;
+
+	pci_read_config_dword(pdev, pos + PCI_REBAR_CAP, &cap);
+	return (cap & PCI_REBAR_CTRL_SIZES_MASK) >>
+		PCI_REBAR_CTRL_SIZES_SHIFT;
+}
+
+/**
+ * pci_rbar_get_current_size - get the current size of a BAR
+ * @dev: PCI device
+ * @bar: BAR to set size to
+ *
+ * Read the size of a BAR from the resizeable BAR config.
+ * Returns size if found or negative error code.
+ */
+int pci_rbar_get_current_size(struct pci_dev *pdev, int bar)
+{
+	u32 ctrl;
+	int pos;
+
+	pos = pci_rbar_find_pos(pdev, bar);
+	if (pos < 0)
+		return pos;
+
+	pci_read_config_dword(pdev, pos + PCI_REBAR_CTRL, &ctrl);
+	return (ctrl & PCI_REBAR_CTRL_BAR_SIZE_MASK) >>
+		PCI_REBAR_CTRL_BAR_SIZE_SHIFT;
+}
+
+/**
+ * pci_rbar_set_size - set a new size for a BAR
+ * @dev: PCI device
+ * @bar: BAR to set size to
+ * @size: new size as defined in the spec (0=1MB, 19=512GB)
+ *
+ * Set the new size of a BAR as defined in the spec.
+ * Returns zero if resizing was successful, error code otherwise.
+ */
+int pci_rbar_set_size(struct pci_dev *pdev, int bar, int size)
+{
+	u32 ctrl;
+	int pos;
+
+	pos = pci_rbar_find_pos(pdev, bar);
+	if (pos < 0)
+		return pos;
+
+	pci_read_config_dword(pdev, pos + PCI_REBAR_CTRL, &ctrl);
+	ctrl &= ~PCI_REBAR_CTRL_BAR_SIZE_MASK;
+	ctrl |= size << PCI_REBAR_CTRL_BAR_SIZE_SHIFT;
+	pci_write_config_dword(pdev, pos + PCI_REBAR_CTRL, ctrl);
+	return 0;
+}
+
+/**
  * pci_swizzle_interrupt_pin - swizzle INTx for device behind bridge
  * @dev: the PCI device
  * @pin: the INTx pin (1=INTA, 2=INTB, 3=INTC, 4=INTD)
diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h
index b3da553..23b75e8 100644
--- a/drivers/pci/pci.h
+++ b/drivers/pci/pci.h
@@ -357,4 +357,12 @@ int acpi_get_rc_resources(struct device *dev, const char *hid, u16 segment,
 			  struct resource *res);
 #endif
 
+u32 pci_rbar_get_possible_sizes(struct pci_dev *pdev, int bar);
+int pci_rbar_get_current_size(struct pci_dev *pdev, int bar);
+int pci_rbar_set_size(struct pci_dev *pdev, int bar, int size);
+static inline u64 pci_rbar_size_to_bytes(int size)
+{
+	return 1ULL << (size + 20);
+}
+
 #endif /* DRIVERS_PCI_H */
diff --git a/include/uapi/linux/pci_regs.h b/include/uapi/linux/pci_regs.h
index 634c9c4..b6bd6e5 100644
--- a/include/uapi/linux/pci_regs.h
+++ b/include/uapi/linux/pci_regs.h
@@ -941,9 +941,16 @@
 #define PCI_SATA_SIZEOF_LONG	16
 
 /* Resizable BARs */
+#define PCI_REBAR_CAP		4	/* capability register */
+#define  PCI_REBAR_CTRL_SIZES_MASK	(0xFFFFF << 4)	/* mask for sizes */
+#define  PCI_REBAR_CTRL_SIZES_SHIFT	4	/* shift for sizes */
 #define PCI_REBAR_CTRL		8	/* control register */
-#define  PCI_REBAR_CTRL_NBAR_MASK	(7 << 5)	/* mask for # bars */
-#define  PCI_REBAR_CTRL_NBAR_SHIFT	5	/* shift for # bars */
+#define  PCI_REBAR_CTRL_BAR_IDX_MASK	(7 << 0)	/* mask for BAR index */
+#define  PCI_REBAR_CTRL_BAR_IDX_SHIFT	0	/* shift for BAR index */
+#define  PCI_REBAR_CTRL_NBAR_MASK	(7 << 5)	/* mask for # BARs */
+#define  PCI_REBAR_CTRL_NBAR_SHIFT	5	/* shift for # BARs */
+#define  PCI_REBAR_CTRL_BAR_SIZE_MASK	(0x1F << 8)	/* mask for BAR size */
+#define  PCI_REBAR_CTRL_BAR_SIZE_SHIFT	8	/* shift for BAR size */
 
 /* Dynamic Power Allocation */
 #define PCI_DPA_CAP		4	/* capability register */
-- 
2.7.4

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

[PATCH v5 3/6] PCI: add functionality for resizing resources v6

[Christian König]

From: Christian König <christian.koenig@amd.com>

This allows device drivers to request resizing their BARs.

The function only tries to reprogram the windows of the bridge directly above
the requesting device and only the BAR of the same type (usually mem, 64bit,
prefetchable). This is done to make sure not to disturb other drivers by
changing the BARs of their devices.

If reprogramming the bridge BAR fails the old status is restored and -ENOSPC
returned to the calling device driver.

v2: rebase on changes in rbar support
v3: style cleanups, fail if memory decoding is enabled or resources
    still allocated, resize all unused bridge BARs,
    drop calling pci_reenable_device
v4: print resources before releasing them, style cleanups,
    use pci_rbar_size_to_bytes, use PCI_RES_TYPE_MASK
v5: use next pointer to simplify loop
v6: move reassigning resources on error to driver side

Signed-off-by: Christian König <christian.koenig@amd.com>
---
 drivers/pci/setup-bus.c | 98 +++++++++++++++++++++++++++++++++++++++++++++++++
 drivers/pci/setup-res.c | 58 +++++++++++++++++++++++++++++
 include/linux/pci.h     |  3 ++
 3 files changed, 159 insertions(+)

diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c
index 451a9c0..2ea872d 100644
--- a/drivers/pci/setup-bus.c
+++ b/drivers/pci/setup-bus.c
@@ -1908,6 +1908,104 @@ void pci_assign_unassigned_bridge_resources(struct pci_dev *bridge)
 }
 EXPORT_SYMBOL_GPL(pci_assign_unassigned_bridge_resources);
 
+int pci_reassign_bridge_resources(struct pci_dev *bridge, unsigned long type)
+{
+	struct pci_dev_resource *dev_res;
+	struct pci_dev *next;
+	LIST_HEAD(saved);
+	LIST_HEAD(added);
+	LIST_HEAD(failed);
+	unsigned int i;
+	int ret;
+
+	/* Walk to the root hub, releasing bridge BARs when possible */
+	next = bridge;
+	do {
+		bridge = next;
+		for (i = PCI_BRIDGE_RESOURCES; i < PCI_BRIDGE_RESOURCE_END;
+		     i++) {
+			struct resource *res = &bridge->resource[i];
+
+			if ((res->flags ^ type) & PCI_RES_TYPE_MASK)
+				continue;
+
+			/* Ignore BARs which are still in use */
+			if (res->child)
+				continue;
+
+			ret = add_to_list(&saved, bridge, res, 0, 0);
+			if (ret)
+				goto cleanup;
+
+			dev_info(&bridge->dev, "BAR %d: releasing %pR\n",
+				 i, res);
+
+			if (res->parent)
+				release_resource(res);
+			res->start = 0;
+			res->end = 0;
+			break;
+		}
+		if (i == PCI_BRIDGE_RESOURCE_END)
+			break;
+
+		next = bridge->bus ? bridge->bus->self : NULL;
+	} while (next);
+
+	if (list_empty(&saved))
+		return -ENOENT;
+
+	__pci_bus_size_bridges(bridge->subordinate, &added);
+	__pci_bridge_assign_resources(bridge, &added, &failed);
+	BUG_ON(!list_empty(&added));
+
+	if (!list_empty(&failed)) {
+		ret = -ENOSPC;
+		goto cleanup;
+	}
+
+	list_for_each_entry(dev_res, &saved, list) {
+		/* Skip the bridge we just assigned resources for. */
+		if (bridge == dev_res->dev)
+			continue;
+
+		bridge = dev_res->dev;
+		pci_setup_bridge(bridge->subordinate);
+	}
+
+	free_list(&saved);
+	return 0;
+
+cleanup:
+	/* restore size and flags */
+	list_for_each_entry(dev_res, &failed, list) {
+		struct resource *res = dev_res->res;
+
+		res->start = dev_res->start;
+		res->end = dev_res->end;
+		res->flags = dev_res->flags;
+	}
+	free_list(&failed);
+
+	/* Revert to the old configuration */
+	list_for_each_entry(dev_res, &saved, list) {
+		struct resource *res = dev_res->res;
+
+		bridge = dev_res->dev;
+		i = res - bridge->resource;
+
+		res->start = dev_res->start;
+		res->end = dev_res->end;
+		res->flags = dev_res->flags;
+
+		pci_claim_resource(bridge, i);
+		pci_setup_bridge(bridge->subordinate);
+	}
+	free_list(&saved);
+
+	return ret;
+}
+
 void pci_assign_unassigned_bus_resources(struct pci_bus *bus)
 {
 	struct pci_dev *dev;
diff --git a/drivers/pci/setup-res.c b/drivers/pci/setup-res.c
index 4bc589e..077c515 100644
--- a/drivers/pci/setup-res.c
+++ b/drivers/pci/setup-res.c
@@ -383,6 +383,64 @@ int pci_reassign_resource(struct pci_dev *dev, int resno, resource_size_t addsiz
 	return 0;
 }
 
+void pci_release_resource(struct pci_dev *dev, int resno)
+{
+	struct resource *res = dev->resource + resno;
+
+	dev_info(&dev->dev, "BAR %d: releasing %pR\n", resno, res);
+	release_resource(res);
+	res->end = resource_size(res) - 1;
+	res->start = 0;
+	res->flags |= IORESOURCE_UNSET;
+}
+EXPORT_SYMBOL(pci_release_resource);
+
+int pci_resize_resource(struct pci_dev *dev, int resno, int size)
+{
+	struct resource *res = dev->resource + resno;
+	int old, ret;
+	u32 sizes;
+	u16 cmd;
+
+	/* Make sure the resource isn't assigned before resizing it. */
+	if (!(res->flags & IORESOURCE_UNSET))
+		return -EBUSY;
+
+	pci_read_config_word(dev, PCI_COMMAND, &cmd);
+	if (cmd & PCI_COMMAND_MEMORY)
+		return -EBUSY;
+
+	sizes = pci_rbar_get_possible_sizes(dev, resno);
+	if (!sizes)
+		return -ENOTSUPP;
+
+	if (!(sizes & BIT(size)))
+		return -EINVAL;
+
+	old = pci_rbar_get_current_size(dev, resno);
+	if (old < 0)
+		return old;
+
+	ret = pci_rbar_set_size(dev, resno, size);
+	if (ret)
+		return ret;
+
+	res->end = res->start + pci_rbar_size_to_bytes(size) - 1;
+
+	/* Check if the new config works by trying to assign everything. */
+	ret = pci_reassign_bridge_resources(dev->bus->self, res->flags);
+	if (ret)
+		goto error_resize;
+
+	return 0;
+
+error_resize:
+	pci_rbar_set_size(dev, resno, old);
+	res->end = res->start + pci_rbar_size_to_bytes(old) - 1;
+	return ret;
+}
+EXPORT_SYMBOL(pci_resize_resource);
+
 int pci_enable_resources(struct pci_dev *dev, int mask)
 {
 	u16 cmd, old_cmd;
diff --git a/include/linux/pci.h b/include/linux/pci.h
index eb3da1a..2d631ad 100644
--- a/include/linux/pci.h
+++ b/include/linux/pci.h
@@ -1068,6 +1068,8 @@ void pci_reset_bridge_secondary_bus(struct pci_dev *dev);
 void pci_update_resource(struct pci_dev *dev, int resno);
 int __must_check pci_assign_resource(struct pci_dev *dev, int i);
 int __must_check pci_reassign_resource(struct pci_dev *dev, int i, resource_size_t add_size, resource_size_t align);
+void pci_release_resource(struct pci_dev *dev, int resno);
+int __must_check pci_resize_resource(struct pci_dev *dev, int i, int size);
 int pci_select_bars(struct pci_dev *dev, unsigned long flags);
 bool pci_device_is_present(struct pci_dev *pdev);
 void pci_ignore_hotplug(struct pci_dev *dev);
@@ -1148,6 +1150,7 @@ void pci_assign_unassigned_resources(void);
 void pci_assign_unassigned_bridge_resources(struct pci_dev *bridge);
 void pci_assign_unassigned_bus_resources(struct pci_bus *bus);
 void pci_assign_unassigned_root_bus_resources(struct pci_bus *bus);
+int pci_reassign_bridge_resources(struct pci_dev *bridge, unsigned long type);
 void pdev_enable_device(struct pci_dev *);
 int pci_enable_resources(struct pci_dev *, int mask);
 void pci_fixup_irqs(u8 (*)(struct pci_dev *, u8 *),
-- 
2.7.4

[PATCH v5 3/6] PCI: add functionality for resizing resources v6

[Christian König]

From: Christian König <christian.koenig@amd.com>

This allows device drivers to request resizing their BARs.

The function only tries to reprogram the windows of the bridge directly above
the requesting device and only the BAR of the same type (usually mem, 64bit,
prefetchable). This is done to make sure not to disturb other drivers by
changing the BARs of their devices.

If reprogramming the bridge BAR fails the old status is restored and -ENOSPC
returned to the calling device driver.

v2: rebase on changes in rbar support
v3: style cleanups, fail if memory decoding is enabled or resources
    still allocated, resize all unused bridge BARs,
    drop calling pci_reenable_device
v4: print resources before releasing them, style cleanups,
    use pci_rbar_size_to_bytes, use PCI_RES_TYPE_MASK
v5: use next pointer to simplify loop
v6: move reassigning resources on error to driver side

Signed-off-by: Christian König <christian.koenig@amd.com>
---
 drivers/pci/setup-bus.c | 98 +++++++++++++++++++++++++++++++++++++++++++++++++
 drivers/pci/setup-res.c | 58 +++++++++++++++++++++++++++++
 include/linux/pci.h     |  3 ++
 3 files changed, 159 insertions(+)

diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c
index 451a9c0..2ea872d 100644
--- a/drivers/pci/setup-bus.c
+++ b/drivers/pci/setup-bus.c
@@ -1908,6 +1908,104 @@ void pci_assign_unassigned_bridge_resources(struct pci_dev *bridge)
 }
 EXPORT_SYMBOL_GPL(pci_assign_unassigned_bridge_resources);
 
+int pci_reassign_bridge_resources(struct pci_dev *bridge, unsigned long type)
+{
+	struct pci_dev_resource *dev_res;
+	struct pci_dev *next;
+	LIST_HEAD(saved);
+	LIST_HEAD(added);
+	LIST_HEAD(failed);
+	unsigned int i;
+	int ret;
+
+	/* Walk to the root hub, releasing bridge BARs when possible */
+	next = bridge;
+	do {
+		bridge = next;
+		for (i = PCI_BRIDGE_RESOURCES; i < PCI_BRIDGE_RESOURCE_END;
+		     i++) {
+			struct resource *res = &bridge->resource[i];
+
+			if ((res->flags ^ type) & PCI_RES_TYPE_MASK)
+				continue;
+
+			/* Ignore BARs which are still in use */
+			if (res->child)
+				continue;
+
+			ret = add_to_list(&saved, bridge, res, 0, 0);
+			if (ret)
+				goto cleanup;
+
+			dev_info(&bridge->dev, "BAR %d: releasing %pR\n",
+				 i, res);
+
+			if (res->parent)
+				release_resource(res);
+			res->start = 0;
+			res->end = 0;
+			break;
+		}
+		if (i == PCI_BRIDGE_RESOURCE_END)
+			break;
+
+		next = bridge->bus ? bridge->bus->self : NULL;
+	} while (next);
+
+	if (list_empty(&saved))
+		return -ENOENT;
+
+	__pci_bus_size_bridges(bridge->subordinate, &added);
+	__pci_bridge_assign_resources(bridge, &added, &failed);
+	BUG_ON(!list_empty(&added));
+
+	if (!list_empty(&failed)) {
+		ret = -ENOSPC;
+		goto cleanup;
+	}
+
+	list_for_each_entry(dev_res, &saved, list) {
+		/* Skip the bridge we just assigned resources for. */
+		if (bridge == dev_res->dev)
+			continue;
+
+		bridge = dev_res->dev;
+		pci_setup_bridge(bridge->subordinate);
+	}
+
+	free_list(&saved);
+	return 0;
+
+cleanup:
+	/* restore size and flags */
+	list_for_each_entry(dev_res, &failed, list) {
+		struct resource *res = dev_res->res;
+
+		res->start = dev_res->start;
+		res->end = dev_res->end;
+		res->flags = dev_res->flags;
+	}
+	free_list(&failed);
+
+	/* Revert to the old configuration */
+	list_for_each_entry(dev_res, &saved, list) {
+		struct resource *res = dev_res->res;
+
+		bridge = dev_res->dev;
+		i = res - bridge->resource;
+
+		res->start = dev_res->start;
+		res->end = dev_res->end;
+		res->flags = dev_res->flags;
+
+		pci_claim_resource(bridge, i);
+		pci_setup_bridge(bridge->subordinate);
+	}
+	free_list(&saved);
+
+	return ret;
+}
+
 void pci_assign_unassigned_bus_resources(struct pci_bus *bus)
 {
 	struct pci_dev *dev;
diff --git a/drivers/pci/setup-res.c b/drivers/pci/setup-res.c
index 4bc589e..077c515 100644
--- a/drivers/pci/setup-res.c
+++ b/drivers/pci/setup-res.c
@@ -383,6 +383,64 @@ int pci_reassign_resource(struct pci_dev *dev, int resno, resource_size_t addsiz
 	return 0;
 }
 
+void pci_release_resource(struct pci_dev *dev, int resno)
+{
+	struct resource *res = dev->resource + resno;
+
+	dev_info(&dev->dev, "BAR %d: releasing %pR\n", resno, res);
+	release_resource(res);
+	res->end = resource_size(res) - 1;
+	res->start = 0;
+	res->flags |= IORESOURCE_UNSET;
+}
+EXPORT_SYMBOL(pci_release_resource);
+
+int pci_resize_resource(struct pci_dev *dev, int resno, int size)
+{
+	struct resource *res = dev->resource + resno;
+	int old, ret;
+	u32 sizes;
+	u16 cmd;
+
+	/* Make sure the resource isn't assigned before resizing it. */
+	if (!(res->flags & IORESOURCE_UNSET))
+		return -EBUSY;
+
+	pci_read_config_word(dev, PCI_COMMAND, &cmd);
+	if (cmd & PCI_COMMAND_MEMORY)
+		return -EBUSY;
+
+	sizes = pci_rbar_get_possible_sizes(dev, resno);
+	if (!sizes)
+		return -ENOTSUPP;
+
+	if (!(sizes & BIT(size)))
+		return -EINVAL;
+
+	old = pci_rbar_get_current_size(dev, resno);
+	if (old < 0)
+		return old;
+
+	ret = pci_rbar_set_size(dev, resno, size);
+	if (ret)
+		return ret;
+
+	res->end = res->start + pci_rbar_size_to_bytes(size) - 1;
+
+	/* Check if the new config works by trying to assign everything. */
+	ret = pci_reassign_bridge_resources(dev->bus->self, res->flags);
+	if (ret)
+		goto error_resize;
+
+	return 0;
+
+error_resize:
+	pci_rbar_set_size(dev, resno, old);
+	res->end = res->start + pci_rbar_size_to_bytes(old) - 1;
+	return ret;
+}
+EXPORT_SYMBOL(pci_resize_resource);
+
 int pci_enable_resources(struct pci_dev *dev, int mask)
 {
 	u16 cmd, old_cmd;
diff --git a/include/linux/pci.h b/include/linux/pci.h
index eb3da1a..2d631ad 100644
--- a/include/linux/pci.h
+++ b/include/linux/pci.h
@@ -1068,6 +1068,8 @@ void pci_reset_bridge_secondary_bus(struct pci_dev *dev);
 void pci_update_resource(struct pci_dev *dev, int resno);
 int __must_check pci_assign_resource(struct pci_dev *dev, int i);
 int __must_check pci_reassign_resource(struct pci_dev *dev, int i, resource_size_t add_size, resource_size_t align);
+void pci_release_resource(struct pci_dev *dev, int resno);
+int __must_check pci_resize_resource(struct pci_dev *dev, int i, int size);
 int pci_select_bars(struct pci_dev *dev, unsigned long flags);
 bool pci_device_is_present(struct pci_dev *pdev);
 void pci_ignore_hotplug(struct pci_dev *dev);
@@ -1148,6 +1150,7 @@ void pci_assign_unassigned_resources(void);
 void pci_assign_unassigned_bridge_resources(struct pci_dev *bridge);
 void pci_assign_unassigned_bus_resources(struct pci_bus *bus);
 void pci_assign_unassigned_root_bus_resources(struct pci_bus *bus);
+int pci_reassign_bridge_resources(struct pci_dev *bridge, unsigned long type);
 void pdev_enable_device(struct pci_dev *);
 int pci_enable_resources(struct pci_dev *, int mask);
 void pci_fixup_irqs(u8 (*)(struct pci_dev *, u8 *),
-- 
2.7.4

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH v5 3/6] PCI: add functionality for resizing resources v6

[Bjorn Helgaas]

On Fri, Jun 09, 2017 at 10:59:44AM +0200, Christian König wrote:
> From: Christian König <christian.koenig@amd.com>
> 
> This allows device drivers to request resizing their BARs.
> 
> The function only tries to reprogram the windows of the bridge directly above
> the requesting device and only the BAR of the same type (usually mem, 64bit,
> prefetchable). This is done to make sure not to disturb other drivers by
> changing the BARs of their devices.
> 
> If reprogramming the bridge BAR fails the old status is restored and -ENOSPC
> returned to the calling device driver.

Can you include an outline of how drivers should use
pci_resize_resource() here, including disabling decoding, releasing
resource(s), resizing, re-enabling decoding, re-reading
pdev->resource[]?

> v2: rebase on changes in rbar support
> v3: style cleanups, fail if memory decoding is enabled or resources
>     still allocated, resize all unused bridge BARs,
>     drop calling pci_reenable_device
> v4: print resources before releasing them, style cleanups,
>     use pci_rbar_size_to_bytes, use PCI_RES_TYPE_MASK
> v5: use next pointer to simplify loop
> v6: move reassigning resources on error to driver side
> 
> Signed-off-by: Christian König <christian.koenig@amd.com>
> ---
>  drivers/pci/setup-bus.c | 98 +++++++++++++++++++++++++++++++++++++++++++++++++
>  drivers/pci/setup-res.c | 58 +++++++++++++++++++++++++++++
>  include/linux/pci.h     |  3 ++
>  3 files changed, 159 insertions(+)
> 
> diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c
> index 451a9c0..2ea872d 100644
> --- a/drivers/pci/setup-bus.c
> +++ b/drivers/pci/setup-bus.c
> @@ -1908,6 +1908,104 @@ void pci_assign_unassigned_bridge_resources(struct pci_dev *bridge)
>  }
>  EXPORT_SYMBOL_GPL(pci_assign_unassigned_bridge_resources);
>  
> +int pci_reassign_bridge_resources(struct pci_dev *bridge, unsigned long type)
> +{
> +	struct pci_dev_resource *dev_res;
> +	struct pci_dev *next;
> +	LIST_HEAD(saved);
> +	LIST_HEAD(added);
> +	LIST_HEAD(failed);
> +	unsigned int i;
> +	int ret;
> +
> +	/* Walk to the root hub, releasing bridge BARs when possible */
> +	next = bridge;
> +	do {
> +		bridge = next;
> +		for (i = PCI_BRIDGE_RESOURCES; i < PCI_BRIDGE_RESOURCE_END;
> +		     i++) {
> +			struct resource *res = &bridge->resource[i];
> +
> +			if ((res->flags ^ type) & PCI_RES_TYPE_MASK)
> +				continue;
> +
> +			/* Ignore BARs which are still in use */
> +			if (res->child)
> +				continue;
> +
> +			ret = add_to_list(&saved, bridge, res, 0, 0);
> +			if (ret)
> +				goto cleanup;
> +
> +			dev_info(&bridge->dev, "BAR %d: releasing %pR\n",
> +				 i, res);
> +
> +			if (res->parent)
> +				release_resource(res);
> +			res->start = 0;
> +			res->end = 0;
> +			break;
> +		}
> +		if (i == PCI_BRIDGE_RESOURCE_END)
> +			break;
> +
> +		next = bridge->bus ? bridge->bus->self : NULL;
> +	} while (next);
> +
> +	if (list_empty(&saved))
> +		return -ENOENT;
> +
> +	__pci_bus_size_bridges(bridge->subordinate, &added);
> +	__pci_bridge_assign_resources(bridge, &added, &failed);
> +	BUG_ON(!list_empty(&added));
> +
> +	if (!list_empty(&failed)) {
> +		ret = -ENOSPC;
> +		goto cleanup;
> +	}
> +
> +	list_for_each_entry(dev_res, &saved, list) {
> +		/* Skip the bridge we just assigned resources for. */
> +		if (bridge == dev_res->dev)
> +			continue;
> +
> +		bridge = dev_res->dev;
> +		pci_setup_bridge(bridge->subordinate);
> +	}
> +
> +	free_list(&saved);
> +	return 0;
> +
> +cleanup:
> +	/* restore size and flags */
> +	list_for_each_entry(dev_res, &failed, list) {
> +		struct resource *res = dev_res->res;
> +
> +		res->start = dev_res->start;
> +		res->end = dev_res->end;
> +		res->flags = dev_res->flags;
> +	}
> +	free_list(&failed);
> +
> +	/* Revert to the old configuration */
> +	list_for_each_entry(dev_res, &saved, list) {
> +		struct resource *res = dev_res->res;
> +
> +		bridge = dev_res->dev;
> +		i = res - bridge->resource;
> +
> +		res->start = dev_res->start;
> +		res->end = dev_res->end;
> +		res->flags = dev_res->flags;
> +
> +		pci_claim_resource(bridge, i);
> +		pci_setup_bridge(bridge->subordinate);
> +	}
> +	free_list(&saved);
> +
> +	return ret;
> +}
> +
>  void pci_assign_unassigned_bus_resources(struct pci_bus *bus)
>  {
>  	struct pci_dev *dev;
> diff --git a/drivers/pci/setup-res.c b/drivers/pci/setup-res.c
> index 4bc589e..077c515 100644
> --- a/drivers/pci/setup-res.c
> +++ b/drivers/pci/setup-res.c
> @@ -383,6 +383,64 @@ int pci_reassign_resource(struct pci_dev *dev, int resno, resource_size_t addsiz
>  	return 0;
>  }
>  
> +void pci_release_resource(struct pci_dev *dev, int resno)
> +{
> +	struct resource *res = dev->resource + resno;
> +
> +	dev_info(&dev->dev, "BAR %d: releasing %pR\n", resno, res);
> +	release_resource(res);
> +	res->end = resource_size(res) - 1;
> +	res->start = 0;
> +	res->flags |= IORESOURCE_UNSET;
> +}
> +EXPORT_SYMBOL(pci_release_resource);
> +
> +int pci_resize_resource(struct pci_dev *dev, int resno, int size)
> +{
> +	struct resource *res = dev->resource + resno;
> +	int old, ret;
> +	u32 sizes;
> +	u16 cmd;
> +
> +	/* Make sure the resource isn't assigned before resizing it. */
> +	if (!(res->flags & IORESOURCE_UNSET))
> +		return -EBUSY;
> +
> +	pci_read_config_word(dev, PCI_COMMAND, &cmd);
> +	if (cmd & PCI_COMMAND_MEMORY)
> +		return -EBUSY;
> +
> +	sizes = pci_rbar_get_possible_sizes(dev, resno);
> +	if (!sizes)
> +		return -ENOTSUPP;
> +
> +	if (!(sizes & BIT(size)))
> +		return -EINVAL;
> +
> +	old = pci_rbar_get_current_size(dev, resno);
> +	if (old < 0)
> +		return old;
> +
> +	ret = pci_rbar_set_size(dev, resno, size);
> +	if (ret)
> +		return ret;
> +
> +	res->end = res->start + pci_rbar_size_to_bytes(size) - 1;
> +
> +	/* Check if the new config works by trying to assign everything. */
> +	ret = pci_reassign_bridge_resources(dev->bus->self, res->flags);
> +	if (ret)
> +		goto error_resize;
> +
> +	return 0;
> +
> +error_resize:
> +	pci_rbar_set_size(dev, resno, old);
> +	res->end = res->start + pci_rbar_size_to_bytes(old) - 1;
> +	return ret;
> +}
> +EXPORT_SYMBOL(pci_resize_resource);
> +
>  int pci_enable_resources(struct pci_dev *dev, int mask)
>  {
>  	u16 cmd, old_cmd;
> diff --git a/include/linux/pci.h b/include/linux/pci.h
> index eb3da1a..2d631ad 100644
> --- a/include/linux/pci.h
> +++ b/include/linux/pci.h
> @@ -1068,6 +1068,8 @@ void pci_reset_bridge_secondary_bus(struct pci_dev *dev);
>  void pci_update_resource(struct pci_dev *dev, int resno);
>  int __must_check pci_assign_resource(struct pci_dev *dev, int i);
>  int __must_check pci_reassign_resource(struct pci_dev *dev, int i, resource_size_t add_size, resource_size_t align);
> +void pci_release_resource(struct pci_dev *dev, int resno);
> +int __must_check pci_resize_resource(struct pci_dev *dev, int i, int size);
>  int pci_select_bars(struct pci_dev *dev, unsigned long flags);
>  bool pci_device_is_present(struct pci_dev *pdev);
>  void pci_ignore_hotplug(struct pci_dev *dev);
> @@ -1148,6 +1150,7 @@ void pci_assign_unassigned_resources(void);
>  void pci_assign_unassigned_bridge_resources(struct pci_dev *bridge);
>  void pci_assign_unassigned_bus_resources(struct pci_bus *bus);
>  void pci_assign_unassigned_root_bus_resources(struct pci_bus *bus);
> +int pci_reassign_bridge_resources(struct pci_dev *bridge, unsigned long type);
>  void pdev_enable_device(struct pci_dev *);
>  int pci_enable_resources(struct pci_dev *, int mask);
>  void pci_fixup_irqs(u8 (*)(struct pci_dev *, u8 *),
> -- 
> 2.7.4
> 

[PATCH v5 4/6] x86/PCI: Enable a 64bit BAR on AMD Family 15h (Models 30h-3fh) Processors v4

[Christian König]

From: Christian König <christian.koenig@amd.com>

Most BIOS don't enable this because of compatibility reasons.

Manually enable a 64bit BAR of 64GB size so that we have
enough room for PCI devices.

v2: style cleanups, increase size, add resource name, set correct flags,
    print message that windows was added
v3: add defines for all the magic numbers, style cleanups
v4: add some comment that the BIOS should actually allow this using
    _PRS and _SRS.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Andy Shevchenko <andy.shevchenko@gmail.com>
---
 arch/x86/pci/fixup.c | 76 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 76 insertions(+)

diff --git a/arch/x86/pci/fixup.c b/arch/x86/pci/fixup.c
index 6d52b94..489e753 100644
--- a/arch/x86/pci/fixup.c
+++ b/arch/x86/pci/fixup.c
@@ -571,3 +571,79 @@ DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_INTEL, 0x2fc0, pci_invalid_bar);
 DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_INTEL, 0x6f60, pci_invalid_bar);
 DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_INTEL, 0x6fa0, pci_invalid_bar);
 DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_INTEL, 0x6fc0, pci_invalid_bar);
+
+#define AMD_141b_MMIO_BASE(x)	(0x80 + (x) * 0x8)
+#define AMD_141b_MMIO_BASE_RE_MASK		BIT(0)
+#define AMD_141b_MMIO_BASE_WE_MASK		BIT(1)
+#define AMD_141b_MMIO_BASE_MMIOBASE_MASK	GENMASK(31,8)
+
+#define AMD_141b_MMIO_LIMIT(x)	(0x84 + (x) * 0x8)
+#define AMD_141b_MMIO_LIMIT_MMIOLIMIT_MASK	GENMASK(31,8)
+
+#define AMD_141b_MMIO_HIGH(x)	(0x180 + (x) * 0x4)
+#define AMD_141b_MMIO_HIGH_MMIOBASE_MASK	GENMASK(7,0)
+#define AMD_141b_MMIO_HIGH_MMIOLIMIT_SHIFT	16
+#define AMD_141b_MMIO_HIGH_MMIOLIMIT_MASK	GENMASK(23,16)
+
+/*
+ * The PCI Firmware Spec, rev 3.2 notes that ACPI should optionally allow
+ * configuring host bridge windows using the _PRS and _SRS methods.
+ *
+ * But this is rarely implemented, so we manually enable a large 64bit BAR for
+ * PCIe device on AMD Family 15h (Models 30h-3fh) Processors here.
+ */
+static void pci_amd_enable_64bit_bar(struct pci_dev *dev)
+{
+	struct resource *res, *conflict;
+	u32 base, limit, high;
+	unsigned i;
+
+	for (i = 0; i < 8; ++i) {
+		pci_read_config_dword(dev, AMD_141b_MMIO_BASE(i), &base);
+		pci_read_config_dword(dev, AMD_141b_MMIO_HIGH(i), &high);
+
+		/* Is this slot free? */
+		if (!(base & (AMD_141b_MMIO_BASE_RE_MASK |
+			      AMD_141b_MMIO_BASE_WE_MASK)))
+			break;
+
+		base >>= 8;
+		base |= high << 24;
+
+		/* Abort if a slot already configures a 64bit BAR. */
+		if (base > 0x10000)
+			return;
+	}
+	if (i == 8)
+		return;
+
+	res = kzalloc(sizeof(*res), GFP_KERNEL);
+	if (!res)
+		return;
+
+	res->name = "PCI Bus 0000:00";
+	res->flags = IORESOURCE_PREFETCH | IORESOURCE_MEM |
+		IORESOURCE_MEM_64 | IORESOURCE_WINDOW;
+	res->start = 0x100000000ull;
+	res->end = 0xfd00000000ull - 1;
+
+	/* Just grab the free area behind system memory for this */
+	while ((conflict = request_resource_conflict(&iomem_resource, res)))
+		res->start = conflict->end + 1;
+
+	dev_info(&dev->dev, "adding root bus resource %pR\n", res);
+
+	base = ((res->start >> 8) & AMD_141b_MMIO_BASE_MMIOBASE_MASK) |
+		AMD_141b_MMIO_BASE_RE_MASK | AMD_141b_MMIO_BASE_WE_MASK;
+	limit = ((res->end + 1) >> 8) & AMD_141b_MMIO_LIMIT_MMIOLIMIT_MASK;
+	high = ((res->start >> 40) & AMD_141b_MMIO_HIGH_MMIOBASE_MASK) |
+		((((res->end + 1) >> 40) << AMD_141b_MMIO_HIGH_MMIOLIMIT_SHIFT)
+		 & AMD_141b_MMIO_HIGH_MMIOLIMIT_MASK);
+
+	pci_write_config_dword(dev, AMD_141b_MMIO_HIGH(i), high);
+	pci_write_config_dword(dev, AMD_141b_MMIO_LIMIT(i), limit);
+	pci_write_config_dword(dev, AMD_141b_MMIO_BASE(i), base);
+
+	pci_bus_add_resource(dev->bus, res, 0);
+}
+DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_AMD, 0x141b, pci_amd_enable_64bit_bar);
-- 
2.7.4

[PATCH v5 4/6] x86/PCI: Enable a 64bit BAR on AMD Family 15h (Models 30h-3fh) Processors v4

[Christian König]

From: Christian König <christian.koenig@amd.com>

Most BIOS don't enable this because of compatibility reasons.

Manually enable a 64bit BAR of 64GB size so that we have
enough room for PCI devices.

v2: style cleanups, increase size, add resource name, set correct flags,
    print message that windows was added
v3: add defines for all the magic numbers, style cleanups
v4: add some comment that the BIOS should actually allow this using
    _PRS and _SRS.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Andy Shevchenko <andy.shevchenko@gmail.com>
---
 arch/x86/pci/fixup.c | 76 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 76 insertions(+)

diff --git a/arch/x86/pci/fixup.c b/arch/x86/pci/fixup.c
index 6d52b94..489e753 100644
--- a/arch/x86/pci/fixup.c
+++ b/arch/x86/pci/fixup.c
@@ -571,3 +571,79 @@ DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_INTEL, 0x2fc0, pci_invalid_bar);
 DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_INTEL, 0x6f60, pci_invalid_bar);
 DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_INTEL, 0x6fa0, pci_invalid_bar);
 DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_INTEL, 0x6fc0, pci_invalid_bar);
+
+#define AMD_141b_MMIO_BASE(x)	(0x80 + (x) * 0x8)
+#define AMD_141b_MMIO_BASE_RE_MASK		BIT(0)
+#define AMD_141b_MMIO_BASE_WE_MASK		BIT(1)
+#define AMD_141b_MMIO_BASE_MMIOBASE_MASK	GENMASK(31,8)
+
+#define AMD_141b_MMIO_LIMIT(x)	(0x84 + (x) * 0x8)
+#define AMD_141b_MMIO_LIMIT_MMIOLIMIT_MASK	GENMASK(31,8)
+
+#define AMD_141b_MMIO_HIGH(x)	(0x180 + (x) * 0x4)
+#define AMD_141b_MMIO_HIGH_MMIOBASE_MASK	GENMASK(7,0)
+#define AMD_141b_MMIO_HIGH_MMIOLIMIT_SHIFT	16
+#define AMD_141b_MMIO_HIGH_MMIOLIMIT_MASK	GENMASK(23,16)
+
+/*
+ * The PCI Firmware Spec, rev 3.2 notes that ACPI should optionally allow
+ * configuring host bridge windows using the _PRS and _SRS methods.
+ *
+ * But this is rarely implemented, so we manually enable a large 64bit BAR for
+ * PCIe device on AMD Family 15h (Models 30h-3fh) Processors here.
+ */
+static void pci_amd_enable_64bit_bar(struct pci_dev *dev)
+{
+	struct resource *res, *conflict;
+	u32 base, limit, high;
+	unsigned i;
+
+	for (i = 0; i < 8; ++i) {
+		pci_read_config_dword(dev, AMD_141b_MMIO_BASE(i), &base);
+		pci_read_config_dword(dev, AMD_141b_MMIO_HIGH(i), &high);
+
+		/* Is this slot free? */
+		if (!(base & (AMD_141b_MMIO_BASE_RE_MASK |
+			      AMD_141b_MMIO_BASE_WE_MASK)))
+			break;
+
+		base >>= 8;
+		base |= high << 24;
+
+		/* Abort if a slot already configures a 64bit BAR. */
+		if (base > 0x10000)
+			return;
+	}
+	if (i == 8)
+		return;
+
+	res = kzalloc(sizeof(*res), GFP_KERNEL);
+	if (!res)
+		return;
+
+	res->name = "PCI Bus 0000:00";
+	res->flags = IORESOURCE_PREFETCH | IORESOURCE_MEM |
+		IORESOURCE_MEM_64 | IORESOURCE_WINDOW;
+	res->start = 0x100000000ull;
+	res->end = 0xfd00000000ull - 1;
+
+	/* Just grab the free area behind system memory for this */
+	while ((conflict = request_resource_conflict(&iomem_resource, res)))
+		res->start = conflict->end + 1;
+
+	dev_info(&dev->dev, "adding root bus resource %pR\n", res);
+
+	base = ((res->start >> 8) & AMD_141b_MMIO_BASE_MMIOBASE_MASK) |
+		AMD_141b_MMIO_BASE_RE_MASK | AMD_141b_MMIO_BASE_WE_MASK;
+	limit = ((res->end + 1) >> 8) & AMD_141b_MMIO_LIMIT_MMIOLIMIT_MASK;
+	high = ((res->start >> 40) & AMD_141b_MMIO_HIGH_MMIOBASE_MASK) |
+		((((res->end + 1) >> 40) << AMD_141b_MMIO_HIGH_MMIOLIMIT_SHIFT)
+		 & AMD_141b_MMIO_HIGH_MMIOLIMIT_MASK);
+
+	pci_write_config_dword(dev, AMD_141b_MMIO_HIGH(i), high);
+	pci_write_config_dword(dev, AMD_141b_MMIO_LIMIT(i), limit);
+	pci_write_config_dword(dev, AMD_141b_MMIO_BASE(i), base);
+
+	pci_bus_add_resource(dev->bus, res, 0);
+}
+DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_AMD, 0x141b, pci_amd_enable_64bit_bar);
-- 
2.7.4

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

[PATCH v5 5/6] drm/amdgpu: move hw generation check into amdgpu_doorbell_init

[Christian König]

From: Christian König <christian.koenig@amd.com>

This way we can savely call it on SI as well.

Signed-off-by: Christian König <christian.koenig@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 17 ++++++++++++-----
 1 file changed, 12 insertions(+), 5 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
index a7d6804..99290af 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
@@ -419,6 +419,15 @@ void amdgpu_pci_config_reset(struct amdgpu_device *adev)
  */
 static int amdgpu_doorbell_init(struct amdgpu_device *adev)
 {
+	/* No doorbell on SI hardware generation */
+	if (adev->asic_type < CHIP_BONAIRE) {
+		adev->doorbell.base = 0;
+		adev->doorbell.size = 0;
+		adev->doorbell.num_doorbells = 0;
+		adev->doorbell.ptr = NULL;
+		return 0;
+	}
+
 	/* doorbell bar mapping */
 	adev->doorbell.base = pci_resource_start(adev->pdev, 2);
 	adev->doorbell.size = pci_resource_len(adev->pdev, 2);
@@ -2136,9 +2145,8 @@ int amdgpu_device_init(struct amdgpu_device *adev,
 	DRM_INFO("register mmio base: 0x%08X\n", (uint32_t)adev->rmmio_base);
 	DRM_INFO("register mmio size: %u\n", (unsigned)adev->rmmio_size);
 
-	if (adev->asic_type >= CHIP_BONAIRE)
-		/* doorbell bar mapping */
-		amdgpu_doorbell_init(adev);
+	/* doorbell bar mapping */
+	amdgpu_doorbell_init(adev);
 
 	/* io port mapping */
 	for (i = 0; i < DEVICE_COUNT_RESOURCE; i++) {
@@ -2335,8 +2343,7 @@ void amdgpu_device_fini(struct amdgpu_device *adev)
 	adev->rio_mem = NULL;
 	iounmap(adev->rmmio);
 	adev->rmmio = NULL;
-	if (adev->asic_type >= CHIP_BONAIRE)
-		amdgpu_doorbell_fini(adev);
+	amdgpu_doorbell_fini(adev);
 	amdgpu_debugfs_regs_cleanup(adev);
 }
 
-- 
2.7.4

[PATCH v5 5/6] drm/amdgpu: move hw generation check into amdgpu_doorbell_init

[Christian König]

From: Christian König <christian.koenig@amd.com>

This way we can savely call it on SI as well.

Signed-off-by: Christian König <christian.koenig@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 17 ++++++++++++-----
 1 file changed, 12 insertions(+), 5 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
index a7d6804..99290af 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
@@ -419,6 +419,15 @@ void amdgpu_pci_config_reset(struct amdgpu_device *adev)
  */
 static int amdgpu_doorbell_init(struct amdgpu_device *adev)
 {
+	/* No doorbell on SI hardware generation */
+	if (adev->asic_type < CHIP_BONAIRE) {
+		adev->doorbell.base = 0;
+		adev->doorbell.size = 0;
+		adev->doorbell.num_doorbells = 0;
+		adev->doorbell.ptr = NULL;
+		return 0;
+	}
+
 	/* doorbell bar mapping */
 	adev->doorbell.base = pci_resource_start(adev->pdev, 2);
 	adev->doorbell.size = pci_resource_len(adev->pdev, 2);
@@ -2136,9 +2145,8 @@ int amdgpu_device_init(struct amdgpu_device *adev,
 	DRM_INFO("register mmio base: 0x%08X\n", (uint32_t)adev->rmmio_base);
 	DRM_INFO("register mmio size: %u\n", (unsigned)adev->rmmio_size);
 
-	if (adev->asic_type >= CHIP_BONAIRE)
-		/* doorbell bar mapping */
-		amdgpu_doorbell_init(adev);
+	/* doorbell bar mapping */
+	amdgpu_doorbell_init(adev);
 
 	/* io port mapping */
 	for (i = 0; i < DEVICE_COUNT_RESOURCE; i++) {
@@ -2335,8 +2343,7 @@ void amdgpu_device_fini(struct amdgpu_device *adev)
 	adev->rio_mem = NULL;
 	iounmap(adev->rmmio);
 	adev->rmmio = NULL;
-	if (adev->asic_type >= CHIP_BONAIRE)
-		amdgpu_doorbell_fini(adev);
+	amdgpu_doorbell_fini(adev);
 	amdgpu_debugfs_regs_cleanup(adev);
 }
 
-- 
2.7.4

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH v5 5/6] drm/amdgpu: move hw generation check into amdgpu_doorbell_init

[Andy Shevchenko]

On Fri, Jun 9, 2017 at 11:59 AM, Christian König
<deathsimple@vodafone.de> wrote:
> From: Christian König <christian.koenig@amd.com>
>
> This way we can savely call it on SI as well.

s/savely/safely

FWIW,
Reviewed-by: Andy Shevchenko <andy.shevchenko@gmail.com>

> Signed-off-by: Christian König <christian.koenig@amd.com>
> ---
>  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 17 ++++++++++++-----
>  1 file changed, 12 insertions(+), 5 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> index a7d6804..99290af 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> @@ -419,6 +419,15 @@ void amdgpu_pci_config_reset(struct amdgpu_device *adev)
>   */
>  static int amdgpu_doorbell_init(struct amdgpu_device *adev)
>  {
> +       /* No doorbell on SI hardware generation */
> +       if (adev->asic_type < CHIP_BONAIRE) {
> +               adev->doorbell.base = 0;
> +               adev->doorbell.size = 0;
> +               adev->doorbell.num_doorbells = 0;
> +               adev->doorbell.ptr = NULL;
> +               return 0;
> +       }
> +
>         /* doorbell bar mapping */
>         adev->doorbell.base = pci_resource_start(adev->pdev, 2);
>         adev->doorbell.size = pci_resource_len(adev->pdev, 2);
> @@ -2136,9 +2145,8 @@ int amdgpu_device_init(struct amdgpu_device *adev,
>         DRM_INFO("register mmio base: 0x%08X\n", (uint32_t)adev->rmmio_base);
>         DRM_INFO("register mmio size: %u\n", (unsigned)adev->rmmio_size);
>
> -       if (adev->asic_type >= CHIP_BONAIRE)
> -               /* doorbell bar mapping */
> -               amdgpu_doorbell_init(adev);
> +       /* doorbell bar mapping */
> +       amdgpu_doorbell_init(adev);
>
>         /* io port mapping */
>         for (i = 0; i < DEVICE_COUNT_RESOURCE; i++) {
> @@ -2335,8 +2343,7 @@ void amdgpu_device_fini(struct amdgpu_device *adev)
>         adev->rio_mem = NULL;
>         iounmap(adev->rmmio);
>         adev->rmmio = NULL;
> -       if (adev->asic_type >= CHIP_BONAIRE)
> -               amdgpu_doorbell_fini(adev);
> +       amdgpu_doorbell_fini(adev);
>         amdgpu_debugfs_regs_cleanup(adev);
>  }
>
> --
> 2.7.4
>



-- 
With Best Regards,
Andy Shevchenko

Re: [PATCH v5 5/6] drm/amdgpu: move hw generation check into amdgpu_doorbell_init

[Andy Shevchenko]

On Fri, Jun 9, 2017 at 11:59 AM, Christian K=C3=B6nig
<deathsimple@vodafone.de> wrote:
> From: Christian K=C3=B6nig <christian.koenig@amd.com>
>
> This way we can savely call it on SI as well.

s/savely/safely

FWIW,
Reviewed-by: Andy Shevchenko <andy.shevchenko@gmail.com>

> Signed-off-by: Christian K=C3=B6nig <christian.koenig@amd.com>
> ---
>  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 17 ++++++++++++-----
>  1 file changed, 12 insertions(+), 5 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm=
/amd/amdgpu/amdgpu_device.c
> index a7d6804..99290af 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> @@ -419,6 +419,15 @@ void amdgpu_pci_config_reset(struct amdgpu_device *a=
dev)
>   */
>  static int amdgpu_doorbell_init(struct amdgpu_device *adev)
>  {
> +       /* No doorbell on SI hardware generation */
> +       if (adev->asic_type < CHIP_BONAIRE) {
> +               adev->doorbell.base =3D 0;
> +               adev->doorbell.size =3D 0;
> +               adev->doorbell.num_doorbells =3D 0;
> +               adev->doorbell.ptr =3D NULL;
> +               return 0;
> +       }
> +
>         /* doorbell bar mapping */
>         adev->doorbell.base =3D pci_resource_start(adev->pdev, 2);
>         adev->doorbell.size =3D pci_resource_len(adev->pdev, 2);
> @@ -2136,9 +2145,8 @@ int amdgpu_device_init(struct amdgpu_device *adev,
>         DRM_INFO("register mmio base: 0x%08X\n", (uint32_t)adev->rmmio_ba=
se);
>         DRM_INFO("register mmio size: %u\n", (unsigned)adev->rmmio_size);
>
> -       if (adev->asic_type >=3D CHIP_BONAIRE)
> -               /* doorbell bar mapping */
> -               amdgpu_doorbell_init(adev);
> +       /* doorbell bar mapping */
> +       amdgpu_doorbell_init(adev);
>
>         /* io port mapping */
>         for (i =3D 0; i < DEVICE_COUNT_RESOURCE; i++) {
> @@ -2335,8 +2343,7 @@ void amdgpu_device_fini(struct amdgpu_device *adev)
>         adev->rio_mem =3D NULL;
>         iounmap(adev->rmmio);
>         adev->rmmio =3D NULL;
> -       if (adev->asic_type >=3D CHIP_BONAIRE)
> -               amdgpu_doorbell_fini(adev);
> +       amdgpu_doorbell_fini(adev);
>         amdgpu_debugfs_regs_cleanup(adev);
>  }
>
> --
> 2.7.4
>



--=20
With Best Regards,
Andy Shevchenko

[PATCH v5 6/6] drm/amdgpu: resize VRAM BAR for CPU access v3

[Christian König]

From: Christian König <christian.koenig@amd.com>

Try to resize BAR0 to let CPU access all of VRAM.

v2: rebased, style cleanups, disable mem decode before resize,
    handle gmc_v9 as well, round size up to power of two.
v3: handle gmc_v6 as well, release and reassign all BARs in the driver.

Signed-off-by: Christian König <christian.koenig@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu.h        |  1 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 40 ++++++++++++++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/gmc_v6_0.c      |  8 +++---
 drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c      |  8 +++---
 drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c      |  8 +++---
 drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c      | 10 +++++---
 6 files changed, 62 insertions(+), 13 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu.h b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
index c6a2ca4..87655e2 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
@@ -1943,6 +1943,7 @@ uint64_t amdgpu_ttm_tt_pte_flags(struct amdgpu_device *adev, struct ttm_tt *ttm,
 				 struct ttm_mem_reg *mem);
 void amdgpu_vram_location(struct amdgpu_device *adev, struct amdgpu_mc *mc, u64 base);
 void amdgpu_gtt_location(struct amdgpu_device *adev, struct amdgpu_mc *mc);
+void amdgpu_resize_bar0(struct amdgpu_device *adev);
 void amdgpu_ttm_set_active_vram_size(struct amdgpu_device *adev, u64 size);
 int amdgpu_ttm_init(struct amdgpu_device *adev);
 void amdgpu_ttm_fini(struct amdgpu_device *adev);
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
index 99290af..f74b79f 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
@@ -709,6 +709,46 @@ void amdgpu_gtt_location(struct amdgpu_device *adev, struct amdgpu_mc *mc)
 			mc->gtt_size >> 20, mc->gtt_start, mc->gtt_end);
 }
 
+/**
+ * amdgpu_resize_bar0 - try to resize BAR0
+ *
+ * @adev: amdgpu_device pointer
+ *
+ * Try to resize BAR0 to make all VRAM CPU accessible.
+ */
+void amdgpu_resize_bar0(struct amdgpu_device *adev)
+{
+	u64 space_needed = roundup_pow_of_two(adev->mc.real_vram_size);
+	u32 rbar_size = order_base_2(((space_needed >> 20) | 1)) - 1;
+	u16 cmd;
+	int r;
+
+	/* Disable memory decoding while we change the BAR addresses and size */
+	pci_read_config_word(adev->pdev, PCI_COMMAND, &cmd);
+	pci_write_config_word(adev->pdev, PCI_COMMAND,
+			      cmd & ~PCI_COMMAND_MEMORY);
+
+	/* Free the VRAM and doorbell BAR, we most likely need to move both. */
+	amdgpu_doorbell_fini(adev);
+	pci_release_resource(adev->pdev, 0);
+	if (adev->asic_type >= CHIP_BONAIRE)
+		pci_release_resource(adev->pdev, 2);
+
+	r = pci_resize_resource(adev->pdev, 0, rbar_size);
+	if (r == -ENOSPC)
+		DRM_INFO("Not enough PCI address space for a large BAR.");
+	else if (r && r != -ENOTSUPP)
+		DRM_ERROR("Problem resizing BAR0 (%d).", r);
+
+	pci_assign_unassigned_bus_resources(adev->pdev->bus);
+	pci_write_config_word(adev->pdev, PCI_COMMAND, cmd);
+
+	/* When the doorbell BAR isn't available we have no chance of
+	 * using the device.
+	 */
+	BUG_ON(amdgpu_doorbell_init(adev));
+}
+
 /*
  * GPU helpers function.
  */
diff --git a/drivers/gpu/drm/amd/amdgpu/gmc_v6_0.c b/drivers/gpu/drm/amd/amdgpu/gmc_v6_0.c
index a33ba60..af3c3c6 100644
--- a/drivers/gpu/drm/amd/amdgpu/gmc_v6_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/gmc_v6_0.c
@@ -334,12 +334,14 @@ static int gmc_v6_0_mc_init(struct amdgpu_device *adev)
 		break;
 	}
 	adev->mc.vram_width = numchan * chansize;
-	/* Could aper size report 0 ? */
-	adev->mc.aper_base = pci_resource_start(adev->pdev, 0);
-	adev->mc.aper_size = pci_resource_len(adev->pdev, 0);
 	/* size in MB on si */
 	adev->mc.mc_vram_size = RREG32(mmCONFIG_MEMSIZE) * 1024ULL * 1024ULL;
 	adev->mc.real_vram_size = RREG32(mmCONFIG_MEMSIZE) * 1024ULL * 1024ULL;
+
+	if (!(adev->flags & AMD_IS_APU))
+		amdgpu_resize_bar0(adev);
+	adev->mc.aper_base = pci_resource_start(adev->pdev, 0);
+	adev->mc.aper_size = pci_resource_len(adev->pdev, 0);
 	adev->mc.visible_vram_size = adev->mc.aper_size;
 
 	/* unless the user had overridden it, set the gart
diff --git a/drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c b/drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c
index 1326c1f..1d9f7a2 100644
--- a/drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c
@@ -372,13 +372,15 @@ static int gmc_v7_0_mc_init(struct amdgpu_device *adev)
 		}
 		adev->mc.vram_width = numchan * chansize;
 	}
-	/* Could aper size report 0 ? */
-	adev->mc.aper_base = pci_resource_start(adev->pdev, 0);
-	adev->mc.aper_size = pci_resource_len(adev->pdev, 0);
 	/* size in MB on si */
 	adev->mc.mc_vram_size = RREG32(mmCONFIG_MEMSIZE) * 1024ULL * 1024ULL;
 	adev->mc.real_vram_size = RREG32(mmCONFIG_MEMSIZE) * 1024ULL * 1024ULL;
 
+	if (!(adev->flags & AMD_IS_APU))
+		amdgpu_resize_bar0(adev);
+	adev->mc.aper_base = pci_resource_start(adev->pdev, 0);
+	adev->mc.aper_size = pci_resource_len(adev->pdev, 0);
+
 #ifdef CONFIG_X86_64
 	if (adev->flags & AMD_IS_APU) {
 		adev->mc.aper_base = ((u64)RREG32(mmMC_VM_FB_OFFSET)) << 22;
diff --git a/drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c b/drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c
index 42e5b55..858153d 100644
--- a/drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c
@@ -534,13 +534,15 @@ static int gmc_v8_0_mc_init(struct amdgpu_device *adev)
 		}
 		adev->mc.vram_width = numchan * chansize;
 	}
-	/* Could aper size report 0 ? */
-	adev->mc.aper_base = pci_resource_start(adev->pdev, 0);
-	adev->mc.aper_size = pci_resource_len(adev->pdev, 0);
 	/* size in MB on si */
 	adev->mc.mc_vram_size = RREG32(mmCONFIG_MEMSIZE) * 1024ULL * 1024ULL;
 	adev->mc.real_vram_size = RREG32(mmCONFIG_MEMSIZE) * 1024ULL * 1024ULL;
 
+	if (!(adev->flags & AMD_IS_APU))
+		amdgpu_resize_bar0(adev);
+	adev->mc.aper_base = pci_resource_start(adev->pdev, 0);
+	adev->mc.aper_size = pci_resource_len(adev->pdev, 0);
+
 #ifdef CONFIG_X86_64
 	if (adev->flags & AMD_IS_APU) {
 		adev->mc.aper_base = ((u64)RREG32(mmMC_VM_FB_OFFSET)) << 22;
diff --git a/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c b/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c
index 68172aa..f2e311d 100644
--- a/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c
@@ -480,17 +480,19 @@ static int gmc_v9_0_mc_init(struct amdgpu_device *adev)
 	}
 	adev->mc.vram_width = numchan * chansize;
 
-	/* Could aper size report 0 ? */
-	adev->mc.aper_base = pci_resource_start(adev->pdev, 0);
-	adev->mc.aper_size = pci_resource_len(adev->pdev, 0);
 	/* size in MB on si */
 	adev->mc.mc_vram_size =
 		((adev->flags & AMD_IS_APU) ? nbio_v7_0_get_memsize(adev) :
 		 nbio_v6_1_get_memsize(adev)) * 1024ULL * 1024ULL;
 	adev->mc.real_vram_size = adev->mc.mc_vram_size;
-	adev->mc.visible_vram_size = adev->mc.aper_size;
+
+	if (!(adev->flags & AMD_IS_APU))
+		amdgpu_resize_bar0(adev);
+	adev->mc.aper_base = pci_resource_start(adev->pdev, 0);
+	adev->mc.aper_size = pci_resource_len(adev->pdev, 0);
 
 	/* In case the PCI BAR is larger than the actual amount of vram */
+	adev->mc.visible_vram_size = adev->mc.aper_size;
 	if (adev->mc.visible_vram_size > adev->mc.real_vram_size)
 		adev->mc.visible_vram_size = adev->mc.real_vram_size;
 
-- 
2.7.4

[PATCH v5 6/6] drm/amdgpu: resize VRAM BAR for CPU access v3

[Christian König]

From: Christian König <christian.koenig@amd.com>

Try to resize BAR0 to let CPU access all of VRAM.

v2: rebased, style cleanups, disable mem decode before resize,
    handle gmc_v9 as well, round size up to power of two.
v3: handle gmc_v6 as well, release and reassign all BARs in the driver.

Signed-off-by: Christian König <christian.koenig@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu.h        |  1 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 40 ++++++++++++++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/gmc_v6_0.c      |  8 +++---
 drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c      |  8 +++---
 drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c      |  8 +++---
 drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c      | 10 +++++---
 6 files changed, 62 insertions(+), 13 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu.h b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
index c6a2ca4..87655e2 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
@@ -1943,6 +1943,7 @@ uint64_t amdgpu_ttm_tt_pte_flags(struct amdgpu_device *adev, struct ttm_tt *ttm,
 				 struct ttm_mem_reg *mem);
 void amdgpu_vram_location(struct amdgpu_device *adev, struct amdgpu_mc *mc, u64 base);
 void amdgpu_gtt_location(struct amdgpu_device *adev, struct amdgpu_mc *mc);
+void amdgpu_resize_bar0(struct amdgpu_device *adev);
 void amdgpu_ttm_set_active_vram_size(struct amdgpu_device *adev, u64 size);
 int amdgpu_ttm_init(struct amdgpu_device *adev);
 void amdgpu_ttm_fini(struct amdgpu_device *adev);
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
index 99290af..f74b79f 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
@@ -709,6 +709,46 @@ void amdgpu_gtt_location(struct amdgpu_device *adev, struct amdgpu_mc *mc)
 			mc->gtt_size >> 20, mc->gtt_start, mc->gtt_end);
 }
 
+/**
+ * amdgpu_resize_bar0 - try to resize BAR0
+ *
+ * @adev: amdgpu_device pointer
+ *
+ * Try to resize BAR0 to make all VRAM CPU accessible.
+ */
+void amdgpu_resize_bar0(struct amdgpu_device *adev)
+{
+	u64 space_needed = roundup_pow_of_two(adev->mc.real_vram_size);
+	u32 rbar_size = order_base_2(((space_needed >> 20) | 1)) - 1;
+	u16 cmd;
+	int r;
+
+	/* Disable memory decoding while we change the BAR addresses and size */
+	pci_read_config_word(adev->pdev, PCI_COMMAND, &cmd);
+	pci_write_config_word(adev->pdev, PCI_COMMAND,
+			      cmd & ~PCI_COMMAND_MEMORY);
+
+	/* Free the VRAM and doorbell BAR, we most likely need to move both. */
+	amdgpu_doorbell_fini(adev);
+	pci_release_resource(adev->pdev, 0);
+	if (adev->asic_type >= CHIP_BONAIRE)
+		pci_release_resource(adev->pdev, 2);
+
+	r = pci_resize_resource(adev->pdev, 0, rbar_size);
+	if (r == -ENOSPC)
+		DRM_INFO("Not enough PCI address space for a large BAR.");
+	else if (r && r != -ENOTSUPP)
+		DRM_ERROR("Problem resizing BAR0 (%d).", r);
+
+	pci_assign_unassigned_bus_resources(adev->pdev->bus);
+	pci_write_config_word(adev->pdev, PCI_COMMAND, cmd);
+
+	/* When the doorbell BAR isn't available we have no chance of
+	 * using the device.
+	 */
+	BUG_ON(amdgpu_doorbell_init(adev));
+}
+
 /*
  * GPU helpers function.
  */
diff --git a/drivers/gpu/drm/amd/amdgpu/gmc_v6_0.c b/drivers/gpu/drm/amd/amdgpu/gmc_v6_0.c
index a33ba60..af3c3c6 100644
--- a/drivers/gpu/drm/amd/amdgpu/gmc_v6_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/gmc_v6_0.c
@@ -334,12 +334,14 @@ static int gmc_v6_0_mc_init(struct amdgpu_device *adev)
 		break;
 	}
 	adev->mc.vram_width = numchan * chansize;
-	/* Could aper size report 0 ? */
-	adev->mc.aper_base = pci_resource_start(adev->pdev, 0);
-	adev->mc.aper_size = pci_resource_len(adev->pdev, 0);
 	/* size in MB on si */
 	adev->mc.mc_vram_size = RREG32(mmCONFIG_MEMSIZE) * 1024ULL * 1024ULL;
 	adev->mc.real_vram_size = RREG32(mmCONFIG_MEMSIZE) * 1024ULL * 1024ULL;
+
+	if (!(adev->flags & AMD_IS_APU))
+		amdgpu_resize_bar0(adev);
+	adev->mc.aper_base = pci_resource_start(adev->pdev, 0);
+	adev->mc.aper_size = pci_resource_len(adev->pdev, 0);
 	adev->mc.visible_vram_size = adev->mc.aper_size;
 
 	/* unless the user had overridden it, set the gart
diff --git a/drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c b/drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c
index 1326c1f..1d9f7a2 100644
--- a/drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c
@@ -372,13 +372,15 @@ static int gmc_v7_0_mc_init(struct amdgpu_device *adev)
 		}
 		adev->mc.vram_width = numchan * chansize;
 	}
-	/* Could aper size report 0 ? */
-	adev->mc.aper_base = pci_resource_start(adev->pdev, 0);
-	adev->mc.aper_size = pci_resource_len(adev->pdev, 0);
 	/* size in MB on si */
 	adev->mc.mc_vram_size = RREG32(mmCONFIG_MEMSIZE) * 1024ULL * 1024ULL;
 	adev->mc.real_vram_size = RREG32(mmCONFIG_MEMSIZE) * 1024ULL * 1024ULL;
 
+	if (!(adev->flags & AMD_IS_APU))
+		amdgpu_resize_bar0(adev);
+	adev->mc.aper_base = pci_resource_start(adev->pdev, 0);
+	adev->mc.aper_size = pci_resource_len(adev->pdev, 0);
+
 #ifdef CONFIG_X86_64
 	if (adev->flags & AMD_IS_APU) {
 		adev->mc.aper_base = ((u64)RREG32(mmMC_VM_FB_OFFSET)) << 22;
diff --git a/drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c b/drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c
index 42e5b55..858153d 100644
--- a/drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c
@@ -534,13 +534,15 @@ static int gmc_v8_0_mc_init(struct amdgpu_device *adev)
 		}
 		adev->mc.vram_width = numchan * chansize;
 	}
-	/* Could aper size report 0 ? */
-	adev->mc.aper_base = pci_resource_start(adev->pdev, 0);
-	adev->mc.aper_size = pci_resource_len(adev->pdev, 0);
 	/* size in MB on si */
 	adev->mc.mc_vram_size = RREG32(mmCONFIG_MEMSIZE) * 1024ULL * 1024ULL;
 	adev->mc.real_vram_size = RREG32(mmCONFIG_MEMSIZE) * 1024ULL * 1024ULL;
 
+	if (!(adev->flags & AMD_IS_APU))
+		amdgpu_resize_bar0(adev);
+	adev->mc.aper_base = pci_resource_start(adev->pdev, 0);
+	adev->mc.aper_size = pci_resource_len(adev->pdev, 0);
+
 #ifdef CONFIG_X86_64
 	if (adev->flags & AMD_IS_APU) {
 		adev->mc.aper_base = ((u64)RREG32(mmMC_VM_FB_OFFSET)) << 22;
diff --git a/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c b/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c
index 68172aa..f2e311d 100644
--- a/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c
@@ -480,17 +480,19 @@ static int gmc_v9_0_mc_init(struct amdgpu_device *adev)
 	}
 	adev->mc.vram_width = numchan * chansize;
 
-	/* Could aper size report 0 ? */
-	adev->mc.aper_base = pci_resource_start(adev->pdev, 0);
-	adev->mc.aper_size = pci_resource_len(adev->pdev, 0);
 	/* size in MB on si */
 	adev->mc.mc_vram_size =
 		((adev->flags & AMD_IS_APU) ? nbio_v7_0_get_memsize(adev) :
 		 nbio_v6_1_get_memsize(adev)) * 1024ULL * 1024ULL;
 	adev->mc.real_vram_size = adev->mc.mc_vram_size;
-	adev->mc.visible_vram_size = adev->mc.aper_size;
+
+	if (!(adev->flags & AMD_IS_APU))
+		amdgpu_resize_bar0(adev);
+	adev->mc.aper_base = pci_resource_start(adev->pdev, 0);
+	adev->mc.aper_size = pci_resource_len(adev->pdev, 0);
 
 	/* In case the PCI BAR is larger than the actual amount of vram */
+	adev->mc.visible_vram_size = adev->mc.aper_size;
 	if (adev->mc.visible_vram_size > adev->mc.real_vram_size)
 		adev->mc.visible_vram_size = adev->mc.real_vram_size;
 
-- 
2.7.4

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH v5 6/6] drm/amdgpu: resize VRAM BAR for CPU access v3

[Bjorn Helgaas]

On Fri, Jun 09, 2017 at 10:59:47AM +0200, Christian König wrote:
> From: Christian König <christian.koenig@amd.com>
> 
> Try to resize BAR0 to let CPU access all of VRAM.
> 
> v2: rebased, style cleanups, disable mem decode before resize,
>     handle gmc_v9 as well, round size up to power of two.
> v3: handle gmc_v6 as well, release and reassign all BARs in the driver.
> 
> Signed-off-by: Christian König <christian.koenig@amd.com>
> ---
>  drivers/gpu/drm/amd/amdgpu/amdgpu.h        |  1 +
>  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 40 ++++++++++++++++++++++++++++++
>  drivers/gpu/drm/amd/amdgpu/gmc_v6_0.c      |  8 +++---
>  drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c      |  8 +++---
>  drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c      |  8 +++---
>  drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c      | 10 +++++---
>  6 files changed, 62 insertions(+), 13 deletions(-)
> 
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu.h b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
> index c6a2ca4..87655e2 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu.h
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
> @@ -1943,6 +1943,7 @@ uint64_t amdgpu_ttm_tt_pte_flags(struct amdgpu_device *adev, struct ttm_tt *ttm,
>  				 struct ttm_mem_reg *mem);
>  void amdgpu_vram_location(struct amdgpu_device *adev, struct amdgpu_mc *mc, u64 base);
>  void amdgpu_gtt_location(struct amdgpu_device *adev, struct amdgpu_mc *mc);
> +void amdgpu_resize_bar0(struct amdgpu_device *adev);
>  void amdgpu_ttm_set_active_vram_size(struct amdgpu_device *adev, u64 size);
>  int amdgpu_ttm_init(struct amdgpu_device *adev);
>  void amdgpu_ttm_fini(struct amdgpu_device *adev);
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> index 99290af..f74b79f 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> @@ -709,6 +709,46 @@ void amdgpu_gtt_location(struct amdgpu_device *adev, struct amdgpu_mc *mc)
>  			mc->gtt_size >> 20, mc->gtt_start, mc->gtt_end);
>  }
>  
> +/**
> + * amdgpu_resize_bar0 - try to resize BAR0
> + *
> + * @adev: amdgpu_device pointer
> + *
> + * Try to resize BAR0 to make all VRAM CPU accessible.
> + */
> +void amdgpu_resize_bar0(struct amdgpu_device *adev)
> +{
> +	u64 space_needed = roundup_pow_of_two(adev->mc.real_vram_size);
> +	u32 rbar_size = order_base_2(((space_needed >> 20) | 1)) - 1;
> +	u16 cmd;
> +	int r;
> +
> +	/* Disable memory decoding while we change the BAR addresses and size */
> +	pci_read_config_word(adev->pdev, PCI_COMMAND, &cmd);
> +	pci_write_config_word(adev->pdev, PCI_COMMAND,
> +			      cmd & ~PCI_COMMAND_MEMORY);
> +
> +	/* Free the VRAM and doorbell BAR, we most likely need to move both. */
> +	amdgpu_doorbell_fini(adev);
> +	pci_release_resource(adev->pdev, 0);
> +	if (adev->asic_type >= CHIP_BONAIRE)
> +		pci_release_resource(adev->pdev, 2);
> +
> +	r = pci_resize_resource(adev->pdev, 0, rbar_size);
> +	if (r == -ENOSPC)
> +		DRM_INFO("Not enough PCI address space for a large BAR.");
> +	else if (r && r != -ENOTSUPP)
> +		DRM_ERROR("Problem resizing BAR0 (%d).", r);
> +
> +	pci_assign_unassigned_bus_resources(adev->pdev->bus);
> +	pci_write_config_word(adev->pdev, PCI_COMMAND, cmd);

I think it's problematic to unconditionally re-enable decoding here
because the pci_assign_unassigned_bus_resources() above may have
failed.  A pci_enable_decoding() interface would be one way to handle
this.

> +
> +	/* When the doorbell BAR isn't available we have no chance of
> +	 * using the device.
> +	 */
> +	BUG_ON(amdgpu_doorbell_init(adev));
> +}
> +
>  /*
>   * GPU helpers function.
>   */
> diff --git a/drivers/gpu/drm/amd/amdgpu/gmc_v6_0.c b/drivers/gpu/drm/amd/amdgpu/gmc_v6_0.c
> index a33ba60..af3c3c6 100644
> --- a/drivers/gpu/drm/amd/amdgpu/gmc_v6_0.c
> +++ b/drivers/gpu/drm/amd/amdgpu/gmc_v6_0.c
> @@ -334,12 +334,14 @@ static int gmc_v6_0_mc_init(struct amdgpu_device *adev)
>  		break;
>  	}
>  	adev->mc.vram_width = numchan * chansize;
> -	/* Could aper size report 0 ? */
> -	adev->mc.aper_base = pci_resource_start(adev->pdev, 0);
> -	adev->mc.aper_size = pci_resource_len(adev->pdev, 0);
>  	/* size in MB on si */
>  	adev->mc.mc_vram_size = RREG32(mmCONFIG_MEMSIZE) * 1024ULL * 1024ULL;
>  	adev->mc.real_vram_size = RREG32(mmCONFIG_MEMSIZE) * 1024ULL * 1024ULL;
> +
> +	if (!(adev->flags & AMD_IS_APU))
> +		amdgpu_resize_bar0(adev);
> +	adev->mc.aper_base = pci_resource_start(adev->pdev, 0);
> +	adev->mc.aper_size = pci_resource_len(adev->pdev, 0);
>  	adev->mc.visible_vram_size = adev->mc.aper_size;
>  
>  	/* unless the user had overridden it, set the gart
> diff --git a/drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c b/drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c
> index 1326c1f..1d9f7a2 100644
> --- a/drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c
> +++ b/drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c
> @@ -372,13 +372,15 @@ static int gmc_v7_0_mc_init(struct amdgpu_device *adev)
>  		}
>  		adev->mc.vram_width = numchan * chansize;
>  	}
> -	/* Could aper size report 0 ? */
> -	adev->mc.aper_base = pci_resource_start(adev->pdev, 0);
> -	adev->mc.aper_size = pci_resource_len(adev->pdev, 0);
>  	/* size in MB on si */
>  	adev->mc.mc_vram_size = RREG32(mmCONFIG_MEMSIZE) * 1024ULL * 1024ULL;
>  	adev->mc.real_vram_size = RREG32(mmCONFIG_MEMSIZE) * 1024ULL * 1024ULL;
>  
> +	if (!(adev->flags & AMD_IS_APU))
> +		amdgpu_resize_bar0(adev);
> +	adev->mc.aper_base = pci_resource_start(adev->pdev, 0);
> +	adev->mc.aper_size = pci_resource_len(adev->pdev, 0);
> +
>  #ifdef CONFIG_X86_64
>  	if (adev->flags & AMD_IS_APU) {
>  		adev->mc.aper_base = ((u64)RREG32(mmMC_VM_FB_OFFSET)) << 22;
> diff --git a/drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c b/drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c
> index 42e5b55..858153d 100644
> --- a/drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c
> +++ b/drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c
> @@ -534,13 +534,15 @@ static int gmc_v8_0_mc_init(struct amdgpu_device *adev)
>  		}
>  		adev->mc.vram_width = numchan * chansize;
>  	}
> -	/* Could aper size report 0 ? */
> -	adev->mc.aper_base = pci_resource_start(adev->pdev, 0);
> -	adev->mc.aper_size = pci_resource_len(adev->pdev, 0);
>  	/* size in MB on si */
>  	adev->mc.mc_vram_size = RREG32(mmCONFIG_MEMSIZE) * 1024ULL * 1024ULL;
>  	adev->mc.real_vram_size = RREG32(mmCONFIG_MEMSIZE) * 1024ULL * 1024ULL;
>  
> +	if (!(adev->flags & AMD_IS_APU))
> +		amdgpu_resize_bar0(adev);
> +	adev->mc.aper_base = pci_resource_start(adev->pdev, 0);
> +	adev->mc.aper_size = pci_resource_len(adev->pdev, 0);
> +
>  #ifdef CONFIG_X86_64
>  	if (adev->flags & AMD_IS_APU) {
>  		adev->mc.aper_base = ((u64)RREG32(mmMC_VM_FB_OFFSET)) << 22;
> diff --git a/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c b/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c
> index 68172aa..f2e311d 100644
> --- a/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c
> +++ b/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c
> @@ -480,17 +480,19 @@ static int gmc_v9_0_mc_init(struct amdgpu_device *adev)
>  	}
>  	adev->mc.vram_width = numchan * chansize;
>  
> -	/* Could aper size report 0 ? */
> -	adev->mc.aper_base = pci_resource_start(adev->pdev, 0);
> -	adev->mc.aper_size = pci_resource_len(adev->pdev, 0);
>  	/* size in MB on si */
>  	adev->mc.mc_vram_size =
>  		((adev->flags & AMD_IS_APU) ? nbio_v7_0_get_memsize(adev) :
>  		 nbio_v6_1_get_memsize(adev)) * 1024ULL * 1024ULL;
>  	adev->mc.real_vram_size = adev->mc.mc_vram_size;
> -	adev->mc.visible_vram_size = adev->mc.aper_size;
> +
> +	if (!(adev->flags & AMD_IS_APU))
> +		amdgpu_resize_bar0(adev);
> +	adev->mc.aper_base = pci_resource_start(adev->pdev, 0);
> +	adev->mc.aper_size = pci_resource_len(adev->pdev, 0);
>  
>  	/* In case the PCI BAR is larger than the actual amount of vram */
> +	adev->mc.visible_vram_size = adev->mc.aper_size;
>  	if (adev->mc.visible_vram_size > adev->mc.real_vram_size)
>  		adev->mc.visible_vram_size = adev->mc.real_vram_size;
>  
> -- 
> 2.7.4
> 

Re: [PATCH v5 6/6] drm/amdgpu: resize VRAM BAR for CPU access v3

[Alex Deucher]

On Fri, Jun 9, 2017 at 4:59 AM, Christian König <deathsimple@vodafone.de> wrote:
> From: Christian König <christian.koenig@amd.com>
>
> Try to resize BAR0 to let CPU access all of VRAM.
>
> v2: rebased, style cleanups, disable mem decode before resize,
>     handle gmc_v9 as well, round size up to power of two.
> v3: handle gmc_v6 as well, release and reassign all BARs in the driver.
>
> Signed-off-by: Christian König <christian.koenig@amd.com>
> ---
>  drivers/gpu/drm/amd/amdgpu/amdgpu.h        |  1 +
>  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 40 ++++++++++++++++++++++++++++++
>  drivers/gpu/drm/amd/amdgpu/gmc_v6_0.c      |  8 +++---
>  drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c      |  8 +++---
>  drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c      |  8 +++---
>  drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c      | 10 +++++---
>  6 files changed, 62 insertions(+), 13 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu.h b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
> index c6a2ca4..87655e2 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu.h
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
> @@ -1943,6 +1943,7 @@ uint64_t amdgpu_ttm_tt_pte_flags(struct amdgpu_device *adev, struct ttm_tt *ttm,
>                                  struct ttm_mem_reg *mem);
>  void amdgpu_vram_location(struct amdgpu_device *adev, struct amdgpu_mc *mc, u64 base);
>  void amdgpu_gtt_location(struct amdgpu_device *adev, struct amdgpu_mc *mc);
> +void amdgpu_resize_bar0(struct amdgpu_device *adev);
>  void amdgpu_ttm_set_active_vram_size(struct amdgpu_device *adev, u64 size);
>  int amdgpu_ttm_init(struct amdgpu_device *adev);
>  void amdgpu_ttm_fini(struct amdgpu_device *adev);
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> index 99290af..f74b79f 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> @@ -709,6 +709,46 @@ void amdgpu_gtt_location(struct amdgpu_device *adev, struct amdgpu_mc *mc)
>                         mc->gtt_size >> 20, mc->gtt_start, mc->gtt_end);
>  }
>
> +/**
> + * amdgpu_resize_bar0 - try to resize BAR0
> + *
> + * @adev: amdgpu_device pointer
> + *
> + * Try to resize BAR0 to make all VRAM CPU accessible.
> + */
> +void amdgpu_resize_bar0(struct amdgpu_device *adev)


Please rename this function to amdgpu_device_resize_fb_bar().
"amdgpu_device" for naming consistency in this file and "fb_bar" in
case the framebuffer bar changes from bar0 to something else in the
future.

Alex

> +{
> +       u64 space_needed = roundup_pow_of_two(adev->mc.real_vram_size);
> +       u32 rbar_size = order_base_2(((space_needed >> 20) | 1)) - 1;
> +       u16 cmd;
> +       int r;
> +
> +       /* Disable memory decoding while we change the BAR addresses and size */
> +       pci_read_config_word(adev->pdev, PCI_COMMAND, &cmd);
> +       pci_write_config_word(adev->pdev, PCI_COMMAND,
> +                             cmd & ~PCI_COMMAND_MEMORY);
> +
> +       /* Free the VRAM and doorbell BAR, we most likely need to move both. */
> +       amdgpu_doorbell_fini(adev);
> +       pci_release_resource(adev->pdev, 0);
> +       if (adev->asic_type >= CHIP_BONAIRE)
> +               pci_release_resource(adev->pdev, 2);
> +
> +       r = pci_resize_resource(adev->pdev, 0, rbar_size);
> +       if (r == -ENOSPC)
> +               DRM_INFO("Not enough PCI address space for a large BAR.");
> +       else if (r && r != -ENOTSUPP)
> +               DRM_ERROR("Problem resizing BAR0 (%d).", r);
> +
> +       pci_assign_unassigned_bus_resources(adev->pdev->bus);
> +       pci_write_config_word(adev->pdev, PCI_COMMAND, cmd);
> +
> +       /* When the doorbell BAR isn't available we have no chance of
> +        * using the device.
> +        */
> +       BUG_ON(amdgpu_doorbell_init(adev));
> +}
> +
>  /*
>   * GPU helpers function.
>   */
> diff --git a/drivers/gpu/drm/amd/amdgpu/gmc_v6_0.c b/drivers/gpu/drm/amd/amdgpu/gmc_v6_0.c
> index a33ba60..af3c3c6 100644
> --- a/drivers/gpu/drm/amd/amdgpu/gmc_v6_0.c
> +++ b/drivers/gpu/drm/amd/amdgpu/gmc_v6_0.c
> @@ -334,12 +334,14 @@ static int gmc_v6_0_mc_init(struct amdgpu_device *adev)
>                 break;
>         }
>         adev->mc.vram_width = numchan * chansize;
> -       /* Could aper size report 0 ? */
> -       adev->mc.aper_base = pci_resource_start(adev->pdev, 0);
> -       adev->mc.aper_size = pci_resource_len(adev->pdev, 0);
>         /* size in MB on si */
>         adev->mc.mc_vram_size = RREG32(mmCONFIG_MEMSIZE) * 1024ULL * 1024ULL;
>         adev->mc.real_vram_size = RREG32(mmCONFIG_MEMSIZE) * 1024ULL * 1024ULL;
> +
> +       if (!(adev->flags & AMD_IS_APU))
> +               amdgpu_resize_bar0(adev);
> +       adev->mc.aper_base = pci_resource_start(adev->pdev, 0);
> +       adev->mc.aper_size = pci_resource_len(adev->pdev, 0);
>         adev->mc.visible_vram_size = adev->mc.aper_size;
>
>         /* unless the user had overridden it, set the gart
> diff --git a/drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c b/drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c
> index 1326c1f..1d9f7a2 100644
> --- a/drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c
> +++ b/drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c
> @@ -372,13 +372,15 @@ static int gmc_v7_0_mc_init(struct amdgpu_device *adev)
>                 }
>                 adev->mc.vram_width = numchan * chansize;
>         }
> -       /* Could aper size report 0 ? */
> -       adev->mc.aper_base = pci_resource_start(adev->pdev, 0);
> -       adev->mc.aper_size = pci_resource_len(adev->pdev, 0);
>         /* size in MB on si */
>         adev->mc.mc_vram_size = RREG32(mmCONFIG_MEMSIZE) * 1024ULL * 1024ULL;
>         adev->mc.real_vram_size = RREG32(mmCONFIG_MEMSIZE) * 1024ULL * 1024ULL;
>
> +       if (!(adev->flags & AMD_IS_APU))
> +               amdgpu_resize_bar0(adev);
> +       adev->mc.aper_base = pci_resource_start(adev->pdev, 0);
> +       adev->mc.aper_size = pci_resource_len(adev->pdev, 0);
> +
>  #ifdef CONFIG_X86_64
>         if (adev->flags & AMD_IS_APU) {
>                 adev->mc.aper_base = ((u64)RREG32(mmMC_VM_FB_OFFSET)) << 22;
> diff --git a/drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c b/drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c
> index 42e5b55..858153d 100644
> --- a/drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c
> +++ b/drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c
> @@ -534,13 +534,15 @@ static int gmc_v8_0_mc_init(struct amdgpu_device *adev)
>                 }
>                 adev->mc.vram_width = numchan * chansize;
>         }
> -       /* Could aper size report 0 ? */
> -       adev->mc.aper_base = pci_resource_start(adev->pdev, 0);
> -       adev->mc.aper_size = pci_resource_len(adev->pdev, 0);
>         /* size in MB on si */
>         adev->mc.mc_vram_size = RREG32(mmCONFIG_MEMSIZE) * 1024ULL * 1024ULL;
>         adev->mc.real_vram_size = RREG32(mmCONFIG_MEMSIZE) * 1024ULL * 1024ULL;
>
> +       if (!(adev->flags & AMD_IS_APU))
> +               amdgpu_resize_bar0(adev);
> +       adev->mc.aper_base = pci_resource_start(adev->pdev, 0);
> +       adev->mc.aper_size = pci_resource_len(adev->pdev, 0);
> +
>  #ifdef CONFIG_X86_64
>         if (adev->flags & AMD_IS_APU) {
>                 adev->mc.aper_base = ((u64)RREG32(mmMC_VM_FB_OFFSET)) << 22;
> diff --git a/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c b/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c
> index 68172aa..f2e311d 100644
> --- a/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c
> +++ b/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c
> @@ -480,17 +480,19 @@ static int gmc_v9_0_mc_init(struct amdgpu_device *adev)
>         }
>         adev->mc.vram_width = numchan * chansize;
>
> -       /* Could aper size report 0 ? */
> -       adev->mc.aper_base = pci_resource_start(adev->pdev, 0);
> -       adev->mc.aper_size = pci_resource_len(adev->pdev, 0);
>         /* size in MB on si */
>         adev->mc.mc_vram_size =
>                 ((adev->flags & AMD_IS_APU) ? nbio_v7_0_get_memsize(adev) :
>                  nbio_v6_1_get_memsize(adev)) * 1024ULL * 1024ULL;
>         adev->mc.real_vram_size = adev->mc.mc_vram_size;
> -       adev->mc.visible_vram_size = adev->mc.aper_size;
> +
> +       if (!(adev->flags & AMD_IS_APU))
> +               amdgpu_resize_bar0(adev);
> +       adev->mc.aper_base = pci_resource_start(adev->pdev, 0);
> +       adev->mc.aper_size = pci_resource_len(adev->pdev, 0);
>
>         /* In case the PCI BAR is larger than the actual amount of vram */
> +       adev->mc.visible_vram_size = adev->mc.aper_size;
>         if (adev->mc.visible_vram_size > adev->mc.real_vram_size)
>                 adev->mc.visible_vram_size = adev->mc.real_vram_size;
>
> --
> 2.7.4
>
> _______________________________________________
> amd-gfx mailing list
> amd-gfx@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH v5 6/6] drm/amdgpu: resize VRAM BAR for CPU access v3

[Alex Deucher]

On Fri, Jun 9, 2017 at 4:59 AM, Christian K=C3=B6nig <deathsimple@vodafone.=
de> wrote:
> From: Christian K=C3=B6nig <christian.koenig@amd.com>
>
> Try to resize BAR0 to let CPU access all of VRAM.
>
> v2: rebased, style cleanups, disable mem decode before resize,
>     handle gmc_v9 as well, round size up to power of two.
> v3: handle gmc_v6 as well, release and reassign all BARs in the driver.
>
> Signed-off-by: Christian K=C3=B6nig <christian.koenig@amd.com>
> ---
>  drivers/gpu/drm/amd/amdgpu/amdgpu.h        |  1 +
>  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 40 ++++++++++++++++++++++++=
++++++
>  drivers/gpu/drm/amd/amdgpu/gmc_v6_0.c      |  8 +++---
>  drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c      |  8 +++---
>  drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c      |  8 +++---
>  drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c      | 10 +++++---
>  6 files changed, 62 insertions(+), 13 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu.h b/drivers/gpu/drm/amd/am=
dgpu/amdgpu.h
> index c6a2ca4..87655e2 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu.h
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
> @@ -1943,6 +1943,7 @@ uint64_t amdgpu_ttm_tt_pte_flags(struct amdgpu_devi=
ce *adev, struct ttm_tt *ttm,
>                                  struct ttm_mem_reg *mem);
>  void amdgpu_vram_location(struct amdgpu_device *adev, struct amdgpu_mc *=
mc, u64 base);
>  void amdgpu_gtt_location(struct amdgpu_device *adev, struct amdgpu_mc *m=
c);
> +void amdgpu_resize_bar0(struct amdgpu_device *adev);
>  void amdgpu_ttm_set_active_vram_size(struct amdgpu_device *adev, u64 siz=
e);
>  int amdgpu_ttm_init(struct amdgpu_device *adev);
>  void amdgpu_ttm_fini(struct amdgpu_device *adev);
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm=
/amd/amdgpu/amdgpu_device.c
> index 99290af..f74b79f 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> @@ -709,6 +709,46 @@ void amdgpu_gtt_location(struct amdgpu_device *adev,=
 struct amdgpu_mc *mc)
>                         mc->gtt_size >> 20, mc->gtt_start, mc->gtt_end);
>  }
>
> +/**
> + * amdgpu_resize_bar0 - try to resize BAR0
> + *
> + * @adev: amdgpu_device pointer
> + *
> + * Try to resize BAR0 to make all VRAM CPU accessible.
> + */
> +void amdgpu_resize_bar0(struct amdgpu_device *adev)


Please rename this function to amdgpu_device_resize_fb_bar().
"amdgpu_device" for naming consistency in this file and "fb_bar" in
case the framebuffer bar changes from bar0 to something else in the
future.

Alex

> +{
> +       u64 space_needed =3D roundup_pow_of_two(adev->mc.real_vram_size);
> +       u32 rbar_size =3D order_base_2(((space_needed >> 20) | 1)) - 1;
> +       u16 cmd;
> +       int r;
> +
> +       /* Disable memory decoding while we change the BAR addresses and =
size */
> +       pci_read_config_word(adev->pdev, PCI_COMMAND, &cmd);
> +       pci_write_config_word(adev->pdev, PCI_COMMAND,
> +                             cmd & ~PCI_COMMAND_MEMORY);
> +
> +       /* Free the VRAM and doorbell BAR, we most likely need to move bo=
th. */
> +       amdgpu_doorbell_fini(adev);
> +       pci_release_resource(adev->pdev, 0);
> +       if (adev->asic_type >=3D CHIP_BONAIRE)
> +               pci_release_resource(adev->pdev, 2);
> +
> +       r =3D pci_resize_resource(adev->pdev, 0, rbar_size);
> +       if (r =3D=3D -ENOSPC)
> +               DRM_INFO("Not enough PCI address space for a large BAR.")=
;
> +       else if (r && r !=3D -ENOTSUPP)
> +               DRM_ERROR("Problem resizing BAR0 (%d).", r);
> +
> +       pci_assign_unassigned_bus_resources(adev->pdev->bus);
> +       pci_write_config_word(adev->pdev, PCI_COMMAND, cmd);
> +
> +       /* When the doorbell BAR isn't available we have no chance of
> +        * using the device.
> +        */
> +       BUG_ON(amdgpu_doorbell_init(adev));
> +}
> +
>  /*
>   * GPU helpers function.
>   */
> diff --git a/drivers/gpu/drm/amd/amdgpu/gmc_v6_0.c b/drivers/gpu/drm/amd/=
amdgpu/gmc_v6_0.c
> index a33ba60..af3c3c6 100644
> --- a/drivers/gpu/drm/amd/amdgpu/gmc_v6_0.c
> +++ b/drivers/gpu/drm/amd/amdgpu/gmc_v6_0.c
> @@ -334,12 +334,14 @@ static int gmc_v6_0_mc_init(struct amdgpu_device *a=
dev)
>                 break;
>         }
>         adev->mc.vram_width =3D numchan * chansize;
> -       /* Could aper size report 0 ? */
> -       adev->mc.aper_base =3D pci_resource_start(adev->pdev, 0);
> -       adev->mc.aper_size =3D pci_resource_len(adev->pdev, 0);
>         /* size in MB on si */
>         adev->mc.mc_vram_size =3D RREG32(mmCONFIG_MEMSIZE) * 1024ULL * 10=
24ULL;
>         adev->mc.real_vram_size =3D RREG32(mmCONFIG_MEMSIZE) * 1024ULL * =
1024ULL;
> +
> +       if (!(adev->flags & AMD_IS_APU))
> +               amdgpu_resize_bar0(adev);
> +       adev->mc.aper_base =3D pci_resource_start(adev->pdev, 0);
> +       adev->mc.aper_size =3D pci_resource_len(adev->pdev, 0);
>         adev->mc.visible_vram_size =3D adev->mc.aper_size;
>
>         /* unless the user had overridden it, set the gart
> diff --git a/drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c b/drivers/gpu/drm/amd/=
amdgpu/gmc_v7_0.c
> index 1326c1f..1d9f7a2 100644
> --- a/drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c
> +++ b/drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c
> @@ -372,13 +372,15 @@ static int gmc_v7_0_mc_init(struct amdgpu_device *a=
dev)
>                 }
>                 adev->mc.vram_width =3D numchan * chansize;
>         }
> -       /* Could aper size report 0 ? */
> -       adev->mc.aper_base =3D pci_resource_start(adev->pdev, 0);
> -       adev->mc.aper_size =3D pci_resource_len(adev->pdev, 0);
>         /* size in MB on si */
>         adev->mc.mc_vram_size =3D RREG32(mmCONFIG_MEMSIZE) * 1024ULL * 10=
24ULL;
>         adev->mc.real_vram_size =3D RREG32(mmCONFIG_MEMSIZE) * 1024ULL * =
1024ULL;
>
> +       if (!(adev->flags & AMD_IS_APU))
> +               amdgpu_resize_bar0(adev);
> +       adev->mc.aper_base =3D pci_resource_start(adev->pdev, 0);
> +       adev->mc.aper_size =3D pci_resource_len(adev->pdev, 0);
> +
>  #ifdef CONFIG_X86_64
>         if (adev->flags & AMD_IS_APU) {
>                 adev->mc.aper_base =3D ((u64)RREG32(mmMC_VM_FB_OFFSET)) <=
< 22;
> diff --git a/drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c b/drivers/gpu/drm/amd/=
amdgpu/gmc_v8_0.c
> index 42e5b55..858153d 100644
> --- a/drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c
> +++ b/drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c
> @@ -534,13 +534,15 @@ static int gmc_v8_0_mc_init(struct amdgpu_device *a=
dev)
>                 }
>                 adev->mc.vram_width =3D numchan * chansize;
>         }
> -       /* Could aper size report 0 ? */
> -       adev->mc.aper_base =3D pci_resource_start(adev->pdev, 0);
> -       adev->mc.aper_size =3D pci_resource_len(adev->pdev, 0);
>         /* size in MB on si */
>         adev->mc.mc_vram_size =3D RREG32(mmCONFIG_MEMSIZE) * 1024ULL * 10=
24ULL;
>         adev->mc.real_vram_size =3D RREG32(mmCONFIG_MEMSIZE) * 1024ULL * =
1024ULL;
>
> +       if (!(adev->flags & AMD_IS_APU))
> +               amdgpu_resize_bar0(adev);
> +       adev->mc.aper_base =3D pci_resource_start(adev->pdev, 0);
> +       adev->mc.aper_size =3D pci_resource_len(adev->pdev, 0);
> +
>  #ifdef CONFIG_X86_64
>         if (adev->flags & AMD_IS_APU) {
>                 adev->mc.aper_base =3D ((u64)RREG32(mmMC_VM_FB_OFFSET)) <=
< 22;
> diff --git a/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c b/drivers/gpu/drm/amd/=
amdgpu/gmc_v9_0.c
> index 68172aa..f2e311d 100644
> --- a/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c
> +++ b/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c
> @@ -480,17 +480,19 @@ static int gmc_v9_0_mc_init(struct amdgpu_device *a=
dev)
>         }
>         adev->mc.vram_width =3D numchan * chansize;
>
> -       /* Could aper size report 0 ? */
> -       adev->mc.aper_base =3D pci_resource_start(adev->pdev, 0);
> -       adev->mc.aper_size =3D pci_resource_len(adev->pdev, 0);
>         /* size in MB on si */
>         adev->mc.mc_vram_size =3D
>                 ((adev->flags & AMD_IS_APU) ? nbio_v7_0_get_memsize(adev)=
 :
>                  nbio_v6_1_get_memsize(adev)) * 1024ULL * 1024ULL;
>         adev->mc.real_vram_size =3D adev->mc.mc_vram_size;
> -       adev->mc.visible_vram_size =3D adev->mc.aper_size;
> +
> +       if (!(adev->flags & AMD_IS_APU))
> +               amdgpu_resize_bar0(adev);
> +       adev->mc.aper_base =3D pci_resource_start(adev->pdev, 0);
> +       adev->mc.aper_size =3D pci_resource_len(adev->pdev, 0);
>
>         /* In case the PCI BAR is larger than the actual amount of vram *=
/
> +       adev->mc.visible_vram_size =3D adev->mc.aper_size;
>         if (adev->mc.visible_vram_size > adev->mc.real_vram_size)
>                 adev->mc.visible_vram_size =3D adev->mc.real_vram_size;
>
> --
> 2.7.4
>
> _______________________________________________
> amd-gfx mailing list
> amd-gfx@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH v5 6/6] drm/amdgpu: resize VRAM BAR for CPU access v3

[Alex Deucher]

On Fri, Jun 9, 2017 at 4:59 AM, Christian König <deathsimple@vodafone.de> wrote:
> From: Christian König <christian.koenig@amd.com>
>
> Try to resize BAR0 to let CPU access all of VRAM.
>
> v2: rebased, style cleanups, disable mem decode before resize,
>     handle gmc_v9 as well, round size up to power of two.
> v3: handle gmc_v6 as well, release and reassign all BARs in the driver.
>
> Signed-off-by: Christian König <christian.koenig@amd.com>
> ---
>  drivers/gpu/drm/amd/amdgpu/amdgpu.h        |  1 +
>  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 40 ++++++++++++++++++++++++++++++
>  drivers/gpu/drm/amd/amdgpu/gmc_v6_0.c      |  8 +++---
>  drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c      |  8 +++---
>  drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c      |  8 +++---
>  drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c      | 10 +++++---
>  6 files changed, 62 insertions(+), 13 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu.h b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
> index c6a2ca4..87655e2 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu.h
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
> @@ -1943,6 +1943,7 @@ uint64_t amdgpu_ttm_tt_pte_flags(struct amdgpu_device *adev, struct ttm_tt *ttm,
>                                  struct ttm_mem_reg *mem);
>  void amdgpu_vram_location(struct amdgpu_device *adev, struct amdgpu_mc *mc, u64 base);
>  void amdgpu_gtt_location(struct amdgpu_device *adev, struct amdgpu_mc *mc);
> +void amdgpu_resize_bar0(struct amdgpu_device *adev);
>  void amdgpu_ttm_set_active_vram_size(struct amdgpu_device *adev, u64 size);
>  int amdgpu_ttm_init(struct amdgpu_device *adev);
>  void amdgpu_ttm_fini(struct amdgpu_device *adev);
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> index 99290af..f74b79f 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> @@ -709,6 +709,46 @@ void amdgpu_gtt_location(struct amdgpu_device *adev, struct amdgpu_mc *mc)
>                         mc->gtt_size >> 20, mc->gtt_start, mc->gtt_end);
>  }
>
> +/**
> + * amdgpu_resize_bar0 - try to resize BAR0
> + *
> + * @adev: amdgpu_device pointer
> + *
> + * Try to resize BAR0 to make all VRAM CPU accessible.
> + */
> +void amdgpu_resize_bar0(struct amdgpu_device *adev)


Please rename this function to amdgpu_device_resize_fb_bar().
"amdgpu_device" for naming consistency in this file and "fb_bar" in
case the framebuffer bar changes from bar0 to something else in the
future.

Alex

> +{
> +       u64 space_needed = roundup_pow_of_two(adev->mc.real_vram_size);
> +       u32 rbar_size = order_base_2(((space_needed >> 20) | 1)) - 1;
> +       u16 cmd;
> +       int r;
> +
> +       /* Disable memory decoding while we change the BAR addresses and size */
> +       pci_read_config_word(adev->pdev, PCI_COMMAND, &cmd);
> +       pci_write_config_word(adev->pdev, PCI_COMMAND,
> +                             cmd & ~PCI_COMMAND_MEMORY);
> +
> +       /* Free the VRAM and doorbell BAR, we most likely need to move both. */
> +       amdgpu_doorbell_fini(adev);
> +       pci_release_resource(adev->pdev, 0);
> +       if (adev->asic_type >= CHIP_BONAIRE)
> +               pci_release_resource(adev->pdev, 2);
> +
> +       r = pci_resize_resource(adev->pdev, 0, rbar_size);
> +       if (r == -ENOSPC)
> +               DRM_INFO("Not enough PCI address space for a large BAR.");
> +       else if (r && r != -ENOTSUPP)
> +               DRM_ERROR("Problem resizing BAR0 (%d).", r);
> +
> +       pci_assign_unassigned_bus_resources(adev->pdev->bus);
> +       pci_write_config_word(adev->pdev, PCI_COMMAND, cmd);
> +
> +       /* When the doorbell BAR isn't available we have no chance of
> +        * using the device.
> +        */
> +       BUG_ON(amdgpu_doorbell_init(adev));
> +}
> +
>  /*
>   * GPU helpers function.
>   */
> diff --git a/drivers/gpu/drm/amd/amdgpu/gmc_v6_0.c b/drivers/gpu/drm/amd/amdgpu/gmc_v6_0.c
> index a33ba60..af3c3c6 100644
> --- a/drivers/gpu/drm/amd/amdgpu/gmc_v6_0.c
> +++ b/drivers/gpu/drm/amd/amdgpu/gmc_v6_0.c
> @@ -334,12 +334,14 @@ static int gmc_v6_0_mc_init(struct amdgpu_device *adev)
>                 break;
>         }
>         adev->mc.vram_width = numchan * chansize;
> -       /* Could aper size report 0 ? */
> -       adev->mc.aper_base = pci_resource_start(adev->pdev, 0);
> -       adev->mc.aper_size = pci_resource_len(adev->pdev, 0);
>         /* size in MB on si */
>         adev->mc.mc_vram_size = RREG32(mmCONFIG_MEMSIZE) * 1024ULL * 1024ULL;
>         adev->mc.real_vram_size = RREG32(mmCONFIG_MEMSIZE) * 1024ULL * 1024ULL;
> +
> +       if (!(adev->flags & AMD_IS_APU))
> +               amdgpu_resize_bar0(adev);
> +       adev->mc.aper_base = pci_resource_start(adev->pdev, 0);
> +       adev->mc.aper_size = pci_resource_len(adev->pdev, 0);
>         adev->mc.visible_vram_size = adev->mc.aper_size;
>
>         /* unless the user had overridden it, set the gart
> diff --git a/drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c b/drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c
> index 1326c1f..1d9f7a2 100644
> --- a/drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c
> +++ b/drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c
> @@ -372,13 +372,15 @@ static int gmc_v7_0_mc_init(struct amdgpu_device *adev)
>                 }
>                 adev->mc.vram_width = numchan * chansize;
>         }
> -       /* Could aper size report 0 ? */
> -       adev->mc.aper_base = pci_resource_start(adev->pdev, 0);
> -       adev->mc.aper_size = pci_resource_len(adev->pdev, 0);
>         /* size in MB on si */
>         adev->mc.mc_vram_size = RREG32(mmCONFIG_MEMSIZE) * 1024ULL * 1024ULL;
>         adev->mc.real_vram_size = RREG32(mmCONFIG_MEMSIZE) * 1024ULL * 1024ULL;
>
> +       if (!(adev->flags & AMD_IS_APU))
> +               amdgpu_resize_bar0(adev);
> +       adev->mc.aper_base = pci_resource_start(adev->pdev, 0);
> +       adev->mc.aper_size = pci_resource_len(adev->pdev, 0);
> +
>  #ifdef CONFIG_X86_64
>         if (adev->flags & AMD_IS_APU) {
>                 adev->mc.aper_base = ((u64)RREG32(mmMC_VM_FB_OFFSET)) << 22;
> diff --git a/drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c b/drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c
> index 42e5b55..858153d 100644
> --- a/drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c
> +++ b/drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c
> @@ -534,13 +534,15 @@ static int gmc_v8_0_mc_init(struct amdgpu_device *adev)
>                 }
>                 adev->mc.vram_width = numchan * chansize;
>         }
> -       /* Could aper size report 0 ? */
> -       adev->mc.aper_base = pci_resource_start(adev->pdev, 0);
> -       adev->mc.aper_size = pci_resource_len(adev->pdev, 0);
>         /* size in MB on si */
>         adev->mc.mc_vram_size = RREG32(mmCONFIG_MEMSIZE) * 1024ULL * 1024ULL;
>         adev->mc.real_vram_size = RREG32(mmCONFIG_MEMSIZE) * 1024ULL * 1024ULL;
>
> +       if (!(adev->flags & AMD_IS_APU))
> +               amdgpu_resize_bar0(adev);
> +       adev->mc.aper_base = pci_resource_start(adev->pdev, 0);
> +       adev->mc.aper_size = pci_resource_len(adev->pdev, 0);
> +
>  #ifdef CONFIG_X86_64
>         if (adev->flags & AMD_IS_APU) {
>                 adev->mc.aper_base = ((u64)RREG32(mmMC_VM_FB_OFFSET)) << 22;
> diff --git a/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c b/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c
> index 68172aa..f2e311d 100644
> --- a/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c
> +++ b/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c
> @@ -480,17 +480,19 @@ static int gmc_v9_0_mc_init(struct amdgpu_device *adev)
>         }
>         adev->mc.vram_width = numchan * chansize;
>
> -       /* Could aper size report 0 ? */
> -       adev->mc.aper_base = pci_resource_start(adev->pdev, 0);
> -       adev->mc.aper_size = pci_resource_len(adev->pdev, 0);
>         /* size in MB on si */
>         adev->mc.mc_vram_size =
>                 ((adev->flags & AMD_IS_APU) ? nbio_v7_0_get_memsize(adev) :
>                  nbio_v6_1_get_memsize(adev)) * 1024ULL * 1024ULL;
>         adev->mc.real_vram_size = adev->mc.mc_vram_size;
> -       adev->mc.visible_vram_size = adev->mc.aper_size;
> +
> +       if (!(adev->flags & AMD_IS_APU))
> +               amdgpu_resize_bar0(adev);
> +       adev->mc.aper_base = pci_resource_start(adev->pdev, 0);
> +       adev->mc.aper_size = pci_resource_len(adev->pdev, 0);
>
>         /* In case the PCI BAR is larger than the actual amount of vram */
> +       adev->mc.visible_vram_size = adev->mc.aper_size;
>         if (adev->mc.visible_vram_size > adev->mc.real_vram_size)
>                 adev->mc.visible_vram_size = adev->mc.real_vram_size;
>
> --
> 2.7.4
>
> _______________________________________________
> amd-gfx mailing list
> amd-gfx@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: Resizeable PCI BAR support V5

[Bjorn Helgaas]

On Fri, Jun 09, 2017 at 10:59:41AM +0200, Christian König wrote:
> Hi everyone,
> 
> This is the fith incarnation of this set of patches. It enables device
> drivers to resize and most likely also relocate the PCI BAR of devices 
> they manage to allow the CPU to access all of the device local memory at once.

I think this is really v7, isn't it?  I see a v6 posted May 9.

> This is very useful for GFX device drivers where the default PCI BAR is only
> about 256MB in size for compatibility reasons, but the device easily have
> multiple gigabyte of local memory.
> 
> Some changes since V4:
> 1. Rebased on 4.11.

I apply patches to topic branches based on -rc1.  I did apply these by
hand, but it's easier if they apply cleanly to -rc1.

> 2. added the rb from Andy Shevchenko to patches which look complete now.
> 3. Move releasing the BAR and reallocating it on error to the driver side.
> 4. Add amdgpu support for GMC V6 hardware generation as well.
> 
> Please review and/or comment,
> Christian.
> 

Re: Resizeable PCI BAR support V5

[Dieter Nützel]

[-- Attachment #1: Type: text/plain, Size: 3388 bytes --]

Hello Christian,

I've running this since you've sent it on-top of amd-staging-4.11. But I 
think my poor little FUJITSU PRIMERGY TX150 S7, Xeon X3470 (Nehalem), 
PCIe 2.0, 24 GB is to old for this stuff...

[    1.066475] pci 0000:05:00.0: VF(n) BAR0 space: [mem 
0x00000000-0x0003ffff 64bit] (contains BAR0 for 16 VFs)
[    1.066489] pci 0000:05:00.0: VF(n) BAR2 space: [mem 
0x00000000-0x003fffff 64bit] (contains BAR2 for 16 VFs)
[    1.121656] pci 0000:00:1c.0: BAR 15: assigned [mem 
0x80000000-0x801fffff 64bit pref]
[    1.121659] pci 0000:00:1c.6: BAR 15: assigned [mem 
0x80200000-0x803fffff 64bit pref]
[    1.121662] pci 0000:01:00.0: BAR 6: assigned [mem 
0xb0120000-0xb013ffff pref]
[    1.121681] pci 0000:05:00.0: BAR 6: assigned [mem 
0xb0280000-0xb02fffff pref]
[    1.121683] pci 0000:05:00.0: BAR 9: no space for [mem size 
0x00400000 64bit]
[    1.121684] pci 0000:05:00.0: BAR 9: failed to assign [mem size 
0x00400000 64bit]
[    1.121685] pci 0000:05:00.0: BAR 7: no space for [mem size 
0x00040000 64bit]
[    1.121687] pci 0000:05:00.0: BAR 7: failed to assign [mem size 
0x00040000 64bit]
[    3.874180] amdgpu 0000:01:00.0: BAR 0: releasing [mem 
0xc0000000-0xcfffffff 64bit pref]
[    3.874182] amdgpu 0000:01:00.0: BAR 2: releasing [mem 
0xb0400000-0xb05fffff 64bit pref]
[    3.874198] pcieport 0000:00:03.0: BAR 15: releasing [mem 
0xb0400000-0xcfffffff 64bit pref]
[    3.874215] pcieport 0000:00:03.0: BAR 15: no space for [mem size 
0x300000000 64bit pref]
[    3.874217] pcieport 0000:00:03.0: BAR 15: failed to assign [mem size 
0x300000000 64bit pref]
[    3.874221] amdgpu 0000:01:00.0: BAR 0: no space for [mem size 
0x200000000 64bit pref]
[    3.874223] amdgpu 0000:01:00.0: BAR 0: failed to assign [mem size 
0x200000000 64bit pref]
[    3.874226] amdgpu 0000:01:00.0: BAR 2: no space for [mem size 
0x00200000 64bit pref]
[    3.874227] amdgpu 0000:01:00.0: BAR 2: failed to assign [mem size 
0x00200000 64bit pref]
[    3.874258] [drm] Not enough PCI address space for a large BAR.
[    3.874261] amdgpu 0000:01:00.0: BAR 0: assigned [mem 
0xc0000000-0xcfffffff 64bit pref]
[    3.874269] amdgpu 0000:01:00.0: BAR 2: assigned [mem 
0xb0400000-0xb05fffff 64bit pref]
[    3.874288] [drm] Detected VRAM RAM=8192M, BAR=256M

Anyway rebase for current amd-staging-4.11 needed.
Find attached dmesg-amd-staging-4.11-1.g7262353-default+.log.xz

Greetings,
Dieter

Am 09.06.2017 10:59, schrieb Christian König:
> Hi everyone,
> 
> This is the fith incarnation of this set of patches. It enables device
> drivers to resize and most likely also relocate the PCI BAR of devices
> they manage to allow the CPU to access all of the device local memory 
> at once.
> 
> This is very useful for GFX device drivers where the default PCI BAR is 
> only
> about 256MB in size for compatibility reasons, but the device easily 
> have
> multiple gigabyte of local memory.
> 
> Some changes since V4:
> 1. Rebased on 4.11.
> 2. added the rb from Andy Shevchenko to patches which look complete 
> now.
> 3. Move releasing the BAR and reallocating it on error to the driver 
> side.
> 4. Add amdgpu support for GMC V6 hardware generation as well.
> 
> Please review and/or comment,
> Christian.
> 
> _______________________________________________
> dri-devel mailing list
> dri-devel@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/dri-devel

[-- Attachment #2: dmesg-amd-staging-4.11-1.g7262353-default+.log.xz --]
[-- Type: application/x-xz, Size: 18008 bytes --]

Re: Resizeable PCI BAR support V5

[Dieter Nützel]

[-- Attachment #1: Type: text/plain, Size: 3388 bytes --]

Hello Christian,

I've running this since you've sent it on-top of amd-staging-4.11. But I 
think my poor little FUJITSU PRIMERGY TX150 S7, Xeon X3470 (Nehalem), 
PCIe 2.0, 24 GB is to old for this stuff...

[    1.066475] pci 0000:05:00.0: VF(n) BAR0 space: [mem 
0x00000000-0x0003ffff 64bit] (contains BAR0 for 16 VFs)
[    1.066489] pci 0000:05:00.0: VF(n) BAR2 space: [mem 
0x00000000-0x003fffff 64bit] (contains BAR2 for 16 VFs)
[    1.121656] pci 0000:00:1c.0: BAR 15: assigned [mem 
0x80000000-0x801fffff 64bit pref]
[    1.121659] pci 0000:00:1c.6: BAR 15: assigned [mem 
0x80200000-0x803fffff 64bit pref]
[    1.121662] pci 0000:01:00.0: BAR 6: assigned [mem 
0xb0120000-0xb013ffff pref]
[    1.121681] pci 0000:05:00.0: BAR 6: assigned [mem 
0xb0280000-0xb02fffff pref]
[    1.121683] pci 0000:05:00.0: BAR 9: no space for [mem size 
0x00400000 64bit]
[    1.121684] pci 0000:05:00.0: BAR 9: failed to assign [mem size 
0x00400000 64bit]
[    1.121685] pci 0000:05:00.0: BAR 7: no space for [mem size 
0x00040000 64bit]
[    1.121687] pci 0000:05:00.0: BAR 7: failed to assign [mem size 
0x00040000 64bit]
[    3.874180] amdgpu 0000:01:00.0: BAR 0: releasing [mem 
0xc0000000-0xcfffffff 64bit pref]
[    3.874182] amdgpu 0000:01:00.0: BAR 2: releasing [mem 
0xb0400000-0xb05fffff 64bit pref]
[    3.874198] pcieport 0000:00:03.0: BAR 15: releasing [mem 
0xb0400000-0xcfffffff 64bit pref]
[    3.874215] pcieport 0000:00:03.0: BAR 15: no space for [mem size 
0x300000000 64bit pref]
[    3.874217] pcieport 0000:00:03.0: BAR 15: failed to assign [mem size 
0x300000000 64bit pref]
[    3.874221] amdgpu 0000:01:00.0: BAR 0: no space for [mem size 
0x200000000 64bit pref]
[    3.874223] amdgpu 0000:01:00.0: BAR 0: failed to assign [mem size 
0x200000000 64bit pref]
[    3.874226] amdgpu 0000:01:00.0: BAR 2: no space for [mem size 
0x00200000 64bit pref]
[    3.874227] amdgpu 0000:01:00.0: BAR 2: failed to assign [mem size 
0x00200000 64bit pref]
[    3.874258] [drm] Not enough PCI address space for a large BAR.
[    3.874261] amdgpu 0000:01:00.0: BAR 0: assigned [mem 
0xc0000000-0xcfffffff 64bit pref]
[    3.874269] amdgpu 0000:01:00.0: BAR 2: assigned [mem 
0xb0400000-0xb05fffff 64bit pref]
[    3.874288] [drm] Detected VRAM RAM=8192M, BAR=256M

Anyway rebase for current amd-staging-4.11 needed.
Find attached dmesg-amd-staging-4.11-1.g7262353-default+.log.xz

Greetings,
Dieter

Am 09.06.2017 10:59, schrieb Christian König:
> Hi everyone,
> 
> This is the fith incarnation of this set of patches. It enables device
> drivers to resize and most likely also relocate the PCI BAR of devices
> they manage to allow the CPU to access all of the device local memory 
> at once.
> 
> This is very useful for GFX device drivers where the default PCI BAR is 
> only
> about 256MB in size for compatibility reasons, but the device easily 
> have
> multiple gigabyte of local memory.
> 
> Some changes since V4:
> 1. Rebased on 4.11.
> 2. added the rb from Andy Shevchenko to patches which look complete 
> now.
> 3. Move releasing the BAR and reallocating it on error to the driver 
> side.
> 4. Add amdgpu support for GMC V6 hardware generation as well.
> 
> Please review and/or comment,
> Christian.
> 
> _______________________________________________
> dri-devel mailing list
> dri-devel@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/dri-devel

[-- Attachment #2: dmesg-amd-staging-4.11-1.g7262353-default+.log.xz --]
[-- Type: application/x-xz, Size: 18008 bytes --]

[-- Attachment #3: Type: text/plain, Size: 160 bytes --]

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: Resizeable PCI BAR support V5

[Christian König]

Hi Dieter,

thanks a lot for testing that.

> But I think my poor little FUJITSU PRIMERGY TX150 S7, Xeon X3470 
> (Nehalem), PCIe 2.0, 24 GB is to old for this stuff... 
Well, actually you only need to figure out how to enable a PCIe window 
above the 4GB limit.

Could be that the BIOS supports this with the ACPI tables (totally 
unlikely) or you could try to dig up the Northbridge documentation for 
this CPU from Intel and use my patch for the AMD CPUs as blueprint how 
to do this on an Intel CPU as well.

Fact is you GFX hardware is perfectly capable of doing this, it's just 
that the BIOS/Motherboard didn't enabled a PCIe window per default to 
avoid problems with 32bit OSes.

Regards,
Christian.

Am 30.06.2017 um 01:51 schrieb Dieter Nützel:
> Hello Christian,
>
> I've running this since you've sent it on-top of amd-staging-4.11. But 
> I think my poor little FUJITSU PRIMERGY TX150 S7, Xeon X3470 
> (Nehalem), PCIe 2.0, 24 GB is to old for this stuff...
>
> [    1.066475] pci 0000:05:00.0: VF(n) BAR0 space: [mem 
> 0x00000000-0x0003ffff 64bit] (contains BAR0 for 16 VFs)
> [    1.066489] pci 0000:05:00.0: VF(n) BAR2 space: [mem 
> 0x00000000-0x003fffff 64bit] (contains BAR2 for 16 VFs)
> [    1.121656] pci 0000:00:1c.0: BAR 15: assigned [mem 
> 0x80000000-0x801fffff 64bit pref]
> [    1.121659] pci 0000:00:1c.6: BAR 15: assigned [mem 
> 0x80200000-0x803fffff 64bit pref]
> [    1.121662] pci 0000:01:00.0: BAR 6: assigned [mem 
> 0xb0120000-0xb013ffff pref]
> [    1.121681] pci 0000:05:00.0: BAR 6: assigned [mem 
> 0xb0280000-0xb02fffff pref]
> [    1.121683] pci 0000:05:00.0: BAR 9: no space for [mem size 
> 0x00400000 64bit]
> [    1.121684] pci 0000:05:00.0: BAR 9: failed to assign [mem size 
> 0x00400000 64bit]
> [    1.121685] pci 0000:05:00.0: BAR 7: no space for [mem size 
> 0x00040000 64bit]
> [    1.121687] pci 0000:05:00.0: BAR 7: failed to assign [mem size 
> 0x00040000 64bit]
> [    3.874180] amdgpu 0000:01:00.0: BAR 0: releasing [mem 
> 0xc0000000-0xcfffffff 64bit pref]
> [    3.874182] amdgpu 0000:01:00.0: BAR 2: releasing [mem 
> 0xb0400000-0xb05fffff 64bit pref]
> [    3.874198] pcieport 0000:00:03.0: BAR 15: releasing [mem 
> 0xb0400000-0xcfffffff 64bit pref]
> [    3.874215] pcieport 0000:00:03.0: BAR 15: no space for [mem size 
> 0x300000000 64bit pref]
> [    3.874217] pcieport 0000:00:03.0: BAR 15: failed to assign [mem 
> size 0x300000000 64bit pref]
> [    3.874221] amdgpu 0000:01:00.0: BAR 0: no space for [mem size 
> 0x200000000 64bit pref]
> [    3.874223] amdgpu 0000:01:00.0: BAR 0: failed to assign [mem size 
> 0x200000000 64bit pref]
> [    3.874226] amdgpu 0000:01:00.0: BAR 2: no space for [mem size 
> 0x00200000 64bit pref]
> [    3.874227] amdgpu 0000:01:00.0: BAR 2: failed to assign [mem size 
> 0x00200000 64bit pref]
> [    3.874258] [drm] Not enough PCI address space for a large BAR.
> [    3.874261] amdgpu 0000:01:00.0: BAR 0: assigned [mem 
> 0xc0000000-0xcfffffff 64bit pref]
> [    3.874269] amdgpu 0000:01:00.0: BAR 2: assigned [mem 
> 0xb0400000-0xb05fffff 64bit pref]
> [    3.874288] [drm] Detected VRAM RAM=8192M, BAR=256M
>
> Anyway rebase for current amd-staging-4.11 needed.
> Find attached dmesg-amd-staging-4.11-1.g7262353-default+.log.xz
>
> Greetings,
> Dieter
>
> Am 09.06.2017 10:59, schrieb Christian König:
>> Hi everyone,
>>
>> This is the fith incarnation of this set of patches. It enables device
>> drivers to resize and most likely also relocate the PCI BAR of devices
>> they manage to allow the CPU to access all of the device local memory 
>> at once.
>>
>> This is very useful for GFX device drivers where the default PCI BAR 
>> is only
>> about 256MB in size for compatibility reasons, but the device easily 
>> have
>> multiple gigabyte of local memory.
>>
>> Some changes since V4:
>> 1. Rebased on 4.11.
>> 2. added the rb from Andy Shevchenko to patches which look complete now.
>> 3. Move releasing the BAR and reallocating it on error to the driver 
>> side.
>> 4. Add amdgpu support for GMC V6 hardware generation as well.
>>
>> Please review and/or comment,
>> Christian.
>>
>> _______________________________________________
>> dri-devel mailing list
>> dri-devel@lists.freedesktop.org
>> https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: Resizeable PCI BAR support V5

[Dieter Nützel]

Hello Christian,

after (long ;-)) vacation, injured wife (bad lumbago/luckily NO disc 
prolapse) on 2cond day @ our target, our daughter's 12th birthday, 
school start for both kids and mostly dad work I'm back...

Do you have a V6 handy.
Will do my fingers dirty if no Intel guy beats me at it.

Greetings,
Dieter

Am 30.06.2017 14:55, schrieb Christian König:
> Hi Dieter,
> 
> thanks a lot for testing that.
> 
>> But I think my poor little FUJITSU PRIMERGY TX150 S7, Xeon X3470 
>> (Nehalem), PCIe 2.0, 24 GB is to old for this stuff...
> Well, actually you only need to figure out how to enable a PCIe window
> above the 4GB limit.
> 
> Could be that the BIOS supports this with the ACPI tables (totally
> unlikely) or you could try to dig up the Northbridge documentation for
> this CPU from Intel and use my patch for the AMD CPUs as blueprint how
> to do this on an Intel CPU as well.
> 
> Fact is you GFX hardware is perfectly capable of doing this, it's just
> that the BIOS/Motherboard didn't enabled a PCIe window per default to
> avoid problems with 32bit OSes.
> 
> Regards,
> Christian.
> 
> Am 30.06.2017 um 01:51 schrieb Dieter Nützel:
>> Hello Christian,
>> 
>> I've running this since you've sent it on-top of amd-staging-4.11. But 
>> I think my poor little FUJITSU PRIMERGY TX150 S7, Xeon X3470 
>> (Nehalem), PCIe 2.0, 24 GB is to old for this stuff...
>> 
>> [    1.066475] pci 0000:05:00.0: VF(n) BAR0 space: [mem 
>> 0x00000000-0x0003ffff 64bit] (contains BAR0 for 16 VFs)
>> [    1.066489] pci 0000:05:00.0: VF(n) BAR2 space: [mem 
>> 0x00000000-0x003fffff 64bit] (contains BAR2 for 16 VFs)
>> [    1.121656] pci 0000:00:1c.0: BAR 15: assigned [mem 
>> 0x80000000-0x801fffff 64bit pref]
>> [    1.121659] pci 0000:00:1c.6: BAR 15: assigned [mem 
>> 0x80200000-0x803fffff 64bit pref]
>> [    1.121662] pci 0000:01:00.0: BAR 6: assigned [mem 
>> 0xb0120000-0xb013ffff pref]
>> [    1.121681] pci 0000:05:00.0: BAR 6: assigned [mem 
>> 0xb0280000-0xb02fffff pref]
>> [    1.121683] pci 0000:05:00.0: BAR 9: no space for [mem size 
>> 0x00400000 64bit]
>> [    1.121684] pci 0000:05:00.0: BAR 9: failed to assign [mem size 
>> 0x00400000 64bit]
>> [    1.121685] pci 0000:05:00.0: BAR 7: no space for [mem size 
>> 0x00040000 64bit]
>> [    1.121687] pci 0000:05:00.0: BAR 7: failed to assign [mem size 
>> 0x00040000 64bit]
>> [    3.874180] amdgpu 0000:01:00.0: BAR 0: releasing [mem 
>> 0xc0000000-0xcfffffff 64bit pref]
>> [    3.874182] amdgpu 0000:01:00.0: BAR 2: releasing [mem 
>> 0xb0400000-0xb05fffff 64bit pref]
>> [    3.874198] pcieport 0000:00:03.0: BAR 15: releasing [mem 
>> 0xb0400000-0xcfffffff 64bit pref]
>> [    3.874215] pcieport 0000:00:03.0: BAR 15: no space for [mem size 
>> 0x300000000 64bit pref]
>> [    3.874217] pcieport 0000:00:03.0: BAR 15: failed to assign [mem 
>> size 0x300000000 64bit pref]
>> [    3.874221] amdgpu 0000:01:00.0: BAR 0: no space for [mem size 
>> 0x200000000 64bit pref]
>> [    3.874223] amdgpu 0000:01:00.0: BAR 0: failed to assign [mem size 
>> 0x200000000 64bit pref]
>> [    3.874226] amdgpu 0000:01:00.0: BAR 2: no space for [mem size 
>> 0x00200000 64bit pref]
>> [    3.874227] amdgpu 0000:01:00.0: BAR 2: failed to assign [mem size 
>> 0x00200000 64bit pref]
>> [    3.874258] [drm] Not enough PCI address space for a large BAR.
>> [    3.874261] amdgpu 0000:01:00.0: BAR 0: assigned [mem 
>> 0xc0000000-0xcfffffff 64bit pref]
>> [    3.874269] amdgpu 0000:01:00.0: BAR 2: assigned [mem 
>> 0xb0400000-0xb05fffff 64bit pref]
>> [    3.874288] [drm] Detected VRAM RAM=8192M, BAR=256M
>> 
>> Anyway rebase for current amd-staging-4.11 needed.
>> Find attached dmesg-amd-staging-4.11-1.g7262353-default+.log.xz
>> 
>> Greetings,
>> Dieter
>> 
>> Am 09.06.2017 10:59, schrieb Christian König:
>>> Hi everyone,
>>> 
>>> This is the fith incarnation of this set of patches. It enables 
>>> device
>>> drivers to resize and most likely also relocate the PCI BAR of 
>>> devices
>>> they manage to allow the CPU to access all of the device local memory 
>>> at once.
>>> 
>>> This is very useful for GFX device drivers where the default PCI BAR 
>>> is only
>>> about 256MB in size for compatibility reasons, but the device easily 
>>> have
>>> multiple gigabyte of local memory.
>>> 
>>> Some changes since V4:
>>> 1. Rebased on 4.11.
>>> 2. added the rb from Andy Shevchenko to patches which look complete 
>>> now.
>>> 3. Move releasing the BAR and reallocating it on error to the driver 
>>> side.
>>> 4. Add amdgpu support for GMC V6 hardware generation as well.
>>> 
>>> Please review and/or comment,
>>> Christian.
>>> 
>>> _______________________________________________
>>> dri-devel mailing list
>>> dri-devel@lists.freedesktop.org
>>> https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: Resizeable PCI BAR support V5

[Dieter Nützel]

Hello Christian,

after (long ;-)) vacation, injured wife (bad lumbago/luckily NO disc 
prolapse) on 2cond day @ our target, our daughter's 12th birthday, 
school start for both kids and mostly dad work I'm back...

Do you have a V6 handy.
Will do my fingers dirty if no Intel guy beats me at it.

Greetings,
Dieter

Am 30.06.2017 14:55, schrieb Christian König:
> Hi Dieter,
> 
> thanks a lot for testing that.
> 
>> But I think my poor little FUJITSU PRIMERGY TX150 S7, Xeon X3470 
>> (Nehalem), PCIe 2.0, 24 GB is to old for this stuff...
> Well, actually you only need to figure out how to enable a PCIe window
> above the 4GB limit.
> 
> Could be that the BIOS supports this with the ACPI tables (totally
> unlikely) or you could try to dig up the Northbridge documentation for
> this CPU from Intel and use my patch for the AMD CPUs as blueprint how
> to do this on an Intel CPU as well.
> 
> Fact is you GFX hardware is perfectly capable of doing this, it's just
> that the BIOS/Motherboard didn't enabled a PCIe window per default to
> avoid problems with 32bit OSes.
> 
> Regards,
> Christian.
> 
> Am 30.06.2017 um 01:51 schrieb Dieter Nützel:
>> Hello Christian,
>> 
>> I've running this since you've sent it on-top of amd-staging-4.11. But 
>> I think my poor little FUJITSU PRIMERGY TX150 S7, Xeon X3470 
>> (Nehalem), PCIe 2.0, 24 GB is to old for this stuff...
>> 
>> [    1.066475] pci 0000:05:00.0: VF(n) BAR0 space: [mem 
>> 0x00000000-0x0003ffff 64bit] (contains BAR0 for 16 VFs)
>> [    1.066489] pci 0000:05:00.0: VF(n) BAR2 space: [mem 
>> 0x00000000-0x003fffff 64bit] (contains BAR2 for 16 VFs)
>> [    1.121656] pci 0000:00:1c.0: BAR 15: assigned [mem 
>> 0x80000000-0x801fffff 64bit pref]
>> [    1.121659] pci 0000:00:1c.6: BAR 15: assigned [mem 
>> 0x80200000-0x803fffff 64bit pref]
>> [    1.121662] pci 0000:01:00.0: BAR 6: assigned [mem 
>> 0xb0120000-0xb013ffff pref]
>> [    1.121681] pci 0000:05:00.0: BAR 6: assigned [mem 
>> 0xb0280000-0xb02fffff pref]
>> [    1.121683] pci 0000:05:00.0: BAR 9: no space for [mem size 
>> 0x00400000 64bit]
>> [    1.121684] pci 0000:05:00.0: BAR 9: failed to assign [mem size 
>> 0x00400000 64bit]
>> [    1.121685] pci 0000:05:00.0: BAR 7: no space for [mem size 
>> 0x00040000 64bit]
>> [    1.121687] pci 0000:05:00.0: BAR 7: failed to assign [mem size 
>> 0x00040000 64bit]
>> [    3.874180] amdgpu 0000:01:00.0: BAR 0: releasing [mem 
>> 0xc0000000-0xcfffffff 64bit pref]
>> [    3.874182] amdgpu 0000:01:00.0: BAR 2: releasing [mem 
>> 0xb0400000-0xb05fffff 64bit pref]
>> [    3.874198] pcieport 0000:00:03.0: BAR 15: releasing [mem 
>> 0xb0400000-0xcfffffff 64bit pref]
>> [    3.874215] pcieport 0000:00:03.0: BAR 15: no space for [mem size 
>> 0x300000000 64bit pref]
>> [    3.874217] pcieport 0000:00:03.0: BAR 15: failed to assign [mem 
>> size 0x300000000 64bit pref]
>> [    3.874221] amdgpu 0000:01:00.0: BAR 0: no space for [mem size 
>> 0x200000000 64bit pref]
>> [    3.874223] amdgpu 0000:01:00.0: BAR 0: failed to assign [mem size 
>> 0x200000000 64bit pref]
>> [    3.874226] amdgpu 0000:01:00.0: BAR 2: no space for [mem size 
>> 0x00200000 64bit pref]
>> [    3.874227] amdgpu 0000:01:00.0: BAR 2: failed to assign [mem size 
>> 0x00200000 64bit pref]
>> [    3.874258] [drm] Not enough PCI address space for a large BAR.
>> [    3.874261] amdgpu 0000:01:00.0: BAR 0: assigned [mem 
>> 0xc0000000-0xcfffffff 64bit pref]
>> [    3.874269] amdgpu 0000:01:00.0: BAR 2: assigned [mem 
>> 0xb0400000-0xb05fffff 64bit pref]
>> [    3.874288] [drm] Detected VRAM RAM=8192M, BAR=256M
>> 
>> Anyway rebase for current amd-staging-4.11 needed.
>> Find attached dmesg-amd-staging-4.11-1.g7262353-default+.log.xz
>> 
>> Greetings,
>> Dieter
>> 
>> Am 09.06.2017 10:59, schrieb Christian König:
>>> Hi everyone,
>>> 
>>> This is the fith incarnation of this set of patches. It enables 
>>> device
>>> drivers to resize and most likely also relocate the PCI BAR of 
>>> devices
>>> they manage to allow the CPU to access all of the device local memory 
>>> at once.
>>> 
>>> This is very useful for GFX device drivers where the default PCI BAR 
>>> is only
>>> about 256MB in size for compatibility reasons, but the device easily 
>>> have
>>> multiple gigabyte of local memory.
>>> 
>>> Some changes since V4:
>>> 1. Rebased on 4.11.
>>> 2. added the rb from Andy Shevchenko to patches which look complete 
>>> now.
>>> 3. Move releasing the BAR and reallocating it on error to the driver 
>>> side.
>>> 4. Add amdgpu support for GMC V6 hardware generation as well.
>>> 
>>> Please review and/or comment,
>>> Christian.
>>> 
>>> _______________________________________________
>>> dri-devel mailing list
>>> dri-devel@lists.freedesktop.org
>>> https://lists.freedesktop.org/mailman/listinfo/dri-devel
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx