* Re: [PATCH 5/5] drm/amdgpu: resize VRAM BAR for CPU access
@ 2017-03-06 21:26 sonofagun
0 siblings, 0 replies; 4+ messages in thread
From: sonofagun @ 2017-03-06 21:26 UTC (permalink / raw)
To: linux-pci
> + DRM_INFO("Not enoigh PCI address space for a large BAR.");
Please fix the typo of the message so that another patch is not needed
in the future!
^ permalink raw reply [flat|nested] 4+ messages in thread
* [PATCH 1/5] PCI: add resizeable BAR infrastructure v2
@ 2017-03-06 11:40 Christian König
2017-03-06 11:40 ` [PATCH 5/5] drm/amdgpu: resize VRAM BAR for CPU access Christian König
0 siblings, 1 reply; 4+ messages in thread
From: Christian König @ 2017-03-06 11:40 UTC (permalink / raw)
To: linux-pci, dri-devel, platform-driver-x86, amd-gfx, linux-kernel
From: Christian König <christian.koenig@amd.com>
Just the defines and helper functions to read the possible sizes of a BAR and
update it's size.
See https://pcisig.com/sites/default/files/specification_documents/ECN_Resizable-BAR_24Apr2008.pdf.
v2: provide read helper as well
Signed-off-by: Christian König <christian.koenig@amd.com>
---
drivers/pci/pci.c | 115 ++++++++++++++++++++++++++++++++++++++++++
include/linux/pci.h | 3 ++
include/uapi/linux/pci_regs.h | 7 +++
3 files changed, 125 insertions(+)
diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
index ba34907..9658aa7 100644
--- a/drivers/pci/pci.c
+++ b/drivers/pci/pci.c
@@ -2944,6 +2944,121 @@ bool pci_acs_path_enabled(struct pci_dev *start,
}
/**
+ * pci_rbar_get_sizes - get possible sizes for BAR
+ * @dev: PCI device
+ * @bar: BAR to query
+ *
+ * Get the possible sizes of a resizeable BAR as bitmask defined in the spec
+ * (bit 0=1MB, bit 19=512GB). Returns 0 if BAR isn't resizeable.
+ */
+u32 pci_rbar_get_sizes(struct pci_dev *pdev, int bar)
+{
+ int pos, nbars;
+ u32 ctrl, cap;
+ int i;
+
+ pos = pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_REBAR);
+ if (!pos)
+ return 0x0;
+
+ pci_read_config_dword(pdev, pos + PCI_REBAR_CTRL, &ctrl);
+ nbars = (ctrl & PCI_REBAR_CTRL_NBAR_MASK) >> PCI_REBAR_CTRL_NBAR_SHIFT;
+
+ for (i = 0; i < nbars; ++i, pos += 8) {
+ int bar_idx;
+
+ pci_read_config_dword(pdev, pos + PCI_REBAR_CTRL, &ctrl);
+ bar_idx = (ctrl & PCI_REBAR_CTRL_BAR_IDX_MASK) >>
+ PCI_REBAR_CTRL_BAR_IDX_SHIFT;
+ if (bar_idx != bar)
+ continue;
+
+ pci_read_config_dword(pdev, pos + PCI_REBAR_CAP, &cap);
+ return (cap & PCI_REBAR_CTRL_SIZES_MASK) >>
+ PCI_REBAR_CTRL_SIZES_SHIFT;
+ }
+
+ return 0x0;
+}
+
+/**
+ * pci_rbar_get_size - get the current size of a BAR
+ * @dev: PCI device
+ * @bar: BAR to set size to
+ *
+ * Read the size of a BAR from the resizeable BAR config.
+ * Returns size if found or negativ error code.
+ */
+int pci_rbar_get_size(struct pci_dev *pdev, int bar)
+{
+ int pos, nbars;
+ u32 ctrl;
+ int i;
+
+ pos = pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_REBAR);
+ if (!pos)
+ return -ENOTSUPP;
+
+ pci_read_config_dword(pdev, pos + PCI_REBAR_CTRL, &ctrl);
+ nbars = (ctrl & PCI_REBAR_CTRL_NBAR_MASK) >> PCI_REBAR_CTRL_NBAR_SHIFT;
+
+ for (i = 0; i < nbars; ++i, pos += 8) {
+ int bar_idx;
+
+ pci_read_config_dword(pdev, pos + PCI_REBAR_CTRL, &ctrl);
+ bar_idx = (ctrl & PCI_REBAR_CTRL_BAR_IDX_MASK) >>
+ PCI_REBAR_CTRL_BAR_IDX_SHIFT;
+ if (bar_idx != bar)
+ continue;
+
+ return (ctrl & PCI_REBAR_CTRL_BAR_SIZE_MASK) >>
+ PCI_REBAR_CTRL_BAR_SIZE_SHIFT;
+ }
+
+ return -ENOENT;
+}
+
+/**
+ * pci_rbar_set_size - set a new size for a BAR
+ * @dev: PCI device
+ * @bar: BAR to set size to
+ * @size: new size as defined in the spec.
+ *
+ * Set the new size of a BAR as defined in the spec (0=1MB, 19=512GB).
+ * Returns true if resizing was successful, false otherwise.
+ */
+bool pci_rbar_set_size(struct pci_dev *pdev, int bar, int size)
+{
+ int pos, nbars;
+ u32 ctrl;
+ int i;
+
+ pos = pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_REBAR);
+ if (!pos)
+ return false;
+
+ pci_read_config_dword(pdev, pos + PCI_REBAR_CTRL, &ctrl);
+ nbars = (ctrl & PCI_REBAR_CTRL_NBAR_MASK) >> PCI_REBAR_CTRL_NBAR_SHIFT;
+
+ for (i = 0; i < nbars; ++i, pos += 8) {
+ int bar_idx;
+
+ pci_read_config_dword(pdev, pos + PCI_REBAR_CTRL, &ctrl);
+ bar_idx = (ctrl & PCI_REBAR_CTRL_BAR_IDX_MASK) >>
+ PCI_REBAR_CTRL_BAR_IDX_SHIFT;
+ if (bar_idx != bar)
+ continue;
+
+ ctrl &= ~PCI_REBAR_CTRL_BAR_SIZE_MASK;
+ ctrl |= size << PCI_REBAR_CTRL_BAR_SIZE_SHIFT;
+ pci_write_config_dword(pdev, pos + PCI_REBAR_CTRL, ctrl);
+ return true;
+ }
+
+ return false;
+}
+
+/**
* pci_swizzle_interrupt_pin - swizzle INTx for device behind bridge
* @dev: the PCI device
* @pin: the INTx pin (1=INTA, 2=INTB, 3=INTC, 4=INTD)
diff --git a/include/linux/pci.h b/include/linux/pci.h
index a38772a..9f26ca4 100644
--- a/include/linux/pci.h
+++ b/include/linux/pci.h
@@ -1946,6 +1946,9 @@ void pci_request_acs(void);
bool pci_acs_enabled(struct pci_dev *pdev, u16 acs_flags);
bool pci_acs_path_enabled(struct pci_dev *start,
struct pci_dev *end, u16 acs_flags);
+u32 pci_rbar_get_sizes(struct pci_dev *pdev, int bar);
+int pci_rbar_get_size(struct pci_dev *pdev, int bar);
+bool pci_rbar_set_size(struct pci_dev *pdev, int bar, int size);
#define PCI_VPD_LRDT 0x80 /* Large Resource Data Type */
#define PCI_VPD_LRDT_ID(x) ((x) | PCI_VPD_LRDT)
diff --git a/include/uapi/linux/pci_regs.h b/include/uapi/linux/pci_regs.h
index e5a2e68..6de29d6 100644
--- a/include/uapi/linux/pci_regs.h
+++ b/include/uapi/linux/pci_regs.h
@@ -932,9 +932,16 @@
#define PCI_SATA_SIZEOF_LONG 16
/* Resizable BARs */
+#define PCI_REBAR_CAP 4 /* capability register */
+#define PCI_REBAR_CTRL_SIZES_MASK (0xFFFFF << 4) /* mask for sizes */
+#define PCI_REBAR_CTRL_SIZES_SHIFT 4 /* shift for sizes */
#define PCI_REBAR_CTRL 8 /* control register */
+#define PCI_REBAR_CTRL_BAR_IDX_MASK (7 << 0) /* mask for bar index */
+#define PCI_REBAR_CTRL_BAR_IDX_SHIFT 0 /* shift for bar index */
#define PCI_REBAR_CTRL_NBAR_MASK (7 << 5) /* mask for # bars */
#define PCI_REBAR_CTRL_NBAR_SHIFT 5 /* shift for # bars */
+#define PCI_REBAR_CTRL_BAR_SIZE_MASK (0x1F << 8) /* mask for bar size */
+#define PCI_REBAR_CTRL_BAR_SIZE_SHIFT 8 /* shift for bar size */
/* Dynamic Power Allocation */
#define PCI_DPA_CAP 4 /* capability register */
--
2.7.4
^ permalink raw reply related [flat|nested] 4+ messages in thread
* [PATCH 5/5] drm/amdgpu: resize VRAM BAR for CPU access
2017-03-06 11:40 [PATCH 1/5] PCI: add resizeable BAR infrastructure v2 Christian König
@ 2017-03-06 11:40 ` Christian König
2017-03-06 12:06 ` Andy Shevchenko
0 siblings, 1 reply; 4+ messages in thread
From: Christian König @ 2017-03-06 11:40 UTC (permalink / raw)
To: linux-pci, dri-devel, platform-driver-x86, amd-gfx, linux-kernel
From: Christian König <christian.koenig@amd.com>
Try to resize BAR0 to let CPU access all of VRAM.
Signed-off-by: Christian König <christian.koenig@amd.com>
---
drivers/gpu/drm/amd/amdgpu/amdgpu.h | 1 +
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 29 +++++++++++++++++++++++++++++
drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c | 8 +++++---
drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c | 8 +++++---
4 files changed, 40 insertions(+), 6 deletions(-)
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu.h b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
index 8a5f8cb..1e888d0 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
@@ -1754,6 +1754,7 @@ uint32_t amdgpu_ttm_tt_pte_flags(struct amdgpu_device *adev, struct ttm_tt *ttm,
struct ttm_mem_reg *mem);
void amdgpu_vram_location(struct amdgpu_device *adev, struct amdgpu_mc *mc, u64 base);
void amdgpu_gtt_location(struct amdgpu_device *adev, struct amdgpu_mc *mc);
+void amdgpu_resize_bar0(struct amdgpu_device *adev);
void amdgpu_ttm_set_active_vram_size(struct amdgpu_device *adev, u64 size);
int amdgpu_ttm_init(struct amdgpu_device *adev);
void amdgpu_ttm_fini(struct amdgpu_device *adev);
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
index a470869..f038195 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
@@ -616,6 +616,35 @@ void amdgpu_gtt_location(struct amdgpu_device *adev, struct amdgpu_mc *mc)
mc->gtt_size >> 20, mc->gtt_start, mc->gtt_end);
}
+/**
+ * amdgpu_resize_bar0 - try to resize BAR0
+ *
+ * @adev: amdgpu_device pointer
+ *
+ * Try to resize BAR0 to make all VRAM CPU accessible.
+ */
+void amdgpu_resize_bar0(struct amdgpu_device *adev)
+{
+ u32 size = max(ilog2(adev->mc.real_vram_size - 1) + 1, 20) - 20;
+ int r;
+
+ r = pci_resize_resource(adev->pdev, 0, size);
+
+ if (r == -ENOTSUPP) {
+ /* The hardware don't support the extension. */
+ return;
+
+ } else if (r == -ENOSPC) {
+ DRM_INFO("Not enoigh PCI address space for a large BAR.");
+ } else if (r) {
+ DRM_ERROR("Problem resizing BAR0 (%d).", r);
+ }
+
+ /* Reinit the doorbell mapping, it is most likely moved as well */
+ amdgpu_doorbell_fini(adev);
+ BUG_ON(amdgpu_doorbell_init(adev));
+}
+
/*
* GPU helpers function.
*/
diff --git a/drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c b/drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c
index 552bf6b..cd5828c 100644
--- a/drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c
@@ -367,13 +367,15 @@ static int gmc_v7_0_mc_init(struct amdgpu_device *adev)
break;
}
adev->mc.vram_width = numchan * chansize;
- /* Could aper size report 0 ? */
- adev->mc.aper_base = pci_resource_start(adev->pdev, 0);
- adev->mc.aper_size = pci_resource_len(adev->pdev, 0);
/* size in MB on si */
adev->mc.mc_vram_size = RREG32(mmCONFIG_MEMSIZE) * 1024ULL * 1024ULL;
adev->mc.real_vram_size = RREG32(mmCONFIG_MEMSIZE) * 1024ULL * 1024ULL;
+ if (!(adev->flags & AMD_IS_APU))
+ amdgpu_resize_bar0(adev);
+ adev->mc.aper_base = pci_resource_start(adev->pdev, 0);
+ adev->mc.aper_size = pci_resource_len(adev->pdev, 0);
+
#ifdef CONFIG_X86_64
if (adev->flags & AMD_IS_APU) {
adev->mc.aper_base = ((u64)RREG32(mmMC_VM_FB_OFFSET)) << 22;
diff --git a/drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c b/drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c
index f2bd016..e277130 100644
--- a/drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c
@@ -459,13 +459,15 @@ static int gmc_v8_0_mc_init(struct amdgpu_device *adev)
break;
}
adev->mc.vram_width = numchan * chansize;
- /* Could aper size report 0 ? */
- adev->mc.aper_base = pci_resource_start(adev->pdev, 0);
- adev->mc.aper_size = pci_resource_len(adev->pdev, 0);
/* size in MB on si */
adev->mc.mc_vram_size = RREG32(mmCONFIG_MEMSIZE) * 1024ULL * 1024ULL;
adev->mc.real_vram_size = RREG32(mmCONFIG_MEMSIZE) * 1024ULL * 1024ULL;
+ if (!(adev->flags & AMD_IS_APU))
+ amdgpu_resize_bar0(adev);
+ adev->mc.aper_base = pci_resource_start(adev->pdev, 0);
+ adev->mc.aper_size = pci_resource_len(adev->pdev, 0);
+
#ifdef CONFIG_X86_64
if (adev->flags & AMD_IS_APU) {
adev->mc.aper_base = ((u64)RREG32(mmMC_VM_FB_OFFSET)) << 22;
--
2.7.4
^ permalink raw reply related [flat|nested] 4+ messages in thread
* Re: [PATCH 5/5] drm/amdgpu: resize VRAM BAR for CPU access
2017-03-06 11:40 ` [PATCH 5/5] drm/amdgpu: resize VRAM BAR for CPU access Christian König
@ 2017-03-06 12:06 ` Andy Shevchenko
2017-03-06 12:34 ` Christian König
0 siblings, 1 reply; 4+ messages in thread
From: Andy Shevchenko @ 2017-03-06 12:06 UTC (permalink / raw)
To: Christian König
Cc: linux-pci, dri-devel, Platform Driver, amd-gfx, linux-kernel
On Mon, Mar 6, 2017 at 1:40 PM, Christian K=C3=B6nig <deathsimple@vodafone.=
de> wrote:
> From: Christian K=C3=B6nig <christian.koenig@amd.com>
>
> Try to resize BAR0 to let CPU access all of VRAM.
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> @@ -616,6 +616,35 @@ void amdgpu_gtt_location(struct amdgpu_device *adev,=
struct amdgpu_mc *mc)
> +void amdgpu_resize_bar0(struct amdgpu_device *adev)
> +{
> + u32 size =3D max(ilog2(adev->mc.real_vram_size - 1) + 1, 20) - 20=
;
Too complicated.
unsigned long =3D fls_long(real_vram_size | BIT(20));
And the result is not a size, right? It's a logarithm from size.
> + int r;
> +
> + r =3D pci_resize_resource(adev->pdev, 0, size);
> +
Redundant line.
> + if (r =3D=3D -ENOTSUPP) {
> + /* The hardware don't support the extension. */
> + return;
> +
> + } else if (r =3D=3D -ENOSPC) {
> + DRM_INFO("Not enoigh PCI address space for a large BAR.")=
;
> + } else if (r) {
> + DRM_ERROR("Problem resizing BAR0 (%d).", r);
> + }
> +
> + /* Reinit the doorbell mapping, it is most likely moved as well *=
/
> + amdgpu_doorbell_fini(adev);
> + BUG_ON(amdgpu_doorbell_init(adev));
No way to recover?!
> +}
> +
--=20
With Best Regards,
Andy Shevchenko
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [PATCH 5/5] drm/amdgpu: resize VRAM BAR for CPU access
2017-03-06 12:06 ` Andy Shevchenko
@ 2017-03-06 12:34 ` Christian König
0 siblings, 0 replies; 4+ messages in thread
From: Christian König @ 2017-03-06 12:34 UTC (permalink / raw)
To: Andy Shevchenko
Cc: linux-pci, dri-devel, Platform Driver, amd-gfx, linux-kernel
Am 06.03.2017 um 13:06 schrieb Andy Shevchenko:
> On Mon, Mar 6, 2017 at 1:40 PM, Christian König <deathsimple@vodafone.de> wrote:
>> From: Christian König <christian.koenig@amd.com>
>>
>> Try to resize BAR0 to let CPU access all of VRAM.
>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
>> @@ -616,6 +616,35 @@ void amdgpu_gtt_location(struct amdgpu_device *adev, struct amdgpu_mc *mc)
>> +void amdgpu_resize_bar0(struct amdgpu_device *adev)
>> +{
>> + u32 size = max(ilog2(adev->mc.real_vram_size - 1) + 1, 20) - 20;
> Too complicated.
>
> unsigned long = fls_long(real_vram_size | BIT(20));
That would round down, not up. We got boards with 6GB VRAM as well and
then need a 8GB BAR.
And the vram size won't fit into a long on 32bit systems. What I really
need is order_base_2 for 64bit values.
But wait a second, thinking more about it we could do
"order_base_2((real_vram_size >> 20) | 1)".
> And the result is not a size, right? It's a logarithm from size.
Yeah, and subtracted by 20. Thought about a better wording as well, but
couldn't come up with something.
"size" is just what the spec uses. How about rbar_size to note that it
is size as the meaning in the RBAR specification?
>
>> + int r;
>> +
>> + r = pci_resize_resource(adev->pdev, 0, size);
>> +
> Redundant line.
>
>> + if (r == -ENOTSUPP) {
>> + /* The hardware don't support the extension. */
>> + return;
>> +
>> + } else if (r == -ENOSPC) {
>> + DRM_INFO("Not enoigh PCI address space for a large BAR.");
>> + } else if (r) {
>> + DRM_ERROR("Problem resizing BAR0 (%d).", r);
>> + }
>> +
>> + /* Reinit the doorbell mapping, it is most likely moved as well */
>> + amdgpu_doorbell_fini(adev);
>> + BUG_ON(amdgpu_doorbell_init(adev));
> No way to recover?!
Nope, I actually thought about calling panic() here instead.
If we hit this we have messed things so badly up that we can't access
the hardware any more, so no way to tell it to shut down or something
like this.
Well, I could completely rewrite the call chain to signal modprobe that
loading the driver didn't worked at all. But that comes pretty near to
calling BUG_ON() as well.
Thanks for the comments,
Christian.
>
>> +}
>> +
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2017-03-06 21:51 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-03-06 21:26 [PATCH 5/5] drm/amdgpu: resize VRAM BAR for CPU access sonofagun
-- strict thread matches above, loose matches on Subject: below --
2017-03-06 11:40 [PATCH 1/5] PCI: add resizeable BAR infrastructure v2 Christian König
2017-03-06 11:40 ` [PATCH 5/5] drm/amdgpu: resize VRAM BAR for CPU access Christian König
2017-03-06 12:06 ` Andy Shevchenko
2017-03-06 12:34 ` Christian König
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).