[PATCH 1/3] drm/amdkfd: Define config HSA_AMD_P2P to support peer-to-peer

[PATCH 1/3] drm/amdkfd: Define config HSA_AMD_P2P to support peer-to-peer

[Ramesh Errabolu]

Extend current kernel config requirements of amdgpu by adding config
HSA_AMD_P2P. Enabling HSA_AMD_P2P is REQUIRED to support peer-to-peer
communication between AMD GPU devices over PCIe bus

Signed-off-by: Ramesh Errabolu <Ramesh.Errabolu@amd.com>
---
 drivers/gpu/drm/amd/amdkfd/Kconfig | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/drivers/gpu/drm/amd/amdkfd/Kconfig b/drivers/gpu/drm/amd/amdkfd/Kconfig
index 8cc0a76ddf9f..93bd4eda0d94 100644
--- a/drivers/gpu/drm/amd/amdkfd/Kconfig
+++ b/drivers/gpu/drm/amd/amdkfd/Kconfig
@@ -25,3 +25,17 @@ config HSA_AMD_SVM
 	  preemptions and one based on page faults. To enable page fault
 	  based memory management on most GFXv9 GPUs, set the module
 	  parameter amdgpu.noretry=0.
+
+config HSA_AMD_P2P
+	bool "HSA kernel driver support for peer-to-peer for AMD GPU devices"
+	depends on HSA_AMD && PCI_P2PDMA && DMABUF_MOVE_NOTIFY
+	help
+	  Enable peer-to-peer (P2P) communication between AMD GPUs over
+	  the PCIe bus. This can improve performance of multi-GPU compute
+	  applications and libraries by enabling GPUs to access data directly
+	  in peer GPUs' memory without intermediate copies in system memory.
+
+	  This P2P feature is only enabled on compatible chipsets, and between
+	  GPUs with large memory BARs that expose the entire VRAM in PCIe bus
+	  address space within the physical address limits of the GPUs.
+
-- 
2.35.1

Re: [PATCH 1/3] drm/amdkfd: Define config HSA_AMD_P2P to support peer-to-peer

[Felix Kuehling]

On 2022-06-03 06:52, Ramesh Errabolu wrote:
> Extend current kernel config requirements of amdgpu by adding config
> HSA_AMD_P2P. Enabling HSA_AMD_P2P is REQUIRED to support peer-to-peer
> communication between AMD GPU devices over PCIe bus
>
> Signed-off-by: Ramesh Errabolu <Ramesh.Errabolu@amd.com>

This patch is

Reviewed-by: Felix Kuehling <Felix.Kuehling@amd.com>


> ---
>   drivers/gpu/drm/amd/amdkfd/Kconfig | 14 ++++++++++++++
>   1 file changed, 14 insertions(+)
>
> diff --git a/drivers/gpu/drm/amd/amdkfd/Kconfig b/drivers/gpu/drm/amd/amdkfd/Kconfig
> index 8cc0a76ddf9f..93bd4eda0d94 100644
> --- a/drivers/gpu/drm/amd/amdkfd/Kconfig
> +++ b/drivers/gpu/drm/amd/amdkfd/Kconfig
> @@ -25,3 +25,17 @@ config HSA_AMD_SVM
>   	  preemptions and one based on page faults. To enable page fault
>   	  based memory management on most GFXv9 GPUs, set the module
>   	  parameter amdgpu.noretry=0.
> +
> +config HSA_AMD_P2P
> +	bool "HSA kernel driver support for peer-to-peer for AMD GPU devices"
> +	depends on HSA_AMD && PCI_P2PDMA && DMABUF_MOVE_NOTIFY
> +	help
> +	  Enable peer-to-peer (P2P) communication between AMD GPUs over
> +	  the PCIe bus. This can improve performance of multi-GPU compute
> +	  applications and libraries by enabling GPUs to access data directly
> +	  in peer GPUs' memory without intermediate copies in system memory.
> +
> +	  This P2P feature is only enabled on compatible chipsets, and between
> +	  GPUs with large memory BARs that expose the entire VRAM in PCIe bus
> +	  address space within the physical address limits of the GPUs.
> +

RE: [PATCH 1/3] drm/amdkfd: Define config HSA_AMD_P2P to support peer-to-peer

[Errabolu, Ramesh]

[AMD Official Use Only - General]

Update the both the description and commit message per review comments.
Will post new patch shortly.

Regards,
Ramesh

-----Original Message-----
From: Kuehling, Felix <Felix.Kuehling@amd.com> 
Sent: Wednesday, June 1, 2022 9:15 PM
To: Errabolu, Ramesh <Ramesh.Errabolu@amd.com>; amd-gfx@lists.freedesktop.org
Subject: Re: [PATCH 1/3] drm/amdkfd: Define config HSA_AMD_P2P to support peer-to-peer

Am 2022-05-31 um 13:02 schrieb Ramesh Errabolu:
> Extend current kernel config requirements of amdgpu by adding config 
> HSA_AMD_P2P. Enabling HSA_AMD_P2P is REQUIRED to support peer-to-peer 
> communication, in both data and control planes, among AMD GPU devices 
> that are connected via PCIe and have large BAR vBIOS
>
> Signed-off-by: Ramesh Errabolu <Ramesh.Errabolu@amd.com>
> ---
>   drivers/gpu/drm/amd/amdkfd/Kconfig | 8 ++++++++
>   1 file changed, 8 insertions(+)
>
> diff --git a/drivers/gpu/drm/amd/amdkfd/Kconfig 
> b/drivers/gpu/drm/amd/amdkfd/Kconfig
> index 8cc0a76ddf9f..26614f5f20ea 100644
> --- a/drivers/gpu/drm/amd/amdkfd/Kconfig
> +++ b/drivers/gpu/drm/amd/amdkfd/Kconfig
> @@ -25,3 +25,11 @@ config HSA_AMD_SVM
>   	  preemptions and one based on page faults. To enable page fault
>   	  based memory management on most GFXv9 GPUs, set the module
>   	  parameter amdgpu.noretry=0.
> +
> +config HSA_AMD_P2P
> +	bool "HSA kernel driver support for peer-to-peer for AMD GPU devices"
> +	depends on HSA_AMD && PCI_P2PDMA && DMABUF_MOVE_NOTIFY
> +	help
> +	  Enable this if you want to access AMD GPU peer devices, in both data
> +	  and control planes, that are connected via PCIe and have large BAR 
> +vBIOS

I have not seen the terms "data plane" and "control plane" used in the context of GPUs. As far as I can tell, this terminology is more common in the context network routing. I think it could cause confusion to introduce these terms without an explanation to users.

The sentence "... if you want to access AMD GPU peer devices ..." seems to address someone writing an application. This help message is meant for users and admins building a kernel, who may want to run compute applications, not for compute application developers.

I would also not mention large-BAR VBIOSes because the BAR can often be resized even with a small-BAR VBIOS.

Therefore I would recommend an alternative text here that avoids uncommon terminology and addresses the concerns of users rather than application developers:

    Enable peer-to-peer (P2P) communication between AMD GPUs over the
    PCIe bus. This can improve performance of multi-GPU compute
    applications and libraries by enabling GPUs to access data directly
    in peer GPUs' memory without intermediate copies in system memory.

    This P2P feature is only enabled on compatible chipsets, and between
    GPUs with large memory BARs that expose the entire VRAM in PCI bus
    address space within the physical address limits of the GPUs.

Regards,
   Felix


> +

Re: [PATCH 1/3] drm/amdkfd: Define config HSA_AMD_P2P to support peer-to-peer

[Felix Kuehling]

Am 2022-05-31 um 13:02 schrieb Ramesh Errabolu:
> Extend current kernel config requirements of amdgpu by adding
> config HSA_AMD_P2P. Enabling HSA_AMD_P2P is REQUIRED to support
> peer-to-peer communication, in both data and control planes, among
> AMD GPU devices that are connected via PCIe and have large BAR vBIOS
>
> Signed-off-by: Ramesh Errabolu <Ramesh.Errabolu@amd.com>
> ---
>   drivers/gpu/drm/amd/amdkfd/Kconfig | 8 ++++++++
>   1 file changed, 8 insertions(+)
>
> diff --git a/drivers/gpu/drm/amd/amdkfd/Kconfig b/drivers/gpu/drm/amd/amdkfd/Kconfig
> index 8cc0a76ddf9f..26614f5f20ea 100644
> --- a/drivers/gpu/drm/amd/amdkfd/Kconfig
> +++ b/drivers/gpu/drm/amd/amdkfd/Kconfig
> @@ -25,3 +25,11 @@ config HSA_AMD_SVM
>   	  preemptions and one based on page faults. To enable page fault
>   	  based memory management on most GFXv9 GPUs, set the module
>   	  parameter amdgpu.noretry=0.
> +
> +config HSA_AMD_P2P
> +	bool "HSA kernel driver support for peer-to-peer for AMD GPU devices"
> +	depends on HSA_AMD && PCI_P2PDMA && DMABUF_MOVE_NOTIFY
> +	help
> +	  Enable this if you want to access AMD GPU peer devices, in both data
> +	  and control planes, that are connected via PCIe and have large BAR vBIOS

I have not seen the terms "data plane" and "control plane" used in the 
context of GPUs. As far as I can tell, this terminology is more common 
in the context network routing. I think it could cause confusion to 
introduce these terms without an explanation to users.

The sentence "... if you want to access AMD GPU peer devices ..." seems 
to address someone writing an application. This help message is meant 
for users and admins building a kernel, who may want to run compute 
applications, not for compute application developers.

I would also not mention large-BAR VBIOSes because the BAR can often be 
resized even with a small-BAR VBIOS.

Therefore I would recommend an alternative text here that avoids 
uncommon terminology and addresses the concerns of users rather than 
application developers:

    Enable peer-to-peer (P2P) communication between AMD GPUs over the
    PCIe bus. This can improve performance of multi-GPU compute
    applications and libraries by enabling GPUs to access data directly
    in peer GPUs' memory without intermediate copies in system memory.

    This P2P feature is only enabled on compatible chipsets, and between
    GPUs with large memory BARs that expose the entire VRAM in PCI bus
    address space within the physical address limits of the GPUs.

Regards,
   Felix


> +

[PATCH 1/3] drm/amdkfd: Define config HSA_AMD_P2P to support peer-to-peer

[Ramesh Errabolu]

Extend current kernel config requirements of amdgpu by adding
config HSA_AMD_P2P. Enabling HSA_AMD_P2P is REQUIRED to support
peer-to-peer communication, in both data and control planes, among
AMD GPU devices that are connected via PCIe and have large BAR vBIOS

Signed-off-by: Ramesh Errabolu <Ramesh.Errabolu@amd.com>
---
 drivers/gpu/drm/amd/amdkfd/Kconfig | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/drivers/gpu/drm/amd/amdkfd/Kconfig b/drivers/gpu/drm/amd/amdkfd/Kconfig
index 8cc0a76ddf9f..26614f5f20ea 100644
--- a/drivers/gpu/drm/amd/amdkfd/Kconfig
+++ b/drivers/gpu/drm/amd/amdkfd/Kconfig
@@ -25,3 +25,11 @@ config HSA_AMD_SVM
 	  preemptions and one based on page faults. To enable page fault
 	  based memory management on most GFXv9 GPUs, set the module
 	  parameter amdgpu.noretry=0.
+
+config HSA_AMD_P2P
+	bool "HSA kernel driver support for peer-to-peer for AMD GPU devices"
+	depends on HSA_AMD && PCI_P2PDMA && DMABUF_MOVE_NOTIFY 
+	help
+	  Enable this if you want to access AMD GPU peer devices, in both data
+	  and control planes, that are connected via PCIe and have large BAR vBIOS
+
-- 
2.35.1