[PATCH 0/6] RFC Support hot device unplug in amdgpu

[PATCH 0/6] RFC Support hot device unplug in amdgpu

[Andrey Grodzovsky]

This RFC is a more of a proof of concept then a fully working solution as there are a few unresolved issues we are hopping to get advise on from people on the mailing list.
Until now extracting a card either by physical extraction (e.g. eGPU with thunderbold connection or by emulation through syfs -> /sys/bus/pci/devices/device_id/remove) 
would cause random crashes in user apps. The random crashes in apps were mostly due to the app having mapped a device backed BO into it's adress space was still 
trying to access the BO while the backing device was gone. 
To answer this first problem Christian suggested to fix the handling of mapped memory in the clients when the device goes away by forcibly unmap all buffers 
the user processes has by clearing their respective VMAs mapping the device BOs. Then when the VMAs try to fill in the page tables again we check in the fault handler 
if the device is removed and if so, return an error. This will generate a SIGBUS to the application which can then cleanly terminate. 
This indeed was done but this in turn created a problem of kernel OOPs were the OOPSes were due to the fact that while the app was terminating because of the SIGBUS 
it would trigger use after free in the driver by calling to accesses device structures that were already released from the pci remove sequence. 
This we handled by introducing a 'flush' seqence during device removal were we wait for drm file reference to drop to 0 meaning all user clients directly using this device terminated. 
With this I was able to cleanly emulate device unplug with X and glxgears running and later emulate device plug back and restart of X and glxgears.

But this use case is only partial and as I see it all the use cases are as follwing and the questions it raises.

1) Application accesses a BO by opening drm file
	1.1) BO is mapped into applications address space (BO is CPU visible) - this one we have a solution for by invaldating BO's CPU mapping casuing SIGBUS 
	     and termination and waiting for drm file refcound to drop to 0 before releasing the device
	1.2) BO is not mapped into applcation address space (BO is CPU invisible) - no solution yet because how we force the application to terminate in this case ?

2) Application accesses a BO by importing a DMA-BUF
	2.1)  BO is mapped into applications address space (BO is CPU visible) - solution is same as 1.1 but instead of waiting for drm file release we wait for the 
	      imported dma-buf's file release
	2.2)  BO is not mapped into applcation address space (BO is CPU invisible) - our solution is to invalidate GPUVM page tables and destroy backing storage for 
              all exported BOs which will in turn casue VM faults in the importing device and then when the importing driver will try to re-attach the imported BO to 
	      update mappings we return -ENODEV in the import hook which hopeffuly will cause the user app to terminate.

3) Applcation opens a drm file or imports a dma-bud and holds a reference but never access any BO or does access but never more after device was unplug - how would we 
   force this applcation to termiante before proceeding with device removal code ? Otherwise the wait in pci remove just hangs for ever.

The attached patches adress 1.1, 2.1 and 2.2, for now only 1.1 fully tested and I am still testing the others but I will be happy for any advise on all the 
described use cases and maybe some alternative and better (more generic) approach to this like maybe obtaining PIDs of relevant processes through some revere 
mapping from device file and exported dma-buf files and send them SIGKILL - would this make more sense or any other method ? 

Patches 1-3 address 1.1
Patch 4 addresses 2.1
Pathces 5-6 address 2.2

Reference: https://gitlab.freedesktop.org/drm/amd/-/issues/1081

Andrey Grodzovsky (6):
  drm/ttm: Add unampping of the entire device address space
  drm/amdgpu: Force unmap all user VMAs on device removal.
  drm/amdgpu: Wait for all user clients
  drm/amdgpu: Wait for all clients importing out dma-bufs.
  drm/ttm: Add destroy flag in TTM BO eviction interface
  drm/amdgpu: Use TTM MMs destroy interface

 drivers/gpu/drm/amd/amdgpu/amdgpu.h         |  3 ++
 drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c |  2 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c  |  7 +++-
 drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c | 27 ++++++++++++-
 drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c     | 22 ++++++++--
 drivers/gpu/drm/amd/amdgpu/amdgpu_job.c     |  9 +++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c     |  4 ++
 drivers/gpu/drm/amd/amdgpu/amdgpu_object.c  | 17 +++++++-
 drivers/gpu/drm/amd/amdgpu/amdgpu_object.h  |  1 +
 drivers/gpu/drm/nouveau/nouveau_drm.c       |  2 +-
 drivers/gpu/drm/qxl/qxl_object.c            |  4 +-
 drivers/gpu/drm/radeon/radeon_object.c      |  2 +-
 drivers/gpu/drm/ttm/ttm_bo.c                | 63 +++++++++++++++++++++--------
 drivers/gpu/drm/vmwgfx/vmwgfx_drv.c         |  6 +--
 include/drm/ttm/ttm_bo_api.h                |  2 +-
 include/drm/ttm/ttm_bo_driver.h             |  2 +
 16 files changed, 139 insertions(+), 34 deletions(-)

-- 
2.7.4

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

[PATCH 0/6] RFC Support hot device unplug in amdgpu

[Andrey Grodzovsky]

This RFC is a more of a proof of concept then a fully working solution as there are a few unresolved issues we are hopping to get advise on from people on the mailing list.
Until now extracting a card either by physical extraction (e.g. eGPU with thunderbold connection or by emulation through syfs -> /sys/bus/pci/devices/device_id/remove) 
would cause random crashes in user apps. The random crashes in apps were mostly due to the app having mapped a device backed BO into it's adress space was still 
trying to access the BO while the backing device was gone. 
To answer this first problem Christian suggested to fix the handling of mapped memory in the clients when the device goes away by forcibly unmap all buffers 
the user processes has by clearing their respective VMAs mapping the device BOs. Then when the VMAs try to fill in the page tables again we check in the fault handler 
if the device is removed and if so, return an error. This will generate a SIGBUS to the application which can then cleanly terminate. 
This indeed was done but this in turn created a problem of kernel OOPs were the OOPSes were due to the fact that while the app was terminating because of the SIGBUS 
it would trigger use after free in the driver by calling to accesses device structures that were already released from the pci remove sequence. 
This we handled by introducing a 'flush' seqence during device removal were we wait for drm file reference to drop to 0 meaning all user clients directly using this device terminated. 
With this I was able to cleanly emulate device unplug with X and glxgears running and later emulate device plug back and restart of X and glxgears.

But this use case is only partial and as I see it all the use cases are as follwing and the questions it raises.

1) Application accesses a BO by opening drm file
	1.1) BO is mapped into applications address space (BO is CPU visible) - this one we have a solution for by invaldating BO's CPU mapping casuing SIGBUS 
	     and termination and waiting for drm file refcound to drop to 0 before releasing the device
	1.2) BO is not mapped into applcation address space (BO is CPU invisible) - no solution yet because how we force the application to terminate in this case ?

2) Application accesses a BO by importing a DMA-BUF
	2.1)  BO is mapped into applications address space (BO is CPU visible) - solution is same as 1.1 but instead of waiting for drm file release we wait for the 
	      imported dma-buf's file release
	2.2)  BO is not mapped into applcation address space (BO is CPU invisible) - our solution is to invalidate GPUVM page tables and destroy backing storage for 
              all exported BOs which will in turn casue VM faults in the importing device and then when the importing driver will try to re-attach the imported BO to 
	      update mappings we return -ENODEV in the import hook which hopeffuly will cause the user app to terminate.

3) Applcation opens a drm file or imports a dma-bud and holds a reference but never access any BO or does access but never more after device was unplug - how would we 
   force this applcation to termiante before proceeding with device removal code ? Otherwise the wait in pci remove just hangs for ever.

The attached patches adress 1.1, 2.1 and 2.2, for now only 1.1 fully tested and I am still testing the others but I will be happy for any advise on all the 
described use cases and maybe some alternative and better (more generic) approach to this like maybe obtaining PIDs of relevant processes through some revere 
mapping from device file and exported dma-buf files and send them SIGKILL - would this make more sense or any other method ? 

Patches 1-3 address 1.1
Patch 4 addresses 2.1
Pathces 5-6 address 2.2

Reference: https://gitlab.freedesktop.org/drm/amd/-/issues/1081

Andrey Grodzovsky (6):
  drm/ttm: Add unampping of the entire device address space
  drm/amdgpu: Force unmap all user VMAs on device removal.
  drm/amdgpu: Wait for all user clients
  drm/amdgpu: Wait for all clients importing out dma-bufs.
  drm/ttm: Add destroy flag in TTM BO eviction interface
  drm/amdgpu: Use TTM MMs destroy interface

 drivers/gpu/drm/amd/amdgpu/amdgpu.h         |  3 ++
 drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c |  2 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c  |  7 +++-
 drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c | 27 ++++++++++++-
 drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c     | 22 ++++++++--
 drivers/gpu/drm/amd/amdgpu/amdgpu_job.c     |  9 +++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c     |  4 ++
 drivers/gpu/drm/amd/amdgpu/amdgpu_object.c  | 17 +++++++-
 drivers/gpu/drm/amd/amdgpu/amdgpu_object.h  |  1 +
 drivers/gpu/drm/nouveau/nouveau_drm.c       |  2 +-
 drivers/gpu/drm/qxl/qxl_object.c            |  4 +-
 drivers/gpu/drm/radeon/radeon_object.c      |  2 +-
 drivers/gpu/drm/ttm/ttm_bo.c                | 63 +++++++++++++++++++++--------
 drivers/gpu/drm/vmwgfx/vmwgfx_drv.c         |  6 +--
 include/drm/ttm/ttm_bo_api.h                |  2 +-
 include/drm/ttm/ttm_bo_driver.h             |  2 +
 16 files changed, 139 insertions(+), 34 deletions(-)

-- 
2.7.4

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

[PATCH 1/6] drm/ttm: Add unampping of the entire device address space

[Andrey Grodzovsky]

Signed-off-by: Andrey Grodzovsky <andrey.grodzovsky@amd.com>
---
 drivers/gpu/drm/ttm/ttm_bo.c    | 22 +++++++++++++++++++++-
 include/drm/ttm/ttm_bo_driver.h |  2 ++
 2 files changed, 23 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/ttm/ttm_bo.c b/drivers/gpu/drm/ttm/ttm_bo.c
index c5b516f..eae61cc 100644
--- a/drivers/gpu/drm/ttm/ttm_bo.c
+++ b/drivers/gpu/drm/ttm/ttm_bo.c
@@ -1750,9 +1750,29 @@ void ttm_bo_unmap_virtual(struct ttm_buffer_object *bo)
 	ttm_bo_unmap_virtual_locked(bo);
 	ttm_mem_io_unlock(man);
 }
+EXPORT_SYMBOL(ttm_bo_unmap_virtual);
 
+void ttm_bo_unmap_virtual_address_space(struct ttm_bo_device *bdev)
+{
+	struct ttm_mem_type_manager *man;
+	int i;
 
-EXPORT_SYMBOL(ttm_bo_unmap_virtual);
+	for (i = 0; i < TTM_NUM_MEM_TYPES; i++) {
+		man = &bdev->man[i];
+		if (man->has_type && man->use_type)
+			ttm_mem_io_lock(man, false);
+	}
+
+	unmap_mapping_range(bdev->dev_mapping, 0, 0 , 1);
+	/*TODO What about ttm_mem_io_free_vm(bo) ? */
+
+	for (i = TTM_NUM_MEM_TYPES - 1; i >= 0; i--) {
+		man = &bdev->man[i];
+		if (man->has_type && man->use_type)
+			ttm_mem_io_unlock(man);
+	}
+}
+EXPORT_SYMBOL(ttm_bo_unmap_virtual_address_space);
 
 int ttm_bo_wait(struct ttm_buffer_object *bo,
 		bool interruptible, bool no_wait)
diff --git a/include/drm/ttm/ttm_bo_driver.h b/include/drm/ttm/ttm_bo_driver.h
index c9e0fd0..3133463 100644
--- a/include/drm/ttm/ttm_bo_driver.h
+++ b/include/drm/ttm/ttm_bo_driver.h
@@ -600,6 +600,8 @@ int ttm_bo_device_init(struct ttm_bo_device *bdev,
  */
 void ttm_bo_unmap_virtual(struct ttm_buffer_object *bo);
 
+void ttm_bo_unmap_virtual_address_space(struct ttm_bo_device *bdev);
+
 /**
  * ttm_bo_unmap_virtual
  *
-- 
2.7.4

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

[PATCH 1/6] drm/ttm: Add unampping of the entire device address space

[Andrey Grodzovsky]

Signed-off-by: Andrey Grodzovsky <andrey.grodzovsky@amd.com>
---
 drivers/gpu/drm/ttm/ttm_bo.c    | 22 +++++++++++++++++++++-
 include/drm/ttm/ttm_bo_driver.h |  2 ++
 2 files changed, 23 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/ttm/ttm_bo.c b/drivers/gpu/drm/ttm/ttm_bo.c
index c5b516f..eae61cc 100644
--- a/drivers/gpu/drm/ttm/ttm_bo.c
+++ b/drivers/gpu/drm/ttm/ttm_bo.c
@@ -1750,9 +1750,29 @@ void ttm_bo_unmap_virtual(struct ttm_buffer_object *bo)
 	ttm_bo_unmap_virtual_locked(bo);
 	ttm_mem_io_unlock(man);
 }
+EXPORT_SYMBOL(ttm_bo_unmap_virtual);
 
+void ttm_bo_unmap_virtual_address_space(struct ttm_bo_device *bdev)
+{
+	struct ttm_mem_type_manager *man;
+	int i;
 
-EXPORT_SYMBOL(ttm_bo_unmap_virtual);
+	for (i = 0; i < TTM_NUM_MEM_TYPES; i++) {
+		man = &bdev->man[i];
+		if (man->has_type && man->use_type)
+			ttm_mem_io_lock(man, false);
+	}
+
+	unmap_mapping_range(bdev->dev_mapping, 0, 0 , 1);
+	/*TODO What about ttm_mem_io_free_vm(bo) ? */
+
+	for (i = TTM_NUM_MEM_TYPES - 1; i >= 0; i--) {
+		man = &bdev->man[i];
+		if (man->has_type && man->use_type)
+			ttm_mem_io_unlock(man);
+	}
+}
+EXPORT_SYMBOL(ttm_bo_unmap_virtual_address_space);
 
 int ttm_bo_wait(struct ttm_buffer_object *bo,
 		bool interruptible, bool no_wait)
diff --git a/include/drm/ttm/ttm_bo_driver.h b/include/drm/ttm/ttm_bo_driver.h
index c9e0fd0..3133463 100644
--- a/include/drm/ttm/ttm_bo_driver.h
+++ b/include/drm/ttm/ttm_bo_driver.h
@@ -600,6 +600,8 @@ int ttm_bo_device_init(struct ttm_bo_device *bdev,
  */
 void ttm_bo_unmap_virtual(struct ttm_buffer_object *bo);
 
+void ttm_bo_unmap_virtual_address_space(struct ttm_bo_device *bdev);
+
 /**
  * ttm_bo_unmap_virtual
  *
-- 
2.7.4

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH 1/6] drm/ttm: Add unampping of the entire device address space

[Christian König]

Am 09.05.20 um 20:51 schrieb Andrey Grodzovsky:
> Signed-off-by: Andrey Grodzovsky <andrey.grodzovsky@amd.com>
> ---
>   drivers/gpu/drm/ttm/ttm_bo.c    | 22 +++++++++++++++++++++-
>   include/drm/ttm/ttm_bo_driver.h |  2 ++
>   2 files changed, 23 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/gpu/drm/ttm/ttm_bo.c b/drivers/gpu/drm/ttm/ttm_bo.c
> index c5b516f..eae61cc 100644
> --- a/drivers/gpu/drm/ttm/ttm_bo.c
> +++ b/drivers/gpu/drm/ttm/ttm_bo.c
> @@ -1750,9 +1750,29 @@ void ttm_bo_unmap_virtual(struct ttm_buffer_object *bo)
>   	ttm_bo_unmap_virtual_locked(bo);
>   	ttm_mem_io_unlock(man);
>   }
> +EXPORT_SYMBOL(ttm_bo_unmap_virtual);
>   
> +void ttm_bo_unmap_virtual_address_space(struct ttm_bo_device *bdev)
> +{
> +	struct ttm_mem_type_manager *man;
> +	int i;
>   
> -EXPORT_SYMBOL(ttm_bo_unmap_virtual);

> +	for (i = 0; i < TTM_NUM_MEM_TYPES; i++) {
> +		man = &bdev->man[i];
> +		if (man->has_type && man->use_type)
> +			ttm_mem_io_lock(man, false);
> +	}

You should drop that it will just result in a deadlock warning for 
Nouveau and has no effect at all.

Apart from that looks good to me,
Christian.

> +
> +	unmap_mapping_range(bdev->dev_mapping, 0, 0 , 1);
> +	/*TODO What about ttm_mem_io_free_vm(bo) ? */
> +
> +	for (i = TTM_NUM_MEM_TYPES - 1; i >= 0; i--) {
> +		man = &bdev->man[i];
> +		if (man->has_type && man->use_type)
> +			ttm_mem_io_unlock(man);
> +	}
> +}
> +EXPORT_SYMBOL(ttm_bo_unmap_virtual_address_space);
>   
>   int ttm_bo_wait(struct ttm_buffer_object *bo,
>   		bool interruptible, bool no_wait)
> diff --git a/include/drm/ttm/ttm_bo_driver.h b/include/drm/ttm/ttm_bo_driver.h
> index c9e0fd0..3133463 100644
> --- a/include/drm/ttm/ttm_bo_driver.h
> +++ b/include/drm/ttm/ttm_bo_driver.h
> @@ -600,6 +600,8 @@ int ttm_bo_device_init(struct ttm_bo_device *bdev,
>    */
>   void ttm_bo_unmap_virtual(struct ttm_buffer_object *bo);
>   
> +void ttm_bo_unmap_virtual_address_space(struct ttm_bo_device *bdev);
> +
>   /**
>    * ttm_bo_unmap_virtual
>    *

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [PATCH 1/6] drm/ttm: Add unampping of the entire device address space

[Christian König]

Am 09.05.20 um 20:51 schrieb Andrey Grodzovsky:
> Signed-off-by: Andrey Grodzovsky <andrey.grodzovsky@amd.com>
> ---
>   drivers/gpu/drm/ttm/ttm_bo.c    | 22 +++++++++++++++++++++-
>   include/drm/ttm/ttm_bo_driver.h |  2 ++
>   2 files changed, 23 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/gpu/drm/ttm/ttm_bo.c b/drivers/gpu/drm/ttm/ttm_bo.c
> index c5b516f..eae61cc 100644
> --- a/drivers/gpu/drm/ttm/ttm_bo.c
> +++ b/drivers/gpu/drm/ttm/ttm_bo.c
> @@ -1750,9 +1750,29 @@ void ttm_bo_unmap_virtual(struct ttm_buffer_object *bo)
>   	ttm_bo_unmap_virtual_locked(bo);
>   	ttm_mem_io_unlock(man);
>   }
> +EXPORT_SYMBOL(ttm_bo_unmap_virtual);
>   
> +void ttm_bo_unmap_virtual_address_space(struct ttm_bo_device *bdev)
> +{
> +	struct ttm_mem_type_manager *man;
> +	int i;
>   
> -EXPORT_SYMBOL(ttm_bo_unmap_virtual);

> +	for (i = 0; i < TTM_NUM_MEM_TYPES; i++) {
> +		man = &bdev->man[i];
> +		if (man->has_type && man->use_type)
> +			ttm_mem_io_lock(man, false);
> +	}

You should drop that it will just result in a deadlock warning for 
Nouveau and has no effect at all.

Apart from that looks good to me,
Christian.

> +
> +	unmap_mapping_range(bdev->dev_mapping, 0, 0 , 1);
> +	/*TODO What about ttm_mem_io_free_vm(bo) ? */
> +
> +	for (i = TTM_NUM_MEM_TYPES - 1; i >= 0; i--) {
> +		man = &bdev->man[i];
> +		if (man->has_type && man->use_type)
> +			ttm_mem_io_unlock(man);
> +	}
> +}
> +EXPORT_SYMBOL(ttm_bo_unmap_virtual_address_space);
>   
>   int ttm_bo_wait(struct ttm_buffer_object *bo,
>   		bool interruptible, bool no_wait)
> diff --git a/include/drm/ttm/ttm_bo_driver.h b/include/drm/ttm/ttm_bo_driver.h
> index c9e0fd0..3133463 100644
> --- a/include/drm/ttm/ttm_bo_driver.h
> +++ b/include/drm/ttm/ttm_bo_driver.h
> @@ -600,6 +600,8 @@ int ttm_bo_device_init(struct ttm_bo_device *bdev,
>    */
>   void ttm_bo_unmap_virtual(struct ttm_buffer_object *bo);
>   
> +void ttm_bo_unmap_virtual_address_space(struct ttm_bo_device *bdev);
> +
>   /**
>    * ttm_bo_unmap_virtual
>    *

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH 1/6] drm/ttm: Add unampping of the entire device address space

[Andrey Grodzovsky]

On 5/11/20 2:45 AM, Christian König wrote:
> Am 09.05.20 um 20:51 schrieb Andrey Grodzovsky:
>> Signed-off-by: Andrey Grodzovsky <andrey.grodzovsky@amd.com>
>> ---
>>   drivers/gpu/drm/ttm/ttm_bo.c    | 22 +++++++++++++++++++++-
>>   include/drm/ttm/ttm_bo_driver.h |  2 ++
>>   2 files changed, 23 insertions(+), 1 deletion(-)
>>
>> diff --git a/drivers/gpu/drm/ttm/ttm_bo.c b/drivers/gpu/drm/ttm/ttm_bo.c
>> index c5b516f..eae61cc 100644
>> --- a/drivers/gpu/drm/ttm/ttm_bo.c
>> +++ b/drivers/gpu/drm/ttm/ttm_bo.c
>> @@ -1750,9 +1750,29 @@ void ttm_bo_unmap_virtual(struct 
>> ttm_buffer_object *bo)
>>       ttm_bo_unmap_virtual_locked(bo);
>>       ttm_mem_io_unlock(man);
>>   }
>> +EXPORT_SYMBOL(ttm_bo_unmap_virtual);
>>   +void ttm_bo_unmap_virtual_address_space(struct ttm_bo_device *bdev)
>> +{
>> +    struct ttm_mem_type_manager *man;
>> +    int i;
>>   -EXPORT_SYMBOL(ttm_bo_unmap_virtual);
>
>> +    for (i = 0; i < TTM_NUM_MEM_TYPES; i++) {
>> +        man = &bdev->man[i];
>> +        if (man->has_type && man->use_type)
>> +            ttm_mem_io_lock(man, false);
>> +    }
>
> You should drop that it will just result in a deadlock warning for 
> Nouveau and has no effect at all.
>
> Apart from that looks good to me,
> Christian.


As I am considering to re-include this in V2 of the patchsets, can you 
clarify please why this will have no effect at all ?

Andrey


>
>> +
>> +    unmap_mapping_range(bdev->dev_mapping, 0, 0 , 1);
>> +    /*TODO What about ttm_mem_io_free_vm(bo) ? */
>> +
>> +    for (i = TTM_NUM_MEM_TYPES - 1; i >= 0; i--) {
>> +        man = &bdev->man[i];
>> +        if (man->has_type && man->use_type)
>> +            ttm_mem_io_unlock(man);
>> +    }
>> +}
>> +EXPORT_SYMBOL(ttm_bo_unmap_virtual_address_space);
>>     int ttm_bo_wait(struct ttm_buffer_object *bo,
>>           bool interruptible, bool no_wait)
>> diff --git a/include/drm/ttm/ttm_bo_driver.h 
>> b/include/drm/ttm/ttm_bo_driver.h
>> index c9e0fd0..3133463 100644
>> --- a/include/drm/ttm/ttm_bo_driver.h
>> +++ b/include/drm/ttm/ttm_bo_driver.h
>> @@ -600,6 +600,8 @@ int ttm_bo_device_init(struct ttm_bo_device *bdev,
>>    */
>>   void ttm_bo_unmap_virtual(struct ttm_buffer_object *bo);
>>   +void ttm_bo_unmap_virtual_address_space(struct ttm_bo_device *bdev);
>> +
>>   /**
>>    * ttm_bo_unmap_virtual
>>    *
>
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [PATCH 1/6] drm/ttm: Add unampping of the entire device address space

[Andrey Grodzovsky]

On 5/11/20 2:45 AM, Christian König wrote:
> Am 09.05.20 um 20:51 schrieb Andrey Grodzovsky:
>> Signed-off-by: Andrey Grodzovsky <andrey.grodzovsky@amd.com>
>> ---
>>   drivers/gpu/drm/ttm/ttm_bo.c    | 22 +++++++++++++++++++++-
>>   include/drm/ttm/ttm_bo_driver.h |  2 ++
>>   2 files changed, 23 insertions(+), 1 deletion(-)
>>
>> diff --git a/drivers/gpu/drm/ttm/ttm_bo.c b/drivers/gpu/drm/ttm/ttm_bo.c
>> index c5b516f..eae61cc 100644
>> --- a/drivers/gpu/drm/ttm/ttm_bo.c
>> +++ b/drivers/gpu/drm/ttm/ttm_bo.c
>> @@ -1750,9 +1750,29 @@ void ttm_bo_unmap_virtual(struct 
>> ttm_buffer_object *bo)
>>       ttm_bo_unmap_virtual_locked(bo);
>>       ttm_mem_io_unlock(man);
>>   }
>> +EXPORT_SYMBOL(ttm_bo_unmap_virtual);
>>   +void ttm_bo_unmap_virtual_address_space(struct ttm_bo_device *bdev)
>> +{
>> +    struct ttm_mem_type_manager *man;
>> +    int i;
>>   -EXPORT_SYMBOL(ttm_bo_unmap_virtual);
>
>> +    for (i = 0; i < TTM_NUM_MEM_TYPES; i++) {
>> +        man = &bdev->man[i];
>> +        if (man->has_type && man->use_type)
>> +            ttm_mem_io_lock(man, false);
>> +    }
>
> You should drop that it will just result in a deadlock warning for 
> Nouveau and has no effect at all.
>
> Apart from that looks good to me,
> Christian.


As I am considering to re-include this in V2 of the patchsets, can you 
clarify please why this will have no effect at all ?

Andrey


>
>> +
>> +    unmap_mapping_range(bdev->dev_mapping, 0, 0 , 1);
>> +    /*TODO What about ttm_mem_io_free_vm(bo) ? */
>> +
>> +    for (i = TTM_NUM_MEM_TYPES - 1; i >= 0; i--) {
>> +        man = &bdev->man[i];
>> +        if (man->has_type && man->use_type)
>> +            ttm_mem_io_unlock(man);
>> +    }
>> +}
>> +EXPORT_SYMBOL(ttm_bo_unmap_virtual_address_space);
>>     int ttm_bo_wait(struct ttm_buffer_object *bo,
>>           bool interruptible, bool no_wait)
>> diff --git a/include/drm/ttm/ttm_bo_driver.h 
>> b/include/drm/ttm/ttm_bo_driver.h
>> index c9e0fd0..3133463 100644
>> --- a/include/drm/ttm/ttm_bo_driver.h
>> +++ b/include/drm/ttm/ttm_bo_driver.h
>> @@ -600,6 +600,8 @@ int ttm_bo_device_init(struct ttm_bo_device *bdev,
>>    */
>>   void ttm_bo_unmap_virtual(struct ttm_buffer_object *bo);
>>   +void ttm_bo_unmap_virtual_address_space(struct ttm_bo_device *bdev);
>> +
>>   /**
>>    * ttm_bo_unmap_virtual
>>    *
>
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH 1/6] drm/ttm: Add unampping of the entire device address space

[Christian König]

Am 05.06.20 um 16:29 schrieb Andrey Grodzovsky:
>
> On 5/11/20 2:45 AM, Christian König wrote:
>> Am 09.05.20 um 20:51 schrieb Andrey Grodzovsky:
>>> Signed-off-by: Andrey Grodzovsky <andrey.grodzovsky@amd.com>
>>> ---
>>>   drivers/gpu/drm/ttm/ttm_bo.c    | 22 +++++++++++++++++++++-
>>>   include/drm/ttm/ttm_bo_driver.h |  2 ++
>>>   2 files changed, 23 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/drivers/gpu/drm/ttm/ttm_bo.c 
>>> b/drivers/gpu/drm/ttm/ttm_bo.c
>>> index c5b516f..eae61cc 100644
>>> --- a/drivers/gpu/drm/ttm/ttm_bo.c
>>> +++ b/drivers/gpu/drm/ttm/ttm_bo.c
>>> @@ -1750,9 +1750,29 @@ void ttm_bo_unmap_virtual(struct 
>>> ttm_buffer_object *bo)
>>>       ttm_bo_unmap_virtual_locked(bo);
>>>       ttm_mem_io_unlock(man);
>>>   }
>>> +EXPORT_SYMBOL(ttm_bo_unmap_virtual);
>>>   +void ttm_bo_unmap_virtual_address_space(struct ttm_bo_device *bdev)
>>> +{
>>> +    struct ttm_mem_type_manager *man;
>>> +    int i;
>>>   -EXPORT_SYMBOL(ttm_bo_unmap_virtual);
>>
>>> +    for (i = 0; i < TTM_NUM_MEM_TYPES; i++) {
>>> +        man = &bdev->man[i];
>>> +        if (man->has_type && man->use_type)
>>> +            ttm_mem_io_lock(man, false);
>>> +    }
>>
>> You should drop that it will just result in a deadlock warning for 
>> Nouveau and has no effect at all.
>>
>> Apart from that looks good to me,
>> Christian.
>
>
> As I am considering to re-include this in V2 of the patchsets, can you 
> clarify please why this will have no effect at all ?

The locks are exclusive for Nouveau to allocate/free the io address space.

Since we don't do this here we don't need the locks.

Christian.

>
> Andrey
>
>
>>
>>> +
>>> +    unmap_mapping_range(bdev->dev_mapping, 0, 0 , 1);
>>> +    /*TODO What about ttm_mem_io_free_vm(bo) ? */
>>> +
>>> +    for (i = TTM_NUM_MEM_TYPES - 1; i >= 0; i--) {
>>> +        man = &bdev->man[i];
>>> +        if (man->has_type && man->use_type)
>>> +            ttm_mem_io_unlock(man);
>>> +    }
>>> +}
>>> +EXPORT_SYMBOL(ttm_bo_unmap_virtual_address_space);
>>>     int ttm_bo_wait(struct ttm_buffer_object *bo,
>>>           bool interruptible, bool no_wait)
>>> diff --git a/include/drm/ttm/ttm_bo_driver.h 
>>> b/include/drm/ttm/ttm_bo_driver.h
>>> index c9e0fd0..3133463 100644
>>> --- a/include/drm/ttm/ttm_bo_driver.h
>>> +++ b/include/drm/ttm/ttm_bo_driver.h
>>> @@ -600,6 +600,8 @@ int ttm_bo_device_init(struct ttm_bo_device *bdev,
>>>    */
>>>   void ttm_bo_unmap_virtual(struct ttm_buffer_object *bo);
>>>   +void ttm_bo_unmap_virtual_address_space(struct ttm_bo_device *bdev);
>>> +
>>>   /**
>>>    * ttm_bo_unmap_virtual
>>>    *
>>

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [PATCH 1/6] drm/ttm: Add unampping of the entire device address space

[Christian König]

Am 05.06.20 um 16:29 schrieb Andrey Grodzovsky:
>
> On 5/11/20 2:45 AM, Christian König wrote:
>> Am 09.05.20 um 20:51 schrieb Andrey Grodzovsky:
>>> Signed-off-by: Andrey Grodzovsky <andrey.grodzovsky@amd.com>
>>> ---
>>>   drivers/gpu/drm/ttm/ttm_bo.c    | 22 +++++++++++++++++++++-
>>>   include/drm/ttm/ttm_bo_driver.h |  2 ++
>>>   2 files changed, 23 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/drivers/gpu/drm/ttm/ttm_bo.c 
>>> b/drivers/gpu/drm/ttm/ttm_bo.c
>>> index c5b516f..eae61cc 100644
>>> --- a/drivers/gpu/drm/ttm/ttm_bo.c
>>> +++ b/drivers/gpu/drm/ttm/ttm_bo.c
>>> @@ -1750,9 +1750,29 @@ void ttm_bo_unmap_virtual(struct 
>>> ttm_buffer_object *bo)
>>>       ttm_bo_unmap_virtual_locked(bo);
>>>       ttm_mem_io_unlock(man);
>>>   }
>>> +EXPORT_SYMBOL(ttm_bo_unmap_virtual);
>>>   +void ttm_bo_unmap_virtual_address_space(struct ttm_bo_device *bdev)
>>> +{
>>> +    struct ttm_mem_type_manager *man;
>>> +    int i;
>>>   -EXPORT_SYMBOL(ttm_bo_unmap_virtual);
>>
>>> +    for (i = 0; i < TTM_NUM_MEM_TYPES; i++) {
>>> +        man = &bdev->man[i];
>>> +        if (man->has_type && man->use_type)
>>> +            ttm_mem_io_lock(man, false);
>>> +    }
>>
>> You should drop that it will just result in a deadlock warning for 
>> Nouveau and has no effect at all.
>>
>> Apart from that looks good to me,
>> Christian.
>
>
> As I am considering to re-include this in V2 of the patchsets, can you 
> clarify please why this will have no effect at all ?

The locks are exclusive for Nouveau to allocate/free the io address space.

Since we don't do this here we don't need the locks.

Christian.

>
> Andrey
>
>
>>
>>> +
>>> +    unmap_mapping_range(bdev->dev_mapping, 0, 0 , 1);
>>> +    /*TODO What about ttm_mem_io_free_vm(bo) ? */
>>> +
>>> +    for (i = TTM_NUM_MEM_TYPES - 1; i >= 0; i--) {
>>> +        man = &bdev->man[i];
>>> +        if (man->has_type && man->use_type)
>>> +            ttm_mem_io_unlock(man);
>>> +    }
>>> +}
>>> +EXPORT_SYMBOL(ttm_bo_unmap_virtual_address_space);
>>>     int ttm_bo_wait(struct ttm_buffer_object *bo,
>>>           bool interruptible, bool no_wait)
>>> diff --git a/include/drm/ttm/ttm_bo_driver.h 
>>> b/include/drm/ttm/ttm_bo_driver.h
>>> index c9e0fd0..3133463 100644
>>> --- a/include/drm/ttm/ttm_bo_driver.h
>>> +++ b/include/drm/ttm/ttm_bo_driver.h
>>> @@ -600,6 +600,8 @@ int ttm_bo_device_init(struct ttm_bo_device *bdev,
>>>    */
>>>   void ttm_bo_unmap_virtual(struct ttm_buffer_object *bo);
>>>   +void ttm_bo_unmap_virtual_address_space(struct ttm_bo_device *bdev);
>>> +
>>>   /**
>>>    * ttm_bo_unmap_virtual
>>>    *
>>

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH 1/6] drm/ttm: Add unampping of the entire device address space

[Andrey Grodzovsky]

On 6/5/20 2:40 PM, Christian König wrote:
> Am 05.06.20 um 16:29 schrieb Andrey Grodzovsky:
>>
>> On 5/11/20 2:45 AM, Christian König wrote:
>>> Am 09.05.20 um 20:51 schrieb Andrey Grodzovsky:
>>>> Signed-off-by: Andrey Grodzovsky <andrey.grodzovsky@amd.com>
>>>> ---
>>>>   drivers/gpu/drm/ttm/ttm_bo.c    | 22 +++++++++++++++++++++-
>>>>   include/drm/ttm/ttm_bo_driver.h |  2 ++
>>>>   2 files changed, 23 insertions(+), 1 deletion(-)
>>>>
>>>> diff --git a/drivers/gpu/drm/ttm/ttm_bo.c 
>>>> b/drivers/gpu/drm/ttm/ttm_bo.c
>>>> index c5b516f..eae61cc 100644
>>>> --- a/drivers/gpu/drm/ttm/ttm_bo.c
>>>> +++ b/drivers/gpu/drm/ttm/ttm_bo.c
>>>> @@ -1750,9 +1750,29 @@ void ttm_bo_unmap_virtual(struct 
>>>> ttm_buffer_object *bo)
>>>>       ttm_bo_unmap_virtual_locked(bo);
>>>>       ttm_mem_io_unlock(man);
>>>>   }
>>>> +EXPORT_SYMBOL(ttm_bo_unmap_virtual);
>>>>   +void ttm_bo_unmap_virtual_address_space(struct ttm_bo_device *bdev)
>>>> +{
>>>> +    struct ttm_mem_type_manager *man;
>>>> +    int i;
>>>>   -EXPORT_SYMBOL(ttm_bo_unmap_virtual);
>>>
>>>> +    for (i = 0; i < TTM_NUM_MEM_TYPES; i++) {
>>>> +        man = &bdev->man[i];
>>>> +        if (man->has_type && man->use_type)
>>>> +            ttm_mem_io_lock(man, false);
>>>> +    }
>>>
>>> You should drop that it will just result in a deadlock warning for 
>>> Nouveau and has no effect at all.
>>>
>>> Apart from that looks good to me,
>>> Christian.
>>
>>
>> As I am considering to re-include this in V2 of the patchsets, can 
>> you clarify please why this will have no effect at all ?
>
> The locks are exclusive for Nouveau to allocate/free the io address 
> space.
>
> Since we don't do this here we don't need the locks.
>
> Christian.


So basically calling unmap_mapping_range doesn't require any extra 
locking around it and whatever locks are taken within the function 
should be enough ?

Andrey


>
>>
>> Andrey
>>
>>
>>>
>>>> +
>>>> +    unmap_mapping_range(bdev->dev_mapping, 0, 0 , 1);
>>>> +    /*TODO What about ttm_mem_io_free_vm(bo) ? */
>>>> +
>>>> +    for (i = TTM_NUM_MEM_TYPES - 1; i >= 0; i--) {
>>>> +        man = &bdev->man[i];
>>>> +        if (man->has_type && man->use_type)
>>>> +            ttm_mem_io_unlock(man);
>>>> +    }
>>>> +}
>>>> +EXPORT_SYMBOL(ttm_bo_unmap_virtual_address_space);
>>>>     int ttm_bo_wait(struct ttm_buffer_object *bo,
>>>>           bool interruptible, bool no_wait)
>>>> diff --git a/include/drm/ttm/ttm_bo_driver.h 
>>>> b/include/drm/ttm/ttm_bo_driver.h
>>>> index c9e0fd0..3133463 100644
>>>> --- a/include/drm/ttm/ttm_bo_driver.h
>>>> +++ b/include/drm/ttm/ttm_bo_driver.h
>>>> @@ -600,6 +600,8 @@ int ttm_bo_device_init(struct ttm_bo_device *bdev,
>>>>    */
>>>>   void ttm_bo_unmap_virtual(struct ttm_buffer_object *bo);
>>>>   +void ttm_bo_unmap_virtual_address_space(struct ttm_bo_device 
>>>> *bdev);
>>>> +
>>>>   /**
>>>>    * ttm_bo_unmap_virtual
>>>>    *
>>>
>
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [PATCH 1/6] drm/ttm: Add unampping of the entire device address space

[Andrey Grodzovsky]

On 6/5/20 2:40 PM, Christian König wrote:
> Am 05.06.20 um 16:29 schrieb Andrey Grodzovsky:
>>
>> On 5/11/20 2:45 AM, Christian König wrote:
>>> Am 09.05.20 um 20:51 schrieb Andrey Grodzovsky:
>>>> Signed-off-by: Andrey Grodzovsky <andrey.grodzovsky@amd.com>
>>>> ---
>>>>   drivers/gpu/drm/ttm/ttm_bo.c    | 22 +++++++++++++++++++++-
>>>>   include/drm/ttm/ttm_bo_driver.h |  2 ++
>>>>   2 files changed, 23 insertions(+), 1 deletion(-)
>>>>
>>>> diff --git a/drivers/gpu/drm/ttm/ttm_bo.c 
>>>> b/drivers/gpu/drm/ttm/ttm_bo.c
>>>> index c5b516f..eae61cc 100644
>>>> --- a/drivers/gpu/drm/ttm/ttm_bo.c
>>>> +++ b/drivers/gpu/drm/ttm/ttm_bo.c
>>>> @@ -1750,9 +1750,29 @@ void ttm_bo_unmap_virtual(struct 
>>>> ttm_buffer_object *bo)
>>>>       ttm_bo_unmap_virtual_locked(bo);
>>>>       ttm_mem_io_unlock(man);
>>>>   }
>>>> +EXPORT_SYMBOL(ttm_bo_unmap_virtual);
>>>>   +void ttm_bo_unmap_virtual_address_space(struct ttm_bo_device *bdev)
>>>> +{
>>>> +    struct ttm_mem_type_manager *man;
>>>> +    int i;
>>>>   -EXPORT_SYMBOL(ttm_bo_unmap_virtual);
>>>
>>>> +    for (i = 0; i < TTM_NUM_MEM_TYPES; i++) {
>>>> +        man = &bdev->man[i];
>>>> +        if (man->has_type && man->use_type)
>>>> +            ttm_mem_io_lock(man, false);
>>>> +    }
>>>
>>> You should drop that it will just result in a deadlock warning for 
>>> Nouveau and has no effect at all.
>>>
>>> Apart from that looks good to me,
>>> Christian.
>>
>>
>> As I am considering to re-include this in V2 of the patchsets, can 
>> you clarify please why this will have no effect at all ?
>
> The locks are exclusive for Nouveau to allocate/free the io address 
> space.
>
> Since we don't do this here we don't need the locks.
>
> Christian.


So basically calling unmap_mapping_range doesn't require any extra 
locking around it and whatever locks are taken within the function 
should be enough ?

Andrey


>
>>
>> Andrey
>>
>>
>>>
>>>> +
>>>> +    unmap_mapping_range(bdev->dev_mapping, 0, 0 , 1);
>>>> +    /*TODO What about ttm_mem_io_free_vm(bo) ? */
>>>> +
>>>> +    for (i = TTM_NUM_MEM_TYPES - 1; i >= 0; i--) {
>>>> +        man = &bdev->man[i];
>>>> +        if (man->has_type && man->use_type)
>>>> +            ttm_mem_io_unlock(man);
>>>> +    }
>>>> +}
>>>> +EXPORT_SYMBOL(ttm_bo_unmap_virtual_address_space);
>>>>     int ttm_bo_wait(struct ttm_buffer_object *bo,
>>>>           bool interruptible, bool no_wait)
>>>> diff --git a/include/drm/ttm/ttm_bo_driver.h 
>>>> b/include/drm/ttm/ttm_bo_driver.h
>>>> index c9e0fd0..3133463 100644
>>>> --- a/include/drm/ttm/ttm_bo_driver.h
>>>> +++ b/include/drm/ttm/ttm_bo_driver.h
>>>> @@ -600,6 +600,8 @@ int ttm_bo_device_init(struct ttm_bo_device *bdev,
>>>>    */
>>>>   void ttm_bo_unmap_virtual(struct ttm_buffer_object *bo);
>>>>   +void ttm_bo_unmap_virtual_address_space(struct ttm_bo_device 
>>>> *bdev);
>>>> +
>>>>   /**
>>>>    * ttm_bo_unmap_virtual
>>>>    *
>>>
>
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

[PATCH 2/6] drm/amdgpu: Force unmap all user VMAs on device removal.

[Andrey Grodzovsky]

Signed-off-by: Andrey Grodzovsky <andrey.grodzovsky@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c |  4 +++-
 drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c    | 14 ++++++++++----
 drivers/gpu/drm/amd/amdgpu/amdgpu_object.c |  4 ++++
 3 files changed, 17 insertions(+), 5 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
index e6978a2..4da52b7 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
@@ -1374,8 +1374,10 @@ amdgpu_device_ip_get_ip_block(struct amdgpu_device *adev,
 			      enum amd_ip_block_type type)
 {
 	int i;
-
 	for (i = 0; i < adev->num_ip_blocks; i++)
+
+
+
 		if (adev->ip_blocks[i].version->type == type)
 			return &adev->ip_blocks[i];
 
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
index 76a6198..ea2b47e 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
@@ -1130,16 +1130,22 @@ static int amdgpu_pci_probe(struct pci_dev *pdev,
 	return ret;
 }
 
+static void amdgpu_force_unmap_user_space_mappings(struct drm_device *dev)
+{
+	struct amdgpu_device *adev = dev->dev_private;
+
+	ttm_bo_unmap_virtual_address_space(&adev->mman.bdev);
+}
+
 static void
 amdgpu_pci_remove(struct pci_dev *pdev)
 {
 	struct drm_device *dev = pci_get_drvdata(pdev);
 
-#ifdef MODULE
-	if (THIS_MODULE->state != MODULE_STATE_GOING)
-#endif
-		DRM_ERROR("Hotplug removal is not supported\n");
 	drm_dev_unplug(dev);
+
+	amdgpu_force_unmap_user_space_mappings(dev);
+
 	amdgpu_driver_unload_kms(dev);
 	pci_disable_device(pdev);
 	pci_set_drvdata(pdev, NULL);
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
index 3d822eb..22afd11 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
@@ -35,6 +35,7 @@
 
 #include <drm/amdgpu_drm.h>
 #include <drm/drm_cache.h>
+#include <drm/drm_drv.h>
 #include "amdgpu.h"
 #include "amdgpu_trace.h"
 #include "amdgpu_amdkfd.h"
@@ -1361,6 +1362,9 @@ int amdgpu_bo_fault_reserve_notify(struct ttm_buffer_object *bo)
 	if (!amdgpu_bo_is_amdgpu_bo(bo))
 		return 0;
 
+	if (drm_dev_is_unplugged(adev->ddev))
+		return -ENODEV;
+
 	abo = ttm_to_amdgpu_bo(bo);
 
 	/* Remember that this BO was accessed by the CPU */
-- 
2.7.4

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

[PATCH 2/6] drm/amdgpu: Force unmap all user VMAs on device removal.

[Andrey Grodzovsky]

Signed-off-by: Andrey Grodzovsky <andrey.grodzovsky@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c |  4 +++-
 drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c    | 14 ++++++++++----
 drivers/gpu/drm/amd/amdgpu/amdgpu_object.c |  4 ++++
 3 files changed, 17 insertions(+), 5 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
index e6978a2..4da52b7 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
@@ -1374,8 +1374,10 @@ amdgpu_device_ip_get_ip_block(struct amdgpu_device *adev,
 			      enum amd_ip_block_type type)
 {
 	int i;
-
 	for (i = 0; i < adev->num_ip_blocks; i++)
+
+
+
 		if (adev->ip_blocks[i].version->type == type)
 			return &adev->ip_blocks[i];
 
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
index 76a6198..ea2b47e 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
@@ -1130,16 +1130,22 @@ static int amdgpu_pci_probe(struct pci_dev *pdev,
 	return ret;
 }
 
+static void amdgpu_force_unmap_user_space_mappings(struct drm_device *dev)
+{
+	struct amdgpu_device *adev = dev->dev_private;
+
+	ttm_bo_unmap_virtual_address_space(&adev->mman.bdev);
+}
+
 static void
 amdgpu_pci_remove(struct pci_dev *pdev)
 {
 	struct drm_device *dev = pci_get_drvdata(pdev);
 
-#ifdef MODULE
-	if (THIS_MODULE->state != MODULE_STATE_GOING)
-#endif
-		DRM_ERROR("Hotplug removal is not supported\n");
 	drm_dev_unplug(dev);
+
+	amdgpu_force_unmap_user_space_mappings(dev);
+
 	amdgpu_driver_unload_kms(dev);
 	pci_disable_device(pdev);
 	pci_set_drvdata(pdev, NULL);
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
index 3d822eb..22afd11 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
@@ -35,6 +35,7 @@
 
 #include <drm/amdgpu_drm.h>
 #include <drm/drm_cache.h>
+#include <drm/drm_drv.h>
 #include "amdgpu.h"
 #include "amdgpu_trace.h"
 #include "amdgpu_amdkfd.h"
@@ -1361,6 +1362,9 @@ int amdgpu_bo_fault_reserve_notify(struct ttm_buffer_object *bo)
 	if (!amdgpu_bo_is_amdgpu_bo(bo))
 		return 0;
 
+	if (drm_dev_is_unplugged(adev->ddev))
+		return -ENODEV;
+
 	abo = ttm_to_amdgpu_bo(bo);
 
 	/* Remember that this BO was accessed by the CPU */
-- 
2.7.4

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH 2/6] drm/amdgpu: Force unmap all user VMAs on device removal.

[Christian König]

Am 09.05.20 um 20:51 schrieb Andrey Grodzovsky:
> Signed-off-by: Andrey Grodzovsky <andrey.grodzovsky@amd.com>
> ---
>   drivers/gpu/drm/amd/amdgpu/amdgpu_device.c |  4 +++-
>   drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c    | 14 ++++++++++----
>   drivers/gpu/drm/amd/amdgpu/amdgpu_object.c |  4 ++++
>   3 files changed, 17 insertions(+), 5 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> index e6978a2..4da52b7 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> @@ -1374,8 +1374,10 @@ amdgpu_device_ip_get_ip_block(struct amdgpu_device *adev,
>   			      enum amd_ip_block_type type)
>   {
>   	int i;
> -
>   	for (i = 0; i < adev->num_ip_blocks; i++)
> +
> +
> +

Unrelated whitespace change.

>   		if (adev->ip_blocks[i].version->type == type)
>   			return &adev->ip_blocks[i];
>   
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
> index 76a6198..ea2b47e 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
> @@ -1130,16 +1130,22 @@ static int amdgpu_pci_probe(struct pci_dev *pdev,
>   	return ret;
>   }
>   
> +static void amdgpu_force_unmap_user_space_mappings(struct drm_device *dev)
> +{
> +	struct amdgpu_device *adev = dev->dev_private;
> +
> +	ttm_bo_unmap_virtual_address_space(&adev->mman.bdev);
> +}
> +

If we really add a function for this we should probably put it into 
amdgpu_ttm.c

>   static void
>   amdgpu_pci_remove(struct pci_dev *pdev)
>   {
>   	struct drm_device *dev = pci_get_drvdata(pdev);
>   
> -#ifdef MODULE
> -	if (THIS_MODULE->state != MODULE_STATE_GOING)
> -#endif
> -		DRM_ERROR("Hotplug removal is not supported\n");

Keep the warning for now, there is a lot of stuff we need to fix first 
before removing that one.

Christian.

>   	drm_dev_unplug(dev);
> +
> +	amdgpu_force_unmap_user_space_mappings(dev);
> +
>   	amdgpu_driver_unload_kms(dev);
>   	pci_disable_device(pdev);
>   	pci_set_drvdata(pdev, NULL);
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
> index 3d822eb..22afd11 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
> @@ -35,6 +35,7 @@
>   
>   #include <drm/amdgpu_drm.h>
>   #include <drm/drm_cache.h>
> +#include <drm/drm_drv.h>
>   #include "amdgpu.h"
>   #include "amdgpu_trace.h"
>   #include "amdgpu_amdkfd.h"
> @@ -1361,6 +1362,9 @@ int amdgpu_bo_fault_reserve_notify(struct ttm_buffer_object *bo)
>   	if (!amdgpu_bo_is_amdgpu_bo(bo))
>   		return 0;
>   
> +	if (drm_dev_is_unplugged(adev->ddev))
> +		return -ENODEV;
> +
>   	abo = ttm_to_amdgpu_bo(bo);
>   
>   	/* Remember that this BO was accessed by the CPU */

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [PATCH 2/6] drm/amdgpu: Force unmap all user VMAs on device removal.

[Christian König]

Am 09.05.20 um 20:51 schrieb Andrey Grodzovsky:
> Signed-off-by: Andrey Grodzovsky <andrey.grodzovsky@amd.com>
> ---
>   drivers/gpu/drm/amd/amdgpu/amdgpu_device.c |  4 +++-
>   drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c    | 14 ++++++++++----
>   drivers/gpu/drm/amd/amdgpu/amdgpu_object.c |  4 ++++
>   3 files changed, 17 insertions(+), 5 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> index e6978a2..4da52b7 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> @@ -1374,8 +1374,10 @@ amdgpu_device_ip_get_ip_block(struct amdgpu_device *adev,
>   			      enum amd_ip_block_type type)
>   {
>   	int i;
> -
>   	for (i = 0; i < adev->num_ip_blocks; i++)
> +
> +
> +

Unrelated whitespace change.

>   		if (adev->ip_blocks[i].version->type == type)
>   			return &adev->ip_blocks[i];
>   
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
> index 76a6198..ea2b47e 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
> @@ -1130,16 +1130,22 @@ static int amdgpu_pci_probe(struct pci_dev *pdev,
>   	return ret;
>   }
>   
> +static void amdgpu_force_unmap_user_space_mappings(struct drm_device *dev)
> +{
> +	struct amdgpu_device *adev = dev->dev_private;
> +
> +	ttm_bo_unmap_virtual_address_space(&adev->mman.bdev);
> +}
> +

If we really add a function for this we should probably put it into 
amdgpu_ttm.c

>   static void
>   amdgpu_pci_remove(struct pci_dev *pdev)
>   {
>   	struct drm_device *dev = pci_get_drvdata(pdev);
>   
> -#ifdef MODULE
> -	if (THIS_MODULE->state != MODULE_STATE_GOING)
> -#endif
> -		DRM_ERROR("Hotplug removal is not supported\n");

Keep the warning for now, there is a lot of stuff we need to fix first 
before removing that one.

Christian.

>   	drm_dev_unplug(dev);
> +
> +	amdgpu_force_unmap_user_space_mappings(dev);
> +
>   	amdgpu_driver_unload_kms(dev);
>   	pci_disable_device(pdev);
>   	pci_set_drvdata(pdev, NULL);
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
> index 3d822eb..22afd11 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
> @@ -35,6 +35,7 @@
>   
>   #include <drm/amdgpu_drm.h>
>   #include <drm/drm_cache.h>
> +#include <drm/drm_drv.h>
>   #include "amdgpu.h"
>   #include "amdgpu_trace.h"
>   #include "amdgpu_amdkfd.h"
> @@ -1361,6 +1362,9 @@ int amdgpu_bo_fault_reserve_notify(struct ttm_buffer_object *bo)
>   	if (!amdgpu_bo_is_amdgpu_bo(bo))
>   		return 0;
>   
> +	if (drm_dev_is_unplugged(adev->ddev))
> +		return -ENODEV;
> +
>   	abo = ttm_to_amdgpu_bo(bo);
>   
>   	/* Remember that this BO was accessed by the CPU */

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

[PATCH 3/6] drm/amdgpu: Wait for all user clients

[Andrey Grodzovsky]

Signed-off-by: Andrey Grodzovsky <andrey.grodzovsky@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu.h        | 2 ++
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 3 +++
 drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c    | 2 ++
 drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c    | 4 ++++
 4 files changed, 11 insertions(+)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu.h b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
index bc1e0fd..79274d5 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
@@ -990,6 +990,8 @@ struct amdgpu_device {
 	char				product_number[16];
 	char				product_name[32];
 	char				serial[16];
+
+	wait_queue_head_t		user_clients_done;
 };
 
 static inline struct amdgpu_device *amdgpu_ttm_adev(struct ttm_bo_device *bdev)
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
index 4da52b7..3bd67cf 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
@@ -3271,6 +3271,9 @@ int amdgpu_device_init(struct amdgpu_device *adev,
 	if (r)
 		dev_err(adev->dev, "amdgpu_pmu_init failed\n");
 
+
+	init_waitqueue_head(&adev->user_clients_done);
+
 	return 0;
 
 failed:
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
index ea2b47e..0531727 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
@@ -1141,10 +1141,12 @@ static void
 amdgpu_pci_remove(struct pci_dev *pdev)
 {
 	struct drm_device *dev = pci_get_drvdata(pdev);
+	struct amdgpu_device *adev = dev->dev_private;
 
 	drm_dev_unplug(dev);
 
 	amdgpu_force_unmap_user_space_mappings(dev);
+	wait_event(adev->user_clients_done, (dev->open_count == 0));
 
 	amdgpu_driver_unload_kms(dev);
 	pci_disable_device(pdev);
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c
index 61fb2ef..d8fc775 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c
@@ -957,8 +957,12 @@ static int amdgpu_info_ioctl(struct drm_device *dev, void *data, struct drm_file
  */
 void amdgpu_driver_lastclose_kms(struct drm_device *dev)
 {
+	struct amdgpu_device *adev = dev->dev_private;
+
 	drm_fb_helper_lastclose(dev);
 	vga_switcheroo_process_delayed_switch();
+
+	wake_up(&adev->user_clients_done);
 }
 
 /**
-- 
2.7.4

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

[PATCH 3/6] drm/amdgpu: Wait for all user clients

[Andrey Grodzovsky]

Signed-off-by: Andrey Grodzovsky <andrey.grodzovsky@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu.h        | 2 ++
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 3 +++
 drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c    | 2 ++
 drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c    | 4 ++++
 4 files changed, 11 insertions(+)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu.h b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
index bc1e0fd..79274d5 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
@@ -990,6 +990,8 @@ struct amdgpu_device {
 	char				product_number[16];
 	char				product_name[32];
 	char				serial[16];
+
+	wait_queue_head_t		user_clients_done;
 };
 
 static inline struct amdgpu_device *amdgpu_ttm_adev(struct ttm_bo_device *bdev)
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
index 4da52b7..3bd67cf 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
@@ -3271,6 +3271,9 @@ int amdgpu_device_init(struct amdgpu_device *adev,
 	if (r)
 		dev_err(adev->dev, "amdgpu_pmu_init failed\n");
 
+
+	init_waitqueue_head(&adev->user_clients_done);
+
 	return 0;
 
 failed:
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
index ea2b47e..0531727 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
@@ -1141,10 +1141,12 @@ static void
 amdgpu_pci_remove(struct pci_dev *pdev)
 {
 	struct drm_device *dev = pci_get_drvdata(pdev);
+	struct amdgpu_device *adev = dev->dev_private;
 
 	drm_dev_unplug(dev);
 
 	amdgpu_force_unmap_user_space_mappings(dev);
+	wait_event(adev->user_clients_done, (dev->open_count == 0));
 
 	amdgpu_driver_unload_kms(dev);
 	pci_disable_device(pdev);
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c
index 61fb2ef..d8fc775 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c
@@ -957,8 +957,12 @@ static int amdgpu_info_ioctl(struct drm_device *dev, void *data, struct drm_file
  */
 void amdgpu_driver_lastclose_kms(struct drm_device *dev)
 {
+	struct amdgpu_device *adev = dev->dev_private;
+
 	drm_fb_helper_lastclose(dev);
 	vga_switcheroo_process_delayed_switch();
+
+	wake_up(&adev->user_clients_done);
 }
 
 /**
-- 
2.7.4

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH 3/6] drm/amdgpu: Wait for all user clients

[Christian König]

A commit message would be nice, apart from that the patch looks clean to me.

But question for Daniel and others: Is that in general the right approach?

It can happen that device removal is delayed indefinitely if userspace 
doesn't close the file descriptors.

Regards,
Christian.

Am 09.05.20 um 20:51 schrieb Andrey Grodzovsky:
> Signed-off-by: Andrey Grodzovsky <andrey.grodzovsky@amd.com>
> ---
>   drivers/gpu/drm/amd/amdgpu/amdgpu.h        | 2 ++
>   drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 3 +++
>   drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c    | 2 ++
>   drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c    | 4 ++++
>   4 files changed, 11 insertions(+)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu.h b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
> index bc1e0fd..79274d5 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu.h
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
> @@ -990,6 +990,8 @@ struct amdgpu_device {
>   	char				product_number[16];
>   	char				product_name[32];
>   	char				serial[16];
> +
> +	wait_queue_head_t		user_clients_done;
>   };
>   
>   static inline struct amdgpu_device *amdgpu_ttm_adev(struct ttm_bo_device *bdev)
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> index 4da52b7..3bd67cf 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> @@ -3271,6 +3271,9 @@ int amdgpu_device_init(struct amdgpu_device *adev,
>   	if (r)
>   		dev_err(adev->dev, "amdgpu_pmu_init failed\n");
>   
> +
> +	init_waitqueue_head(&adev->user_clients_done);
> +
>   	return 0;
>   
>   failed:
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
> index ea2b47e..0531727 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
> @@ -1141,10 +1141,12 @@ static void
>   amdgpu_pci_remove(struct pci_dev *pdev)
>   {
>   	struct drm_device *dev = pci_get_drvdata(pdev);
> +	struct amdgpu_device *adev = dev->dev_private;
>   
>   	drm_dev_unplug(dev);
>   
>   	amdgpu_force_unmap_user_space_mappings(dev);
> +	wait_event(adev->user_clients_done, (dev->open_count == 0));
>   
>   	amdgpu_driver_unload_kms(dev);
>   	pci_disable_device(pdev);
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c
> index 61fb2ef..d8fc775 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c
> @@ -957,8 +957,12 @@ static int amdgpu_info_ioctl(struct drm_device *dev, void *data, struct drm_file
>    */
>   void amdgpu_driver_lastclose_kms(struct drm_device *dev)
>   {
> +	struct amdgpu_device *adev = dev->dev_private;
> +
>   	drm_fb_helper_lastclose(dev);
>   	vga_switcheroo_process_delayed_switch();
> +
> +	wake_up(&adev->user_clients_done);
>   }
>   
>   /**

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [PATCH 3/6] drm/amdgpu: Wait for all user clients

[Christian König]

A commit message would be nice, apart from that the patch looks clean to me.

But question for Daniel and others: Is that in general the right approach?

It can happen that device removal is delayed indefinitely if userspace 
doesn't close the file descriptors.

Regards,
Christian.

Am 09.05.20 um 20:51 schrieb Andrey Grodzovsky:
> Signed-off-by: Andrey Grodzovsky <andrey.grodzovsky@amd.com>
> ---
>   drivers/gpu/drm/amd/amdgpu/amdgpu.h        | 2 ++
>   drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 3 +++
>   drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c    | 2 ++
>   drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c    | 4 ++++
>   4 files changed, 11 insertions(+)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu.h b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
> index bc1e0fd..79274d5 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu.h
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
> @@ -990,6 +990,8 @@ struct amdgpu_device {
>   	char				product_number[16];
>   	char				product_name[32];
>   	char				serial[16];
> +
> +	wait_queue_head_t		user_clients_done;
>   };
>   
>   static inline struct amdgpu_device *amdgpu_ttm_adev(struct ttm_bo_device *bdev)
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> index 4da52b7..3bd67cf 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> @@ -3271,6 +3271,9 @@ int amdgpu_device_init(struct amdgpu_device *adev,
>   	if (r)
>   		dev_err(adev->dev, "amdgpu_pmu_init failed\n");
>   
> +
> +	init_waitqueue_head(&adev->user_clients_done);
> +
>   	return 0;
>   
>   failed:
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
> index ea2b47e..0531727 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
> @@ -1141,10 +1141,12 @@ static void
>   amdgpu_pci_remove(struct pci_dev *pdev)
>   {
>   	struct drm_device *dev = pci_get_drvdata(pdev);
> +	struct amdgpu_device *adev = dev->dev_private;
>   
>   	drm_dev_unplug(dev);
>   
>   	amdgpu_force_unmap_user_space_mappings(dev);
> +	wait_event(adev->user_clients_done, (dev->open_count == 0));
>   
>   	amdgpu_driver_unload_kms(dev);
>   	pci_disable_device(pdev);
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c
> index 61fb2ef..d8fc775 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c
> @@ -957,8 +957,12 @@ static int amdgpu_info_ioctl(struct drm_device *dev, void *data, struct drm_file
>    */
>   void amdgpu_driver_lastclose_kms(struct drm_device *dev)
>   {
> +	struct amdgpu_device *adev = dev->dev_private;
> +
>   	drm_fb_helper_lastclose(dev);
>   	vga_switcheroo_process_delayed_switch();
> +
> +	wake_up(&adev->user_clients_done);
>   }
>   
>   /**

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

[PATCH 4/6] drm/amdgpu: Wait for all clients importing out dma-bufs.

[Andrey Grodzovsky]

Also avoid GPU recovery if device is unplagged

Signed-off-by: Andrey Grodzovsky <andrey.grodzovsky@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu.h         |  1 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c | 27 +++++++++++++++++++++++++--
 drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c     |  4 +++-
 drivers/gpu/drm/amd/amdgpu/amdgpu_job.c     |  9 +++++++++
 4 files changed, 38 insertions(+), 3 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu.h b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
index 79274d5..f212622 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
@@ -992,6 +992,7 @@ struct amdgpu_device {
 	char				serial[16];
 
 	wait_queue_head_t		user_clients_done;
+	atomic_t 			exported_dma_bufs_count;
 };
 
 static inline struct amdgpu_device *amdgpu_ttm_adev(struct ttm_bo_device *bdev)
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c
index ffeb20f..479ff98 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c
@@ -36,6 +36,7 @@
 #include "amdgpu_gem.h"
 #include "amdgpu_dma_buf.h"
 #include <drm/amdgpu_drm.h>
+#include <drm/drm_drv.h>
 #include <linux/dma-buf.h>
 #include <linux/dma-fence-array.h>
 
@@ -116,6 +117,7 @@ int amdgpu_gem_prime_mmap(struct drm_gem_object *obj,
 		return ret;
 
 	ret = ttm_bo_mmap(vma->vm_file, vma, &adev->mman.bdev);
+
 	drm_vma_node_revoke(&obj->vma_node, vma->vm_file->private_data);
 
 	return ret;
@@ -179,6 +181,9 @@ static int amdgpu_dma_buf_attach(struct dma_buf *dmabuf,
 	struct amdgpu_device *adev = amdgpu_ttm_adev(bo->tbo.bdev);
 	int r;
 
+	if (drm_dev_is_unplugged(adev->ddev))
+		return -ENODEV;
+
 	if (attach->dev->driver == adev->dev->driver)
 		return 0;
 
@@ -363,6 +368,19 @@ static int amdgpu_dma_buf_begin_cpu_access(struct dma_buf *dma_buf,
 	return ret;
 }
 
+
+static void amdgpu_dma_buf_release(struct dma_buf *dma_buf)
+{
+	struct amdgpu_bo *bo = gem_to_amdgpu_bo(dma_buf->priv);
+	struct amdgpu_device *adev = amdgpu_ttm_adev(bo->tbo.bdev);
+
+	drm_gem_dmabuf_release(dma_buf);
+
+	atomic_dec(&adev->exported_dma_bufs_count);
+	wake_up(&adev->user_clients_done);
+
+}
+
 const struct dma_buf_ops amdgpu_dmabuf_ops = {
 	.attach = amdgpu_dma_buf_attach,
 	.detach = amdgpu_dma_buf_detach,
@@ -370,13 +388,14 @@ const struct dma_buf_ops amdgpu_dmabuf_ops = {
 	.unpin = amdgpu_dma_buf_unpin,
 	.map_dma_buf = amdgpu_dma_buf_map,
 	.unmap_dma_buf = amdgpu_dma_buf_unmap,
-	.release = drm_gem_dmabuf_release,
+	.release = amdgpu_dma_buf_release,
 	.begin_cpu_access = amdgpu_dma_buf_begin_cpu_access,
 	.mmap = drm_gem_dmabuf_mmap,
 	.vmap = drm_gem_dmabuf_vmap,
 	.vunmap = drm_gem_dmabuf_vunmap,
 };
 
+
 /**
  * amdgpu_gem_prime_export - &drm_driver.gem_prime_export implementation
  * @gobj: GEM BO
@@ -391,6 +410,7 @@ struct dma_buf *amdgpu_gem_prime_export(struct drm_gem_object *gobj,
 					int flags)
 {
 	struct amdgpu_bo *bo = gem_to_amdgpu_bo(gobj);
+	struct amdgpu_device *adev = amdgpu_ttm_adev(bo->tbo.bdev);
 	struct dma_buf *buf;
 
 	if (amdgpu_ttm_tt_get_usermm(bo->tbo.ttm) ||
@@ -398,8 +418,10 @@ struct dma_buf *amdgpu_gem_prime_export(struct drm_gem_object *gobj,
 		return ERR_PTR(-EPERM);
 
 	buf = drm_gem_prime_export(gobj, flags);
-	if (!IS_ERR(buf))
+	if (!IS_ERR(buf)) {
 		buf->ops = &amdgpu_dmabuf_ops;
+		atomic_inc(&((adev)->exported_dma_bufs_count));
+	}
 
 	return buf;
 }
@@ -558,5 +580,6 @@ struct drm_gem_object *amdgpu_gem_prime_import(struct drm_device *dev,
 
 	get_dma_buf(dma_buf);
 	obj->import_attach = attach;
+
 	return obj;
 }
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
index 0531727..11410a9 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
@@ -1146,7 +1146,9 @@ amdgpu_pci_remove(struct pci_dev *pdev)
 	drm_dev_unplug(dev);
 
 	amdgpu_force_unmap_user_space_mappings(dev);
-	wait_event(adev->user_clients_done, (dev->open_count == 0));
+	wait_event(adev->user_clients_done,
+		   !atomic_read(&dev->open_count) &&
+		   !atomic_read(&adev->exported_dma_bufs_count));
 
 	amdgpu_driver_unload_kms(dev);
 	pci_disable_device(pdev);
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c
index 4720718..20cf36d 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c
@@ -28,6 +28,9 @@
 #include "amdgpu.h"
 #include "amdgpu_trace.h"
 
+#include <drm/drm_drv.h>
+
+
 static void amdgpu_job_timedout(struct drm_sched_job *s_job)
 {
 	struct amdgpu_ring *ring = to_amdgpu_ring(s_job->sched);
@@ -37,6 +40,12 @@ static void amdgpu_job_timedout(struct drm_sched_job *s_job)
 
 	memset(&ti, 0, sizeof(struct amdgpu_task_info));
 
+
+	if (drm_dev_is_unplugged(adev->ddev)) {
+		DRM_WARN("amdgpu_job_timedout - device is unplugged, skiping!");
+		return;
+	}
+
 	if (amdgpu_ring_soft_recovery(ring, job->vmid, s_job->s_fence->parent)) {
 		DRM_ERROR("ring %s timeout, but soft recovered\n",
 			  s_job->sched->name);
-- 
2.7.4

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

[PATCH 4/6] drm/amdgpu: Wait for all clients importing out dma-bufs.

[Andrey Grodzovsky]

Also avoid GPU recovery if device is unplagged

Signed-off-by: Andrey Grodzovsky <andrey.grodzovsky@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu.h         |  1 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c | 27 +++++++++++++++++++++++++--
 drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c     |  4 +++-
 drivers/gpu/drm/amd/amdgpu/amdgpu_job.c     |  9 +++++++++
 4 files changed, 38 insertions(+), 3 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu.h b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
index 79274d5..f212622 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
@@ -992,6 +992,7 @@ struct amdgpu_device {
 	char				serial[16];
 
 	wait_queue_head_t		user_clients_done;
+	atomic_t 			exported_dma_bufs_count;
 };
 
 static inline struct amdgpu_device *amdgpu_ttm_adev(struct ttm_bo_device *bdev)
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c
index ffeb20f..479ff98 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c
@@ -36,6 +36,7 @@
 #include "amdgpu_gem.h"
 #include "amdgpu_dma_buf.h"
 #include <drm/amdgpu_drm.h>
+#include <drm/drm_drv.h>
 #include <linux/dma-buf.h>
 #include <linux/dma-fence-array.h>
 
@@ -116,6 +117,7 @@ int amdgpu_gem_prime_mmap(struct drm_gem_object *obj,
 		return ret;
 
 	ret = ttm_bo_mmap(vma->vm_file, vma, &adev->mman.bdev);
+
 	drm_vma_node_revoke(&obj->vma_node, vma->vm_file->private_data);
 
 	return ret;
@@ -179,6 +181,9 @@ static int amdgpu_dma_buf_attach(struct dma_buf *dmabuf,
 	struct amdgpu_device *adev = amdgpu_ttm_adev(bo->tbo.bdev);
 	int r;
 
+	if (drm_dev_is_unplugged(adev->ddev))
+		return -ENODEV;
+
 	if (attach->dev->driver == adev->dev->driver)
 		return 0;
 
@@ -363,6 +368,19 @@ static int amdgpu_dma_buf_begin_cpu_access(struct dma_buf *dma_buf,
 	return ret;
 }
 
+
+static void amdgpu_dma_buf_release(struct dma_buf *dma_buf)
+{
+	struct amdgpu_bo *bo = gem_to_amdgpu_bo(dma_buf->priv);
+	struct amdgpu_device *adev = amdgpu_ttm_adev(bo->tbo.bdev);
+
+	drm_gem_dmabuf_release(dma_buf);
+
+	atomic_dec(&adev->exported_dma_bufs_count);
+	wake_up(&adev->user_clients_done);
+
+}
+
 const struct dma_buf_ops amdgpu_dmabuf_ops = {
 	.attach = amdgpu_dma_buf_attach,
 	.detach = amdgpu_dma_buf_detach,
@@ -370,13 +388,14 @@ const struct dma_buf_ops amdgpu_dmabuf_ops = {
 	.unpin = amdgpu_dma_buf_unpin,
 	.map_dma_buf = amdgpu_dma_buf_map,
 	.unmap_dma_buf = amdgpu_dma_buf_unmap,
-	.release = drm_gem_dmabuf_release,
+	.release = amdgpu_dma_buf_release,
 	.begin_cpu_access = amdgpu_dma_buf_begin_cpu_access,
 	.mmap = drm_gem_dmabuf_mmap,
 	.vmap = drm_gem_dmabuf_vmap,
 	.vunmap = drm_gem_dmabuf_vunmap,
 };
 
+
 /**
  * amdgpu_gem_prime_export - &drm_driver.gem_prime_export implementation
  * @gobj: GEM BO
@@ -391,6 +410,7 @@ struct dma_buf *amdgpu_gem_prime_export(struct drm_gem_object *gobj,
 					int flags)
 {
 	struct amdgpu_bo *bo = gem_to_amdgpu_bo(gobj);
+	struct amdgpu_device *adev = amdgpu_ttm_adev(bo->tbo.bdev);
 	struct dma_buf *buf;
 
 	if (amdgpu_ttm_tt_get_usermm(bo->tbo.ttm) ||
@@ -398,8 +418,10 @@ struct dma_buf *amdgpu_gem_prime_export(struct drm_gem_object *gobj,
 		return ERR_PTR(-EPERM);
 
 	buf = drm_gem_prime_export(gobj, flags);
-	if (!IS_ERR(buf))
+	if (!IS_ERR(buf)) {
 		buf->ops = &amdgpu_dmabuf_ops;
+		atomic_inc(&((adev)->exported_dma_bufs_count));
+	}
 
 	return buf;
 }
@@ -558,5 +580,6 @@ struct drm_gem_object *amdgpu_gem_prime_import(struct drm_device *dev,
 
 	get_dma_buf(dma_buf);
 	obj->import_attach = attach;
+
 	return obj;
 }
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
index 0531727..11410a9 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
@@ -1146,7 +1146,9 @@ amdgpu_pci_remove(struct pci_dev *pdev)
 	drm_dev_unplug(dev);
 
 	amdgpu_force_unmap_user_space_mappings(dev);
-	wait_event(adev->user_clients_done, (dev->open_count == 0));
+	wait_event(adev->user_clients_done,
+		   !atomic_read(&dev->open_count) &&
+		   !atomic_read(&adev->exported_dma_bufs_count));
 
 	amdgpu_driver_unload_kms(dev);
 	pci_disable_device(pdev);
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c
index 4720718..20cf36d 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c
@@ -28,6 +28,9 @@
 #include "amdgpu.h"
 #include "amdgpu_trace.h"
 
+#include <drm/drm_drv.h>
+
+
 static void amdgpu_job_timedout(struct drm_sched_job *s_job)
 {
 	struct amdgpu_ring *ring = to_amdgpu_ring(s_job->sched);
@@ -37,6 +40,12 @@ static void amdgpu_job_timedout(struct drm_sched_job *s_job)
 
 	memset(&ti, 0, sizeof(struct amdgpu_task_info));
 
+
+	if (drm_dev_is_unplugged(adev->ddev)) {
+		DRM_WARN("amdgpu_job_timedout - device is unplugged, skiping!");
+		return;
+	}
+
 	if (amdgpu_ring_soft_recovery(ring, job->vmid, s_job->s_fence->parent)) {
 		DRM_ERROR("ring %s timeout, but soft recovered\n",
 			  s_job->sched->name);
-- 
2.7.4

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

[PATCH 5/6] drm/ttm: Add destroy flag in TTM BO eviction interface

[Andrey Grodzovsky]

This will allow to invalidate, destroy backing storage and notify users
of BOs when device is unpluged.

Signed-off-by: Andrey Grodzovsky <andrey.grodzovsky@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c |  2 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_object.c  |  2 +-
 drivers/gpu/drm/nouveau/nouveau_drm.c       |  2 +-
 drivers/gpu/drm/qxl/qxl_object.c            |  4 +--
 drivers/gpu/drm/radeon/radeon_object.c      |  2 +-
 drivers/gpu/drm/ttm/ttm_bo.c                | 41 ++++++++++++++++++-----------
 drivers/gpu/drm/vmwgfx/vmwgfx_drv.c         |  6 ++---
 include/drm/ttm/ttm_bo_api.h                |  2 +-
 8 files changed, 35 insertions(+), 26 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c
index 1a4894f..f96c6d1 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c
@@ -1145,7 +1145,7 @@ static int amdgpu_debugfs_evict_gtt(struct seq_file *m, void *data)
 	if (r < 0)
 		return r;
 
-	seq_printf(m, "(%d)\n", ttm_bo_evict_mm(&adev->mman.bdev, TTM_PL_TT));
+	seq_printf(m, "(%d)\n", ttm_bo_evict_mm(&adev->mman.bdev, TTM_PL_TT, false));
 
 	pm_runtime_mark_last_busy(dev->dev);
 	pm_runtime_put_autosuspend(dev->dev);
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
index 22afd11..82d43d0 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
@@ -1046,7 +1046,7 @@ int amdgpu_bo_evict_vram(struct amdgpu_device *adev)
 		return 0;
 	}
 #endif
-	return ttm_bo_evict_mm(&adev->mman.bdev, TTM_PL_VRAM);
+	return ttm_bo_evict_mm(&adev->mman.bdev, TTM_PL_VRAM, false);
 }
 
 static const char *amdgpu_vram_names[] = {
diff --git a/drivers/gpu/drm/nouveau/nouveau_drm.c b/drivers/gpu/drm/nouveau/nouveau_drm.c
index 6b1629c..f3eea89 100644
--- a/drivers/gpu/drm/nouveau/nouveau_drm.c
+++ b/drivers/gpu/drm/nouveau/nouveau_drm.c
@@ -759,7 +759,7 @@ nouveau_do_suspend(struct drm_device *dev, bool runtime)
 	}
 
 	NV_DEBUG(drm, "evicting buffers...\n");
-	ttm_bo_evict_mm(&drm->ttm.bdev, TTM_PL_VRAM);
+	ttm_bo_evict_mm(&drm->ttm.bdev, TTM_PL_VRAM, false);
 
 	NV_DEBUG(drm, "waiting for kernel channels to go idle...\n");
 	if (drm->cechan) {
diff --git a/drivers/gpu/drm/qxl/qxl_object.c b/drivers/gpu/drm/qxl/qxl_object.c
index ab72dc3..45bb89b 100644
--- a/drivers/gpu/drm/qxl/qxl_object.c
+++ b/drivers/gpu/drm/qxl/qxl_object.c
@@ -359,10 +359,10 @@ int qxl_bo_check_id(struct qxl_device *qdev, struct qxl_bo *bo)
 
 int qxl_surf_evict(struct qxl_device *qdev)
 {
-	return ttm_bo_evict_mm(&qdev->mman.bdev, TTM_PL_PRIV);
+	return ttm_bo_evict_mm(&qdev->mman.bdev, TTM_PL_PRIV, false);
 }
 
 int qxl_vram_evict(struct qxl_device *qdev)
 {
-	return ttm_bo_evict_mm(&qdev->mman.bdev, TTM_PL_VRAM);
+	return ttm_bo_evict_mm(&qdev->mman.bdev, TTM_PL_VRAM, false);
 }
diff --git a/drivers/gpu/drm/radeon/radeon_object.c b/drivers/gpu/drm/radeon/radeon_object.c
index 140d94c..3eeeb8d 100644
--- a/drivers/gpu/drm/radeon/radeon_object.c
+++ b/drivers/gpu/drm/radeon/radeon_object.c
@@ -429,7 +429,7 @@ int radeon_bo_evict_vram(struct radeon_device *rdev)
 			return 0;
 	}
 #endif
-	return ttm_bo_evict_mm(&rdev->mman.bdev, TTM_PL_VRAM);
+	return ttm_bo_evict_mm(&rdev->mman.bdev, TTM_PL_VRAM, false);
 }
 
 void radeon_bo_force_delete(struct radeon_device *rdev)
diff --git a/drivers/gpu/drm/ttm/ttm_bo.c b/drivers/gpu/drm/ttm/ttm_bo.c
index eae61cc..a17f87c2 100644
--- a/drivers/gpu/drm/ttm/ttm_bo.c
+++ b/drivers/gpu/drm/ttm/ttm_bo.c
@@ -42,6 +42,7 @@
 #include <linux/module.h>
 #include <linux/atomic.h>
 #include <linux/dma-resv.h>
+#include <drm/drm_drv.h>
 
 static void ttm_bo_global_kobj_release(struct kobject *kobj);
 
@@ -652,7 +653,8 @@ void ttm_bo_unlock_delayed_workqueue(struct ttm_bo_device *bdev, int resched)
 EXPORT_SYMBOL(ttm_bo_unlock_delayed_workqueue);
 
 static int ttm_bo_evict(struct ttm_buffer_object *bo,
-			struct ttm_operation_ctx *ctx)
+			struct ttm_operation_ctx *ctx,
+			bool destroy)
 {
 	struct ttm_bo_device *bdev = bo->bdev;
 	struct ttm_mem_reg evict_mem;
@@ -665,18 +667,23 @@ static int ttm_bo_evict(struct ttm_buffer_object *bo,
 	placement.num_busy_placement = 0;
 	bdev->driver->evict_flags(bo, &placement);
 
-	if (!placement.num_placement && !placement.num_busy_placement) {
+	evict_mem = bo->mem;
+	evict_mem.mm_node = NULL;
+	evict_mem.bus.io_reserved_vm = false;
+	evict_mem.bus.io_reserved_count = 0;
+
+	if (destroy || !placement.num_placement && !placement.num_busy_placement) {
 		ret = ttm_bo_pipeline_gutting(bo);
 		if (ret)
 			return ret;
 
-		return ttm_tt_create(bo, false);
-	}
+		ret = ttm_tt_create(bo, false);
 
-	evict_mem = bo->mem;
-	evict_mem.mm_node = NULL;
-	evict_mem.bus.io_reserved_vm = false;
-	evict_mem.bus.io_reserved_count = 0;
+		if (bdev->driver->move_notify)
+			bdev->driver->move_notify(bo, true, &evict_mem);
+
+		return ret;
+	}
 
 	ret = ttm_bo_mem_space(bo, &placement, &evict_mem, ctx);
 	if (ret) {
@@ -785,7 +792,8 @@ static int ttm_mem_evict_first(struct ttm_bo_device *bdev,
 			       uint32_t mem_type,
 			       const struct ttm_place *place,
 			       struct ttm_operation_ctx *ctx,
-			       struct ww_acquire_ctx *ticket)
+			       struct ww_acquire_ctx *ticket,
+			       bool destroy)
 {
 	struct ttm_buffer_object *bo = NULL, *busy_bo = NULL;
 	struct ttm_mem_type_manager *man = &bdev->man[mem_type];
@@ -846,7 +854,7 @@ static int ttm_mem_evict_first(struct ttm_bo_device *bdev,
 
 	spin_unlock(&ttm_bo_glob.lru_lock);
 
-	ret = ttm_bo_evict(bo, ctx);
+	ret = ttm_bo_evict(bo, ctx, destroy);
 	if (locked)
 		ttm_bo_unreserve(bo);
 
@@ -919,7 +927,7 @@ static int ttm_bo_mem_force_space(struct ttm_buffer_object *bo,
 		if (mem->mm_node)
 			break;
 		ret = ttm_mem_evict_first(bdev, mem->mem_type, place, ctx,
-					  ticket);
+					  ticket, false);
 		if (unlikely(ret != 0))
 			return ret;
 	} while (1);
@@ -1428,7 +1436,8 @@ int ttm_bo_create(struct ttm_bo_device *bdev,
 EXPORT_SYMBOL(ttm_bo_create);
 
 static int ttm_bo_force_list_clean(struct ttm_bo_device *bdev,
-				   unsigned mem_type)
+				   unsigned mem_type,
+				   bool destroy)
 {
 	struct ttm_operation_ctx ctx = {
 		.interruptible = false,
@@ -1450,7 +1459,7 @@ static int ttm_bo_force_list_clean(struct ttm_bo_device *bdev,
 		while (!list_empty(&man->lru[i])) {
 			spin_unlock(&glob->lru_lock);
 			ret = ttm_mem_evict_first(bdev, mem_type, NULL, &ctx,
-						  NULL);
+						  NULL, destroy);
 			if (ret)
 				return ret;
 			spin_lock(&glob->lru_lock);
@@ -1494,7 +1503,7 @@ int ttm_bo_clean_mm(struct ttm_bo_device *bdev, unsigned mem_type)
 
 	ret = 0;
 	if (mem_type > 0) {
-		ret = ttm_bo_force_list_clean(bdev, mem_type);
+		ret = ttm_bo_force_list_clean(bdev, mem_type, false);
 		if (ret) {
 			pr_err("Cleanup eviction failed\n");
 			return ret;
@@ -1510,7 +1519,7 @@ int ttm_bo_clean_mm(struct ttm_bo_device *bdev, unsigned mem_type)
 }
 EXPORT_SYMBOL(ttm_bo_clean_mm);
 
-int ttm_bo_evict_mm(struct ttm_bo_device *bdev, unsigned mem_type)
+int ttm_bo_evict_mm(struct ttm_bo_device *bdev, unsigned mem_type, bool destroy)
 {
 	struct ttm_mem_type_manager *man = &bdev->man[mem_type];
 
@@ -1524,7 +1533,7 @@ int ttm_bo_evict_mm(struct ttm_bo_device *bdev, unsigned mem_type)
 		return 0;
 	}
 
-	return ttm_bo_force_list_clean(bdev, mem_type);
+	return ttm_bo_force_list_clean(bdev, mem_type, destroy);
 }
 EXPORT_SYMBOL(ttm_bo_evict_mm);
 
diff --git a/drivers/gpu/drm/vmwgfx/vmwgfx_drv.c b/drivers/gpu/drm/vmwgfx/vmwgfx_drv.c
index 71e45b56..350064f 100644
--- a/drivers/gpu/drm/vmwgfx/vmwgfx_drv.c
+++ b/drivers/gpu/drm/vmwgfx/vmwgfx_drv.c
@@ -468,7 +468,7 @@ static int vmw_request_device(struct vmw_private *dev_priv)
 	if (dev_priv->cman)
 		vmw_cmdbuf_remove_pool(dev_priv->cman);
 	if (dev_priv->has_mob) {
-		(void) ttm_bo_evict_mm(&dev_priv->bdev, VMW_PL_MOB);
+		(void) ttm_bo_evict_mm(&dev_priv->bdev, VMW_PL_MOB, false);
 		vmw_otables_takedown(dev_priv);
 	}
 	if (dev_priv->cman)
@@ -501,7 +501,7 @@ static void vmw_release_device_early(struct vmw_private *dev_priv)
 		vmw_cmdbuf_remove_pool(dev_priv->cman);
 
 	if (dev_priv->has_mob) {
-		ttm_bo_evict_mm(&dev_priv->bdev, VMW_PL_MOB);
+		ttm_bo_evict_mm(&dev_priv->bdev, VMW_PL_MOB, false);
 		vmw_otables_takedown(dev_priv);
 	}
 }
@@ -1227,7 +1227,7 @@ void vmw_svga_disable(struct vmw_private *dev_priv)
 	if (dev_priv->bdev.man[TTM_PL_VRAM].use_type) {
 		dev_priv->bdev.man[TTM_PL_VRAM].use_type = false;
 		spin_unlock(&dev_priv->svga_lock);
-		if (ttm_bo_evict_mm(&dev_priv->bdev, TTM_PL_VRAM))
+		if (ttm_bo_evict_mm(&dev_priv->bdev, TTM_PL_VRAM, false))
 			DRM_ERROR("Failed evicting VRAM buffers.\n");
 		vmw_write(dev_priv, SVGA_REG_ENABLE,
 			  SVGA_REG_ENABLE_HIDE |
diff --git a/include/drm/ttm/ttm_bo_api.h b/include/drm/ttm/ttm_bo_api.h
index b9bc1b0..9d57b8c 100644
--- a/include/drm/ttm/ttm_bo_api.h
+++ b/include/drm/ttm/ttm_bo_api.h
@@ -597,7 +597,7 @@ int ttm_bo_clean_mm(struct ttm_bo_device *bdev, unsigned mem_type);
  * -ERESTARTSYS: The call was interrupted by a signal while waiting to
  * evict a buffer.
  */
-int ttm_bo_evict_mm(struct ttm_bo_device *bdev, unsigned mem_type);
+int ttm_bo_evict_mm(struct ttm_bo_device *bdev, unsigned mem_type, bool destroy);
 
 /**
  * ttm_kmap_obj_virtual
-- 
2.7.4

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

[PATCH 5/6] drm/ttm: Add destroy flag in TTM BO eviction interface

[Andrey Grodzovsky]

This will allow to invalidate, destroy backing storage and notify users
of BOs when device is unpluged.

Signed-off-by: Andrey Grodzovsky <andrey.grodzovsky@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c |  2 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_object.c  |  2 +-
 drivers/gpu/drm/nouveau/nouveau_drm.c       |  2 +-
 drivers/gpu/drm/qxl/qxl_object.c            |  4 +--
 drivers/gpu/drm/radeon/radeon_object.c      |  2 +-
 drivers/gpu/drm/ttm/ttm_bo.c                | 41 ++++++++++++++++++-----------
 drivers/gpu/drm/vmwgfx/vmwgfx_drv.c         |  6 ++---
 include/drm/ttm/ttm_bo_api.h                |  2 +-
 8 files changed, 35 insertions(+), 26 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c
index 1a4894f..f96c6d1 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c
@@ -1145,7 +1145,7 @@ static int amdgpu_debugfs_evict_gtt(struct seq_file *m, void *data)
 	if (r < 0)
 		return r;
 
-	seq_printf(m, "(%d)\n", ttm_bo_evict_mm(&adev->mman.bdev, TTM_PL_TT));
+	seq_printf(m, "(%d)\n", ttm_bo_evict_mm(&adev->mman.bdev, TTM_PL_TT, false));
 
 	pm_runtime_mark_last_busy(dev->dev);
 	pm_runtime_put_autosuspend(dev->dev);
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
index 22afd11..82d43d0 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
@@ -1046,7 +1046,7 @@ int amdgpu_bo_evict_vram(struct amdgpu_device *adev)
 		return 0;
 	}
 #endif
-	return ttm_bo_evict_mm(&adev->mman.bdev, TTM_PL_VRAM);
+	return ttm_bo_evict_mm(&adev->mman.bdev, TTM_PL_VRAM, false);
 }
 
 static const char *amdgpu_vram_names[] = {
diff --git a/drivers/gpu/drm/nouveau/nouveau_drm.c b/drivers/gpu/drm/nouveau/nouveau_drm.c
index 6b1629c..f3eea89 100644
--- a/drivers/gpu/drm/nouveau/nouveau_drm.c
+++ b/drivers/gpu/drm/nouveau/nouveau_drm.c
@@ -759,7 +759,7 @@ nouveau_do_suspend(struct drm_device *dev, bool runtime)
 	}
 
 	NV_DEBUG(drm, "evicting buffers...\n");
-	ttm_bo_evict_mm(&drm->ttm.bdev, TTM_PL_VRAM);
+	ttm_bo_evict_mm(&drm->ttm.bdev, TTM_PL_VRAM, false);
 
 	NV_DEBUG(drm, "waiting for kernel channels to go idle...\n");
 	if (drm->cechan) {
diff --git a/drivers/gpu/drm/qxl/qxl_object.c b/drivers/gpu/drm/qxl/qxl_object.c
index ab72dc3..45bb89b 100644
--- a/drivers/gpu/drm/qxl/qxl_object.c
+++ b/drivers/gpu/drm/qxl/qxl_object.c
@@ -359,10 +359,10 @@ int qxl_bo_check_id(struct qxl_device *qdev, struct qxl_bo *bo)
 
 int qxl_surf_evict(struct qxl_device *qdev)
 {
-	return ttm_bo_evict_mm(&qdev->mman.bdev, TTM_PL_PRIV);
+	return ttm_bo_evict_mm(&qdev->mman.bdev, TTM_PL_PRIV, false);
 }
 
 int qxl_vram_evict(struct qxl_device *qdev)
 {
-	return ttm_bo_evict_mm(&qdev->mman.bdev, TTM_PL_VRAM);
+	return ttm_bo_evict_mm(&qdev->mman.bdev, TTM_PL_VRAM, false);
 }
diff --git a/drivers/gpu/drm/radeon/radeon_object.c b/drivers/gpu/drm/radeon/radeon_object.c
index 140d94c..3eeeb8d 100644
--- a/drivers/gpu/drm/radeon/radeon_object.c
+++ b/drivers/gpu/drm/radeon/radeon_object.c
@@ -429,7 +429,7 @@ int radeon_bo_evict_vram(struct radeon_device *rdev)
 			return 0;
 	}
 #endif
-	return ttm_bo_evict_mm(&rdev->mman.bdev, TTM_PL_VRAM);
+	return ttm_bo_evict_mm(&rdev->mman.bdev, TTM_PL_VRAM, false);
 }
 
 void radeon_bo_force_delete(struct radeon_device *rdev)
diff --git a/drivers/gpu/drm/ttm/ttm_bo.c b/drivers/gpu/drm/ttm/ttm_bo.c
index eae61cc..a17f87c2 100644
--- a/drivers/gpu/drm/ttm/ttm_bo.c
+++ b/drivers/gpu/drm/ttm/ttm_bo.c
@@ -42,6 +42,7 @@
 #include <linux/module.h>
 #include <linux/atomic.h>
 #include <linux/dma-resv.h>
+#include <drm/drm_drv.h>
 
 static void ttm_bo_global_kobj_release(struct kobject *kobj);
 
@@ -652,7 +653,8 @@ void ttm_bo_unlock_delayed_workqueue(struct ttm_bo_device *bdev, int resched)
 EXPORT_SYMBOL(ttm_bo_unlock_delayed_workqueue);
 
 static int ttm_bo_evict(struct ttm_buffer_object *bo,
-			struct ttm_operation_ctx *ctx)
+			struct ttm_operation_ctx *ctx,
+			bool destroy)
 {
 	struct ttm_bo_device *bdev = bo->bdev;
 	struct ttm_mem_reg evict_mem;
@@ -665,18 +667,23 @@ static int ttm_bo_evict(struct ttm_buffer_object *bo,
 	placement.num_busy_placement = 0;
 	bdev->driver->evict_flags(bo, &placement);
 
-	if (!placement.num_placement && !placement.num_busy_placement) {
+	evict_mem = bo->mem;
+	evict_mem.mm_node = NULL;
+	evict_mem.bus.io_reserved_vm = false;
+	evict_mem.bus.io_reserved_count = 0;
+
+	if (destroy || !placement.num_placement && !placement.num_busy_placement) {
 		ret = ttm_bo_pipeline_gutting(bo);
 		if (ret)
 			return ret;
 
-		return ttm_tt_create(bo, false);
-	}
+		ret = ttm_tt_create(bo, false);
 
-	evict_mem = bo->mem;
-	evict_mem.mm_node = NULL;
-	evict_mem.bus.io_reserved_vm = false;
-	evict_mem.bus.io_reserved_count = 0;
+		if (bdev->driver->move_notify)
+			bdev->driver->move_notify(bo, true, &evict_mem);
+
+		return ret;
+	}
 
 	ret = ttm_bo_mem_space(bo, &placement, &evict_mem, ctx);
 	if (ret) {
@@ -785,7 +792,8 @@ static int ttm_mem_evict_first(struct ttm_bo_device *bdev,
 			       uint32_t mem_type,
 			       const struct ttm_place *place,
 			       struct ttm_operation_ctx *ctx,
-			       struct ww_acquire_ctx *ticket)
+			       struct ww_acquire_ctx *ticket,
+			       bool destroy)
 {
 	struct ttm_buffer_object *bo = NULL, *busy_bo = NULL;
 	struct ttm_mem_type_manager *man = &bdev->man[mem_type];
@@ -846,7 +854,7 @@ static int ttm_mem_evict_first(struct ttm_bo_device *bdev,
 
 	spin_unlock(&ttm_bo_glob.lru_lock);
 
-	ret = ttm_bo_evict(bo, ctx);
+	ret = ttm_bo_evict(bo, ctx, destroy);
 	if (locked)
 		ttm_bo_unreserve(bo);
 
@@ -919,7 +927,7 @@ static int ttm_bo_mem_force_space(struct ttm_buffer_object *bo,
 		if (mem->mm_node)
 			break;
 		ret = ttm_mem_evict_first(bdev, mem->mem_type, place, ctx,
-					  ticket);
+					  ticket, false);
 		if (unlikely(ret != 0))
 			return ret;
 	} while (1);
@@ -1428,7 +1436,8 @@ int ttm_bo_create(struct ttm_bo_device *bdev,
 EXPORT_SYMBOL(ttm_bo_create);
 
 static int ttm_bo_force_list_clean(struct ttm_bo_device *bdev,
-				   unsigned mem_type)
+				   unsigned mem_type,
+				   bool destroy)
 {
 	struct ttm_operation_ctx ctx = {
 		.interruptible = false,
@@ -1450,7 +1459,7 @@ static int ttm_bo_force_list_clean(struct ttm_bo_device *bdev,
 		while (!list_empty(&man->lru[i])) {
 			spin_unlock(&glob->lru_lock);
 			ret = ttm_mem_evict_first(bdev, mem_type, NULL, &ctx,
-						  NULL);
+						  NULL, destroy);
 			if (ret)
 				return ret;
 			spin_lock(&glob->lru_lock);
@@ -1494,7 +1503,7 @@ int ttm_bo_clean_mm(struct ttm_bo_device *bdev, unsigned mem_type)
 
 	ret = 0;
 	if (mem_type > 0) {
-		ret = ttm_bo_force_list_clean(bdev, mem_type);
+		ret = ttm_bo_force_list_clean(bdev, mem_type, false);
 		if (ret) {
 			pr_err("Cleanup eviction failed\n");
 			return ret;
@@ -1510,7 +1519,7 @@ int ttm_bo_clean_mm(struct ttm_bo_device *bdev, unsigned mem_type)
 }
 EXPORT_SYMBOL(ttm_bo_clean_mm);
 
-int ttm_bo_evict_mm(struct ttm_bo_device *bdev, unsigned mem_type)
+int ttm_bo_evict_mm(struct ttm_bo_device *bdev, unsigned mem_type, bool destroy)
 {
 	struct ttm_mem_type_manager *man = &bdev->man[mem_type];
 
@@ -1524,7 +1533,7 @@ int ttm_bo_evict_mm(struct ttm_bo_device *bdev, unsigned mem_type)
 		return 0;
 	}
 
-	return ttm_bo_force_list_clean(bdev, mem_type);
+	return ttm_bo_force_list_clean(bdev, mem_type, destroy);
 }
 EXPORT_SYMBOL(ttm_bo_evict_mm);
 
diff --git a/drivers/gpu/drm/vmwgfx/vmwgfx_drv.c b/drivers/gpu/drm/vmwgfx/vmwgfx_drv.c
index 71e45b56..350064f 100644
--- a/drivers/gpu/drm/vmwgfx/vmwgfx_drv.c
+++ b/drivers/gpu/drm/vmwgfx/vmwgfx_drv.c
@@ -468,7 +468,7 @@ static int vmw_request_device(struct vmw_private *dev_priv)
 	if (dev_priv->cman)
 		vmw_cmdbuf_remove_pool(dev_priv->cman);
 	if (dev_priv->has_mob) {
-		(void) ttm_bo_evict_mm(&dev_priv->bdev, VMW_PL_MOB);
+		(void) ttm_bo_evict_mm(&dev_priv->bdev, VMW_PL_MOB, false);
 		vmw_otables_takedown(dev_priv);
 	}
 	if (dev_priv->cman)
@@ -501,7 +501,7 @@ static void vmw_release_device_early(struct vmw_private *dev_priv)
 		vmw_cmdbuf_remove_pool(dev_priv->cman);
 
 	if (dev_priv->has_mob) {
-		ttm_bo_evict_mm(&dev_priv->bdev, VMW_PL_MOB);
+		ttm_bo_evict_mm(&dev_priv->bdev, VMW_PL_MOB, false);
 		vmw_otables_takedown(dev_priv);
 	}
 }
@@ -1227,7 +1227,7 @@ void vmw_svga_disable(struct vmw_private *dev_priv)
 	if (dev_priv->bdev.man[TTM_PL_VRAM].use_type) {
 		dev_priv->bdev.man[TTM_PL_VRAM].use_type = false;
 		spin_unlock(&dev_priv->svga_lock);
-		if (ttm_bo_evict_mm(&dev_priv->bdev, TTM_PL_VRAM))
+		if (ttm_bo_evict_mm(&dev_priv->bdev, TTM_PL_VRAM, false))
 			DRM_ERROR("Failed evicting VRAM buffers.\n");
 		vmw_write(dev_priv, SVGA_REG_ENABLE,
 			  SVGA_REG_ENABLE_HIDE |
diff --git a/include/drm/ttm/ttm_bo_api.h b/include/drm/ttm/ttm_bo_api.h
index b9bc1b0..9d57b8c 100644
--- a/include/drm/ttm/ttm_bo_api.h
+++ b/include/drm/ttm/ttm_bo_api.h
@@ -597,7 +597,7 @@ int ttm_bo_clean_mm(struct ttm_bo_device *bdev, unsigned mem_type);
  * -ERESTARTSYS: The call was interrupted by a signal while waiting to
  * evict a buffer.
  */
-int ttm_bo_evict_mm(struct ttm_bo_device *bdev, unsigned mem_type);
+int ttm_bo_evict_mm(struct ttm_bo_device *bdev, unsigned mem_type, bool destroy);
 
 /**
  * ttm_kmap_obj_virtual
-- 
2.7.4

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH 5/6] drm/ttm: Add destroy flag in TTM BO eviction interface

[Christian König]

While this looks rather nice it is most likely the wrong approach since 
it doesn't handles other concurrent evictions.

It would most likely be better to add a check calling 
drm_dev_is_unplugged() into amdgpu_bo_move().

Regards,
Christian.

Am 09.05.20 um 20:51 schrieb Andrey Grodzovsky:
> This will allow to invalidate, destroy backing storage and notify users
> of BOs when device is unpluged.
>
> Signed-off-by: Andrey Grodzovsky <andrey.grodzovsky@amd.com>
> ---
>   drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c |  2 +-
>   drivers/gpu/drm/amd/amdgpu/amdgpu_object.c  |  2 +-
>   drivers/gpu/drm/nouveau/nouveau_drm.c       |  2 +-
>   drivers/gpu/drm/qxl/qxl_object.c            |  4 +--
>   drivers/gpu/drm/radeon/radeon_object.c      |  2 +-
>   drivers/gpu/drm/ttm/ttm_bo.c                | 41 ++++++++++++++++++-----------
>   drivers/gpu/drm/vmwgfx/vmwgfx_drv.c         |  6 ++---
>   include/drm/ttm/ttm_bo_api.h                |  2 +-
>   8 files changed, 35 insertions(+), 26 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c
> index 1a4894f..f96c6d1 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c
> @@ -1145,7 +1145,7 @@ static int amdgpu_debugfs_evict_gtt(struct seq_file *m, void *data)
>   	if (r < 0)
>   		return r;
>   
> -	seq_printf(m, "(%d)\n", ttm_bo_evict_mm(&adev->mman.bdev, TTM_PL_TT));
> +	seq_printf(m, "(%d)\n", ttm_bo_evict_mm(&adev->mman.bdev, TTM_PL_TT, false));
>   
>   	pm_runtime_mark_last_busy(dev->dev);
>   	pm_runtime_put_autosuspend(dev->dev);
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
> index 22afd11..82d43d0 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
> @@ -1046,7 +1046,7 @@ int amdgpu_bo_evict_vram(struct amdgpu_device *adev)
>   		return 0;
>   	}
>   #endif
> -	return ttm_bo_evict_mm(&adev->mman.bdev, TTM_PL_VRAM);
> +	return ttm_bo_evict_mm(&adev->mman.bdev, TTM_PL_VRAM, false);
>   }
>   
>   static const char *amdgpu_vram_names[] = {
> diff --git a/drivers/gpu/drm/nouveau/nouveau_drm.c b/drivers/gpu/drm/nouveau/nouveau_drm.c
> index 6b1629c..f3eea89 100644
> --- a/drivers/gpu/drm/nouveau/nouveau_drm.c
> +++ b/drivers/gpu/drm/nouveau/nouveau_drm.c
> @@ -759,7 +759,7 @@ nouveau_do_suspend(struct drm_device *dev, bool runtime)
>   	}
>   
>   	NV_DEBUG(drm, "evicting buffers...\n");
> -	ttm_bo_evict_mm(&drm->ttm.bdev, TTM_PL_VRAM);
> +	ttm_bo_evict_mm(&drm->ttm.bdev, TTM_PL_VRAM, false);
>   
>   	NV_DEBUG(drm, "waiting for kernel channels to go idle...\n");
>   	if (drm->cechan) {
> diff --git a/drivers/gpu/drm/qxl/qxl_object.c b/drivers/gpu/drm/qxl/qxl_object.c
> index ab72dc3..45bb89b 100644
> --- a/drivers/gpu/drm/qxl/qxl_object.c
> +++ b/drivers/gpu/drm/qxl/qxl_object.c
> @@ -359,10 +359,10 @@ int qxl_bo_check_id(struct qxl_device *qdev, struct qxl_bo *bo)
>   
>   int qxl_surf_evict(struct qxl_device *qdev)
>   {
> -	return ttm_bo_evict_mm(&qdev->mman.bdev, TTM_PL_PRIV);
> +	return ttm_bo_evict_mm(&qdev->mman.bdev, TTM_PL_PRIV, false);
>   }
>   
>   int qxl_vram_evict(struct qxl_device *qdev)
>   {
> -	return ttm_bo_evict_mm(&qdev->mman.bdev, TTM_PL_VRAM);
> +	return ttm_bo_evict_mm(&qdev->mman.bdev, TTM_PL_VRAM, false);
>   }
> diff --git a/drivers/gpu/drm/radeon/radeon_object.c b/drivers/gpu/drm/radeon/radeon_object.c
> index 140d94c..3eeeb8d 100644
> --- a/drivers/gpu/drm/radeon/radeon_object.c
> +++ b/drivers/gpu/drm/radeon/radeon_object.c
> @@ -429,7 +429,7 @@ int radeon_bo_evict_vram(struct radeon_device *rdev)
>   			return 0;
>   	}
>   #endif
> -	return ttm_bo_evict_mm(&rdev->mman.bdev, TTM_PL_VRAM);
> +	return ttm_bo_evict_mm(&rdev->mman.bdev, TTM_PL_VRAM, false);
>   }
>   
>   void radeon_bo_force_delete(struct radeon_device *rdev)
> diff --git a/drivers/gpu/drm/ttm/ttm_bo.c b/drivers/gpu/drm/ttm/ttm_bo.c
> index eae61cc..a17f87c2 100644
> --- a/drivers/gpu/drm/ttm/ttm_bo.c
> +++ b/drivers/gpu/drm/ttm/ttm_bo.c
> @@ -42,6 +42,7 @@
>   #include <linux/module.h>
>   #include <linux/atomic.h>
>   #include <linux/dma-resv.h>
> +#include <drm/drm_drv.h>
>   
>   static void ttm_bo_global_kobj_release(struct kobject *kobj);
>   
> @@ -652,7 +653,8 @@ void ttm_bo_unlock_delayed_workqueue(struct ttm_bo_device *bdev, int resched)
>   EXPORT_SYMBOL(ttm_bo_unlock_delayed_workqueue);
>   
>   static int ttm_bo_evict(struct ttm_buffer_object *bo,
> -			struct ttm_operation_ctx *ctx)
> +			struct ttm_operation_ctx *ctx,
> +			bool destroy)
>   {
>   	struct ttm_bo_device *bdev = bo->bdev;
>   	struct ttm_mem_reg evict_mem;
> @@ -665,18 +667,23 @@ static int ttm_bo_evict(struct ttm_buffer_object *bo,
>   	placement.num_busy_placement = 0;
>   	bdev->driver->evict_flags(bo, &placement);
>   
> -	if (!placement.num_placement && !placement.num_busy_placement) {
> +	evict_mem = bo->mem;
> +	evict_mem.mm_node = NULL;
> +	evict_mem.bus.io_reserved_vm = false;
> +	evict_mem.bus.io_reserved_count = 0;
> +
> +	if (destroy || !placement.num_placement && !placement.num_busy_placement) {
>   		ret = ttm_bo_pipeline_gutting(bo);
>   		if (ret)
>   			return ret;
>   
> -		return ttm_tt_create(bo, false);
> -	}
> +		ret = ttm_tt_create(bo, false);
>   
> -	evict_mem = bo->mem;
> -	evict_mem.mm_node = NULL;
> -	evict_mem.bus.io_reserved_vm = false;
> -	evict_mem.bus.io_reserved_count = 0;
> +		if (bdev->driver->move_notify)
> +			bdev->driver->move_notify(bo, true, &evict_mem);
> +
> +		return ret;
> +	}
>   
>   	ret = ttm_bo_mem_space(bo, &placement, &evict_mem, ctx);
>   	if (ret) {
> @@ -785,7 +792,8 @@ static int ttm_mem_evict_first(struct ttm_bo_device *bdev,
>   			       uint32_t mem_type,
>   			       const struct ttm_place *place,
>   			       struct ttm_operation_ctx *ctx,
> -			       struct ww_acquire_ctx *ticket)
> +			       struct ww_acquire_ctx *ticket,
> +			       bool destroy)
>   {
>   	struct ttm_buffer_object *bo = NULL, *busy_bo = NULL;
>   	struct ttm_mem_type_manager *man = &bdev->man[mem_type];
> @@ -846,7 +854,7 @@ static int ttm_mem_evict_first(struct ttm_bo_device *bdev,
>   
>   	spin_unlock(&ttm_bo_glob.lru_lock);
>   
> -	ret = ttm_bo_evict(bo, ctx);
> +	ret = ttm_bo_evict(bo, ctx, destroy);
>   	if (locked)
>   		ttm_bo_unreserve(bo);
>   
> @@ -919,7 +927,7 @@ static int ttm_bo_mem_force_space(struct ttm_buffer_object *bo,
>   		if (mem->mm_node)
>   			break;
>   		ret = ttm_mem_evict_first(bdev, mem->mem_type, place, ctx,
> -					  ticket);
> +					  ticket, false);
>   		if (unlikely(ret != 0))
>   			return ret;
>   	} while (1);
> @@ -1428,7 +1436,8 @@ int ttm_bo_create(struct ttm_bo_device *bdev,
>   EXPORT_SYMBOL(ttm_bo_create);
>   
>   static int ttm_bo_force_list_clean(struct ttm_bo_device *bdev,
> -				   unsigned mem_type)
> +				   unsigned mem_type,
> +				   bool destroy)
>   {
>   	struct ttm_operation_ctx ctx = {
>   		.interruptible = false,
> @@ -1450,7 +1459,7 @@ static int ttm_bo_force_list_clean(struct ttm_bo_device *bdev,
>   		while (!list_empty(&man->lru[i])) {
>   			spin_unlock(&glob->lru_lock);
>   			ret = ttm_mem_evict_first(bdev, mem_type, NULL, &ctx,
> -						  NULL);
> +						  NULL, destroy);
>   			if (ret)
>   				return ret;
>   			spin_lock(&glob->lru_lock);
> @@ -1494,7 +1503,7 @@ int ttm_bo_clean_mm(struct ttm_bo_device *bdev, unsigned mem_type)
>   
>   	ret = 0;
>   	if (mem_type > 0) {
> -		ret = ttm_bo_force_list_clean(bdev, mem_type);
> +		ret = ttm_bo_force_list_clean(bdev, mem_type, false);
>   		if (ret) {
>   			pr_err("Cleanup eviction failed\n");
>   			return ret;
> @@ -1510,7 +1519,7 @@ int ttm_bo_clean_mm(struct ttm_bo_device *bdev, unsigned mem_type)
>   }
>   EXPORT_SYMBOL(ttm_bo_clean_mm);
>   
> -int ttm_bo_evict_mm(struct ttm_bo_device *bdev, unsigned mem_type)
> +int ttm_bo_evict_mm(struct ttm_bo_device *bdev, unsigned mem_type, bool destroy)
>   {
>   	struct ttm_mem_type_manager *man = &bdev->man[mem_type];
>   
> @@ -1524,7 +1533,7 @@ int ttm_bo_evict_mm(struct ttm_bo_device *bdev, unsigned mem_type)
>   		return 0;
>   	}
>   
> -	return ttm_bo_force_list_clean(bdev, mem_type);
> +	return ttm_bo_force_list_clean(bdev, mem_type, destroy);
>   }
>   EXPORT_SYMBOL(ttm_bo_evict_mm);
>   
> diff --git a/drivers/gpu/drm/vmwgfx/vmwgfx_drv.c b/drivers/gpu/drm/vmwgfx/vmwgfx_drv.c
> index 71e45b56..350064f 100644
> --- a/drivers/gpu/drm/vmwgfx/vmwgfx_drv.c
> +++ b/drivers/gpu/drm/vmwgfx/vmwgfx_drv.c
> @@ -468,7 +468,7 @@ static int vmw_request_device(struct vmw_private *dev_priv)
>   	if (dev_priv->cman)
>   		vmw_cmdbuf_remove_pool(dev_priv->cman);
>   	if (dev_priv->has_mob) {
> -		(void) ttm_bo_evict_mm(&dev_priv->bdev, VMW_PL_MOB);
> +		(void) ttm_bo_evict_mm(&dev_priv->bdev, VMW_PL_MOB, false);
>   		vmw_otables_takedown(dev_priv);
>   	}
>   	if (dev_priv->cman)
> @@ -501,7 +501,7 @@ static void vmw_release_device_early(struct vmw_private *dev_priv)
>   		vmw_cmdbuf_remove_pool(dev_priv->cman);
>   
>   	if (dev_priv->has_mob) {
> -		ttm_bo_evict_mm(&dev_priv->bdev, VMW_PL_MOB);
> +		ttm_bo_evict_mm(&dev_priv->bdev, VMW_PL_MOB, false);
>   		vmw_otables_takedown(dev_priv);
>   	}
>   }
> @@ -1227,7 +1227,7 @@ void vmw_svga_disable(struct vmw_private *dev_priv)
>   	if (dev_priv->bdev.man[TTM_PL_VRAM].use_type) {
>   		dev_priv->bdev.man[TTM_PL_VRAM].use_type = false;
>   		spin_unlock(&dev_priv->svga_lock);
> -		if (ttm_bo_evict_mm(&dev_priv->bdev, TTM_PL_VRAM))
> +		if (ttm_bo_evict_mm(&dev_priv->bdev, TTM_PL_VRAM, false))
>   			DRM_ERROR("Failed evicting VRAM buffers.\n");
>   		vmw_write(dev_priv, SVGA_REG_ENABLE,
>   			  SVGA_REG_ENABLE_HIDE |
> diff --git a/include/drm/ttm/ttm_bo_api.h b/include/drm/ttm/ttm_bo_api.h
> index b9bc1b0..9d57b8c 100644
> --- a/include/drm/ttm/ttm_bo_api.h
> +++ b/include/drm/ttm/ttm_bo_api.h
> @@ -597,7 +597,7 @@ int ttm_bo_clean_mm(struct ttm_bo_device *bdev, unsigned mem_type);
>    * -ERESTARTSYS: The call was interrupted by a signal while waiting to
>    * evict a buffer.
>    */
> -int ttm_bo_evict_mm(struct ttm_bo_device *bdev, unsigned mem_type);
> +int ttm_bo_evict_mm(struct ttm_bo_device *bdev, unsigned mem_type, bool destroy);
>   
>   /**
>    * ttm_kmap_obj_virtual

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [PATCH 5/6] drm/ttm: Add destroy flag in TTM BO eviction interface

[Christian König]

While this looks rather nice it is most likely the wrong approach since 
it doesn't handles other concurrent evictions.

It would most likely be better to add a check calling 
drm_dev_is_unplugged() into amdgpu_bo_move().

Regards,
Christian.

Am 09.05.20 um 20:51 schrieb Andrey Grodzovsky:
> This will allow to invalidate, destroy backing storage and notify users
> of BOs when device is unpluged.
>
> Signed-off-by: Andrey Grodzovsky <andrey.grodzovsky@amd.com>
> ---
>   drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c |  2 +-
>   drivers/gpu/drm/amd/amdgpu/amdgpu_object.c  |  2 +-
>   drivers/gpu/drm/nouveau/nouveau_drm.c       |  2 +-
>   drivers/gpu/drm/qxl/qxl_object.c            |  4 +--
>   drivers/gpu/drm/radeon/radeon_object.c      |  2 +-
>   drivers/gpu/drm/ttm/ttm_bo.c                | 41 ++++++++++++++++++-----------
>   drivers/gpu/drm/vmwgfx/vmwgfx_drv.c         |  6 ++---
>   include/drm/ttm/ttm_bo_api.h                |  2 +-
>   8 files changed, 35 insertions(+), 26 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c
> index 1a4894f..f96c6d1 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c
> @@ -1145,7 +1145,7 @@ static int amdgpu_debugfs_evict_gtt(struct seq_file *m, void *data)
>   	if (r < 0)
>   		return r;
>   
> -	seq_printf(m, "(%d)\n", ttm_bo_evict_mm(&adev->mman.bdev, TTM_PL_TT));
> +	seq_printf(m, "(%d)\n", ttm_bo_evict_mm(&adev->mman.bdev, TTM_PL_TT, false));
>   
>   	pm_runtime_mark_last_busy(dev->dev);
>   	pm_runtime_put_autosuspend(dev->dev);
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
> index 22afd11..82d43d0 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
> @@ -1046,7 +1046,7 @@ int amdgpu_bo_evict_vram(struct amdgpu_device *adev)
>   		return 0;
>   	}
>   #endif
> -	return ttm_bo_evict_mm(&adev->mman.bdev, TTM_PL_VRAM);
> +	return ttm_bo_evict_mm(&adev->mman.bdev, TTM_PL_VRAM, false);
>   }
>   
>   static const char *amdgpu_vram_names[] = {
> diff --git a/drivers/gpu/drm/nouveau/nouveau_drm.c b/drivers/gpu/drm/nouveau/nouveau_drm.c
> index 6b1629c..f3eea89 100644
> --- a/drivers/gpu/drm/nouveau/nouveau_drm.c
> +++ b/drivers/gpu/drm/nouveau/nouveau_drm.c
> @@ -759,7 +759,7 @@ nouveau_do_suspend(struct drm_device *dev, bool runtime)
>   	}
>   
>   	NV_DEBUG(drm, "evicting buffers...\n");
> -	ttm_bo_evict_mm(&drm->ttm.bdev, TTM_PL_VRAM);
> +	ttm_bo_evict_mm(&drm->ttm.bdev, TTM_PL_VRAM, false);
>   
>   	NV_DEBUG(drm, "waiting for kernel channels to go idle...\n");
>   	if (drm->cechan) {
> diff --git a/drivers/gpu/drm/qxl/qxl_object.c b/drivers/gpu/drm/qxl/qxl_object.c
> index ab72dc3..45bb89b 100644
> --- a/drivers/gpu/drm/qxl/qxl_object.c
> +++ b/drivers/gpu/drm/qxl/qxl_object.c
> @@ -359,10 +359,10 @@ int qxl_bo_check_id(struct qxl_device *qdev, struct qxl_bo *bo)
>   
>   int qxl_surf_evict(struct qxl_device *qdev)
>   {
> -	return ttm_bo_evict_mm(&qdev->mman.bdev, TTM_PL_PRIV);
> +	return ttm_bo_evict_mm(&qdev->mman.bdev, TTM_PL_PRIV, false);
>   }
>   
>   int qxl_vram_evict(struct qxl_device *qdev)
>   {
> -	return ttm_bo_evict_mm(&qdev->mman.bdev, TTM_PL_VRAM);
> +	return ttm_bo_evict_mm(&qdev->mman.bdev, TTM_PL_VRAM, false);
>   }
> diff --git a/drivers/gpu/drm/radeon/radeon_object.c b/drivers/gpu/drm/radeon/radeon_object.c
> index 140d94c..3eeeb8d 100644
> --- a/drivers/gpu/drm/radeon/radeon_object.c
> +++ b/drivers/gpu/drm/radeon/radeon_object.c
> @@ -429,7 +429,7 @@ int radeon_bo_evict_vram(struct radeon_device *rdev)
>   			return 0;
>   	}
>   #endif
> -	return ttm_bo_evict_mm(&rdev->mman.bdev, TTM_PL_VRAM);
> +	return ttm_bo_evict_mm(&rdev->mman.bdev, TTM_PL_VRAM, false);
>   }
>   
>   void radeon_bo_force_delete(struct radeon_device *rdev)
> diff --git a/drivers/gpu/drm/ttm/ttm_bo.c b/drivers/gpu/drm/ttm/ttm_bo.c
> index eae61cc..a17f87c2 100644
> --- a/drivers/gpu/drm/ttm/ttm_bo.c
> +++ b/drivers/gpu/drm/ttm/ttm_bo.c
> @@ -42,6 +42,7 @@
>   #include <linux/module.h>
>   #include <linux/atomic.h>
>   #include <linux/dma-resv.h>
> +#include <drm/drm_drv.h>
>   
>   static void ttm_bo_global_kobj_release(struct kobject *kobj);
>   
> @@ -652,7 +653,8 @@ void ttm_bo_unlock_delayed_workqueue(struct ttm_bo_device *bdev, int resched)
>   EXPORT_SYMBOL(ttm_bo_unlock_delayed_workqueue);
>   
>   static int ttm_bo_evict(struct ttm_buffer_object *bo,
> -			struct ttm_operation_ctx *ctx)
> +			struct ttm_operation_ctx *ctx,
> +			bool destroy)
>   {
>   	struct ttm_bo_device *bdev = bo->bdev;
>   	struct ttm_mem_reg evict_mem;
> @@ -665,18 +667,23 @@ static int ttm_bo_evict(struct ttm_buffer_object *bo,
>   	placement.num_busy_placement = 0;
>   	bdev->driver->evict_flags(bo, &placement);
>   
> -	if (!placement.num_placement && !placement.num_busy_placement) {
> +	evict_mem = bo->mem;
> +	evict_mem.mm_node = NULL;
> +	evict_mem.bus.io_reserved_vm = false;
> +	evict_mem.bus.io_reserved_count = 0;
> +
> +	if (destroy || !placement.num_placement && !placement.num_busy_placement) {
>   		ret = ttm_bo_pipeline_gutting(bo);
>   		if (ret)
>   			return ret;
>   
> -		return ttm_tt_create(bo, false);
> -	}
> +		ret = ttm_tt_create(bo, false);
>   
> -	evict_mem = bo->mem;
> -	evict_mem.mm_node = NULL;
> -	evict_mem.bus.io_reserved_vm = false;
> -	evict_mem.bus.io_reserved_count = 0;
> +		if (bdev->driver->move_notify)
> +			bdev->driver->move_notify(bo, true, &evict_mem);
> +
> +		return ret;
> +	}
>   
>   	ret = ttm_bo_mem_space(bo, &placement, &evict_mem, ctx);
>   	if (ret) {
> @@ -785,7 +792,8 @@ static int ttm_mem_evict_first(struct ttm_bo_device *bdev,
>   			       uint32_t mem_type,
>   			       const struct ttm_place *place,
>   			       struct ttm_operation_ctx *ctx,
> -			       struct ww_acquire_ctx *ticket)
> +			       struct ww_acquire_ctx *ticket,
> +			       bool destroy)
>   {
>   	struct ttm_buffer_object *bo = NULL, *busy_bo = NULL;
>   	struct ttm_mem_type_manager *man = &bdev->man[mem_type];
> @@ -846,7 +854,7 @@ static int ttm_mem_evict_first(struct ttm_bo_device *bdev,
>   
>   	spin_unlock(&ttm_bo_glob.lru_lock);
>   
> -	ret = ttm_bo_evict(bo, ctx);
> +	ret = ttm_bo_evict(bo, ctx, destroy);
>   	if (locked)
>   		ttm_bo_unreserve(bo);
>   
> @@ -919,7 +927,7 @@ static int ttm_bo_mem_force_space(struct ttm_buffer_object *bo,
>   		if (mem->mm_node)
>   			break;
>   		ret = ttm_mem_evict_first(bdev, mem->mem_type, place, ctx,
> -					  ticket);
> +					  ticket, false);
>   		if (unlikely(ret != 0))
>   			return ret;
>   	} while (1);
> @@ -1428,7 +1436,8 @@ int ttm_bo_create(struct ttm_bo_device *bdev,
>   EXPORT_SYMBOL(ttm_bo_create);
>   
>   static int ttm_bo_force_list_clean(struct ttm_bo_device *bdev,
> -				   unsigned mem_type)
> +				   unsigned mem_type,
> +				   bool destroy)
>   {
>   	struct ttm_operation_ctx ctx = {
>   		.interruptible = false,
> @@ -1450,7 +1459,7 @@ static int ttm_bo_force_list_clean(struct ttm_bo_device *bdev,
>   		while (!list_empty(&man->lru[i])) {
>   			spin_unlock(&glob->lru_lock);
>   			ret = ttm_mem_evict_first(bdev, mem_type, NULL, &ctx,
> -						  NULL);
> +						  NULL, destroy);
>   			if (ret)
>   				return ret;
>   			spin_lock(&glob->lru_lock);
> @@ -1494,7 +1503,7 @@ int ttm_bo_clean_mm(struct ttm_bo_device *bdev, unsigned mem_type)
>   
>   	ret = 0;
>   	if (mem_type > 0) {
> -		ret = ttm_bo_force_list_clean(bdev, mem_type);
> +		ret = ttm_bo_force_list_clean(bdev, mem_type, false);
>   		if (ret) {
>   			pr_err("Cleanup eviction failed\n");
>   			return ret;
> @@ -1510,7 +1519,7 @@ int ttm_bo_clean_mm(struct ttm_bo_device *bdev, unsigned mem_type)
>   }
>   EXPORT_SYMBOL(ttm_bo_clean_mm);
>   
> -int ttm_bo_evict_mm(struct ttm_bo_device *bdev, unsigned mem_type)
> +int ttm_bo_evict_mm(struct ttm_bo_device *bdev, unsigned mem_type, bool destroy)
>   {
>   	struct ttm_mem_type_manager *man = &bdev->man[mem_type];
>   
> @@ -1524,7 +1533,7 @@ int ttm_bo_evict_mm(struct ttm_bo_device *bdev, unsigned mem_type)
>   		return 0;
>   	}
>   
> -	return ttm_bo_force_list_clean(bdev, mem_type);
> +	return ttm_bo_force_list_clean(bdev, mem_type, destroy);
>   }
>   EXPORT_SYMBOL(ttm_bo_evict_mm);
>   
> diff --git a/drivers/gpu/drm/vmwgfx/vmwgfx_drv.c b/drivers/gpu/drm/vmwgfx/vmwgfx_drv.c
> index 71e45b56..350064f 100644
> --- a/drivers/gpu/drm/vmwgfx/vmwgfx_drv.c
> +++ b/drivers/gpu/drm/vmwgfx/vmwgfx_drv.c
> @@ -468,7 +468,7 @@ static int vmw_request_device(struct vmw_private *dev_priv)
>   	if (dev_priv->cman)
>   		vmw_cmdbuf_remove_pool(dev_priv->cman);
>   	if (dev_priv->has_mob) {
> -		(void) ttm_bo_evict_mm(&dev_priv->bdev, VMW_PL_MOB);
> +		(void) ttm_bo_evict_mm(&dev_priv->bdev, VMW_PL_MOB, false);
>   		vmw_otables_takedown(dev_priv);
>   	}
>   	if (dev_priv->cman)
> @@ -501,7 +501,7 @@ static void vmw_release_device_early(struct vmw_private *dev_priv)
>   		vmw_cmdbuf_remove_pool(dev_priv->cman);
>   
>   	if (dev_priv->has_mob) {
> -		ttm_bo_evict_mm(&dev_priv->bdev, VMW_PL_MOB);
> +		ttm_bo_evict_mm(&dev_priv->bdev, VMW_PL_MOB, false);
>   		vmw_otables_takedown(dev_priv);
>   	}
>   }
> @@ -1227,7 +1227,7 @@ void vmw_svga_disable(struct vmw_private *dev_priv)
>   	if (dev_priv->bdev.man[TTM_PL_VRAM].use_type) {
>   		dev_priv->bdev.man[TTM_PL_VRAM].use_type = false;
>   		spin_unlock(&dev_priv->svga_lock);
> -		if (ttm_bo_evict_mm(&dev_priv->bdev, TTM_PL_VRAM))
> +		if (ttm_bo_evict_mm(&dev_priv->bdev, TTM_PL_VRAM, false))
>   			DRM_ERROR("Failed evicting VRAM buffers.\n");
>   		vmw_write(dev_priv, SVGA_REG_ENABLE,
>   			  SVGA_REG_ENABLE_HIDE |
> diff --git a/include/drm/ttm/ttm_bo_api.h b/include/drm/ttm/ttm_bo_api.h
> index b9bc1b0..9d57b8c 100644
> --- a/include/drm/ttm/ttm_bo_api.h
> +++ b/include/drm/ttm/ttm_bo_api.h
> @@ -597,7 +597,7 @@ int ttm_bo_clean_mm(struct ttm_bo_device *bdev, unsigned mem_type);
>    * -ERESTARTSYS: The call was interrupted by a signal while waiting to
>    * evict a buffer.
>    */
> -int ttm_bo_evict_mm(struct ttm_bo_device *bdev, unsigned mem_type);
> +int ttm_bo_evict_mm(struct ttm_bo_device *bdev, unsigned mem_type, bool destroy);
>   
>   /**
>    * ttm_kmap_obj_virtual

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH 5/6] drm/ttm: Add destroy flag in TTM BO eviction interface

[Thomas Hellström (Intel)]

On 5/9/20 8:51 PM, Andrey Grodzovsky wrote:
> This will allow to invalidate, destroy backing storage and notify users
> of BOs when device is unpluged.
>
> Signed-off-by: Andrey Grodzovsky <andrey.grodzovsky@amd.com>

Please add a motivation in the commit message and use imperative wording 
("Allow to invalidate..." instead of "This will allow to")

s /unpluged/unplugged/


> ---
>   drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c |  2 +-
>   drivers/gpu/drm/amd/amdgpu/amdgpu_object.c  |  2 +-
>   drivers/gpu/drm/nouveau/nouveau_drm.c       |  2 +-
>   drivers/gpu/drm/qxl/qxl_object.c            |  4 +--
>   drivers/gpu/drm/radeon/radeon_object.c      |  2 +-
>   drivers/gpu/drm/ttm/ttm_bo.c                | 41 ++++++++++++++++++-----------
>   drivers/gpu/drm/vmwgfx/vmwgfx_drv.c         |  6 ++---
>   include/drm/ttm/ttm_bo_api.h                |  2 +-
>   8 files changed, 35 insertions(+), 26 deletions(-)
>
> diff --git a/include/drm/ttm/ttm_bo_api.h b/include/drm/ttm/ttm_bo_api.h
> index b9bc1b0..9d57b8c 100644
> --- a/include/drm/ttm/ttm_bo_api.h
> +++ b/include/drm/ttm/ttm_bo_api.h
> @@ -597,7 +597,7 @@ int ttm_bo_clean_mm(struct ttm_bo_device *bdev, unsigned mem_type);
>    * -ERESTARTSYS: The call was interrupted by a signal while waiting to
>    * evict a buffer.
>    */

Please also update the function documentation.

> -int ttm_bo_evict_mm(struct ttm_bo_device *bdev, unsigned mem_type);
> +int ttm_bo_evict_mm(struct ttm_bo_device *bdev, unsigned mem_type, bool destroy);
>   
>   /**
>    * ttm_kmap_obj_virtual


Thanks,

Thomas


_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [PATCH 5/6] drm/ttm: Add destroy flag in TTM BO eviction interface

[Thomas Hellström (Intel)]

On 5/9/20 8:51 PM, Andrey Grodzovsky wrote:
> This will allow to invalidate, destroy backing storage and notify users
> of BOs when device is unpluged.
>
> Signed-off-by: Andrey Grodzovsky <andrey.grodzovsky@amd.com>

Please add a motivation in the commit message and use imperative wording 
("Allow to invalidate..." instead of "This will allow to")

s /unpluged/unplugged/


> ---
>   drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c |  2 +-
>   drivers/gpu/drm/amd/amdgpu/amdgpu_object.c  |  2 +-
>   drivers/gpu/drm/nouveau/nouveau_drm.c       |  2 +-
>   drivers/gpu/drm/qxl/qxl_object.c            |  4 +--
>   drivers/gpu/drm/radeon/radeon_object.c      |  2 +-
>   drivers/gpu/drm/ttm/ttm_bo.c                | 41 ++++++++++++++++++-----------
>   drivers/gpu/drm/vmwgfx/vmwgfx_drv.c         |  6 ++---
>   include/drm/ttm/ttm_bo_api.h                |  2 +-
>   8 files changed, 35 insertions(+), 26 deletions(-)
>
> diff --git a/include/drm/ttm/ttm_bo_api.h b/include/drm/ttm/ttm_bo_api.h
> index b9bc1b0..9d57b8c 100644
> --- a/include/drm/ttm/ttm_bo_api.h
> +++ b/include/drm/ttm/ttm_bo_api.h
> @@ -597,7 +597,7 @@ int ttm_bo_clean_mm(struct ttm_bo_device *bdev, unsigned mem_type);
>    * -ERESTARTSYS: The call was interrupted by a signal while waiting to
>    * evict a buffer.
>    */

Please also update the function documentation.

> -int ttm_bo_evict_mm(struct ttm_bo_device *bdev, unsigned mem_type);
> +int ttm_bo_evict_mm(struct ttm_bo_device *bdev, unsigned mem_type, bool destroy);
>   
>   /**
>    * ttm_kmap_obj_virtual


Thanks,

Thomas


_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH 5/6] drm/ttm: Add destroy flag in TTM BO eviction interface

[Andrey Grodzovsky]

On 6/10/20 6:25 AM, Thomas Hellström (Intel) wrote:
>
> On 5/9/20 8:51 PM, Andrey Grodzovsky wrote:
>> This will allow to invalidate, destroy backing storage and notify users
>> of BOs when device is unpluged.
>>
>> Signed-off-by: Andrey Grodzovsky <andrey.grodzovsky@amd.com>
>
> Please add a motivation in the commit message and use imperative 
> wording ("Allow to invalidate..." instead of "This will allow to")
>
> s /unpluged/unplugged/


I am not sure yet this patch is needed in V2

Andrey


>
>
>> ---
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c |  2 +-
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_object.c  |  2 +-
>>   drivers/gpu/drm/nouveau/nouveau_drm.c       |  2 +-
>>   drivers/gpu/drm/qxl/qxl_object.c            |  4 +--
>>   drivers/gpu/drm/radeon/radeon_object.c      |  2 +-
>>   drivers/gpu/drm/ttm/ttm_bo.c                | 41 
>> ++++++++++++++++++-----------
>>   drivers/gpu/drm/vmwgfx/vmwgfx_drv.c         |  6 ++---
>>   include/drm/ttm/ttm_bo_api.h                |  2 +-
>>   8 files changed, 35 insertions(+), 26 deletions(-)
>>
>> diff --git a/include/drm/ttm/ttm_bo_api.h b/include/drm/ttm/ttm_bo_api.h
>> index b9bc1b0..9d57b8c 100644
>> --- a/include/drm/ttm/ttm_bo_api.h
>> +++ b/include/drm/ttm/ttm_bo_api.h
>> @@ -597,7 +597,7 @@ int ttm_bo_clean_mm(struct ttm_bo_device *bdev, 
>> unsigned mem_type);
>>    * -ERESTARTSYS: The call was interrupted by a signal while waiting to
>>    * evict a buffer.
>>    */
>
> Please also update the function documentation.
>
>> -int ttm_bo_evict_mm(struct ttm_bo_device *bdev, unsigned mem_type);
>> +int ttm_bo_evict_mm(struct ttm_bo_device *bdev, unsigned mem_type, 
>> bool destroy);
>>     /**
>>    * ttm_kmap_obj_virtual
>
>
> Thanks,
>
> Thomas
>
>
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [PATCH 5/6] drm/ttm: Add destroy flag in TTM BO eviction interface

[Andrey Grodzovsky]

On 6/10/20 6:25 AM, Thomas Hellström (Intel) wrote:
>
> On 5/9/20 8:51 PM, Andrey Grodzovsky wrote:
>> This will allow to invalidate, destroy backing storage and notify users
>> of BOs when device is unpluged.
>>
>> Signed-off-by: Andrey Grodzovsky <andrey.grodzovsky@amd.com>
>
> Please add a motivation in the commit message and use imperative 
> wording ("Allow to invalidate..." instead of "This will allow to")
>
> s /unpluged/unplugged/


I am not sure yet this patch is needed in V2

Andrey


>
>
>> ---
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c |  2 +-
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_object.c  |  2 +-
>>   drivers/gpu/drm/nouveau/nouveau_drm.c       |  2 +-
>>   drivers/gpu/drm/qxl/qxl_object.c            |  4 +--
>>   drivers/gpu/drm/radeon/radeon_object.c      |  2 +-
>>   drivers/gpu/drm/ttm/ttm_bo.c                | 41 
>> ++++++++++++++++++-----------
>>   drivers/gpu/drm/vmwgfx/vmwgfx_drv.c         |  6 ++---
>>   include/drm/ttm/ttm_bo_api.h                |  2 +-
>>   8 files changed, 35 insertions(+), 26 deletions(-)
>>
>> diff --git a/include/drm/ttm/ttm_bo_api.h b/include/drm/ttm/ttm_bo_api.h
>> index b9bc1b0..9d57b8c 100644
>> --- a/include/drm/ttm/ttm_bo_api.h
>> +++ b/include/drm/ttm/ttm_bo_api.h
>> @@ -597,7 +597,7 @@ int ttm_bo_clean_mm(struct ttm_bo_device *bdev, 
>> unsigned mem_type);
>>    * -ERESTARTSYS: The call was interrupted by a signal while waiting to
>>    * evict a buffer.
>>    */
>
> Please also update the function documentation.
>
>> -int ttm_bo_evict_mm(struct ttm_bo_device *bdev, unsigned mem_type);
>> +int ttm_bo_evict_mm(struct ttm_bo_device *bdev, unsigned mem_type, 
>> bool destroy);
>>     /**
>>    * ttm_kmap_obj_virtual
>
>
> Thanks,
>
> Thomas
>
>
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

[PATCH 6/6] drm/amdgpu: Use TTM MMs destroy interface

[Andrey Grodzovsky]

Signed-off-by: Andrey Grodzovsky <andrey.grodzovsky@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c    |  4 ++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_object.c | 11 +++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_object.h |  1 +
 3 files changed, 16 insertions(+)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
index 11410a9..338b946 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
@@ -1145,7 +1145,11 @@ amdgpu_pci_remove(struct pci_dev *pdev)
 
 	drm_dev_unplug(dev);
 
+	/* Will trigger SIGBUS on all subsequent CPU accesses */
 	amdgpu_force_unmap_user_space_mappings(dev);
+
+	/* Will trigger VMFAULTs on all subsequent GPU successes */
+	amdgpu_bo_destroy_mms(adev);
 	wait_event(adev->user_clients_done,
 		   !atomic_read(&dev->open_count) &&
 		   !atomic_read(&adev->exported_dma_bufs_count));
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
index 82d43d0..7eac00b 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
@@ -1049,6 +1049,17 @@ int amdgpu_bo_evict_vram(struct amdgpu_device *adev)
 	return ttm_bo_evict_mm(&adev->mman.bdev, TTM_PL_VRAM, false);
 }
 
+int amdgpu_bo_destroy_mms(struct amdgpu_device *adev)
+{
+	int r;
+
+	r = ttm_bo_evict_mm(&adev->mman.bdev, TTM_PL_VRAM, true);
+	if (!r)
+		return ttm_bo_evict_mm(&adev->mman.bdev, TTM_PL_TT, true);
+
+	return r;
+}
+
 static const char *amdgpu_vram_names[] = {
 	"UNKNOWN",
 	"GDDR1",
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.h
index 7d41f7b..4892265 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.h
@@ -269,6 +269,7 @@ int amdgpu_bo_pin_restricted(struct amdgpu_bo *bo, u32 domain,
 			     u64 min_offset, u64 max_offset);
 int amdgpu_bo_unpin(struct amdgpu_bo *bo);
 int amdgpu_bo_evict_vram(struct amdgpu_device *adev);
+int amdgpu_bo_destroy_mms(struct amdgpu_device *adev);
 int amdgpu_bo_init(struct amdgpu_device *adev);
 int amdgpu_bo_late_init(struct amdgpu_device *adev);
 void amdgpu_bo_fini(struct amdgpu_device *adev);
-- 
2.7.4

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

[PATCH 6/6] drm/amdgpu: Use TTM MMs destroy interface

[Andrey Grodzovsky]

Signed-off-by: Andrey Grodzovsky <andrey.grodzovsky@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c    |  4 ++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_object.c | 11 +++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_object.h |  1 +
 3 files changed, 16 insertions(+)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
index 11410a9..338b946 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
@@ -1145,7 +1145,11 @@ amdgpu_pci_remove(struct pci_dev *pdev)
 
 	drm_dev_unplug(dev);
 
+	/* Will trigger SIGBUS on all subsequent CPU accesses */
 	amdgpu_force_unmap_user_space_mappings(dev);
+
+	/* Will trigger VMFAULTs on all subsequent GPU successes */
+	amdgpu_bo_destroy_mms(adev);
 	wait_event(adev->user_clients_done,
 		   !atomic_read(&dev->open_count) &&
 		   !atomic_read(&adev->exported_dma_bufs_count));
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
index 82d43d0..7eac00b 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
@@ -1049,6 +1049,17 @@ int amdgpu_bo_evict_vram(struct amdgpu_device *adev)
 	return ttm_bo_evict_mm(&adev->mman.bdev, TTM_PL_VRAM, false);
 }
 
+int amdgpu_bo_destroy_mms(struct amdgpu_device *adev)
+{
+	int r;
+
+	r = ttm_bo_evict_mm(&adev->mman.bdev, TTM_PL_VRAM, true);
+	if (!r)
+		return ttm_bo_evict_mm(&adev->mman.bdev, TTM_PL_TT, true);
+
+	return r;
+}
+
 static const char *amdgpu_vram_names[] = {
 	"UNKNOWN",
 	"GDDR1",
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.h
index 7d41f7b..4892265 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.h
@@ -269,6 +269,7 @@ int amdgpu_bo_pin_restricted(struct amdgpu_bo *bo, u32 domain,
 			     u64 min_offset, u64 max_offset);
 int amdgpu_bo_unpin(struct amdgpu_bo *bo);
 int amdgpu_bo_evict_vram(struct amdgpu_device *adev);
+int amdgpu_bo_destroy_mms(struct amdgpu_device *adev);
 int amdgpu_bo_init(struct amdgpu_device *adev);
 int amdgpu_bo_late_init(struct amdgpu_device *adev);
 void amdgpu_bo_fini(struct amdgpu_device *adev);
-- 
2.7.4

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH 0/6] RFC Support hot device unplug in amdgpu

[Pekka Paalanen]

[-- Attachment #1.1: Type: text/plain, Size: 8341 bytes --]

On Sat, 9 May 2020 14:51:44 -0400
Andrey Grodzovsky <andrey.grodzovsky@amd.com> wrote:

> This RFC is a more of a proof of concept then a fully working
> solution as there are a few unresolved issues we are hopping to get
> advise on from people on the mailing list. Until now extracting a
> card either by physical extraction (e.g. eGPU with thunderbold
> connection or by emulation through syfs
> -> /sys/bus/pci/devices/device_id/remove) would cause random crashes
> in user apps. The random crashes in apps were mostly due to the app
> having mapped a device backed BO into it's adress space was still
> trying to access the BO while the backing device was gone. To answer
> this first problem Christian suggested to fix the handling of mapped
> memory in the clients when the device goes away by forcibly unmap all
> buffers the user processes has by clearing their respective VMAs
> mapping the device BOs. Then when the VMAs try to fill in the page
> tables again we check in the fault handler if the device is removed
> and if so, return an error. This will generate a SIGBUS to the
> application which can then cleanly terminate. This indeed was done
> but this in turn created a problem of kernel OOPs were the OOPSes
> were due to the fact that while the app was terminating because of
> the SIGBUS it would trigger use after free in the driver by calling
> to accesses device structures that were already released from the pci
> remove sequence. This we handled by introducing a 'flush' seqence
> during device removal were we wait for drm file reference to drop to
> 0 meaning all user clients directly using this device terminated.
> With this I was able to cleanly emulate device unplug with X and
> glxgears running and later emulate device plug back and restart of X
> and glxgears.
> 
> But this use case is only partial and as I see it all the use cases
> are as follwing and the questions it raises.
> 
> 1) Application accesses a BO by opening drm file
> 	1.1) BO is mapped into applications address space (BO is CPU visible) - this one we have a solution for by invaldating BO's CPU mapping casuing SIGBUS 
> 	     and termination and waiting for drm file refcound to drop to 0 before releasing the device
> 	1.2) BO is not mapped into applcation address space (BO is CPU invisible) - no solution yet because how we force the application to terminate in this case ?
> 
> 2) Application accesses a BO by importing a DMA-BUF
> 	2.1)  BO is mapped into applications address space (BO is CPU visible) - solution is same as 1.1 but instead of waiting for drm file release we wait for the 
> 	      imported dma-buf's file release
> 	2.2)  BO is not mapped into applcation address space (BO is CPU invisible) - our solution is to invalidate GPUVM page tables and destroy backing storage for 
>               all exported BOs which will in turn casue VM faults in the importing device and then when the importing driver will try to re-attach the imported BO to 
> 	      update mappings we return -ENODEV in the import hook which hopeffuly will cause the user app to terminate.
> 
> 3) Applcation opens a drm file or imports a dma-bud and holds a reference but never access any BO or does access but never more after device was unplug - how would we 
>    force this applcation to termiante before proceeding with device removal code ? Otherwise the wait in pci remove just hangs for ever.
> 
> The attached patches adress 1.1, 2.1 and 2.2, for now only 1.1 fully tested and I am still testing the others but I will be happy for any advise on all the 
> described use cases and maybe some alternative and better (more generic) approach to this like maybe obtaining PIDs of relevant processes through some revere 
> mapping from device file and exported dma-buf files and send them SIGKILL - would this make more sense or any other method ? 
> 
> Patches 1-3 address 1.1
> Patch 4 addresses 2.1
> Pathces 5-6 address 2.2
> 
> Reference: https://gitlab.freedesktop.org/drm/amd/-/issues/1081

Hi,

how did you come up with the goal "make applications terminate"? Is
that your end goal, or is it just step 1 of many on the road of
supporting device hot-unplug?

Why do you want to terminate also applications that don't "need" to
terminate? Why hunt them down? I'm referring to your points 1.2, 2.2
and 3.

From an end user perspective, I believe making applications terminate
is not helpful at all. Your display server still disappears, which
means all your apps are forced to quit, and you lose your desktop. I do
understand that a graceful termination is better than a hard lockup,
but not much.

When I've talked about DRM device hot-unplug with Daniel Vetter, our
shared opinion seems to be that the unplug should not outright kill any
programs that are prepared to handle errors, that is, functions or
ioctls that return a success code can return an error, and then it is
up for the application to decide how to handle that. The end goal must
not be to terminate all applications that had something to do with the
device. At the very least the display server must survive.

The rough idea on how that should work is that DRM ioctls start
returning errors and all mmaps are replaced with something harmless
that does not cause a SIGBUS. Userspace can handle the errors if it
wants to, and display servers will react to the device removed uevent
if not earlier.

Why deliberately avoid raising SIGBUS? Because it is such a huge pain
to handle due to the archaic design of how signals are delivered. Most
of existing userspace is also not prepared to handle SIGBUS anywhere.

The problem of handling SIGBUS at all is that a process can only have a
single signal handler per signal, but the process may comprise of
multiple components that cannot cooperate on signal catching: Mesa GPU
drivers, GUI toolkits, and the application itself may all do some
things that would require handling SIGBUS if removing a DRM device
raised it. For Mesa to cooperate with SIGBUS handling with the other
components in the process, we'd need some whole new APIs, an EGL
extension and maybe Vulkan extension too. The process may also have
threads, which are really painful with signals. What if you need to
handle the SIGBUS differently in different threads?

Hence, mmaps should be replaced with something harmless, maybe
something that reads back all zeros and ignores writes. The application
will learn later that the DRM device is gone. Sending it a SIGBUS on
the spot when it accesses an mmap does not help: the memory is gone
already - if you didn't have a backup of the contents, you're not going
to make one now.

My point here is, are you designing things to specifically only
terminate processes, or will you leave room in the design to improve the
implementation towards a proper handling of DRM device hot-unplug?


Thanks,
pq


> 
> Andrey Grodzovsky (6):
>   drm/ttm: Add unampping of the entire device address space
>   drm/amdgpu: Force unmap all user VMAs on device removal.
>   drm/amdgpu: Wait for all user clients
>   drm/amdgpu: Wait for all clients importing out dma-bufs.
>   drm/ttm: Add destroy flag in TTM BO eviction interface
>   drm/amdgpu: Use TTM MMs destroy interface
> 
>  drivers/gpu/drm/amd/amdgpu/amdgpu.h         |  3 ++
>  drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c |  2 +-
>  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c  |  7 +++-
>  drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c | 27 ++++++++++++-
>  drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c     | 22 ++++++++--
>  drivers/gpu/drm/amd/amdgpu/amdgpu_job.c     |  9 +++++
>  drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c     |  4 ++
>  drivers/gpu/drm/amd/amdgpu/amdgpu_object.c  | 17 +++++++-
>  drivers/gpu/drm/amd/amdgpu/amdgpu_object.h  |  1 +
>  drivers/gpu/drm/nouveau/nouveau_drm.c       |  2 +-
>  drivers/gpu/drm/qxl/qxl_object.c            |  4 +-
>  drivers/gpu/drm/radeon/radeon_object.c      |  2 +-
>  drivers/gpu/drm/ttm/ttm_bo.c                | 63 +++++++++++++++++++++--------
>  drivers/gpu/drm/vmwgfx/vmwgfx_drv.c         |  6 +--
>  include/drm/ttm/ttm_bo_api.h                |  2 +-
>  include/drm/ttm/ttm_bo_driver.h             |  2 +
>  16 files changed, 139 insertions(+), 34 deletions(-)
> 


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [PATCH 0/6] RFC Support hot device unplug in amdgpu

[Pekka Paalanen]

[-- Attachment #1.1: Type: text/plain, Size: 8341 bytes --]

On Sat, 9 May 2020 14:51:44 -0400
Andrey Grodzovsky <andrey.grodzovsky@amd.com> wrote:

> This RFC is a more of a proof of concept then a fully working
> solution as there are a few unresolved issues we are hopping to get
> advise on from people on the mailing list. Until now extracting a
> card either by physical extraction (e.g. eGPU with thunderbold
> connection or by emulation through syfs
> -> /sys/bus/pci/devices/device_id/remove) would cause random crashes
> in user apps. The random crashes in apps were mostly due to the app
> having mapped a device backed BO into it's adress space was still
> trying to access the BO while the backing device was gone. To answer
> this first problem Christian suggested to fix the handling of mapped
> memory in the clients when the device goes away by forcibly unmap all
> buffers the user processes has by clearing their respective VMAs
> mapping the device BOs. Then when the VMAs try to fill in the page
> tables again we check in the fault handler if the device is removed
> and if so, return an error. This will generate a SIGBUS to the
> application which can then cleanly terminate. This indeed was done
> but this in turn created a problem of kernel OOPs were the OOPSes
> were due to the fact that while the app was terminating because of
> the SIGBUS it would trigger use after free in the driver by calling
> to accesses device structures that were already released from the pci
> remove sequence. This we handled by introducing a 'flush' seqence
> during device removal were we wait for drm file reference to drop to
> 0 meaning all user clients directly using this device terminated.
> With this I was able to cleanly emulate device unplug with X and
> glxgears running and later emulate device plug back and restart of X
> and glxgears.
> 
> But this use case is only partial and as I see it all the use cases
> are as follwing and the questions it raises.
> 
> 1) Application accesses a BO by opening drm file
> 	1.1) BO is mapped into applications address space (BO is CPU visible) - this one we have a solution for by invaldating BO's CPU mapping casuing SIGBUS 
> 	     and termination and waiting for drm file refcound to drop to 0 before releasing the device
> 	1.2) BO is not mapped into applcation address space (BO is CPU invisible) - no solution yet because how we force the application to terminate in this case ?
> 
> 2) Application accesses a BO by importing a DMA-BUF
> 	2.1)  BO is mapped into applications address space (BO is CPU visible) - solution is same as 1.1 but instead of waiting for drm file release we wait for the 
> 	      imported dma-buf's file release
> 	2.2)  BO is not mapped into applcation address space (BO is CPU invisible) - our solution is to invalidate GPUVM page tables and destroy backing storage for 
>               all exported BOs which will in turn casue VM faults in the importing device and then when the importing driver will try to re-attach the imported BO to 
> 	      update mappings we return -ENODEV in the import hook which hopeffuly will cause the user app to terminate.
> 
> 3) Applcation opens a drm file or imports a dma-bud and holds a reference but never access any BO or does access but never more after device was unplug - how would we 
>    force this applcation to termiante before proceeding with device removal code ? Otherwise the wait in pci remove just hangs for ever.
> 
> The attached patches adress 1.1, 2.1 and 2.2, for now only 1.1 fully tested and I am still testing the others but I will be happy for any advise on all the 
> described use cases and maybe some alternative and better (more generic) approach to this like maybe obtaining PIDs of relevant processes through some revere 
> mapping from device file and exported dma-buf files and send them SIGKILL - would this make more sense or any other method ? 
> 
> Patches 1-3 address 1.1
> Patch 4 addresses 2.1
> Pathces 5-6 address 2.2
> 
> Reference: https://gitlab.freedesktop.org/drm/amd/-/issues/1081

Hi,

how did you come up with the goal "make applications terminate"? Is
that your end goal, or is it just step 1 of many on the road of
supporting device hot-unplug?

Why do you want to terminate also applications that don't "need" to
terminate? Why hunt them down? I'm referring to your points 1.2, 2.2
and 3.

From an end user perspective, I believe making applications terminate
is not helpful at all. Your display server still disappears, which
means all your apps are forced to quit, and you lose your desktop. I do
understand that a graceful termination is better than a hard lockup,
but not much.

When I've talked about DRM device hot-unplug with Daniel Vetter, our
shared opinion seems to be that the unplug should not outright kill any
programs that are prepared to handle errors, that is, functions or
ioctls that return a success code can return an error, and then it is
up for the application to decide how to handle that. The end goal must
not be to terminate all applications that had something to do with the
device. At the very least the display server must survive.

The rough idea on how that should work is that DRM ioctls start
returning errors and all mmaps are replaced with something harmless
that does not cause a SIGBUS. Userspace can handle the errors if it
wants to, and display servers will react to the device removed uevent
if not earlier.

Why deliberately avoid raising SIGBUS? Because it is such a huge pain
to handle due to the archaic design of how signals are delivered. Most
of existing userspace is also not prepared to handle SIGBUS anywhere.

The problem of handling SIGBUS at all is that a process can only have a
single signal handler per signal, but the process may comprise of
multiple components that cannot cooperate on signal catching: Mesa GPU
drivers, GUI toolkits, and the application itself may all do some
things that would require handling SIGBUS if removing a DRM device
raised it. For Mesa to cooperate with SIGBUS handling with the other
components in the process, we'd need some whole new APIs, an EGL
extension and maybe Vulkan extension too. The process may also have
threads, which are really painful with signals. What if you need to
handle the SIGBUS differently in different threads?

Hence, mmaps should be replaced with something harmless, maybe
something that reads back all zeros and ignores writes. The application
will learn later that the DRM device is gone. Sending it a SIGBUS on
the spot when it accesses an mmap does not help: the memory is gone
already - if you didn't have a backup of the contents, you're not going
to make one now.

My point here is, are you designing things to specifically only
terminate processes, or will you leave room in the design to improve the
implementation towards a proper handling of DRM device hot-unplug?


Thanks,
pq


> 
> Andrey Grodzovsky (6):
>   drm/ttm: Add unampping of the entire device address space
>   drm/amdgpu: Force unmap all user VMAs on device removal.
>   drm/amdgpu: Wait for all user clients
>   drm/amdgpu: Wait for all clients importing out dma-bufs.
>   drm/ttm: Add destroy flag in TTM BO eviction interface
>   drm/amdgpu: Use TTM MMs destroy interface
> 
>  drivers/gpu/drm/amd/amdgpu/amdgpu.h         |  3 ++
>  drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c |  2 +-
>  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c  |  7 +++-
>  drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c | 27 ++++++++++++-
>  drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c     | 22 ++++++++--
>  drivers/gpu/drm/amd/amdgpu/amdgpu_job.c     |  9 +++++
>  drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c     |  4 ++
>  drivers/gpu/drm/amd/amdgpu/amdgpu_object.c  | 17 +++++++-
>  drivers/gpu/drm/amd/amdgpu/amdgpu_object.h  |  1 +
>  drivers/gpu/drm/nouveau/nouveau_drm.c       |  2 +-
>  drivers/gpu/drm/qxl/qxl_object.c            |  4 +-
>  drivers/gpu/drm/radeon/radeon_object.c      |  2 +-
>  drivers/gpu/drm/ttm/ttm_bo.c                | 63 +++++++++++++++++++++--------
>  drivers/gpu/drm/vmwgfx/vmwgfx_drv.c         |  6 +--
>  include/drm/ttm/ttm_bo_api.h                |  2 +-
>  include/drm/ttm/ttm_bo_driver.h             |  2 +
>  16 files changed, 139 insertions(+), 34 deletions(-)
> 


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[-- Attachment #2: Type: text/plain, Size: 154 bytes --]

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH 0/6] RFC Support hot device unplug in amdgpu

[Christian König]

Am 11.05.20 um 11:26 schrieb Pekka Paalanen:
> On Sat, 9 May 2020 14:51:44 -0400
> Andrey Grodzovsky <andrey.grodzovsky@amd.com> wrote:
>
>> This RFC is a more of a proof of concept then a fully working
>> solution as there are a few unresolved issues we are hopping to get
>> advise on from people on the mailing list. Until now extracting a
>> card either by physical extraction (e.g. eGPU with thunderbold
>> connection or by emulation through syfs
>> -> /sys/bus/pci/devices/device_id/remove) would cause random crashes
>> in user apps. The random crashes in apps were mostly due to the app
>> having mapped a device backed BO into it's adress space was still
>> trying to access the BO while the backing device was gone. To answer
>> this first problem Christian suggested to fix the handling of mapped
>> memory in the clients when the device goes away by forcibly unmap all
>> buffers the user processes has by clearing their respective VMAs
>> mapping the device BOs. Then when the VMAs try to fill in the page
>> tables again we check in the fault handler if the device is removed
>> and if so, return an error. This will generate a SIGBUS to the
>> application which can then cleanly terminate. This indeed was done
>> but this in turn created a problem of kernel OOPs were the OOPSes
>> were due to the fact that while the app was terminating because of
>> the SIGBUS it would trigger use after free in the driver by calling
>> to accesses device structures that were already released from the pci
>> remove sequence. This we handled by introducing a 'flush' seqence
>> during device removal were we wait for drm file reference to drop to
>> 0 meaning all user clients directly using this device terminated.
>> With this I was able to cleanly emulate device unplug with X and
>> glxgears running and later emulate device plug back and restart of X
>> and glxgears.
>>
>> But this use case is only partial and as I see it all the use cases
>> are as follwing and the questions it raises.
>>
>> 1) Application accesses a BO by opening drm file
>> 	1.1) BO is mapped into applications address space (BO is CPU visible) - this one we have a solution for by invaldating BO's CPU mapping casuing SIGBUS
>> 	     and termination and waiting for drm file refcound to drop to 0 before releasing the device
>> 	1.2) BO is not mapped into applcation address space (BO is CPU invisible) - no solution yet because how we force the application to terminate in this case ?
>>
>> 2) Application accesses a BO by importing a DMA-BUF
>> 	2.1)  BO is mapped into applications address space (BO is CPU visible) - solution is same as 1.1 but instead of waiting for drm file release we wait for the
>> 	      imported dma-buf's file release
>> 	2.2)  BO is not mapped into applcation address space (BO is CPU invisible) - our solution is to invalidate GPUVM page tables and destroy backing storage for
>>                all exported BOs which will in turn casue VM faults in the importing device and then when the importing driver will try to re-attach the imported BO to
>> 	      update mappings we return -ENODEV in the import hook which hopeffuly will cause the user app to terminate.
>>
>> 3) Applcation opens a drm file or imports a dma-bud and holds a reference but never access any BO or does access but never more after device was unplug - how would we
>>     force this applcation to termiante before proceeding with device removal code ? Otherwise the wait in pci remove just hangs for ever.
>>
>> The attached patches adress 1.1, 2.1 and 2.2, for now only 1.1 fully tested and I am still testing the others but I will be happy for any advise on all the
>> described use cases and maybe some alternative and better (more generic) approach to this like maybe obtaining PIDs of relevant processes through some revere
>> mapping from device file and exported dma-buf files and send them SIGKILL - would this make more sense or any other method ?
>>
>> Patches 1-3 address 1.1
>> Patch 4 addresses 2.1
>> Pathces 5-6 address 2.2
>>
>> Reference: https://gitlab.freedesktop.org/drm/amd/-/issues/1081
> Hi,
>
> how did you come up with the goal "make applications terminate"? Is
> that your end goal, or is it just step 1 of many on the road of
> supporting device hot-unplug?
>
> Why do you want to terminate also applications that don't "need" to
> terminate? Why hunt them down? I'm referring to your points 1.2, 2.2
> and 3.

Yeah, that is a known limitation. For now the whole idea is to terminate 
the programs using the device as soon as possible.

>  From an end user perspective, I believe making applications terminate
> is not helpful at all. Your display server still disappears, which
> means all your apps are forced to quit, and you lose your desktop. I do
> understand that a graceful termination is better than a hard lockup,
> but not much.

This is not for a desktop use case at all.

Regards,
Christian.

>
> When I've talked about DRM device hot-unplug with Daniel Vetter, our
> shared opinion seems to be that the unplug should not outright kill any
> programs that are prepared to handle errors, that is, functions or
> ioctls that return a success code can return an error, and then it is
> up for the application to decide how to handle that. The end goal must
> not be to terminate all applications that had something to do with the
> device. At the very least the display server must survive.
>
> The rough idea on how that should work is that DRM ioctls start
> returning errors and all mmaps are replaced with something harmless
> that does not cause a SIGBUS. Userspace can handle the errors if it
> wants to, and display servers will react to the device removed uevent
> if not earlier.
>
> Why deliberately avoid raising SIGBUS? Because it is such a huge pain
> to handle due to the archaic design of how signals are delivered. Most
> of existing userspace is also not prepared to handle SIGBUS anywhere.
>
> The problem of handling SIGBUS at all is that a process can only have a
> single signal handler per signal, but the process may comprise of
> multiple components that cannot cooperate on signal catching: Mesa GPU
> drivers, GUI toolkits, and the application itself may all do some
> things that would require handling SIGBUS if removing a DRM device
> raised it. For Mesa to cooperate with SIGBUS handling with the other
> components in the process, we'd need some whole new APIs, an EGL
> extension and maybe Vulkan extension too. The process may also have
> threads, which are really painful with signals. What if you need to
> handle the SIGBUS differently in different threads?
>
> Hence, mmaps should be replaced with something harmless, maybe
> something that reads back all zeros and ignores writes. The application
> will learn later that the DRM device is gone. Sending it a SIGBUS on
> the spot when it accesses an mmap does not help: the memory is gone
> already - if you didn't have a backup of the contents, you're not going
> to make one now.
>
> My point here is, are you designing things to specifically only
> terminate processes, or will you leave room in the design to improve the
> implementation towards a proper handling of DRM device hot-unplug?
>
>
> Thanks,
> pq
>
>
>> Andrey Grodzovsky (6):
>>    drm/ttm: Add unampping of the entire device address space
>>    drm/amdgpu: Force unmap all user VMAs on device removal.
>>    drm/amdgpu: Wait for all user clients
>>    drm/amdgpu: Wait for all clients importing out dma-bufs.
>>    drm/ttm: Add destroy flag in TTM BO eviction interface
>>    drm/amdgpu: Use TTM MMs destroy interface
>>
>>   drivers/gpu/drm/amd/amdgpu/amdgpu.h         |  3 ++
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c |  2 +-
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_device.c  |  7 +++-
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c | 27 ++++++++++++-
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c     | 22 ++++++++--
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_job.c     |  9 +++++
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c     |  4 ++
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_object.c  | 17 +++++++-
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_object.h  |  1 +
>>   drivers/gpu/drm/nouveau/nouveau_drm.c       |  2 +-
>>   drivers/gpu/drm/qxl/qxl_object.c            |  4 +-
>>   drivers/gpu/drm/radeon/radeon_object.c      |  2 +-
>>   drivers/gpu/drm/ttm/ttm_bo.c                | 63 +++++++++++++++++++++--------
>>   drivers/gpu/drm/vmwgfx/vmwgfx_drv.c         |  6 +--
>>   include/drm/ttm/ttm_bo_api.h                |  2 +-
>>   include/drm/ttm/ttm_bo_driver.h             |  2 +
>>   16 files changed, 139 insertions(+), 34 deletions(-)
>>

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [PATCH 0/6] RFC Support hot device unplug in amdgpu

[Christian König]

Am 11.05.20 um 11:26 schrieb Pekka Paalanen:
> On Sat, 9 May 2020 14:51:44 -0400
> Andrey Grodzovsky <andrey.grodzovsky@amd.com> wrote:
>
>> This RFC is a more of a proof of concept then a fully working
>> solution as there are a few unresolved issues we are hopping to get
>> advise on from people on the mailing list. Until now extracting a
>> card either by physical extraction (e.g. eGPU with thunderbold
>> connection or by emulation through syfs
>> -> /sys/bus/pci/devices/device_id/remove) would cause random crashes
>> in user apps. The random crashes in apps were mostly due to the app
>> having mapped a device backed BO into it's adress space was still
>> trying to access the BO while the backing device was gone. To answer
>> this first problem Christian suggested to fix the handling of mapped
>> memory in the clients when the device goes away by forcibly unmap all
>> buffers the user processes has by clearing their respective VMAs
>> mapping the device BOs. Then when the VMAs try to fill in the page
>> tables again we check in the fault handler if the device is removed
>> and if so, return an error. This will generate a SIGBUS to the
>> application which can then cleanly terminate. This indeed was done
>> but this in turn created a problem of kernel OOPs were the OOPSes
>> were due to the fact that while the app was terminating because of
>> the SIGBUS it would trigger use after free in the driver by calling
>> to accesses device structures that were already released from the pci
>> remove sequence. This we handled by introducing a 'flush' seqence
>> during device removal were we wait for drm file reference to drop to
>> 0 meaning all user clients directly using this device terminated.
>> With this I was able to cleanly emulate device unplug with X and
>> glxgears running and later emulate device plug back and restart of X
>> and glxgears.
>>
>> But this use case is only partial and as I see it all the use cases
>> are as follwing and the questions it raises.
>>
>> 1) Application accesses a BO by opening drm file
>> 	1.1) BO is mapped into applications address space (BO is CPU visible) - this one we have a solution for by invaldating BO's CPU mapping casuing SIGBUS
>> 	     and termination and waiting for drm file refcound to drop to 0 before releasing the device
>> 	1.2) BO is not mapped into applcation address space (BO is CPU invisible) - no solution yet because how we force the application to terminate in this case ?
>>
>> 2) Application accesses a BO by importing a DMA-BUF
>> 	2.1)  BO is mapped into applications address space (BO is CPU visible) - solution is same as 1.1 but instead of waiting for drm file release we wait for the
>> 	      imported dma-buf's file release
>> 	2.2)  BO is not mapped into applcation address space (BO is CPU invisible) - our solution is to invalidate GPUVM page tables and destroy backing storage for
>>                all exported BOs which will in turn casue VM faults in the importing device and then when the importing driver will try to re-attach the imported BO to
>> 	      update mappings we return -ENODEV in the import hook which hopeffuly will cause the user app to terminate.
>>
>> 3) Applcation opens a drm file or imports a dma-bud and holds a reference but never access any BO or does access but never more after device was unplug - how would we
>>     force this applcation to termiante before proceeding with device removal code ? Otherwise the wait in pci remove just hangs for ever.
>>
>> The attached patches adress 1.1, 2.1 and 2.2, for now only 1.1 fully tested and I am still testing the others but I will be happy for any advise on all the
>> described use cases and maybe some alternative and better (more generic) approach to this like maybe obtaining PIDs of relevant processes through some revere
>> mapping from device file and exported dma-buf files and send them SIGKILL - would this make more sense or any other method ?
>>
>> Patches 1-3 address 1.1
>> Patch 4 addresses 2.1
>> Pathces 5-6 address 2.2
>>
>> Reference: https://gitlab.freedesktop.org/drm/amd/-/issues/1081
> Hi,
>
> how did you come up with the goal "make applications terminate"? Is
> that your end goal, or is it just step 1 of many on the road of
> supporting device hot-unplug?
>
> Why do you want to terminate also applications that don't "need" to
> terminate? Why hunt them down? I'm referring to your points 1.2, 2.2
> and 3.

Yeah, that is a known limitation. For now the whole idea is to terminate 
the programs using the device as soon as possible.

>  From an end user perspective, I believe making applications terminate
> is not helpful at all. Your display server still disappears, which
> means all your apps are forced to quit, and you lose your desktop. I do
> understand that a graceful termination is better than a hard lockup,
> but not much.

This is not for a desktop use case at all.

Regards,
Christian.

>
> When I've talked about DRM device hot-unplug with Daniel Vetter, our
> shared opinion seems to be that the unplug should not outright kill any
> programs that are prepared to handle errors, that is, functions or
> ioctls that return a success code can return an error, and then it is
> up for the application to decide how to handle that. The end goal must
> not be to terminate all applications that had something to do with the
> device. At the very least the display server must survive.
>
> The rough idea on how that should work is that DRM ioctls start
> returning errors and all mmaps are replaced with something harmless
> that does not cause a SIGBUS. Userspace can handle the errors if it
> wants to, and display servers will react to the device removed uevent
> if not earlier.
>
> Why deliberately avoid raising SIGBUS? Because it is such a huge pain
> to handle due to the archaic design of how signals are delivered. Most
> of existing userspace is also not prepared to handle SIGBUS anywhere.
>
> The problem of handling SIGBUS at all is that a process can only have a
> single signal handler per signal, but the process may comprise of
> multiple components that cannot cooperate on signal catching: Mesa GPU
> drivers, GUI toolkits, and the application itself may all do some
> things that would require handling SIGBUS if removing a DRM device
> raised it. For Mesa to cooperate with SIGBUS handling with the other
> components in the process, we'd need some whole new APIs, an EGL
> extension and maybe Vulkan extension too. The process may also have
> threads, which are really painful with signals. What if you need to
> handle the SIGBUS differently in different threads?
>
> Hence, mmaps should be replaced with something harmless, maybe
> something that reads back all zeros and ignores writes. The application
> will learn later that the DRM device is gone. Sending it a SIGBUS on
> the spot when it accesses an mmap does not help: the memory is gone
> already - if you didn't have a backup of the contents, you're not going
> to make one now.
>
> My point here is, are you designing things to specifically only
> terminate processes, or will you leave room in the design to improve the
> implementation towards a proper handling of DRM device hot-unplug?
>
>
> Thanks,
> pq
>
>
>> Andrey Grodzovsky (6):
>>    drm/ttm: Add unampping of the entire device address space
>>    drm/amdgpu: Force unmap all user VMAs on device removal.
>>    drm/amdgpu: Wait for all user clients
>>    drm/amdgpu: Wait for all clients importing out dma-bufs.
>>    drm/ttm: Add destroy flag in TTM BO eviction interface
>>    drm/amdgpu: Use TTM MMs destroy interface
>>
>>   drivers/gpu/drm/amd/amdgpu/amdgpu.h         |  3 ++
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c |  2 +-
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_device.c  |  7 +++-
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c | 27 ++++++++++++-
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c     | 22 ++++++++--
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_job.c     |  9 +++++
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c     |  4 ++
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_object.c  | 17 +++++++-
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_object.h  |  1 +
>>   drivers/gpu/drm/nouveau/nouveau_drm.c       |  2 +-
>>   drivers/gpu/drm/qxl/qxl_object.c            |  4 +-
>>   drivers/gpu/drm/radeon/radeon_object.c      |  2 +-
>>   drivers/gpu/drm/ttm/ttm_bo.c                | 63 +++++++++++++++++++++--------
>>   drivers/gpu/drm/vmwgfx/vmwgfx_drv.c         |  6 +--
>>   include/drm/ttm/ttm_bo_api.h                |  2 +-
>>   include/drm/ttm/ttm_bo_driver.h             |  2 +
>>   16 files changed, 139 insertions(+), 34 deletions(-)
>>

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH 0/6] RFC Support hot device unplug in amdgpu

[Andrey Grodzovsky]

[-- Attachment #1.1: Type: text/plain, Size: 9978 bytes --]


On 5/11/20 5:26 AM, Pekka Paalanen wrote:
> On Sat, 9 May 2020 14:51:44 -0400
> Andrey Grodzovsky <andrey.grodzovsky@amd.com> wrote:
>
>> This RFC is a more of a proof of concept then a fully working
>> solution as there are a few unresolved issues we are hopping to get
>> advise on from people on the mailing list. Until now extracting a
>> card either by physical extraction (e.g. eGPU with thunderbold
>> connection or by emulation through syfs
>> -> /sys/bus/pci/devices/device_id/remove) would cause random crashes
>> in user apps. The random crashes in apps were mostly due to the app
>> having mapped a device backed BO into it's adress space was still
>> trying to access the BO while the backing device was gone. To answer
>> this first problem Christian suggested to fix the handling of mapped
>> memory in the clients when the device goes away by forcibly unmap all
>> buffers the user processes has by clearing their respective VMAs
>> mapping the device BOs. Then when the VMAs try to fill in the page
>> tables again we check in the fault handler if the device is removed
>> and if so, return an error. This will generate a SIGBUS to the
>> application which can then cleanly terminate. This indeed was done
>> but this in turn created a problem of kernel OOPs were the OOPSes
>> were due to the fact that while the app was terminating because of
>> the SIGBUS it would trigger use after free in the driver by calling
>> to accesses device structures that were already released from the pci
>> remove sequence. This we handled by introducing a 'flush' seqence
>> during device removal were we wait for drm file reference to drop to
>> 0 meaning all user clients directly using this device terminated.
>> With this I was able to cleanly emulate device unplug with X and
>> glxgears running and later emulate device plug back and restart of X
>> and glxgears.
>>
>> But this use case is only partial and as I see it all the use cases
>> are as follwing and the questions it raises.
>>
>> 1) Application accesses a BO by opening drm file
>> 	1.1) BO is mapped into applications address space (BO is CPU visible) - this one we have a solution for by invaldating BO's CPU mapping casuing SIGBUS
>> 	     and termination and waiting for drm file refcound to drop to 0 before releasing the device
>> 	1.2) BO is not mapped into applcation address space (BO is CPU invisible) - no solution yet because how we force the application to terminate in this case ?
>>
>> 2) Application accesses a BO by importing a DMA-BUF
>> 	2.1)  BO is mapped into applications address space (BO is CPU visible) - solution is same as 1.1 but instead of waiting for drm file release we wait for the
>> 	      imported dma-buf's file release
>> 	2.2)  BO is not mapped into applcation address space (BO is CPU invisible) - our solution is to invalidate GPUVM page tables and destroy backing storage for
>>                all exported BOs which will in turn casue VM faults in the importing device and then when the importing driver will try to re-attach the imported BO to
>> 	      update mappings we return -ENODEV in the import hook which hopeffuly will cause the user app to terminate.
>>
>> 3) Applcation opens a drm file or imports a dma-bud and holds a reference but never access any BO or does access but never more after device was unplug - how would we
>>     force this applcation to termiante before proceeding with device removal code ? Otherwise the wait in pci remove just hangs for ever.
>>
>> The attached patches adress 1.1, 2.1 and 2.2, for now only 1.1 fully tested and I am still testing the others but I will be happy for any advise on all the
>> described use cases and maybe some alternative and better (more generic) approach to this like maybe obtaining PIDs of relevant processes through some revere
>> mapping from device file and exported dma-buf files and send them SIGKILL - would this make more sense or any other method ?
>>
>> Patches 1-3 address 1.1
>> Patch 4 addresses 2.1
>> Pathces 5-6 address 2.2
>>
>> Reference: https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.freedesktop.org%2Fdrm%2Famd%2F-%2Fissues%2F1081&amp;data=02%7C01%7Candrey.grodzovsky%40amd.com%7C6f92386d0dd444de4fe608d7f58d5ae9%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637247860388177520&amp;sdata=xg3zrilEwSCR7icmkKVVzZwiI11XvmGR%2Bca8nOWBiDM%3D&amp;reserved=0
> Hi,
>
> how did you come up with the goal "make applications terminate"? Is
> that your end goal, or is it just step 1 of many on the road of
> supporting device hot-unplug?


Just as an effort to improve the current situation where we have 
unexpected random crashes following device removal.

>
> Why do you want to terminate also applications that don't "need" to
> terminate? Why hunt them down? I'm referring to your points 1.2, 2.2
> and 3.


Because when those applications do exit and since they hold a reference 
to drm device through their open device file descriptor or dma-buf file 
descriptor we end up in use after free situation where during pci remove 
we already released everything but now last drm_dev_put release callback 
is trying to access those released structures. Any way, as you and 
Daniel pointed out forcing termination is a bad approach.  Seems we need 
to actually keep all the drm structures around until the very last 
device reference is dropped while in the meatime returning error code 
for any new IOCTLs and rerouting any  page fault to zero page.

Thanks  for the detailed response.

Andrey


>
>  From an end user perspective, I believe making applications terminate
> is not helpful at all. Your display server still disappears, which
> means all your apps are forced to quit, and you lose your desktop. I do
> understand that a graceful termination is better than a hard lockup,
> but not much.
>
> When I've talked about DRM device hot-unplug with Daniel Vetter, our
> shared opinion seems to be that the unplug should not outright kill any
> programs that are prepared to handle errors, that is, functions or
> ioctls that return a success code can return an error, and then it is
> up for the application to decide how to handle that. The end goal must
> not be to terminate all applications that had something to do with the
> device. At the very least the display server must survive.
>
> The rough idea on how that should work is that DRM ioctls start
> returning errors and all mmaps are replaced with something harmless
> that does not cause a SIGBUS. Userspace can handle the errors if it
> wants to, and display servers will react to the device removed uevent
> if not earlier.
>
> Why deliberately avoid raising SIGBUS? Because it is such a huge pain
> to handle due to the archaic design of how signals are delivered. Most
> of existing userspace is also not prepared to handle SIGBUS anywhere.
>
> The problem of handling SIGBUS at all is that a process can only have a
> single signal handler per signal, but the process may comprise of
> multiple components that cannot cooperate on signal catching: Mesa GPU
> drivers, GUI toolkits, and the application itself may all do some
> things that would require handling SIGBUS if removing a DRM device
> raised it. For Mesa to cooperate with SIGBUS handling with the other
> components in the process, we'd need some whole new APIs, an EGL
> extension and maybe Vulkan extension too. The process may also have
> threads, which are really painful with signals. What if you need to
> handle the SIGBUS differently in different threads?
>
> Hence, mmaps should be replaced with something harmless, maybe
> something that reads back all zeros and ignores writes. The application
> will learn later that the DRM device is gone. Sending it a SIGBUS on
> the spot when it accesses an mmap does not help: the memory is gone
> already - if you didn't have a backup of the contents, you're not going
> to make one now.
>
> My point here is, are you designing things to specifically only
> terminate processes, or will you leave room in the design to improve the
> implementation towards a proper handling of DRM device hot-unplug?
>
>
> Thanks,
> pq
>
>
>> Andrey Grodzovsky (6):
>>    drm/ttm: Add unampping of the entire device address space
>>    drm/amdgpu: Force unmap all user VMAs on device removal.
>>    drm/amdgpu: Wait for all user clients
>>    drm/amdgpu: Wait for all clients importing out dma-bufs.
>>    drm/ttm: Add destroy flag in TTM BO eviction interface
>>    drm/amdgpu: Use TTM MMs destroy interface
>>
>>   drivers/gpu/drm/amd/amdgpu/amdgpu.h         |  3 ++
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c |  2 +-
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_device.c  |  7 +++-
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c | 27 ++++++++++++-
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c     | 22 ++++++++--
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_job.c     |  9 +++++
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c     |  4 ++
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_object.c  | 17 +++++++-
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_object.h  |  1 +
>>   drivers/gpu/drm/nouveau/nouveau_drm.c       |  2 +-
>>   drivers/gpu/drm/qxl/qxl_object.c            |  4 +-
>>   drivers/gpu/drm/radeon/radeon_object.c      |  2 +-
>>   drivers/gpu/drm/ttm/ttm_bo.c                | 63 +++++++++++++++++++++--------
>>   drivers/gpu/drm/vmwgfx/vmwgfx_drv.c         |  6 +--
>>   include/drm/ttm/ttm_bo_api.h                |  2 +-
>>   include/drm/ttm/ttm_bo_driver.h             |  2 +
>>   16 files changed, 139 insertions(+), 34 deletions(-)
>>
>
> _______________________________________________
> amd-gfx mailing list
> amd-gfx@lists.freedesktop.org
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Fmailman%2Flistinfo%2Famd-gfx&amp;data=02%7C01%7Candrey.grodzovsky%40amd.com%7C6f92386d0dd444de4fe608d7f58d5ae9%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637247860388197434&amp;sdata=Unqh9pySrEsPeAFLxzmI0deAlPF29%2FfXLMdSl8Jsvgo%3D&amp;reserved=0

[-- Attachment #1.2: Type: text/html, Size: 12179 bytes --]

[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [PATCH 0/6] RFC Support hot device unplug in amdgpu

[Andrey Grodzovsky]

[-- Attachment #1.1: Type: text/plain, Size: 9978 bytes --]


On 5/11/20 5:26 AM, Pekka Paalanen wrote:
> On Sat, 9 May 2020 14:51:44 -0400
> Andrey Grodzovsky <andrey.grodzovsky@amd.com> wrote:
>
>> This RFC is a more of a proof of concept then a fully working
>> solution as there are a few unresolved issues we are hopping to get
>> advise on from people on the mailing list. Until now extracting a
>> card either by physical extraction (e.g. eGPU with thunderbold
>> connection or by emulation through syfs
>> -> /sys/bus/pci/devices/device_id/remove) would cause random crashes
>> in user apps. The random crashes in apps were mostly due to the app
>> having mapped a device backed BO into it's adress space was still
>> trying to access the BO while the backing device was gone. To answer
>> this first problem Christian suggested to fix the handling of mapped
>> memory in the clients when the device goes away by forcibly unmap all
>> buffers the user processes has by clearing their respective VMAs
>> mapping the device BOs. Then when the VMAs try to fill in the page
>> tables again we check in the fault handler if the device is removed
>> and if so, return an error. This will generate a SIGBUS to the
>> application which can then cleanly terminate. This indeed was done
>> but this in turn created a problem of kernel OOPs were the OOPSes
>> were due to the fact that while the app was terminating because of
>> the SIGBUS it would trigger use after free in the driver by calling
>> to accesses device structures that were already released from the pci
>> remove sequence. This we handled by introducing a 'flush' seqence
>> during device removal were we wait for drm file reference to drop to
>> 0 meaning all user clients directly using this device terminated.
>> With this I was able to cleanly emulate device unplug with X and
>> glxgears running and later emulate device plug back and restart of X
>> and glxgears.
>>
>> But this use case is only partial and as I see it all the use cases
>> are as follwing and the questions it raises.
>>
>> 1) Application accesses a BO by opening drm file
>> 	1.1) BO is mapped into applications address space (BO is CPU visible) - this one we have a solution for by invaldating BO's CPU mapping casuing SIGBUS
>> 	     and termination and waiting for drm file refcound to drop to 0 before releasing the device
>> 	1.2) BO is not mapped into applcation address space (BO is CPU invisible) - no solution yet because how we force the application to terminate in this case ?
>>
>> 2) Application accesses a BO by importing a DMA-BUF
>> 	2.1)  BO is mapped into applications address space (BO is CPU visible) - solution is same as 1.1 but instead of waiting for drm file release we wait for the
>> 	      imported dma-buf's file release
>> 	2.2)  BO is not mapped into applcation address space (BO is CPU invisible) - our solution is to invalidate GPUVM page tables and destroy backing storage for
>>                all exported BOs which will in turn casue VM faults in the importing device and then when the importing driver will try to re-attach the imported BO to
>> 	      update mappings we return -ENODEV in the import hook which hopeffuly will cause the user app to terminate.
>>
>> 3) Applcation opens a drm file or imports a dma-bud and holds a reference but never access any BO or does access but never more after device was unplug - how would we
>>     force this applcation to termiante before proceeding with device removal code ? Otherwise the wait in pci remove just hangs for ever.
>>
>> The attached patches adress 1.1, 2.1 and 2.2, for now only 1.1 fully tested and I am still testing the others but I will be happy for any advise on all the
>> described use cases and maybe some alternative and better (more generic) approach to this like maybe obtaining PIDs of relevant processes through some revere
>> mapping from device file and exported dma-buf files and send them SIGKILL - would this make more sense or any other method ?
>>
>> Patches 1-3 address 1.1
>> Patch 4 addresses 2.1
>> Pathces 5-6 address 2.2
>>
>> Reference: https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.freedesktop.org%2Fdrm%2Famd%2F-%2Fissues%2F1081&amp;data=02%7C01%7Candrey.grodzovsky%40amd.com%7C6f92386d0dd444de4fe608d7f58d5ae9%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637247860388177520&amp;sdata=xg3zrilEwSCR7icmkKVVzZwiI11XvmGR%2Bca8nOWBiDM%3D&amp;reserved=0
> Hi,
>
> how did you come up with the goal "make applications terminate"? Is
> that your end goal, or is it just step 1 of many on the road of
> supporting device hot-unplug?


Just as an effort to improve the current situation where we have 
unexpected random crashes following device removal.

>
> Why do you want to terminate also applications that don't "need" to
> terminate? Why hunt them down? I'm referring to your points 1.2, 2.2
> and 3.


Because when those applications do exit and since they hold a reference 
to drm device through their open device file descriptor or dma-buf file 
descriptor we end up in use after free situation where during pci remove 
we already released everything but now last drm_dev_put release callback 
is trying to access those released structures. Any way, as you and 
Daniel pointed out forcing termination is a bad approach.  Seems we need 
to actually keep all the drm structures around until the very last 
device reference is dropped while in the meatime returning error code 
for any new IOCTLs and rerouting any  page fault to zero page.

Thanks  for the detailed response.

Andrey


>
>  From an end user perspective, I believe making applications terminate
> is not helpful at all. Your display server still disappears, which
> means all your apps are forced to quit, and you lose your desktop. I do
> understand that a graceful termination is better than a hard lockup,
> but not much.
>
> When I've talked about DRM device hot-unplug with Daniel Vetter, our
> shared opinion seems to be that the unplug should not outright kill any
> programs that are prepared to handle errors, that is, functions or
> ioctls that return a success code can return an error, and then it is
> up for the application to decide how to handle that. The end goal must
> not be to terminate all applications that had something to do with the
> device. At the very least the display server must survive.
>
> The rough idea on how that should work is that DRM ioctls start
> returning errors and all mmaps are replaced with something harmless
> that does not cause a SIGBUS. Userspace can handle the errors if it
> wants to, and display servers will react to the device removed uevent
> if not earlier.
>
> Why deliberately avoid raising SIGBUS? Because it is such a huge pain
> to handle due to the archaic design of how signals are delivered. Most
> of existing userspace is also not prepared to handle SIGBUS anywhere.
>
> The problem of handling SIGBUS at all is that a process can only have a
> single signal handler per signal, but the process may comprise of
> multiple components that cannot cooperate on signal catching: Mesa GPU
> drivers, GUI toolkits, and the application itself may all do some
> things that would require handling SIGBUS if removing a DRM device
> raised it. For Mesa to cooperate with SIGBUS handling with the other
> components in the process, we'd need some whole new APIs, an EGL
> extension and maybe Vulkan extension too. The process may also have
> threads, which are really painful with signals. What if you need to
> handle the SIGBUS differently in different threads?
>
> Hence, mmaps should be replaced with something harmless, maybe
> something that reads back all zeros and ignores writes. The application
> will learn later that the DRM device is gone. Sending it a SIGBUS on
> the spot when it accesses an mmap does not help: the memory is gone
> already - if you didn't have a backup of the contents, you're not going
> to make one now.
>
> My point here is, are you designing things to specifically only
> terminate processes, or will you leave room in the design to improve the
> implementation towards a proper handling of DRM device hot-unplug?
>
>
> Thanks,
> pq
>
>
>> Andrey Grodzovsky (6):
>>    drm/ttm: Add unampping of the entire device address space
>>    drm/amdgpu: Force unmap all user VMAs on device removal.
>>    drm/amdgpu: Wait for all user clients
>>    drm/amdgpu: Wait for all clients importing out dma-bufs.
>>    drm/ttm: Add destroy flag in TTM BO eviction interface
>>    drm/amdgpu: Use TTM MMs destroy interface
>>
>>   drivers/gpu/drm/amd/amdgpu/amdgpu.h         |  3 ++
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c |  2 +-
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_device.c  |  7 +++-
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c | 27 ++++++++++++-
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c     | 22 ++++++++--
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_job.c     |  9 +++++
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c     |  4 ++
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_object.c  | 17 +++++++-
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_object.h  |  1 +
>>   drivers/gpu/drm/nouveau/nouveau_drm.c       |  2 +-
>>   drivers/gpu/drm/qxl/qxl_object.c            |  4 +-
>>   drivers/gpu/drm/radeon/radeon_object.c      |  2 +-
>>   drivers/gpu/drm/ttm/ttm_bo.c                | 63 +++++++++++++++++++++--------
>>   drivers/gpu/drm/vmwgfx/vmwgfx_drv.c         |  6 +--
>>   include/drm/ttm/ttm_bo_api.h                |  2 +-
>>   include/drm/ttm/ttm_bo_driver.h             |  2 +
>>   16 files changed, 139 insertions(+), 34 deletions(-)
>>
>
> _______________________________________________
> amd-gfx mailing list
> amd-gfx@lists.freedesktop.org
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Fmailman%2Flistinfo%2Famd-gfx&amp;data=02%7C01%7Candrey.grodzovsky%40amd.com%7C6f92386d0dd444de4fe608d7f58d5ae9%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637247860388197434&amp;sdata=Unqh9pySrEsPeAFLxzmI0deAlPF29%2FfXLMdSl8Jsvgo%3D&amp;reserved=0

[-- Attachment #1.2: Type: text/html, Size: 12179 bytes --]

[-- Attachment #2: Type: text/plain, Size: 154 bytes --]

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH 0/6] RFC Support hot device unplug in amdgpu

[Daniel Vetter]

On Sat, May 09, 2020 at 02:51:44PM -0400, Andrey Grodzovsky wrote:
> This RFC is a more of a proof of concept then a fully working solution as there are a few unresolved issues we are hopping to get advise on from people on the mailing list.
> Until now extracting a card either by physical extraction (e.g. eGPU with thunderbold connection or by emulation through syfs -> /sys/bus/pci/devices/device_id/remove) 
> would cause random crashes in user apps. The random crashes in apps were mostly due to the app having mapped a device backed BO into it's adress space was still 
> trying to access the BO while the backing device was gone. 
> To answer this first problem Christian suggested to fix the handling of mapped memory in the clients when the device goes away by forcibly unmap all buffers 
> the user processes has by clearing their respective VMAs mapping the device BOs. Then when the VMAs try to fill in the page tables again we check in the fault handler 
> if the device is removed and if so, return an error. This will generate a SIGBUS to the application which can then cleanly terminate. 
> This indeed was done but this in turn created a problem of kernel OOPs were the OOPSes were due to the fact that while the app was terminating because of the SIGBUS 
> it would trigger use after free in the driver by calling to accesses device structures that were already released from the pci remove sequence. 
> This we handled by introducing a 'flush' seqence during device removal were we wait for drm file reference to drop to 0 meaning all user clients directly using this device terminated. 
> With this I was able to cleanly emulate device unplug with X and glxgears running and later emulate device plug back and restart of X and glxgears.
> 
> But this use case is only partial and as I see it all the use cases are as follwing and the questions it raises.
> 
> 1) Application accesses a BO by opening drm file
> 	1.1) BO is mapped into applications address space (BO is CPU visible) - this one we have a solution for by invaldating BO's CPU mapping casuing SIGBUS 
> 	     and termination and waiting for drm file refcound to drop to 0 before releasing the device
> 	1.2) BO is not mapped into applcation address space (BO is CPU invisible) - no solution yet because how we force the application to terminate in this case ?
> 
> 2) Application accesses a BO by importing a DMA-BUF
> 	2.1)  BO is mapped into applications address space (BO is CPU visible) - solution is same as 1.1 but instead of waiting for drm file release we wait for the 
> 	      imported dma-buf's file release
> 	2.2)  BO is not mapped into applcation address space (BO is CPU invisible) - our solution is to invalidate GPUVM page tables and destroy backing storage for 
>               all exported BOs which will in turn casue VM faults in the importing device and then when the importing driver will try to re-attach the imported BO to 
> 	      update mappings we return -ENODEV in the import hook which hopeffuly will cause the user app to terminate.
> 
> 3) Applcation opens a drm file or imports a dma-bud and holds a reference but never access any BO or does access but never more after device was unplug - how would we 
>    force this applcation to termiante before proceeding with device removal code ? Otherwise the wait in pci remove just hangs for ever.
> 
> The attached patches adress 1.1, 2.1 and 2.2, for now only 1.1 fully tested and I am still testing the others but I will be happy for any advise on all the 
> described use cases and maybe some alternative and better (more generic) approach to this like maybe obtaining PIDs of relevant processes through some revere 
> mapping from device file and exported dma-buf files and send them SIGKILL - would this make more sense or any other method ? 
> 
> Patches 1-3 address 1.1
> Patch 4 addresses 2.1
> Pathces 5-6 address 2.2
> 
> Reference: https://gitlab.freedesktop.org/drm/amd/-/issues/1081

So we've been working on this problem for a few years already (but it's
still not solved), I think you could have saved yourselfs some typing.

Bunch of things:
- we can't wait for userspace in the hotunplug handlers, that might never
  happen. The correct way is to untangle the lifetime of your hw driver
  for a specific struct pci_device from the drm_device lifetime.
  Infrastructure is all there now, see drm_dev_get/put, drm_dev_unplug and
  drm_dev_enter/exit. A bunch of usb/spi drivers use this 100% correctly
  now, so there's examples. Plus kerneldoc explains stuff.

- for a big driver like amdgpu doing this split up is going to be
  horrendously complex. I know, we've done it for i915, at least
  partially. I strongly recommend that you're using devm_ for managing hw
  related resources (iomap, irq, ...) as much as possible.

  For drm_device resources (mostly structures and everything related to
  that) we've just merged the drmm_ managed resources framework. There's
  some more work to be done there for various kms objects, but you can at
  least somewhat avoid tedious handrolling for everything internal
  already.

  Don't ever use devm_kzalloc and friends, I've looked at hundreds of uses
  of this in drm, they're all wrong.

- dma-buf is hilarious (and atm unfixed), dma-fence is even worse. In
  theory they're already refcounted and all and so should work, in
  practice I think we need to refcount the underlying drm_device with
  drm_dev_get/put to avoid the worst fall-out.

- One unfortunate thing with drm_dev_unplug is that the driver core is
  very opinionated and doesn't tell you whether it's a hotunplug or a
  driver unload. In the former case trying to shut down hw just wastes
  time (and might hit driver bugs), in the latter case driver engineers
  very much expect everything to be shut down.

  Right now you can only have one or the other, so this needs a module
  option hack or similar (default to the correct hotunplug behaviour for
  users).

- SIGBUS is better than crashing the kernel, but it's not even close for
  users. They still lose everything because everything crashes because in
  my experience, in practice, no one ever handles errors. There's a few
  more things on top:

  - sighandlers are global, which means only the app can use it. You can't
    use it in e.g. mesa. They're also not composable, so if you have on
    sighandler for gpu1 and a 2nd one for gpu2 (could be different vendor)
    it's all sadness. Hence "usersapce will handle SIGBUS" wont work.

  - worse, neither vk nor gl (to my knowledge) have a concept of events
    for when the gpu died. The only stuff you have is things like
    arb_robustness which says a) everything continues as if nothing
    happened b) there's a function where you can ask whether your gl
    context and all the textures/buffers are toast.

    I think that's about the only hotunplug application model we can
    realistically expect applications to support. That means _all_ errors
    need to be silently eaten by either mesa or the kernel. On i915 the
    model (also for terminally wedged gpu hangs) is that all ioctl keep
    working, mmaps keep working, and execbuf gives you an -EIO (which mesa
    eats silently iirc for arb_robustness).

  Conclusion is that SIGBUS is imo a no-go, and the only option we have is
  that a) mmaps fully keep working, doable for shmem or b) we put some
  fake memory in there (for vram or whatever), maybe even only a single
  page for all fake memory.

- you probably want arb_robustness and similar stuff in userspace as a
  first step.

tldr;
- refcounting, not waiting for userspace
- nothing can fail because userspace wont handle it

That's at least my take on this mess, and what we've been pushing for over
the past few years. For kms-only drm_driver we should have achieved that
by now (plus/minus maybe some issues for dma-buf/fences, but kms-only
dma-buf/fences are simple enough that maybe we don't go boom yet).

For big gpus with rendering I think best next step would be to type up a
reasonable Gran Plan (into Documentation/gpu/todo.rst) with all the issues
and likely solutions. And then bikeshed that, since the above is just my
take on all this.

Cheers, Daniel

> 
> Andrey Grodzovsky (6):
>   drm/ttm: Add unampping of the entire device address space
>   drm/amdgpu: Force unmap all user VMAs on device removal.
>   drm/amdgpu: Wait for all user clients
>   drm/amdgpu: Wait for all clients importing out dma-bufs.
>   drm/ttm: Add destroy flag in TTM BO eviction interface
>   drm/amdgpu: Use TTM MMs destroy interface
> 
>  drivers/gpu/drm/amd/amdgpu/amdgpu.h         |  3 ++
>  drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c |  2 +-
>  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c  |  7 +++-
>  drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c | 27 ++++++++++++-
>  drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c     | 22 ++++++++--
>  drivers/gpu/drm/amd/amdgpu/amdgpu_job.c     |  9 +++++
>  drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c     |  4 ++
>  drivers/gpu/drm/amd/amdgpu/amdgpu_object.c  | 17 +++++++-
>  drivers/gpu/drm/amd/amdgpu/amdgpu_object.h  |  1 +
>  drivers/gpu/drm/nouveau/nouveau_drm.c       |  2 +-
>  drivers/gpu/drm/qxl/qxl_object.c            |  4 +-
>  drivers/gpu/drm/radeon/radeon_object.c      |  2 +-
>  drivers/gpu/drm/ttm/ttm_bo.c                | 63 +++++++++++++++++++++--------
>  drivers/gpu/drm/vmwgfx/vmwgfx_drv.c         |  6 +--
>  include/drm/ttm/ttm_bo_api.h                |  2 +-
>  include/drm/ttm/ttm_bo_driver.h             |  2 +
>  16 files changed, 139 insertions(+), 34 deletions(-)
> 
> -- 
> 2.7.4
> 

-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [PATCH 0/6] RFC Support hot device unplug in amdgpu

[Daniel Vetter]

On Sat, May 09, 2020 at 02:51:44PM -0400, Andrey Grodzovsky wrote:
> This RFC is a more of a proof of concept then a fully working solution as there are a few unresolved issues we are hopping to get advise on from people on the mailing list.
> Until now extracting a card either by physical extraction (e.g. eGPU with thunderbold connection or by emulation through syfs -> /sys/bus/pci/devices/device_id/remove) 
> would cause random crashes in user apps. The random crashes in apps were mostly due to the app having mapped a device backed BO into it's adress space was still 
> trying to access the BO while the backing device was gone. 
> To answer this first problem Christian suggested to fix the handling of mapped memory in the clients when the device goes away by forcibly unmap all buffers 
> the user processes has by clearing their respective VMAs mapping the device BOs. Then when the VMAs try to fill in the page tables again we check in the fault handler 
> if the device is removed and if so, return an error. This will generate a SIGBUS to the application which can then cleanly terminate. 
> This indeed was done but this in turn created a problem of kernel OOPs were the OOPSes were due to the fact that while the app was terminating because of the SIGBUS 
> it would trigger use after free in the driver by calling to accesses device structures that were already released from the pci remove sequence. 
> This we handled by introducing a 'flush' seqence during device removal were we wait for drm file reference to drop to 0 meaning all user clients directly using this device terminated. 
> With this I was able to cleanly emulate device unplug with X and glxgears running and later emulate device plug back and restart of X and glxgears.
> 
> But this use case is only partial and as I see it all the use cases are as follwing and the questions it raises.
> 
> 1) Application accesses a BO by opening drm file
> 	1.1) BO is mapped into applications address space (BO is CPU visible) - this one we have a solution for by invaldating BO's CPU mapping casuing SIGBUS 
> 	     and termination and waiting for drm file refcound to drop to 0 before releasing the device
> 	1.2) BO is not mapped into applcation address space (BO is CPU invisible) - no solution yet because how we force the application to terminate in this case ?
> 
> 2) Application accesses a BO by importing a DMA-BUF
> 	2.1)  BO is mapped into applications address space (BO is CPU visible) - solution is same as 1.1 but instead of waiting for drm file release we wait for the 
> 	      imported dma-buf's file release
> 	2.2)  BO is not mapped into applcation address space (BO is CPU invisible) - our solution is to invalidate GPUVM page tables and destroy backing storage for 
>               all exported BOs which will in turn casue VM faults in the importing device and then when the importing driver will try to re-attach the imported BO to 
> 	      update mappings we return -ENODEV in the import hook which hopeffuly will cause the user app to terminate.
> 
> 3) Applcation opens a drm file or imports a dma-bud and holds a reference but never access any BO or does access but never more after device was unplug - how would we 
>    force this applcation to termiante before proceeding with device removal code ? Otherwise the wait in pci remove just hangs for ever.
> 
> The attached patches adress 1.1, 2.1 and 2.2, for now only 1.1 fully tested and I am still testing the others but I will be happy for any advise on all the 
> described use cases and maybe some alternative and better (more generic) approach to this like maybe obtaining PIDs of relevant processes through some revere 
> mapping from device file and exported dma-buf files and send them SIGKILL - would this make more sense or any other method ? 
> 
> Patches 1-3 address 1.1
> Patch 4 addresses 2.1
> Pathces 5-6 address 2.2
> 
> Reference: https://gitlab.freedesktop.org/drm/amd/-/issues/1081

So we've been working on this problem for a few years already (but it's
still not solved), I think you could have saved yourselfs some typing.

Bunch of things:
- we can't wait for userspace in the hotunplug handlers, that might never
  happen. The correct way is to untangle the lifetime of your hw driver
  for a specific struct pci_device from the drm_device lifetime.
  Infrastructure is all there now, see drm_dev_get/put, drm_dev_unplug and
  drm_dev_enter/exit. A bunch of usb/spi drivers use this 100% correctly
  now, so there's examples. Plus kerneldoc explains stuff.

- for a big driver like amdgpu doing this split up is going to be
  horrendously complex. I know, we've done it for i915, at least
  partially. I strongly recommend that you're using devm_ for managing hw
  related resources (iomap, irq, ...) as much as possible.

  For drm_device resources (mostly structures and everything related to
  that) we've just merged the drmm_ managed resources framework. There's
  some more work to be done there for various kms objects, but you can at
  least somewhat avoid tedious handrolling for everything internal
  already.

  Don't ever use devm_kzalloc and friends, I've looked at hundreds of uses
  of this in drm, they're all wrong.

- dma-buf is hilarious (and atm unfixed), dma-fence is even worse. In
  theory they're already refcounted and all and so should work, in
  practice I think we need to refcount the underlying drm_device with
  drm_dev_get/put to avoid the worst fall-out.

- One unfortunate thing with drm_dev_unplug is that the driver core is
  very opinionated and doesn't tell you whether it's a hotunplug or a
  driver unload. In the former case trying to shut down hw just wastes
  time (and might hit driver bugs), in the latter case driver engineers
  very much expect everything to be shut down.

  Right now you can only have one or the other, so this needs a module
  option hack or similar (default to the correct hotunplug behaviour for
  users).

- SIGBUS is better than crashing the kernel, but it's not even close for
  users. They still lose everything because everything crashes because in
  my experience, in practice, no one ever handles errors. There's a few
  more things on top:

  - sighandlers are global, which means only the app can use it. You can't
    use it in e.g. mesa. They're also not composable, so if you have on
    sighandler for gpu1 and a 2nd one for gpu2 (could be different vendor)
    it's all sadness. Hence "usersapce will handle SIGBUS" wont work.

  - worse, neither vk nor gl (to my knowledge) have a concept of events
    for when the gpu died. The only stuff you have is things like
    arb_robustness which says a) everything continues as if nothing
    happened b) there's a function where you can ask whether your gl
    context and all the textures/buffers are toast.

    I think that's about the only hotunplug application model we can
    realistically expect applications to support. That means _all_ errors
    need to be silently eaten by either mesa or the kernel. On i915 the
    model (also for terminally wedged gpu hangs) is that all ioctl keep
    working, mmaps keep working, and execbuf gives you an -EIO (which mesa
    eats silently iirc for arb_robustness).

  Conclusion is that SIGBUS is imo a no-go, and the only option we have is
  that a) mmaps fully keep working, doable for shmem or b) we put some
  fake memory in there (for vram or whatever), maybe even only a single
  page for all fake memory.

- you probably want arb_robustness and similar stuff in userspace as a
  first step.

tldr;
- refcounting, not waiting for userspace
- nothing can fail because userspace wont handle it

That's at least my take on this mess, and what we've been pushing for over
the past few years. For kms-only drm_driver we should have achieved that
by now (plus/minus maybe some issues for dma-buf/fences, but kms-only
dma-buf/fences are simple enough that maybe we don't go boom yet).

For big gpus with rendering I think best next step would be to type up a
reasonable Gran Plan (into Documentation/gpu/todo.rst) with all the issues
and likely solutions. And then bikeshed that, since the above is just my
take on all this.

Cheers, Daniel

> 
> Andrey Grodzovsky (6):
>   drm/ttm: Add unampping of the entire device address space
>   drm/amdgpu: Force unmap all user VMAs on device removal.
>   drm/amdgpu: Wait for all user clients
>   drm/amdgpu: Wait for all clients importing out dma-bufs.
>   drm/ttm: Add destroy flag in TTM BO eviction interface
>   drm/amdgpu: Use TTM MMs destroy interface
> 
>  drivers/gpu/drm/amd/amdgpu/amdgpu.h         |  3 ++
>  drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c |  2 +-
>  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c  |  7 +++-
>  drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c | 27 ++++++++++++-
>  drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c     | 22 ++++++++--
>  drivers/gpu/drm/amd/amdgpu/amdgpu_job.c     |  9 +++++
>  drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c     |  4 ++
>  drivers/gpu/drm/amd/amdgpu/amdgpu_object.c  | 17 +++++++-
>  drivers/gpu/drm/amd/amdgpu/amdgpu_object.h  |  1 +
>  drivers/gpu/drm/nouveau/nouveau_drm.c       |  2 +-
>  drivers/gpu/drm/qxl/qxl_object.c            |  4 +-
>  drivers/gpu/drm/radeon/radeon_object.c      |  2 +-
>  drivers/gpu/drm/ttm/ttm_bo.c                | 63 +++++++++++++++++++++--------
>  drivers/gpu/drm/vmwgfx/vmwgfx_drv.c         |  6 +--
>  include/drm/ttm/ttm_bo_api.h                |  2 +-
>  include/drm/ttm/ttm_bo_driver.h             |  2 +
>  16 files changed, 139 insertions(+), 34 deletions(-)
> 
> -- 
> 2.7.4
> 

-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH 0/6] RFC Support hot device unplug in amdgpu

[Chris Wilson]

Quoting Daniel Vetter (2020-05-11 10:54:33)
>   - worse, neither vk nor gl (to my knowledge) have a concept of events
>     for when the gpu died. The only stuff you have is things like
>     arb_robustness which says a) everything continues as if nothing
>     happened b) there's a function where you can ask whether your gl
>     context and all the textures/buffers are toast.

Vulkan/DX12 arrived after eGPU, and there is at least the concept of
VK_ERROR_DEVICE_LOST. Mainly used at the moment after a GPU hang and
loss of context.

https://www.khronos.org/registry/vulkan/specs/1.2-extensions/html/vkspec.html#devsandqueues-lost-device
-Chris
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [PATCH 0/6] RFC Support hot device unplug in amdgpu

[Chris Wilson]

Quoting Daniel Vetter (2020-05-11 10:54:33)
>   - worse, neither vk nor gl (to my knowledge) have a concept of events
>     for when the gpu died. The only stuff you have is things like
>     arb_robustness which says a) everything continues as if nothing
>     happened b) there's a function where you can ask whether your gl
>     context and all the textures/buffers are toast.

Vulkan/DX12 arrived after eGPU, and there is at least the concept of
VK_ERROR_DEVICE_LOST. Mainly used at the moment after a GPU hang and
loss of context.

https://www.khronos.org/registry/vulkan/specs/1.2-extensions/html/vkspec.html#devsandqueues-lost-device
-Chris
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH 0/6] RFC Support hot device unplug in amdgpu

[Daniel Vetter]

On Mon, May 11, 2020 at 11:19:30AM +0100, Chris Wilson wrote:
> Quoting Daniel Vetter (2020-05-11 10:54:33)
> >   - worse, neither vk nor gl (to my knowledge) have a concept of events
> >     for when the gpu died. The only stuff you have is things like
> >     arb_robustness which says a) everything continues as if nothing
> >     happened b) there's a function where you can ask whether your gl
> >     context and all the textures/buffers are toast.
> 
> Vulkan/DX12 arrived after eGPU, and there is at least the concept of
> VK_ERROR_DEVICE_LOST. Mainly used at the moment after a GPU hang and
> loss of context.
> 
> https://www.khronos.org/registry/vulkan/specs/1.2-extensions/html/vkspec.html#devsandqueues-lost-device

Ah cool, so -EIO on some ioctls, with silencing it in the gl driver and
passing it on for the vk driver should be ok. Assuming vk frameworks
bother to implement the *may* thing. I'm assuming if the validation
midlayer doesn't inject this, it's untested and firework will ensue.

But then more direct path to fireworks is what vk is all about :-)
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [PATCH 0/6] RFC Support hot device unplug in amdgpu

[Daniel Vetter]

On Mon, May 11, 2020 at 11:19:30AM +0100, Chris Wilson wrote:
> Quoting Daniel Vetter (2020-05-11 10:54:33)
> >   - worse, neither vk nor gl (to my knowledge) have a concept of events
> >     for when the gpu died. The only stuff you have is things like
> >     arb_robustness which says a) everything continues as if nothing
> >     happened b) there's a function where you can ask whether your gl
> >     context and all the textures/buffers are toast.
> 
> Vulkan/DX12 arrived after eGPU, and there is at least the concept of
> VK_ERROR_DEVICE_LOST. Mainly used at the moment after a GPU hang and
> loss of context.
> 
> https://www.khronos.org/registry/vulkan/specs/1.2-extensions/html/vkspec.html#devsandqueues-lost-device

Ah cool, so -EIO on some ioctls, with silencing it in the gl driver and
passing it on for the vk driver should be ok. Assuming vk frameworks
bother to implement the *may* thing. I'm assuming if the validation
midlayer doesn't inject this, it's untested and firework will ensue.

But then more direct path to fireworks is what vk is all about :-)
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH 0/6] RFC Support hot device unplug in amdgpu

[Daniel Vetter]

On Mon, May 11, 2020 at 11:54:33AM +0200, Daniel Vetter wrote:
> On Sat, May 09, 2020 at 02:51:44PM -0400, Andrey Grodzovsky wrote:
> > This RFC is a more of a proof of concept then a fully working solution as there are a few unresolved issues we are hopping to get advise on from people on the mailing list.
> > Until now extracting a card either by physical extraction (e.g. eGPU with thunderbold connection or by emulation through syfs -> /sys/bus/pci/devices/device_id/remove) 
> > would cause random crashes in user apps. The random crashes in apps were mostly due to the app having mapped a device backed BO into it's adress space was still 
> > trying to access the BO while the backing device was gone. 
> > To answer this first problem Christian suggested to fix the handling of mapped memory in the clients when the device goes away by forcibly unmap all buffers 
> > the user processes has by clearing their respective VMAs mapping the device BOs. Then when the VMAs try to fill in the page tables again we check in the fault handler 
> > if the device is removed and if so, return an error. This will generate a SIGBUS to the application which can then cleanly terminate. 
> > This indeed was done but this in turn created a problem of kernel OOPs were the OOPSes were due to the fact that while the app was terminating because of the SIGBUS 
> > it would trigger use after free in the driver by calling to accesses device structures that were already released from the pci remove sequence. 
> > This we handled by introducing a 'flush' seqence during device removal were we wait for drm file reference to drop to 0 meaning all user clients directly using this device terminated. 
> > With this I was able to cleanly emulate device unplug with X and glxgears running and later emulate device plug back and restart of X and glxgears.
> > 
> > But this use case is only partial and as I see it all the use cases are as follwing and the questions it raises.
> > 
> > 1) Application accesses a BO by opening drm file
> > 	1.1) BO is mapped into applications address space (BO is CPU visible) - this one we have a solution for by invaldating BO's CPU mapping casuing SIGBUS 
> > 	     and termination and waiting for drm file refcound to drop to 0 before releasing the device
> > 	1.2) BO is not mapped into applcation address space (BO is CPU invisible) - no solution yet because how we force the application to terminate in this case ?
> > 
> > 2) Application accesses a BO by importing a DMA-BUF
> > 	2.1)  BO is mapped into applications address space (BO is CPU visible) - solution is same as 1.1 but instead of waiting for drm file release we wait for the 
> > 	      imported dma-buf's file release
> > 	2.2)  BO is not mapped into applcation address space (BO is CPU invisible) - our solution is to invalidate GPUVM page tables and destroy backing storage for 
> >               all exported BOs which will in turn casue VM faults in the importing device and then when the importing driver will try to re-attach the imported BO to 
> > 	      update mappings we return -ENODEV in the import hook which hopeffuly will cause the user app to terminate.
> > 
> > 3) Applcation opens a drm file or imports a dma-bud and holds a reference but never access any BO or does access but never more after device was unplug - how would we 
> >    force this applcation to termiante before proceeding with device removal code ? Otherwise the wait in pci remove just hangs for ever.
> > 
> > The attached patches adress 1.1, 2.1 and 2.2, for now only 1.1 fully tested and I am still testing the others but I will be happy for any advise on all the 
> > described use cases and maybe some alternative and better (more generic) approach to this like maybe obtaining PIDs of relevant processes through some revere 
> > mapping from device file and exported dma-buf files and send them SIGKILL - would this make more sense or any other method ? 
> > 
> > Patches 1-3 address 1.1
> > Patch 4 addresses 2.1
> > Pathces 5-6 address 2.2
> > 
> > Reference: https://gitlab.freedesktop.org/drm/amd/-/issues/1081
> 
> So we've been working on this problem for a few years already (but it's
> still not solved), I think you could have saved yourselfs some typing.
> 
> Bunch of things:
> - we can't wait for userspace in the hotunplug handlers, that might never
>   happen. The correct way is to untangle the lifetime of your hw driver
>   for a specific struct pci_device from the drm_device lifetime.
>   Infrastructure is all there now, see drm_dev_get/put, drm_dev_unplug and
>   drm_dev_enter/exit. A bunch of usb/spi drivers use this 100% correctly
>   now, so there's examples. Plus kerneldoc explains stuff.
> 
> - for a big driver like amdgpu doing this split up is going to be
>   horrendously complex. I know, we've done it for i915, at least
>   partially. I strongly recommend that you're using devm_ for managing hw
>   related resources (iomap, irq, ...) as much as possible.
> 
>   For drm_device resources (mostly structures and everything related to
>   that) we've just merged the drmm_ managed resources framework. There's
>   some more work to be done there for various kms objects, but you can at
>   least somewhat avoid tedious handrolling for everything internal
>   already.
> 
>   Don't ever use devm_kzalloc and friends, I've looked at hundreds of uses
>   of this in drm, they're all wrong.
> 
> - dma-buf is hilarious (and atm unfixed), dma-fence is even worse. In
>   theory they're already refcounted and all and so should work, in
>   practice I think we need to refcount the underlying drm_device with
>   drm_dev_get/put to avoid the worst fall-out.

oh I forgot one, since it's new since the last time we've seriously
discussed this: p2p dma-buf

But that /should/ be handleable with the move_notify callback. Assuming we
don't have any bugs anywhere, and the importer can indeed get rid of all
its mapping, always.

But for completeness probably need this one, just to keep it noted.
-Daniel

> 
> - One unfortunate thing with drm_dev_unplug is that the driver core is
>   very opinionated and doesn't tell you whether it's a hotunplug or a
>   driver unload. In the former case trying to shut down hw just wastes
>   time (and might hit driver bugs), in the latter case driver engineers
>   very much expect everything to be shut down.
> 
>   Right now you can only have one or the other, so this needs a module
>   option hack or similar (default to the correct hotunplug behaviour for
>   users).
> 
> - SIGBUS is better than crashing the kernel, but it's not even close for
>   users. They still lose everything because everything crashes because in
>   my experience, in practice, no one ever handles errors. There's a few
>   more things on top:
> 
>   - sighandlers are global, which means only the app can use it. You can't
>     use it in e.g. mesa. They're also not composable, so if you have on
>     sighandler for gpu1 and a 2nd one for gpu2 (could be different vendor)
>     it's all sadness. Hence "usersapce will handle SIGBUS" wont work.
> 
>   - worse, neither vk nor gl (to my knowledge) have a concept of events
>     for when the gpu died. The only stuff you have is things like
>     arb_robustness which says a) everything continues as if nothing
>     happened b) there's a function where you can ask whether your gl
>     context and all the textures/buffers are toast.
> 
>     I think that's about the only hotunplug application model we can
>     realistically expect applications to support. That means _all_ errors
>     need to be silently eaten by either mesa or the kernel. On i915 the
>     model (also for terminally wedged gpu hangs) is that all ioctl keep
>     working, mmaps keep working, and execbuf gives you an -EIO (which mesa
>     eats silently iirc for arb_robustness).
> 
>   Conclusion is that SIGBUS is imo a no-go, and the only option we have is
>   that a) mmaps fully keep working, doable for shmem or b) we put some
>   fake memory in there (for vram or whatever), maybe even only a single
>   page for all fake memory.
> 
> - you probably want arb_robustness and similar stuff in userspace as a
>   first step.
> 
> tldr;
> - refcounting, not waiting for userspace
> - nothing can fail because userspace wont handle it
> 
> That's at least my take on this mess, and what we've been pushing for over
> the past few years. For kms-only drm_driver we should have achieved that
> by now (plus/minus maybe some issues for dma-buf/fences, but kms-only
> dma-buf/fences are simple enough that maybe we don't go boom yet).
> 
> For big gpus with rendering I think best next step would be to type up a
> reasonable Gran Plan (into Documentation/gpu/todo.rst) with all the issues
> and likely solutions. And then bikeshed that, since the above is just my
> take on all this.
> 
> Cheers, Daniel
> 
> > 
> > Andrey Grodzovsky (6):
> >   drm/ttm: Add unampping of the entire device address space
> >   drm/amdgpu: Force unmap all user VMAs on device removal.
> >   drm/amdgpu: Wait for all user clients
> >   drm/amdgpu: Wait for all clients importing out dma-bufs.
> >   drm/ttm: Add destroy flag in TTM BO eviction interface
> >   drm/amdgpu: Use TTM MMs destroy interface
> > 
> >  drivers/gpu/drm/amd/amdgpu/amdgpu.h         |  3 ++
> >  drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c |  2 +-
> >  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c  |  7 +++-
> >  drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c | 27 ++++++++++++-
> >  drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c     | 22 ++++++++--
> >  drivers/gpu/drm/amd/amdgpu/amdgpu_job.c     |  9 +++++
> >  drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c     |  4 ++
> >  drivers/gpu/drm/amd/amdgpu/amdgpu_object.c  | 17 +++++++-
> >  drivers/gpu/drm/amd/amdgpu/amdgpu_object.h  |  1 +
> >  drivers/gpu/drm/nouveau/nouveau_drm.c       |  2 +-
> >  drivers/gpu/drm/qxl/qxl_object.c            |  4 +-
> >  drivers/gpu/drm/radeon/radeon_object.c      |  2 +-
> >  drivers/gpu/drm/ttm/ttm_bo.c                | 63 +++++++++++++++++++++--------
> >  drivers/gpu/drm/vmwgfx/vmwgfx_drv.c         |  6 +--
> >  include/drm/ttm/ttm_bo_api.h                |  2 +-
> >  include/drm/ttm/ttm_bo_driver.h             |  2 +
> >  16 files changed, 139 insertions(+), 34 deletions(-)
> > 
> > -- 
> > 2.7.4
> > 
> 
> -- 
> Daniel Vetter
> Software Engineer, Intel Corporation
> http://blog.ffwll.ch

-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [PATCH 0/6] RFC Support hot device unplug in amdgpu

[Daniel Vetter]

On Mon, May 11, 2020 at 11:54:33AM +0200, Daniel Vetter wrote:
> On Sat, May 09, 2020 at 02:51:44PM -0400, Andrey Grodzovsky wrote:
> > This RFC is a more of a proof of concept then a fully working solution as there are a few unresolved issues we are hopping to get advise on from people on the mailing list.
> > Until now extracting a card either by physical extraction (e.g. eGPU with thunderbold connection or by emulation through syfs -> /sys/bus/pci/devices/device_id/remove) 
> > would cause random crashes in user apps. The random crashes in apps were mostly due to the app having mapped a device backed BO into it's adress space was still 
> > trying to access the BO while the backing device was gone. 
> > To answer this first problem Christian suggested to fix the handling of mapped memory in the clients when the device goes away by forcibly unmap all buffers 
> > the user processes has by clearing their respective VMAs mapping the device BOs. Then when the VMAs try to fill in the page tables again we check in the fault handler 
> > if the device is removed and if so, return an error. This will generate a SIGBUS to the application which can then cleanly terminate. 
> > This indeed was done but this in turn created a problem of kernel OOPs were the OOPSes were due to the fact that while the app was terminating because of the SIGBUS 
> > it would trigger use after free in the driver by calling to accesses device structures that were already released from the pci remove sequence. 
> > This we handled by introducing a 'flush' seqence during device removal were we wait for drm file reference to drop to 0 meaning all user clients directly using this device terminated. 
> > With this I was able to cleanly emulate device unplug with X and glxgears running and later emulate device plug back and restart of X and glxgears.
> > 
> > But this use case is only partial and as I see it all the use cases are as follwing and the questions it raises.
> > 
> > 1) Application accesses a BO by opening drm file
> > 	1.1) BO is mapped into applications address space (BO is CPU visible) - this one we have a solution for by invaldating BO's CPU mapping casuing SIGBUS 
> > 	     and termination and waiting for drm file refcound to drop to 0 before releasing the device
> > 	1.2) BO is not mapped into applcation address space (BO is CPU invisible) - no solution yet because how we force the application to terminate in this case ?
> > 
> > 2) Application accesses a BO by importing a DMA-BUF
> > 	2.1)  BO is mapped into applications address space (BO is CPU visible) - solution is same as 1.1 but instead of waiting for drm file release we wait for the 
> > 	      imported dma-buf's file release
> > 	2.2)  BO is not mapped into applcation address space (BO is CPU invisible) - our solution is to invalidate GPUVM page tables and destroy backing storage for 
> >               all exported BOs which will in turn casue VM faults in the importing device and then when the importing driver will try to re-attach the imported BO to 
> > 	      update mappings we return -ENODEV in the import hook which hopeffuly will cause the user app to terminate.
> > 
> > 3) Applcation opens a drm file or imports a dma-bud and holds a reference but never access any BO or does access but never more after device was unplug - how would we 
> >    force this applcation to termiante before proceeding with device removal code ? Otherwise the wait in pci remove just hangs for ever.
> > 
> > The attached patches adress 1.1, 2.1 and 2.2, for now only 1.1 fully tested and I am still testing the others but I will be happy for any advise on all the 
> > described use cases and maybe some alternative and better (more generic) approach to this like maybe obtaining PIDs of relevant processes through some revere 
> > mapping from device file and exported dma-buf files and send them SIGKILL - would this make more sense or any other method ? 
> > 
> > Patches 1-3 address 1.1
> > Patch 4 addresses 2.1
> > Pathces 5-6 address 2.2
> > 
> > Reference: https://gitlab.freedesktop.org/drm/amd/-/issues/1081
> 
> So we've been working on this problem for a few years already (but it's
> still not solved), I think you could have saved yourselfs some typing.
> 
> Bunch of things:
> - we can't wait for userspace in the hotunplug handlers, that might never
>   happen. The correct way is to untangle the lifetime of your hw driver
>   for a specific struct pci_device from the drm_device lifetime.
>   Infrastructure is all there now, see drm_dev_get/put, drm_dev_unplug and
>   drm_dev_enter/exit. A bunch of usb/spi drivers use this 100% correctly
>   now, so there's examples. Plus kerneldoc explains stuff.
> 
> - for a big driver like amdgpu doing this split up is going to be
>   horrendously complex. I know, we've done it for i915, at least
>   partially. I strongly recommend that you're using devm_ for managing hw
>   related resources (iomap, irq, ...) as much as possible.
> 
>   For drm_device resources (mostly structures and everything related to
>   that) we've just merged the drmm_ managed resources framework. There's
>   some more work to be done there for various kms objects, but you can at
>   least somewhat avoid tedious handrolling for everything internal
>   already.
> 
>   Don't ever use devm_kzalloc and friends, I've looked at hundreds of uses
>   of this in drm, they're all wrong.
> 
> - dma-buf is hilarious (and atm unfixed), dma-fence is even worse. In
>   theory they're already refcounted and all and so should work, in
>   practice I think we need to refcount the underlying drm_device with
>   drm_dev_get/put to avoid the worst fall-out.

oh I forgot one, since it's new since the last time we've seriously
discussed this: p2p dma-buf

But that /should/ be handleable with the move_notify callback. Assuming we
don't have any bugs anywhere, and the importer can indeed get rid of all
its mapping, always.

But for completeness probably need this one, just to keep it noted.
-Daniel

> 
> - One unfortunate thing with drm_dev_unplug is that the driver core is
>   very opinionated and doesn't tell you whether it's a hotunplug or a
>   driver unload. In the former case trying to shut down hw just wastes
>   time (and might hit driver bugs), in the latter case driver engineers
>   very much expect everything to be shut down.
> 
>   Right now you can only have one or the other, so this needs a module
>   option hack or similar (default to the correct hotunplug behaviour for
>   users).
> 
> - SIGBUS is better than crashing the kernel, but it's not even close for
>   users. They still lose everything because everything crashes because in
>   my experience, in practice, no one ever handles errors. There's a few
>   more things on top:
> 
>   - sighandlers are global, which means only the app can use it. You can't
>     use it in e.g. mesa. They're also not composable, so if you have on
>     sighandler for gpu1 and a 2nd one for gpu2 (could be different vendor)
>     it's all sadness. Hence "usersapce will handle SIGBUS" wont work.
> 
>   - worse, neither vk nor gl (to my knowledge) have a concept of events
>     for when the gpu died. The only stuff you have is things like
>     arb_robustness which says a) everything continues as if nothing
>     happened b) there's a function where you can ask whether your gl
>     context and all the textures/buffers are toast.
> 
>     I think that's about the only hotunplug application model we can
>     realistically expect applications to support. That means _all_ errors
>     need to be silently eaten by either mesa or the kernel. On i915 the
>     model (also for terminally wedged gpu hangs) is that all ioctl keep
>     working, mmaps keep working, and execbuf gives you an -EIO (which mesa
>     eats silently iirc for arb_robustness).
> 
>   Conclusion is that SIGBUS is imo a no-go, and the only option we have is
>   that a) mmaps fully keep working, doable for shmem or b) we put some
>   fake memory in there (for vram or whatever), maybe even only a single
>   page for all fake memory.
> 
> - you probably want arb_robustness and similar stuff in userspace as a
>   first step.
> 
> tldr;
> - refcounting, not waiting for userspace
> - nothing can fail because userspace wont handle it
> 
> That's at least my take on this mess, and what we've been pushing for over
> the past few years. For kms-only drm_driver we should have achieved that
> by now (plus/minus maybe some issues for dma-buf/fences, but kms-only
> dma-buf/fences are simple enough that maybe we don't go boom yet).
> 
> For big gpus with rendering I think best next step would be to type up a
> reasonable Gran Plan (into Documentation/gpu/todo.rst) with all the issues
> and likely solutions. And then bikeshed that, since the above is just my
> take on all this.
> 
> Cheers, Daniel
> 
> > 
> > Andrey Grodzovsky (6):
> >   drm/ttm: Add unampping of the entire device address space
> >   drm/amdgpu: Force unmap all user VMAs on device removal.
> >   drm/amdgpu: Wait for all user clients
> >   drm/amdgpu: Wait for all clients importing out dma-bufs.
> >   drm/ttm: Add destroy flag in TTM BO eviction interface
> >   drm/amdgpu: Use TTM MMs destroy interface
> > 
> >  drivers/gpu/drm/amd/amdgpu/amdgpu.h         |  3 ++
> >  drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c |  2 +-
> >  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c  |  7 +++-
> >  drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c | 27 ++++++++++++-
> >  drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c     | 22 ++++++++--
> >  drivers/gpu/drm/amd/amdgpu/amdgpu_job.c     |  9 +++++
> >  drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c     |  4 ++
> >  drivers/gpu/drm/amd/amdgpu/amdgpu_object.c  | 17 +++++++-
> >  drivers/gpu/drm/amd/amdgpu/amdgpu_object.h  |  1 +
> >  drivers/gpu/drm/nouveau/nouveau_drm.c       |  2 +-
> >  drivers/gpu/drm/qxl/qxl_object.c            |  4 +-
> >  drivers/gpu/drm/radeon/radeon_object.c      |  2 +-
> >  drivers/gpu/drm/ttm/ttm_bo.c                | 63 +++++++++++++++++++++--------
> >  drivers/gpu/drm/vmwgfx/vmwgfx_drv.c         |  6 +--
> >  include/drm/ttm/ttm_bo_api.h                |  2 +-
> >  include/drm/ttm/ttm_bo_driver.h             |  2 +
> >  16 files changed, 139 insertions(+), 34 deletions(-)
> > 
> > -- 
> > 2.7.4
> > 
> 
> -- 
> Daniel Vetter
> Software Engineer, Intel Corporation
> http://blog.ffwll.ch

-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH 0/6] RFC Support hot device unplug in amdgpu

[Christian König]

Am 11.05.20 um 13:19 schrieb Daniel Vetter:
> On Mon, May 11, 2020 at 11:54:33AM +0200, Daniel Vetter wrote:
>> On Sat, May 09, 2020 at 02:51:44PM -0400, Andrey Grodzovsky wrote:
>>> This RFC is a more of a proof of concept then a fully working solution as there are a few unresolved issues we are hopping to get advise on from people on the mailing list.
>>> Until now extracting a card either by physical extraction (e.g. eGPU with thunderbold connection or by emulation through syfs -> /sys/bus/pci/devices/device_id/remove)
>>> would cause random crashes in user apps. The random crashes in apps were mostly due to the app having mapped a device backed BO into it's adress space was still
>>> trying to access the BO while the backing device was gone.
>>> To answer this first problem Christian suggested to fix the handling of mapped memory in the clients when the device goes away by forcibly unmap all buffers
>>> the user processes has by clearing their respective VMAs mapping the device BOs. Then when the VMAs try to fill in the page tables again we check in the fault handler
>>> if the device is removed and if so, return an error. This will generate a SIGBUS to the application which can then cleanly terminate.
>>> This indeed was done but this in turn created a problem of kernel OOPs were the OOPSes were due to the fact that while the app was terminating because of the SIGBUS
>>> it would trigger use after free in the driver by calling to accesses device structures that were already released from the pci remove sequence.
>>> This we handled by introducing a 'flush' seqence during device removal were we wait for drm file reference to drop to 0 meaning all user clients directly using this device terminated.
>>> With this I was able to cleanly emulate device unplug with X and glxgears running and later emulate device plug back and restart of X and glxgears.
>>>
>>> But this use case is only partial and as I see it all the use cases are as follwing and the questions it raises.
>>>
>>> 1) Application accesses a BO by opening drm file
>>> 	1.1) BO is mapped into applications address space (BO is CPU visible) - this one we have a solution for by invaldating BO's CPU mapping casuing SIGBUS
>>> 	     and termination and waiting for drm file refcound to drop to 0 before releasing the device
>>> 	1.2) BO is not mapped into applcation address space (BO is CPU invisible) - no solution yet because how we force the application to terminate in this case ?
>>>
>>> 2) Application accesses a BO by importing a DMA-BUF
>>> 	2.1)  BO is mapped into applications address space (BO is CPU visible) - solution is same as 1.1 but instead of waiting for drm file release we wait for the
>>> 	      imported dma-buf's file release
>>> 	2.2)  BO is not mapped into applcation address space (BO is CPU invisible) - our solution is to invalidate GPUVM page tables and destroy backing storage for
>>>                all exported BOs which will in turn casue VM faults in the importing device and then when the importing driver will try to re-attach the imported BO to
>>> 	      update mappings we return -ENODEV in the import hook which hopeffuly will cause the user app to terminate.
>>>
>>> 3) Applcation opens a drm file or imports a dma-bud and holds a reference but never access any BO or does access but never more after device was unplug - how would we
>>>     force this applcation to termiante before proceeding with device removal code ? Otherwise the wait in pci remove just hangs for ever.
>>>
>>> The attached patches adress 1.1, 2.1 and 2.2, for now only 1.1 fully tested and I am still testing the others but I will be happy for any advise on all the
>>> described use cases and maybe some alternative and better (more generic) approach to this like maybe obtaining PIDs of relevant processes through some revere
>>> mapping from device file and exported dma-buf files and send them SIGKILL - would this make more sense or any other method ?
>>>
>>> Patches 1-3 address 1.1
>>> Patch 4 addresses 2.1
>>> Pathces 5-6 address 2.2
>>>
>>> Reference: https://gitlab.freedesktop.org/drm/amd/-/issues/1081
>> So we've been working on this problem for a few years already (but it's
>> still not solved), I think you could have saved yourselfs some typing.
>>
>> Bunch of things:
>> - we can't wait for userspace in the hotunplug handlers, that might never
>>    happen. The correct way is to untangle the lifetime of your hw driver
>>    for a specific struct pci_device from the drm_device lifetime.
>>    Infrastructure is all there now, see drm_dev_get/put, drm_dev_unplug and
>>    drm_dev_enter/exit. A bunch of usb/spi drivers use this 100% correctly
>>    now, so there's examples. Plus kerneldoc explains stuff.

That's exactly what we tried first and I expected that this is a 
necessity. Ok so back to the drawing board for this.

>>
>> - for a big driver like amdgpu doing this split up is going to be
>>    horrendously complex. I know, we've done it for i915, at least
>>    partially. I strongly recommend that you're using devm_ for managing hw
>>    related resources (iomap, irq, ...) as much as possible.
>>
>>    For drm_device resources (mostly structures and everything related to
>>    that) we've just merged the drmm_ managed resources framework. There's
>>    some more work to be done there for various kms objects, but you can at
>>    least somewhat avoid tedious handrolling for everything internal
>>    already.
>>
>>    Don't ever use devm_kzalloc and friends, I've looked at hundreds of uses
>>    of this in drm, they're all wrong.
>>
>> - dma-buf is hilarious (and atm unfixed), dma-fence is even worse. In
>>    theory they're already refcounted and all and so should work, in
>>    practice I think we need to refcount the underlying drm_device with
>>    drm_dev_get/put to avoid the worst fall-out.
> oh I forgot one, since it's new since the last time we've seriously
> discussed this: p2p dma-buf
>
> But that /should/ be handleable with the move_notify callback. Assuming we
> don't have any bugs anywhere, and the importer can indeed get rid of all
> its mapping, always.

Yeah, already noted that as well in the internal discussion.

>
> But for completeness probably need this one, just to keep it noted.
> -Daniel
>
>> - One unfortunate thing with drm_dev_unplug is that the driver core is
>>    very opinionated and doesn't tell you whether it's a hotunplug or a
>>    driver unload. In the former case trying to shut down hw just wastes
>>    time (and might hit driver bugs), in the latter case driver engineers
>>    very much expect everything to be shut down.
>>
>>    Right now you can only have one or the other, so this needs a module
>>    option hack or similar (default to the correct hotunplug behaviour for
>>    users).
>>
>> - SIGBUS is better than crashing the kernel, but it's not even close for
>>    users. They still lose everything because everything crashes because in
>>    my experience, in practice, no one ever handles errors. There's a few
>>    more things on top:
>>
>>    - sighandlers are global, which means only the app can use it. You can't
>>      use it in e.g. mesa. They're also not composable, so if you have on
>>      sighandler for gpu1 and a 2nd one for gpu2 (could be different vendor)
>>      it's all sadness. Hence "usersapce will handle SIGBUS" wont work.
>>
>>    - worse, neither vk nor gl (to my knowledge) have a concept of events
>>      for when the gpu died. The only stuff you have is things like
>>      arb_robustness which says a) everything continues as if nothing
>>      happened b) there's a function where you can ask whether your gl
>>      context and all the textures/buffers are toast.
>>
>>      I think that's about the only hotunplug application model we can
>>      realistically expect applications to support. That means _all_ errors
>>      need to be silently eaten by either mesa or the kernel. On i915 the
>>      model (also for terminally wedged gpu hangs) is that all ioctl keep
>>      working, mmaps keep working, and execbuf gives you an -EIO (which mesa
>>      eats silently iirc for arb_robustness).
>>
>>    Conclusion is that SIGBUS is imo a no-go, and the only option we have is
>>    that a) mmaps fully keep working, doable for shmem or b) we put some
>>    fake memory in there (for vram or whatever), maybe even only a single
>>    page for all fake memory.

Ok, good to know.

So to summarize no application termination, but instead redirect all 
memory access to a dummy page.

 From the IOCTLs we return -ENODEV instead of -EIO. Is that a problem?

Thanks for the comments,
Christian.

>>
>> - you probably want arb_robustness and similar stuff in userspace as a
>>    first step.
>>
>> tldr;
>> - refcounting, not waiting for userspace
>> - nothing can fail because userspace wont handle it
>>
>> That's at least my take on this mess, and what we've been pushing for over
>> the past few years. For kms-only drm_driver we should have achieved that
>> by now (plus/minus maybe some issues for dma-buf/fences, but kms-only
>> dma-buf/fences are simple enough that maybe we don't go boom yet).
>>
>> For big gpus with rendering I think best next step would be to type up a
>> reasonable Gran Plan (into Documentation/gpu/todo.rst) with all the issues
>> and likely solutions. And then bikeshed that, since the above is just my
>> take on all this.
>>
>> Cheers, Daniel
>>
>>> Andrey Grodzovsky (6):
>>>    drm/ttm: Add unampping of the entire device address space
>>>    drm/amdgpu: Force unmap all user VMAs on device removal.
>>>    drm/amdgpu: Wait for all user clients
>>>    drm/amdgpu: Wait for all clients importing out dma-bufs.
>>>    drm/ttm: Add destroy flag in TTM BO eviction interface
>>>    drm/amdgpu: Use TTM MMs destroy interface
>>>
>>>   drivers/gpu/drm/amd/amdgpu/amdgpu.h         |  3 ++
>>>   drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c |  2 +-
>>>   drivers/gpu/drm/amd/amdgpu/amdgpu_device.c  |  7 +++-
>>>   drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c | 27 ++++++++++++-
>>>   drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c     | 22 ++++++++--
>>>   drivers/gpu/drm/amd/amdgpu/amdgpu_job.c     |  9 +++++
>>>   drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c     |  4 ++
>>>   drivers/gpu/drm/amd/amdgpu/amdgpu_object.c  | 17 +++++++-
>>>   drivers/gpu/drm/amd/amdgpu/amdgpu_object.h  |  1 +
>>>   drivers/gpu/drm/nouveau/nouveau_drm.c       |  2 +-
>>>   drivers/gpu/drm/qxl/qxl_object.c            |  4 +-
>>>   drivers/gpu/drm/radeon/radeon_object.c      |  2 +-
>>>   drivers/gpu/drm/ttm/ttm_bo.c                | 63 +++++++++++++++++++++--------
>>>   drivers/gpu/drm/vmwgfx/vmwgfx_drv.c         |  6 +--
>>>   include/drm/ttm/ttm_bo_api.h                |  2 +-
>>>   include/drm/ttm/ttm_bo_driver.h             |  2 +
>>>   16 files changed, 139 insertions(+), 34 deletions(-)
>>>
>>> -- 
>>> 2.7.4
>>>
>> -- 
>> Daniel Vetter
>> Software Engineer, Intel Corporation
>> http://blog.ffwll.ch

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [PATCH 0/6] RFC Support hot device unplug in amdgpu

[Christian König]

Am 11.05.20 um 13:19 schrieb Daniel Vetter:
> On Mon, May 11, 2020 at 11:54:33AM +0200, Daniel Vetter wrote:
>> On Sat, May 09, 2020 at 02:51:44PM -0400, Andrey Grodzovsky wrote:
>>> This RFC is a more of a proof of concept then a fully working solution as there are a few unresolved issues we are hopping to get advise on from people on the mailing list.
>>> Until now extracting a card either by physical extraction (e.g. eGPU with thunderbold connection or by emulation through syfs -> /sys/bus/pci/devices/device_id/remove)
>>> would cause random crashes in user apps. The random crashes in apps were mostly due to the app having mapped a device backed BO into it's adress space was still
>>> trying to access the BO while the backing device was gone.
>>> To answer this first problem Christian suggested to fix the handling of mapped memory in the clients when the device goes away by forcibly unmap all buffers
>>> the user processes has by clearing their respective VMAs mapping the device BOs. Then when the VMAs try to fill in the page tables again we check in the fault handler
>>> if the device is removed and if so, return an error. This will generate a SIGBUS to the application which can then cleanly terminate.
>>> This indeed was done but this in turn created a problem of kernel OOPs were the OOPSes were due to the fact that while the app was terminating because of the SIGBUS
>>> it would trigger use after free in the driver by calling to accesses device structures that were already released from the pci remove sequence.
>>> This we handled by introducing a 'flush' seqence during device removal were we wait for drm file reference to drop to 0 meaning all user clients directly using this device terminated.
>>> With this I was able to cleanly emulate device unplug with X and glxgears running and later emulate device plug back and restart of X and glxgears.
>>>
>>> But this use case is only partial and as I see it all the use cases are as follwing and the questions it raises.
>>>
>>> 1) Application accesses a BO by opening drm file
>>> 	1.1) BO is mapped into applications address space (BO is CPU visible) - this one we have a solution for by invaldating BO's CPU mapping casuing SIGBUS
>>> 	     and termination and waiting for drm file refcound to drop to 0 before releasing the device
>>> 	1.2) BO is not mapped into applcation address space (BO is CPU invisible) - no solution yet because how we force the application to terminate in this case ?
>>>
>>> 2) Application accesses a BO by importing a DMA-BUF
>>> 	2.1)  BO is mapped into applications address space (BO is CPU visible) - solution is same as 1.1 but instead of waiting for drm file release we wait for the
>>> 	      imported dma-buf's file release
>>> 	2.2)  BO is not mapped into applcation address space (BO is CPU invisible) - our solution is to invalidate GPUVM page tables and destroy backing storage for
>>>                all exported BOs which will in turn casue VM faults in the importing device and then when the importing driver will try to re-attach the imported BO to
>>> 	      update mappings we return -ENODEV in the import hook which hopeffuly will cause the user app to terminate.
>>>
>>> 3) Applcation opens a drm file or imports a dma-bud and holds a reference but never access any BO or does access but never more after device was unplug - how would we
>>>     force this applcation to termiante before proceeding with device removal code ? Otherwise the wait in pci remove just hangs for ever.
>>>
>>> The attached patches adress 1.1, 2.1 and 2.2, for now only 1.1 fully tested and I am still testing the others but I will be happy for any advise on all the
>>> described use cases and maybe some alternative and better (more generic) approach to this like maybe obtaining PIDs of relevant processes through some revere
>>> mapping from device file and exported dma-buf files and send them SIGKILL - would this make more sense or any other method ?
>>>
>>> Patches 1-3 address 1.1
>>> Patch 4 addresses 2.1
>>> Pathces 5-6 address 2.2
>>>
>>> Reference: https://gitlab.freedesktop.org/drm/amd/-/issues/1081
>> So we've been working on this problem for a few years already (but it's
>> still not solved), I think you could have saved yourselfs some typing.
>>
>> Bunch of things:
>> - we can't wait for userspace in the hotunplug handlers, that might never
>>    happen. The correct way is to untangle the lifetime of your hw driver
>>    for a specific struct pci_device from the drm_device lifetime.
>>    Infrastructure is all there now, see drm_dev_get/put, drm_dev_unplug and
>>    drm_dev_enter/exit. A bunch of usb/spi drivers use this 100% correctly
>>    now, so there's examples. Plus kerneldoc explains stuff.

That's exactly what we tried first and I expected that this is a 
necessity. Ok so back to the drawing board for this.

>>
>> - for a big driver like amdgpu doing this split up is going to be
>>    horrendously complex. I know, we've done it for i915, at least
>>    partially. I strongly recommend that you're using devm_ for managing hw
>>    related resources (iomap, irq, ...) as much as possible.
>>
>>    For drm_device resources (mostly structures and everything related to
>>    that) we've just merged the drmm_ managed resources framework. There's
>>    some more work to be done there for various kms objects, but you can at
>>    least somewhat avoid tedious handrolling for everything internal
>>    already.
>>
>>    Don't ever use devm_kzalloc and friends, I've looked at hundreds of uses
>>    of this in drm, they're all wrong.
>>
>> - dma-buf is hilarious (and atm unfixed), dma-fence is even worse. In
>>    theory they're already refcounted and all and so should work, in
>>    practice I think we need to refcount the underlying drm_device with
>>    drm_dev_get/put to avoid the worst fall-out.
> oh I forgot one, since it's new since the last time we've seriously
> discussed this: p2p dma-buf
>
> But that /should/ be handleable with the move_notify callback. Assuming we
> don't have any bugs anywhere, and the importer can indeed get rid of all
> its mapping, always.

Yeah, already noted that as well in the internal discussion.

>
> But for completeness probably need this one, just to keep it noted.
> -Daniel
>
>> - One unfortunate thing with drm_dev_unplug is that the driver core is
>>    very opinionated and doesn't tell you whether it's a hotunplug or a
>>    driver unload. In the former case trying to shut down hw just wastes
>>    time (and might hit driver bugs), in the latter case driver engineers
>>    very much expect everything to be shut down.
>>
>>    Right now you can only have one or the other, so this needs a module
>>    option hack or similar (default to the correct hotunplug behaviour for
>>    users).
>>
>> - SIGBUS is better than crashing the kernel, but it's not even close for
>>    users. They still lose everything because everything crashes because in
>>    my experience, in practice, no one ever handles errors. There's a few
>>    more things on top:
>>
>>    - sighandlers are global, which means only the app can use it. You can't
>>      use it in e.g. mesa. They're also not composable, so if you have on
>>      sighandler for gpu1 and a 2nd one for gpu2 (could be different vendor)
>>      it's all sadness. Hence "usersapce will handle SIGBUS" wont work.
>>
>>    - worse, neither vk nor gl (to my knowledge) have a concept of events
>>      for when the gpu died. The only stuff you have is things like
>>      arb_robustness which says a) everything continues as if nothing
>>      happened b) there's a function where you can ask whether your gl
>>      context and all the textures/buffers are toast.
>>
>>      I think that's about the only hotunplug application model we can
>>      realistically expect applications to support. That means _all_ errors
>>      need to be silently eaten by either mesa or the kernel. On i915 the
>>      model (also for terminally wedged gpu hangs) is that all ioctl keep
>>      working, mmaps keep working, and execbuf gives you an -EIO (which mesa
>>      eats silently iirc for arb_robustness).
>>
>>    Conclusion is that SIGBUS is imo a no-go, and the only option we have is
>>    that a) mmaps fully keep working, doable for shmem or b) we put some
>>    fake memory in there (for vram or whatever), maybe even only a single
>>    page for all fake memory.

Ok, good to know.

So to summarize no application termination, but instead redirect all 
memory access to a dummy page.

 From the IOCTLs we return -ENODEV instead of -EIO. Is that a problem?

Thanks for the comments,
Christian.

>>
>> - you probably want arb_robustness and similar stuff in userspace as a
>>    first step.
>>
>> tldr;
>> - refcounting, not waiting for userspace
>> - nothing can fail because userspace wont handle it
>>
>> That's at least my take on this mess, and what we've been pushing for over
>> the past few years. For kms-only drm_driver we should have achieved that
>> by now (plus/minus maybe some issues for dma-buf/fences, but kms-only
>> dma-buf/fences are simple enough that maybe we don't go boom yet).
>>
>> For big gpus with rendering I think best next step would be to type up a
>> reasonable Gran Plan (into Documentation/gpu/todo.rst) with all the issues
>> and likely solutions. And then bikeshed that, since the above is just my
>> take on all this.
>>
>> Cheers, Daniel
>>
>>> Andrey Grodzovsky (6):
>>>    drm/ttm: Add unampping of the entire device address space
>>>    drm/amdgpu: Force unmap all user VMAs on device removal.
>>>    drm/amdgpu: Wait for all user clients
>>>    drm/amdgpu: Wait for all clients importing out dma-bufs.
>>>    drm/ttm: Add destroy flag in TTM BO eviction interface
>>>    drm/amdgpu: Use TTM MMs destroy interface
>>>
>>>   drivers/gpu/drm/amd/amdgpu/amdgpu.h         |  3 ++
>>>   drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c |  2 +-
>>>   drivers/gpu/drm/amd/amdgpu/amdgpu_device.c  |  7 +++-
>>>   drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c | 27 ++++++++++++-
>>>   drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c     | 22 ++++++++--
>>>   drivers/gpu/drm/amd/amdgpu/amdgpu_job.c     |  9 +++++
>>>   drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c     |  4 ++
>>>   drivers/gpu/drm/amd/amdgpu/amdgpu_object.c  | 17 +++++++-
>>>   drivers/gpu/drm/amd/amdgpu/amdgpu_object.h  |  1 +
>>>   drivers/gpu/drm/nouveau/nouveau_drm.c       |  2 +-
>>>   drivers/gpu/drm/qxl/qxl_object.c            |  4 +-
>>>   drivers/gpu/drm/radeon/radeon_object.c      |  2 +-
>>>   drivers/gpu/drm/ttm/ttm_bo.c                | 63 +++++++++++++++++++++--------
>>>   drivers/gpu/drm/vmwgfx/vmwgfx_drv.c         |  6 +--
>>>   include/drm/ttm/ttm_bo_api.h                |  2 +-
>>>   include/drm/ttm/ttm_bo_driver.h             |  2 +
>>>   16 files changed, 139 insertions(+), 34 deletions(-)
>>>
>>> -- 
>>> 2.7.4
>>>
>> -- 
>> Daniel Vetter
>> Software Engineer, Intel Corporation
>> http://blog.ffwll.ch

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH 0/6] RFC Support hot device unplug in amdgpu

[Daniel Vetter]

On Mon, May 11, 2020 at 2:34 PM Christian König
<ckoenig.leichtzumerken@gmail.com> wrote:
>
> Am 11.05.20 um 13:19 schrieb Daniel Vetter:
> > On Mon, May 11, 2020 at 11:54:33AM +0200, Daniel Vetter wrote:
> >> On Sat, May 09, 2020 at 02:51:44PM -0400, Andrey Grodzovsky wrote:
> >>> This RFC is a more of a proof of concept then a fully working solution as there are a few unresolved issues we are hopping to get advise on from people on the mailing list.
> >>> Until now extracting a card either by physical extraction (e.g. eGPU with thunderbold connection or by emulation through syfs -> /sys/bus/pci/devices/device_id/remove)
> >>> would cause random crashes in user apps. The random crashes in apps were mostly due to the app having mapped a device backed BO into it's adress space was still
> >>> trying to access the BO while the backing device was gone.
> >>> To answer this first problem Christian suggested to fix the handling of mapped memory in the clients when the device goes away by forcibly unmap all buffers
> >>> the user processes has by clearing their respective VMAs mapping the device BOs. Then when the VMAs try to fill in the page tables again we check in the fault handler
> >>> if the device is removed and if so, return an error. This will generate a SIGBUS to the application which can then cleanly terminate.
> >>> This indeed was done but this in turn created a problem of kernel OOPs were the OOPSes were due to the fact that while the app was terminating because of the SIGBUS
> >>> it would trigger use after free in the driver by calling to accesses device structures that were already released from the pci remove sequence.
> >>> This we handled by introducing a 'flush' seqence during device removal were we wait for drm file reference to drop to 0 meaning all user clients directly using this device terminated.
> >>> With this I was able to cleanly emulate device unplug with X and glxgears running and later emulate device plug back and restart of X and glxgears.
> >>>
> >>> But this use case is only partial and as I see it all the use cases are as follwing and the questions it raises.
> >>>
> >>> 1) Application accesses a BO by opening drm file
> >>>     1.1) BO is mapped into applications address space (BO is CPU visible) - this one we have a solution for by invaldating BO's CPU mapping casuing SIGBUS
> >>>          and termination and waiting for drm file refcound to drop to 0 before releasing the device
> >>>     1.2) BO is not mapped into applcation address space (BO is CPU invisible) - no solution yet because how we force the application to terminate in this case ?
> >>>
> >>> 2) Application accesses a BO by importing a DMA-BUF
> >>>     2.1)  BO is mapped into applications address space (BO is CPU visible) - solution is same as 1.1 but instead of waiting for drm file release we wait for the
> >>>           imported dma-buf's file release
> >>>     2.2)  BO is not mapped into applcation address space (BO is CPU invisible) - our solution is to invalidate GPUVM page tables and destroy backing storage for
> >>>                all exported BOs which will in turn casue VM faults in the importing device and then when the importing driver will try to re-attach the imported BO to
> >>>           update mappings we return -ENODEV in the import hook which hopeffuly will cause the user app to terminate.
> >>>
> >>> 3) Applcation opens a drm file or imports a dma-bud and holds a reference but never access any BO or does access but never more after device was unplug - how would we
> >>>     force this applcation to termiante before proceeding with device removal code ? Otherwise the wait in pci remove just hangs for ever.
> >>>
> >>> The attached patches adress 1.1, 2.1 and 2.2, for now only 1.1 fully tested and I am still testing the others but I will be happy for any advise on all the
> >>> described use cases and maybe some alternative and better (more generic) approach to this like maybe obtaining PIDs of relevant processes through some revere
> >>> mapping from device file and exported dma-buf files and send them SIGKILL - would this make more sense or any other method ?
> >>>
> >>> Patches 1-3 address 1.1
> >>> Patch 4 addresses 2.1
> >>> Pathces 5-6 address 2.2
> >>>
> >>> Reference: https://gitlab.freedesktop.org/drm/amd/-/issues/1081
> >> So we've been working on this problem for a few years already (but it's
> >> still not solved), I think you could have saved yourselfs some typing.
> >>
> >> Bunch of things:
> >> - we can't wait for userspace in the hotunplug handlers, that might never
> >>    happen. The correct way is to untangle the lifetime of your hw driver
> >>    for a specific struct pci_device from the drm_device lifetime.
> >>    Infrastructure is all there now, see drm_dev_get/put, drm_dev_unplug and
> >>    drm_dev_enter/exit. A bunch of usb/spi drivers use this 100% correctly
> >>    now, so there's examples. Plus kerneldoc explains stuff.
>
> That's exactly what we tried first and I expected that this is a
> necessity. Ok so back to the drawing board for this.
>
> >>
> >> - for a big driver like amdgpu doing this split up is going to be
> >>    horrendously complex. I know, we've done it for i915, at least
> >>    partially. I strongly recommend that you're using devm_ for managing hw
> >>    related resources (iomap, irq, ...) as much as possible.
> >>
> >>    For drm_device resources (mostly structures and everything related to
> >>    that) we've just merged the drmm_ managed resources framework. There's
> >>    some more work to be done there for various kms objects, but you can at
> >>    least somewhat avoid tedious handrolling for everything internal
> >>    already.
> >>
> >>    Don't ever use devm_kzalloc and friends, I've looked at hundreds of uses
> >>    of this in drm, they're all wrong.
> >>
> >> - dma-buf is hilarious (and atm unfixed), dma-fence is even worse. In
> >>    theory they're already refcounted and all and so should work, in
> >>    practice I think we need to refcount the underlying drm_device with
> >>    drm_dev_get/put to avoid the worst fall-out.
> > oh I forgot one, since it's new since the last time we've seriously
> > discussed this: p2p dma-buf
> >
> > But that /should/ be handleable with the move_notify callback. Assuming we
> > don't have any bugs anywhere, and the importer can indeed get rid of all
> > its mapping, always.
>
> Yeah, already noted that as well in the internal discussion.
>
> >
> > But for completeness probably need this one, just to keep it noted.
> > -Daniel
> >
> >> - One unfortunate thing with drm_dev_unplug is that the driver core is
> >>    very opinionated and doesn't tell you whether it's a hotunplug or a
> >>    driver unload. In the former case trying to shut down hw just wastes
> >>    time (and might hit driver bugs), in the latter case driver engineers
> >>    very much expect everything to be shut down.
> >>
> >>    Right now you can only have one or the other, so this needs a module
> >>    option hack or similar (default to the correct hotunplug behaviour for
> >>    users).
> >>
> >> - SIGBUS is better than crashing the kernel, but it's not even close for
> >>    users. They still lose everything because everything crashes because in
> >>    my experience, in practice, no one ever handles errors. There's a few
> >>    more things on top:
> >>
> >>    - sighandlers are global, which means only the app can use it. You can't
> >>      use it in e.g. mesa. They're also not composable, so if you have on
> >>      sighandler for gpu1 and a 2nd one for gpu2 (could be different vendor)
> >>      it's all sadness. Hence "usersapce will handle SIGBUS" wont work.
> >>
> >>    - worse, neither vk nor gl (to my knowledge) have a concept of events
> >>      for when the gpu died. The only stuff you have is things like
> >>      arb_robustness which says a) everything continues as if nothing
> >>      happened b) there's a function where you can ask whether your gl
> >>      context and all the textures/buffers are toast.
> >>
> >>      I think that's about the only hotunplug application model we can
> >>      realistically expect applications to support. That means _all_ errors
> >>      need to be silently eaten by either mesa or the kernel. On i915 the
> >>      model (also for terminally wedged gpu hangs) is that all ioctl keep
> >>      working, mmaps keep working, and execbuf gives you an -EIO (which mesa
> >>      eats silently iirc for arb_robustness).
> >>
> >>    Conclusion is that SIGBUS is imo a no-go, and the only option we have is
> >>    that a) mmaps fully keep working, doable for shmem or b) we put some
> >>    fake memory in there (for vram or whatever), maybe even only a single
> >>    page for all fake memory.
>
> Ok, good to know.
>
> So to summarize no application termination, but instead redirect all
> memory access to a dummy page.
>
>  From the IOCTLs we return -ENODEV instead of -EIO. Is that a problem?

For atomic I think it'd be good if we're consistent across drivers.

For render ioctl all that matters is that you end up implementing
arb_robusteness/vk device removal/whatever else custom interface HSA
has/... correctly. So entirely up to a discussion between amdgpu and
radeonsi folks I'd say.
-Daniel

> Thanks for the comments,
> Christian.
>
> >>
> >> - you probably want arb_robustness and similar stuff in userspace as a
> >>    first step.
> >>
> >> tldr;
> >> - refcounting, not waiting for userspace
> >> - nothing can fail because userspace wont handle it
> >>
> >> That's at least my take on this mess, and what we've been pushing for over
> >> the past few years. For kms-only drm_driver we should have achieved that
> >> by now (plus/minus maybe some issues for dma-buf/fences, but kms-only
> >> dma-buf/fences are simple enough that maybe we don't go boom yet).
> >>
> >> For big gpus with rendering I think best next step would be to type up a
> >> reasonable Gran Plan (into Documentation/gpu/todo.rst) with all the issues
> >> and likely solutions. And then bikeshed that, since the above is just my
> >> take on all this.
> >>
> >> Cheers, Daniel
> >>
> >>> Andrey Grodzovsky (6):
> >>>    drm/ttm: Add unampping of the entire device address space
> >>>    drm/amdgpu: Force unmap all user VMAs on device removal.
> >>>    drm/amdgpu: Wait for all user clients
> >>>    drm/amdgpu: Wait for all clients importing out dma-bufs.
> >>>    drm/ttm: Add destroy flag in TTM BO eviction interface
> >>>    drm/amdgpu: Use TTM MMs destroy interface
> >>>
> >>>   drivers/gpu/drm/amd/amdgpu/amdgpu.h         |  3 ++
> >>>   drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c |  2 +-
> >>>   drivers/gpu/drm/amd/amdgpu/amdgpu_device.c  |  7 +++-
> >>>   drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c | 27 ++++++++++++-
> >>>   drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c     | 22 ++++++++--
> >>>   drivers/gpu/drm/amd/amdgpu/amdgpu_job.c     |  9 +++++
> >>>   drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c     |  4 ++
> >>>   drivers/gpu/drm/amd/amdgpu/amdgpu_object.c  | 17 +++++++-
> >>>   drivers/gpu/drm/amd/amdgpu/amdgpu_object.h  |  1 +
> >>>   drivers/gpu/drm/nouveau/nouveau_drm.c       |  2 +-
> >>>   drivers/gpu/drm/qxl/qxl_object.c            |  4 +-
> >>>   drivers/gpu/drm/radeon/radeon_object.c      |  2 +-
> >>>   drivers/gpu/drm/ttm/ttm_bo.c                | 63 +++++++++++++++++++++--------
> >>>   drivers/gpu/drm/vmwgfx/vmwgfx_drv.c         |  6 +--
> >>>   include/drm/ttm/ttm_bo_api.h                |  2 +-
> >>>   include/drm/ttm/ttm_bo_driver.h             |  2 +
> >>>   16 files changed, 139 insertions(+), 34 deletions(-)
> >>>
> >>> --
> >>> 2.7.4
> >>>
> >> --
> >> Daniel Vetter
> >> Software Engineer, Intel Corporation
> >> http://blog.ffwll.ch
>


-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [PATCH 0/6] RFC Support hot device unplug in amdgpu

[Daniel Vetter]

On Mon, May 11, 2020 at 2:34 PM Christian König
<ckoenig.leichtzumerken@gmail.com> wrote:
>
> Am 11.05.20 um 13:19 schrieb Daniel Vetter:
> > On Mon, May 11, 2020 at 11:54:33AM +0200, Daniel Vetter wrote:
> >> On Sat, May 09, 2020 at 02:51:44PM -0400, Andrey Grodzovsky wrote:
> >>> This RFC is a more of a proof of concept then a fully working solution as there are a few unresolved issues we are hopping to get advise on from people on the mailing list.
> >>> Until now extracting a card either by physical extraction (e.g. eGPU with thunderbold connection or by emulation through syfs -> /sys/bus/pci/devices/device_id/remove)
> >>> would cause random crashes in user apps. The random crashes in apps were mostly due to the app having mapped a device backed BO into it's adress space was still
> >>> trying to access the BO while the backing device was gone.
> >>> To answer this first problem Christian suggested to fix the handling of mapped memory in the clients when the device goes away by forcibly unmap all buffers
> >>> the user processes has by clearing their respective VMAs mapping the device BOs. Then when the VMAs try to fill in the page tables again we check in the fault handler
> >>> if the device is removed and if so, return an error. This will generate a SIGBUS to the application which can then cleanly terminate.
> >>> This indeed was done but this in turn created a problem of kernel OOPs were the OOPSes were due to the fact that while the app was terminating because of the SIGBUS
> >>> it would trigger use after free in the driver by calling to accesses device structures that were already released from the pci remove sequence.
> >>> This we handled by introducing a 'flush' seqence during device removal were we wait for drm file reference to drop to 0 meaning all user clients directly using this device terminated.
> >>> With this I was able to cleanly emulate device unplug with X and glxgears running and later emulate device plug back and restart of X and glxgears.
> >>>
> >>> But this use case is only partial and as I see it all the use cases are as follwing and the questions it raises.
> >>>
> >>> 1) Application accesses a BO by opening drm file
> >>>     1.1) BO is mapped into applications address space (BO is CPU visible) - this one we have a solution for by invaldating BO's CPU mapping casuing SIGBUS
> >>>          and termination and waiting for drm file refcound to drop to 0 before releasing the device
> >>>     1.2) BO is not mapped into applcation address space (BO is CPU invisible) - no solution yet because how we force the application to terminate in this case ?
> >>>
> >>> 2) Application accesses a BO by importing a DMA-BUF
> >>>     2.1)  BO is mapped into applications address space (BO is CPU visible) - solution is same as 1.1 but instead of waiting for drm file release we wait for the
> >>>           imported dma-buf's file release
> >>>     2.2)  BO is not mapped into applcation address space (BO is CPU invisible) - our solution is to invalidate GPUVM page tables and destroy backing storage for
> >>>                all exported BOs which will in turn casue VM faults in the importing device and then when the importing driver will try to re-attach the imported BO to
> >>>           update mappings we return -ENODEV in the import hook which hopeffuly will cause the user app to terminate.
> >>>
> >>> 3) Applcation opens a drm file or imports a dma-bud and holds a reference but never access any BO or does access but never more after device was unplug - how would we
> >>>     force this applcation to termiante before proceeding with device removal code ? Otherwise the wait in pci remove just hangs for ever.
> >>>
> >>> The attached patches adress 1.1, 2.1 and 2.2, for now only 1.1 fully tested and I am still testing the others but I will be happy for any advise on all the
> >>> described use cases and maybe some alternative and better (more generic) approach to this like maybe obtaining PIDs of relevant processes through some revere
> >>> mapping from device file and exported dma-buf files and send them SIGKILL - would this make more sense or any other method ?
> >>>
> >>> Patches 1-3 address 1.1
> >>> Patch 4 addresses 2.1
> >>> Pathces 5-6 address 2.2
> >>>
> >>> Reference: https://gitlab.freedesktop.org/drm/amd/-/issues/1081
> >> So we've been working on this problem for a few years already (but it's
> >> still not solved), I think you could have saved yourselfs some typing.
> >>
> >> Bunch of things:
> >> - we can't wait for userspace in the hotunplug handlers, that might never
> >>    happen. The correct way is to untangle the lifetime of your hw driver
> >>    for a specific struct pci_device from the drm_device lifetime.
> >>    Infrastructure is all there now, see drm_dev_get/put, drm_dev_unplug and
> >>    drm_dev_enter/exit. A bunch of usb/spi drivers use this 100% correctly
> >>    now, so there's examples. Plus kerneldoc explains stuff.
>
> That's exactly what we tried first and I expected that this is a
> necessity. Ok so back to the drawing board for this.
>
> >>
> >> - for a big driver like amdgpu doing this split up is going to be
> >>    horrendously complex. I know, we've done it for i915, at least
> >>    partially. I strongly recommend that you're using devm_ for managing hw
> >>    related resources (iomap, irq, ...) as much as possible.
> >>
> >>    For drm_device resources (mostly structures and everything related to
> >>    that) we've just merged the drmm_ managed resources framework. There's
> >>    some more work to be done there for various kms objects, but you can at
> >>    least somewhat avoid tedious handrolling for everything internal
> >>    already.
> >>
> >>    Don't ever use devm_kzalloc and friends, I've looked at hundreds of uses
> >>    of this in drm, they're all wrong.
> >>
> >> - dma-buf is hilarious (and atm unfixed), dma-fence is even worse. In
> >>    theory they're already refcounted and all and so should work, in
> >>    practice I think we need to refcount the underlying drm_device with
> >>    drm_dev_get/put to avoid the worst fall-out.
> > oh I forgot one, since it's new since the last time we've seriously
> > discussed this: p2p dma-buf
> >
> > But that /should/ be handleable with the move_notify callback. Assuming we
> > don't have any bugs anywhere, and the importer can indeed get rid of all
> > its mapping, always.
>
> Yeah, already noted that as well in the internal discussion.
>
> >
> > But for completeness probably need this one, just to keep it noted.
> > -Daniel
> >
> >> - One unfortunate thing with drm_dev_unplug is that the driver core is
> >>    very opinionated and doesn't tell you whether it's a hotunplug or a
> >>    driver unload. In the former case trying to shut down hw just wastes
> >>    time (and might hit driver bugs), in the latter case driver engineers
> >>    very much expect everything to be shut down.
> >>
> >>    Right now you can only have one or the other, so this needs a module
> >>    option hack or similar (default to the correct hotunplug behaviour for
> >>    users).
> >>
> >> - SIGBUS is better than crashing the kernel, but it's not even close for
> >>    users. They still lose everything because everything crashes because in
> >>    my experience, in practice, no one ever handles errors. There's a few
> >>    more things on top:
> >>
> >>    - sighandlers are global, which means only the app can use it. You can't
> >>      use it in e.g. mesa. They're also not composable, so if you have on
> >>      sighandler for gpu1 and a 2nd one for gpu2 (could be different vendor)
> >>      it's all sadness. Hence "usersapce will handle SIGBUS" wont work.
> >>
> >>    - worse, neither vk nor gl (to my knowledge) have a concept of events
> >>      for when the gpu died. The only stuff you have is things like
> >>      arb_robustness which says a) everything continues as if nothing
> >>      happened b) there's a function where you can ask whether your gl
> >>      context and all the textures/buffers are toast.
> >>
> >>      I think that's about the only hotunplug application model we can
> >>      realistically expect applications to support. That means _all_ errors
> >>      need to be silently eaten by either mesa or the kernel. On i915 the
> >>      model (also for terminally wedged gpu hangs) is that all ioctl keep
> >>      working, mmaps keep working, and execbuf gives you an -EIO (which mesa
> >>      eats silently iirc for arb_robustness).
> >>
> >>    Conclusion is that SIGBUS is imo a no-go, and the only option we have is
> >>    that a) mmaps fully keep working, doable for shmem or b) we put some
> >>    fake memory in there (for vram or whatever), maybe even only a single
> >>    page for all fake memory.
>
> Ok, good to know.
>
> So to summarize no application termination, but instead redirect all
> memory access to a dummy page.
>
>  From the IOCTLs we return -ENODEV instead of -EIO. Is that a problem?

For atomic I think it'd be good if we're consistent across drivers.

For render ioctl all that matters is that you end up implementing
arb_robusteness/vk device removal/whatever else custom interface HSA
has/... correctly. So entirely up to a discussion between amdgpu and
radeonsi folks I'd say.
-Daniel

> Thanks for the comments,
> Christian.
>
> >>
> >> - you probably want arb_robustness and similar stuff in userspace as a
> >>    first step.
> >>
> >> tldr;
> >> - refcounting, not waiting for userspace
> >> - nothing can fail because userspace wont handle it
> >>
> >> That's at least my take on this mess, and what we've been pushing for over
> >> the past few years. For kms-only drm_driver we should have achieved that
> >> by now (plus/minus maybe some issues for dma-buf/fences, but kms-only
> >> dma-buf/fences are simple enough that maybe we don't go boom yet).
> >>
> >> For big gpus with rendering I think best next step would be to type up a
> >> reasonable Gran Plan (into Documentation/gpu/todo.rst) with all the issues
> >> and likely solutions. And then bikeshed that, since the above is just my
> >> take on all this.
> >>
> >> Cheers, Daniel
> >>
> >>> Andrey Grodzovsky (6):
> >>>    drm/ttm: Add unampping of the entire device address space
> >>>    drm/amdgpu: Force unmap all user VMAs on device removal.
> >>>    drm/amdgpu: Wait for all user clients
> >>>    drm/amdgpu: Wait for all clients importing out dma-bufs.
> >>>    drm/ttm: Add destroy flag in TTM BO eviction interface
> >>>    drm/amdgpu: Use TTM MMs destroy interface
> >>>
> >>>   drivers/gpu/drm/amd/amdgpu/amdgpu.h         |  3 ++
> >>>   drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c |  2 +-
> >>>   drivers/gpu/drm/amd/amdgpu/amdgpu_device.c  |  7 +++-
> >>>   drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c | 27 ++++++++++++-
> >>>   drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c     | 22 ++++++++--
> >>>   drivers/gpu/drm/amd/amdgpu/amdgpu_job.c     |  9 +++++
> >>>   drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c     |  4 ++
> >>>   drivers/gpu/drm/amd/amdgpu/amdgpu_object.c  | 17 +++++++-
> >>>   drivers/gpu/drm/amd/amdgpu/amdgpu_object.h  |  1 +
> >>>   drivers/gpu/drm/nouveau/nouveau_drm.c       |  2 +-
> >>>   drivers/gpu/drm/qxl/qxl_object.c            |  4 +-
> >>>   drivers/gpu/drm/radeon/radeon_object.c      |  2 +-
> >>>   drivers/gpu/drm/ttm/ttm_bo.c                | 63 +++++++++++++++++++++--------
> >>>   drivers/gpu/drm/vmwgfx/vmwgfx_drv.c         |  6 +--
> >>>   include/drm/ttm/ttm_bo_api.h                |  2 +-
> >>>   include/drm/ttm/ttm_bo_driver.h             |  2 +
> >>>   16 files changed, 139 insertions(+), 34 deletions(-)
> >>>
> >>> --
> >>> 2.7.4
> >>>
> >> --
> >> Daniel Vetter
> >> Software Engineer, Intel Corporation
> >> http://blog.ffwll.ch
>


-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH 0/6] RFC Support hot device unplug in amdgpu

[Lukas Wunner]

On Mon, May 11, 2020 at 11:54:33AM +0200, Daniel Vetter wrote:
> - One unfortunate thing with drm_dev_unplug is that the driver core is
>   very opinionated and doesn't tell you whether it's a hotunplug or a
>   driver unload. In the former case trying to shut down hw just wastes
>   time (and might hit driver bugs), in the latter case driver engineers
>   very much expect everything to be shut down.

You can get that information at the PCI bus level with
pci_dev_is_disconnected().  The flag queried by this function is set
upon hot removal.  Be aware however that the device is guaranteed to
be unreachable if the function returns true, but the converse is NOT
guaranteed, i.e. the function may return false even though the device
has just gone away.

Those somewhat difficult semantics are one of the reasons why some
people are skeptical of the function's merits (notably Greg KH).
See this LWN article for more information:

https://lwn.net/Articles/767885/
(scroll down to the "Surprise removal" section)

I've suggested to Greg a few years back that we should have a flag
at the device level to indicate whether it's gone, not just at the bus
level.  That way the property could be expressed regardless of the bus
used.  It would facilitate the feature you're missing, that the driver
core tells you whether it's a surprise removal or not.  Unfortunately
Greg rejected the idea.

Thanks,

Lukas
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [PATCH 0/6] RFC Support hot device unplug in amdgpu

[Lukas Wunner]

On Mon, May 11, 2020 at 11:54:33AM +0200, Daniel Vetter wrote:
> - One unfortunate thing with drm_dev_unplug is that the driver core is
>   very opinionated and doesn't tell you whether it's a hotunplug or a
>   driver unload. In the former case trying to shut down hw just wastes
>   time (and might hit driver bugs), in the latter case driver engineers
>   very much expect everything to be shut down.

You can get that information at the PCI bus level with
pci_dev_is_disconnected().  The flag queried by this function is set
upon hot removal.  Be aware however that the device is guaranteed to
be unreachable if the function returns true, but the converse is NOT
guaranteed, i.e. the function may return false even though the device
has just gone away.

Those somewhat difficult semantics are one of the reasons why some
people are skeptical of the function's merits (notably Greg KH).
See this LWN article for more information:

https://lwn.net/Articles/767885/
(scroll down to the "Surprise removal" section)

I've suggested to Greg a few years back that we should have a flag
at the device level to indicate whether it's gone, not just at the bus
level.  That way the property could be expressed regardless of the bus
used.  It would facilitate the feature you're missing, that the driver
core tells you whether it's a surprise removal or not.  Unfortunately
Greg rejected the idea.

Thanks,

Lukas
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH 0/6] RFC Support hot device unplug in amdgpu

[Daniel Vetter]

On Mon, May 11, 2020 at 1:43 PM Lukas Wunner <lukas@wunner.de> wrote:
>
> On Mon, May 11, 2020 at 11:54:33AM +0200, Daniel Vetter wrote:
> > - One unfortunate thing with drm_dev_unplug is that the driver core is
> >   very opinionated and doesn't tell you whether it's a hotunplug or a
> >   driver unload. In the former case trying to shut down hw just wastes
> >   time (and might hit driver bugs), in the latter case driver engineers
> >   very much expect everything to be shut down.
>
> You can get that information at the PCI bus level with
> pci_dev_is_disconnected().  The flag queried by this function is set
> upon hot removal.  Be aware however that the device is guaranteed to
> be unreachable if the function returns true, but the converse is NOT
> guaranteed, i.e. the function may return false even though the device
> has just gone away.
>
> Those somewhat difficult semantics are one of the reasons why some
> people are skeptical of the function's merits (notably Greg KH).
> See this LWN article for more information:
>
> https://lwn.net/Articles/767885/
> (scroll down to the "Surprise removal" section)
>
> I've suggested to Greg a few years back that we should have a flag
> at the device level to indicate whether it's gone, not just at the bus
> level.  That way the property could be expressed regardless of the bus
> used.  It would facilitate the feature you're missing, that the driver
> core tells you whether it's a surprise removal or not.  Unfortunately
> Greg rejected the idea.

Ok, so at least for pci devices you could do something like

if (pci_dev_is_disconnected())
    drm_dev_unplug();
else
    drm_dev_unregister();

In the ->remove callback and both users and developers should be
happy. I guess for other drivers like usb/spi just yanking the cable
for driver hacking is good enough - loss of power should also reset
the device :-)
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [PATCH 0/6] RFC Support hot device unplug in amdgpu

[Daniel Vetter]

On Mon, May 11, 2020 at 1:43 PM Lukas Wunner <lukas@wunner.de> wrote:
>
> On Mon, May 11, 2020 at 11:54:33AM +0200, Daniel Vetter wrote:
> > - One unfortunate thing with drm_dev_unplug is that the driver core is
> >   very opinionated and doesn't tell you whether it's a hotunplug or a
> >   driver unload. In the former case trying to shut down hw just wastes
> >   time (and might hit driver bugs), in the latter case driver engineers
> >   very much expect everything to be shut down.
>
> You can get that information at the PCI bus level with
> pci_dev_is_disconnected().  The flag queried by this function is set
> upon hot removal.  Be aware however that the device is guaranteed to
> be unreachable if the function returns true, but the converse is NOT
> guaranteed, i.e. the function may return false even though the device
> has just gone away.
>
> Those somewhat difficult semantics are one of the reasons why some
> people are skeptical of the function's merits (notably Greg KH).
> See this LWN article for more information:
>
> https://lwn.net/Articles/767885/
> (scroll down to the "Surprise removal" section)
>
> I've suggested to Greg a few years back that we should have a flag
> at the device level to indicate whether it's gone, not just at the bus
> level.  That way the property could be expressed regardless of the bus
> used.  It would facilitate the feature you're missing, that the driver
> core tells you whether it's a surprise removal or not.  Unfortunately
> Greg rejected the idea.

Ok, so at least for pci devices you could do something like

if (pci_dev_is_disconnected())
    drm_dev_unplug();
else
    drm_dev_unregister();

In the ->remove callback and both users and developers should be
happy. I guess for other drivers like usb/spi just yanking the cable
for driver hacking is good enough - loss of power should also reset
the device :-)
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH 0/6] RFC Support hot device unplug in amdgpu

[Lukas Wunner]

On Mon, May 11, 2020 at 02:21:57PM +0200, Daniel Vetter wrote:
> On Mon, May 11, 2020 at 1:43 PM Lukas Wunner <lukas@wunner.de> wrote:
> > On Mon, May 11, 2020 at 11:54:33AM +0200, Daniel Vetter wrote:
> > > - One unfortunate thing with drm_dev_unplug is that the driver core is
> > >   very opinionated and doesn't tell you whether it's a hotunplug or a
> > >   driver unload. In the former case trying to shut down hw just wastes
> > >   time (and might hit driver bugs), in the latter case driver engineers
> > >   very much expect everything to be shut down.
> > 
> > You can get that information at the PCI bus level with
> > pci_dev_is_disconnected().
> 
> Ok, so at least for pci devices you could do something like
> 
> if (pci_dev_is_disconnected())
>     drm_dev_unplug();
> else
>     drm_dev_unregister();
>
> In the ->remove callback and both users and developers should be
> happy.

Basically yes.  But if the driver is unbound e.g. via sysfs and the
device is hot-removed while it is being unbound, that approach fails.

So you'll need checks for pci_dev_is_disconnected() further below in
the call stack as well to avoid unpleasant side effects such as unduly
delaying unbinding or ending up in infinite loops when reading "all ones"
from PCI BARs, etc.

It may also be worth checking for pci_dev_is_disconnected() in ioctls
as well and directly returning -ENODEV, though of course that suffers
from the same race.  (The device may disappear after the check for
pci_dev_is_disconnected(), or it may have already disappeared but
pciehp hasn't updated the device's channel state yet.)

Thanks,

Lukas
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [PATCH 0/6] RFC Support hot device unplug in amdgpu

[Lukas Wunner]

On Mon, May 11, 2020 at 02:21:57PM +0200, Daniel Vetter wrote:
> On Mon, May 11, 2020 at 1:43 PM Lukas Wunner <lukas@wunner.de> wrote:
> > On Mon, May 11, 2020 at 11:54:33AM +0200, Daniel Vetter wrote:
> > > - One unfortunate thing with drm_dev_unplug is that the driver core is
> > >   very opinionated and doesn't tell you whether it's a hotunplug or a
> > >   driver unload. In the former case trying to shut down hw just wastes
> > >   time (and might hit driver bugs), in the latter case driver engineers
> > >   very much expect everything to be shut down.
> > 
> > You can get that information at the PCI bus level with
> > pci_dev_is_disconnected().
> 
> Ok, so at least for pci devices you could do something like
> 
> if (pci_dev_is_disconnected())
>     drm_dev_unplug();
> else
>     drm_dev_unregister();
>
> In the ->remove callback and both users and developers should be
> happy.

Basically yes.  But if the driver is unbound e.g. via sysfs and the
device is hot-removed while it is being unbound, that approach fails.

So you'll need checks for pci_dev_is_disconnected() further below in
the call stack as well to avoid unpleasant side effects such as unduly
delaying unbinding or ending up in infinite loops when reading "all ones"
from PCI BARs, etc.

It may also be worth checking for pci_dev_is_disconnected() in ioctls
as well and directly returning -ENODEV, though of course that suffers
from the same race.  (The device may disappear after the check for
pci_dev_is_disconnected(), or it may have already disappeared but
pciehp hasn't updated the device's channel state yet.)

Thanks,

Lukas
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH 0/6] RFC Support hot device unplug in amdgpu

[Daniel Vetter]

On Mon, May 11, 2020 at 4:08 PM Lukas Wunner <lukas@wunner.de> wrote:
>
> On Mon, May 11, 2020 at 02:21:57PM +0200, Daniel Vetter wrote:
> > On Mon, May 11, 2020 at 1:43 PM Lukas Wunner <lukas@wunner.de> wrote:
> > > On Mon, May 11, 2020 at 11:54:33AM +0200, Daniel Vetter wrote:
> > > > - One unfortunate thing with drm_dev_unplug is that the driver core is
> > > >   very opinionated and doesn't tell you whether it's a hotunplug or a
> > > >   driver unload. In the former case trying to shut down hw just wastes
> > > >   time (and might hit driver bugs), in the latter case driver engineers
> > > >   very much expect everything to be shut down.
> > >
> > > You can get that information at the PCI bus level with
> > > pci_dev_is_disconnected().
> >
> > Ok, so at least for pci devices you could do something like
> >
> > if (pci_dev_is_disconnected())
> >     drm_dev_unplug();
> > else
> >     drm_dev_unregister();
> >
> > In the ->remove callback and both users and developers should be
> > happy.
>
> Basically yes.  But if the driver is unbound e.g. via sysfs and the
> device is hot-removed while it is being unbound, that approach fails.
>
> So you'll need checks for pci_dev_is_disconnected() further below in
> the call stack as well to avoid unpleasant side effects such as unduly
> delaying unbinding or ending up in infinite loops when reading "all ones"
> from PCI BARs, etc.
>
> It may also be worth checking for pci_dev_is_disconnected() in ioctls
> as well and directly returning -ENODEV, though of course that suffers
> from the same race.  (The device may disappear after the check for
> pci_dev_is_disconnected(), or it may have already disappeared but
> pciehp hasn't updated the device's channel state yet.)

I guess we could do a drm_pci_dev_enter which combines drm_dev_enter +
pci_dev_is_connected. Not perfect, but well then the only real
solution is just unconditionaly drm_dev_unplug in ->remove. I think if
we do an additional developer_mode module parameter, and if that's not
explicitly set, ignore the pci_dev_is_disconnected and just always
call drm_dev_unplug() that would be about as good as it gets.
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [PATCH 0/6] RFC Support hot device unplug in amdgpu

[Daniel Vetter]

On Mon, May 11, 2020 at 4:08 PM Lukas Wunner <lukas@wunner.de> wrote:
>
> On Mon, May 11, 2020 at 02:21:57PM +0200, Daniel Vetter wrote:
> > On Mon, May 11, 2020 at 1:43 PM Lukas Wunner <lukas@wunner.de> wrote:
> > > On Mon, May 11, 2020 at 11:54:33AM +0200, Daniel Vetter wrote:
> > > > - One unfortunate thing with drm_dev_unplug is that the driver core is
> > > >   very opinionated and doesn't tell you whether it's a hotunplug or a
> > > >   driver unload. In the former case trying to shut down hw just wastes
> > > >   time (and might hit driver bugs), in the latter case driver engineers
> > > >   very much expect everything to be shut down.
> > >
> > > You can get that information at the PCI bus level with
> > > pci_dev_is_disconnected().
> >
> > Ok, so at least for pci devices you could do something like
> >
> > if (pci_dev_is_disconnected())
> >     drm_dev_unplug();
> > else
> >     drm_dev_unregister();
> >
> > In the ->remove callback and both users and developers should be
> > happy.
>
> Basically yes.  But if the driver is unbound e.g. via sysfs and the
> device is hot-removed while it is being unbound, that approach fails.
>
> So you'll need checks for pci_dev_is_disconnected() further below in
> the call stack as well to avoid unpleasant side effects such as unduly
> delaying unbinding or ending up in infinite loops when reading "all ones"
> from PCI BARs, etc.
>
> It may also be worth checking for pci_dev_is_disconnected() in ioctls
> as well and directly returning -ENODEV, though of course that suffers
> from the same race.  (The device may disappear after the check for
> pci_dev_is_disconnected(), or it may have already disappeared but
> pciehp hasn't updated the device's channel state yet.)

I guess we could do a drm_pci_dev_enter which combines drm_dev_enter +
pci_dev_is_connected. Not perfect, but well then the only real
solution is just unconditionaly drm_dev_unplug in ->remove. I think if
we do an additional developer_mode module parameter, and if that's not
explicitly set, ignore the pci_dev_is_disconnected and just always
call drm_dev_unplug() that would be about as good as it gets.
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH 0/6] RFC Support hot device unplug in amdgpu

[Andrey Grodzovsky]

On 5/11/20 5:54 AM, Daniel Vetter wrote:
> On Sat, May 09, 2020 at 02:51:44PM -0400, Andrey Grodzovsky wrote:
>> This RFC is a more of a proof of concept then a fully working solution as there are a few unresolved issues we are hopping to get advise on from people on the mailing list.
>> Until now extracting a card either by physical extraction (e.g. eGPU with thunderbold connection or by emulation through syfs -> /sys/bus/pci/devices/device_id/remove)
>> would cause random crashes in user apps. The random crashes in apps were mostly due to the app having mapped a device backed BO into it's adress space was still
>> trying to access the BO while the backing device was gone.
>> To answer this first problem Christian suggested to fix the handling of mapped memory in the clients when the device goes away by forcibly unmap all buffers
>> the user processes has by clearing their respective VMAs mapping the device BOs. Then when the VMAs try to fill in the page tables again we check in the fault handler
>> if the device is removed and if so, return an error. This will generate a SIGBUS to the application which can then cleanly terminate.
>> This indeed was done but this in turn created a problem of kernel OOPs were the OOPSes were due to the fact that while the app was terminating because of the SIGBUS
>> it would trigger use after free in the driver by calling to accesses device structures that were already released from the pci remove sequence.
>> This we handled by introducing a 'flush' seqence during device removal were we wait for drm file reference to drop to 0 meaning all user clients directly using this device terminated.
>> With this I was able to cleanly emulate device unplug with X and glxgears running and later emulate device plug back and restart of X and glxgears.
>>
>> But this use case is only partial and as I see it all the use cases are as follwing and the questions it raises.
>>
>> 1) Application accesses a BO by opening drm file
>> 	1.1) BO is mapped into applications address space (BO is CPU visible) - this one we have a solution for by invaldating BO's CPU mapping casuing SIGBUS
>> 	     and termination and waiting for drm file refcound to drop to 0 before releasing the device
>> 	1.2) BO is not mapped into applcation address space (BO is CPU invisible) - no solution yet because how we force the application to terminate in this case ?
>>
>> 2) Application accesses a BO by importing a DMA-BUF
>> 	2.1)  BO is mapped into applications address space (BO is CPU visible) - solution is same as 1.1 but instead of waiting for drm file release we wait for the
>> 	      imported dma-buf's file release
>> 	2.2)  BO is not mapped into applcation address space (BO is CPU invisible) - our solution is to invalidate GPUVM page tables and destroy backing storage for
>>                all exported BOs which will in turn casue VM faults in the importing device and then when the importing driver will try to re-attach the imported BO to
>> 	      update mappings we return -ENODEV in the import hook which hopeffuly will cause the user app to terminate.
>>
>> 3) Applcation opens a drm file or imports a dma-bud and holds a reference but never access any BO or does access but never more after device was unplug - how would we
>>     force this applcation to termiante before proceeding with device removal code ? Otherwise the wait in pci remove just hangs for ever.
>>
>> The attached patches adress 1.1, 2.1 and 2.2, for now only 1.1 fully tested and I am still testing the others but I will be happy for any advise on all the
>> described use cases and maybe some alternative and better (more generic) approach to this like maybe obtaining PIDs of relevant processes through some revere
>> mapping from device file and exported dma-buf files and send them SIGKILL - would this make more sense or any other method ?
>>
>> Patches 1-3 address 1.1
>> Patch 4 addresses 2.1
>> Pathces 5-6 address 2.2
>>
>> Reference: https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.freedesktop.org%2Fdrm%2Famd%2F-%2Fissues%2F1081&data=02%7C01%7Candrey.grodzovsky%40amd.com%7Cf6eec90e9da144cb772a08d7f5921ec2%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637247880251844517&sdata=QBGIbm1KLysglvRAvoiek8jBcNLE%2B4J7gVGDAbZD5Jw%3D&reserved=0
> So we've been working on this problem for a few years already (but it's
> still not solved), I think you could have saved yourselfs some typing.
>
> Bunch of things:
> - we can't wait for userspace in the hotunplug handlers, that might never
>    happen. The correct way is to untangle the lifetime of your hw driver
>    for a specific struct pci_device from the drm_device lifetime.
>    Infrastructure is all there now, see drm_dev_get/put, drm_dev_unplug and
>    drm_dev_enter/exit.

this

To be sure I understood you - do you mean that we should 
disable/shutdown any HW related stuff such as interrupts disable, any 
shutdown related device registers programming and io regions unmapping 
during pci remove sequence (in our case amdgpu_pci_remove) while keeping 
all the drm/amdgpu structures around in memory until drm_dev_put 
refocunt drop to 0 and &drm_driver.release is called thus avoiding any 
user after free oopses when last user reference is dropped ?

Is there any point in doing any HW programming to shutdown device if 
device is already removed anyway (i assume that if driver hook for pci 
remove  is called and it's a physical remove the device is already gone, 
no ?)

What happens if drm_dev_put doesn't drop to 0 before the device is 
plugged back into the system ? In this case i have duplicates of all 
device structures in the system. Do you expect this to be not a problem 
or if it is it's up to me to resolve i guess ?


> A bunch of usb/spi drivers use this 100% correctly
>    now, so there's examples. Plus kerneldoc explains stuff.


Would you say tiny drm drivers are a good example ?


>
> - for a big driver like amdgpu doing this split up is going to be
>    horrendously complex. I know, we've done it for i915, at least
>    partially.


Can you point me to relevant code/commits for i915 ?


> I strongly recommend that you're using devm_ for managing hw
>    related resources (iomap, irq, ...) as much as possible.


 From what i saw, in DRM devres implementation  amounts to using 
devm_drm_dev_init/devm_drm_dev_init_release - is that what you mean ? If 
so i see that devm_drm_dev_init_release just calls drm_dev_put, 
drm_dev_unplug ends up calling devm_drm_dev_init_release through the 
devres infrastructure - We already call drm_dev_unplug in 
amdgpu_pci_remove, we also directly call drm_dev_put there so i am not 
clear what's the added value of using devm here ?


>
>    For drm_device resources (mostly structures and everything related to
>    that) we've just merged the drmm_ managed resources framework. There's
>    some more work to be done there for various kms objects, but you can at
>    least somewhat avoid tedious handrolling for everything internal
>    already.


I can't find drmm in the code, can you point me please ?


>
>    Don't ever use devm_kzalloc and friends, I've looked at hundreds of uses
>    of this in drm, they're all wrong.
>
> - dma-buf is hilarious (and atm unfixed), dma-fence is even worse. In
>    theory they're already refcounted and all and so should work, in
>    practice I think we need to refcount the underlying drm_device with
>    drm_dev_get/put to avoid the worst fall-out.
>
> - One unfortunate thing with drm_dev_unplug is that the driver core is
>    very opinionated and doesn't tell you whether it's a hotunplug or a
>    driver unload. In the former case trying to shut down hw just wastes
>    time (and might hit driver bugs), in the latter case driver engineers
>    very much expect everything to be shut down.
>
>    Right now you can only have one or the other, so this needs a module
>    option hack or similar (default to the correct hotunplug behaviour for
>    users).
>
> - SIGBUS is better than crashing the kernel, but it's not even close for
>    users. They still lose everything because everything crashes because in
>    my experience, in practice, no one ever handles errors. There's a few
>    more things on top:
>
>    - sighandlers are global, which means only the app can use it. You can't
>      use it in e.g. mesa. They're also not composable, so if you have on
>      sighandler for gpu1 and a 2nd one for gpu2 (could be different vendor)
>      it's all sadness. Hence "usersapce will handle SIGBUS" wont work.
>
>    - worse, neither vk nor gl (to my knowledge) have a concept of events
>      for when the gpu died. The only stuff you have is things like
>      arb_robustness which says a) everything continues as if nothing
>      happened b) there's a function where you can ask whether your gl
>      context and all the textures/buffers are toast.
>
>      I think that's about the only hotunplug application model we can
>      realistically expect applications to support. That means _all_ errors
>      need to be silently eaten by either mesa or the kernel. On i915 the
>      model (also for terminally wedged gpu hangs) is that all ioctl keep
>      working, mmaps keep working, and execbuf gives you an -EIO (which mesa
>      eats silently iirc for arb_robustness).
>
>    Conclusion is that SIGBUS is imo a no-go, and the only option we have is
>    that a) mmaps fully keep working, doable for shmem or b) we put some
>    fake memory in there (for vram or whatever), maybe even only a single
>    page for all fake memory.
>
> - you probably want arb_robustness and similar stuff in userspace as a
>    first step.
>
> tldr;
> - refcounting, not waiting for userspace
> - nothing can fail because userspace wont handle it


For nothing can fail i see in tiny drm driver examples (e.g. 
ili9225_pipe_enable) that for any function which is about to do HW 
programming they check for drm_dev_enter and silently return if device 
is not present - is that what you mean, that I should pepper all of 
amdgpu code such that any function that ends up doing some HW 
programming be guarded with drm_dev_enter/exit silently returning in 
case of device is gone ?

Thanks a lot for your detailed response.

Andrey


>
> That's at least my take on this mess, and what we've been pushing for over
> the past few years. For kms-only drm_driver we should have achieved that
> by now (plus/minus maybe some issues for dma-buf/fences, but kms-only
> dma-buf/fences are simple enough that maybe we don't go boom yet).
>
> For big gpus with rendering I think best next step would be to type up a
> reasonable Gran Plan (into Documentation/gpu/todo.rst) with all the issues
> and likely solutions. And then bikeshed that, since the above is just my
> take on all this.
>
> Cheers, Daniel
>
>> Andrey Grodzovsky (6):
>>    drm/ttm: Add unampping of the entire device address space
>>    drm/amdgpu: Force unmap all user VMAs on device removal.
>>    drm/amdgpu: Wait for all user clients
>>    drm/amdgpu: Wait for all clients importing out dma-bufs.
>>    drm/ttm: Add destroy flag in TTM BO eviction interface
>>    drm/amdgpu: Use TTM MMs destroy interface
>>
>>   drivers/gpu/drm/amd/amdgpu/amdgpu.h         |  3 ++
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c |  2 +-
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_device.c  |  7 +++-
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c | 27 ++++++++++++-
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c     | 22 ++++++++--
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_job.c     |  9 +++++
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c     |  4 ++
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_object.c  | 17 +++++++-
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_object.h  |  1 +
>>   drivers/gpu/drm/nouveau/nouveau_drm.c       |  2 +-
>>   drivers/gpu/drm/qxl/qxl_object.c            |  4 +-
>>   drivers/gpu/drm/radeon/radeon_object.c      |  2 +-
>>   drivers/gpu/drm/ttm/ttm_bo.c                | 63 +++++++++++++++++++++--------
>>   drivers/gpu/drm/vmwgfx/vmwgfx_drv.c         |  6 +--
>>   include/drm/ttm/ttm_bo_api.h                |  2 +-
>>   include/drm/ttm/ttm_bo_driver.h             |  2 +
>>   16 files changed, 139 insertions(+), 34 deletions(-)
>>
>> -- 
>> 2.7.4
>>
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [PATCH 0/6] RFC Support hot device unplug in amdgpu

[Andrey Grodzovsky]

On 5/11/20 5:54 AM, Daniel Vetter wrote:
> On Sat, May 09, 2020 at 02:51:44PM -0400, Andrey Grodzovsky wrote:
>> This RFC is a more of a proof of concept then a fully working solution as there are a few unresolved issues we are hopping to get advise on from people on the mailing list.
>> Until now extracting a card either by physical extraction (e.g. eGPU with thunderbold connection or by emulation through syfs -> /sys/bus/pci/devices/device_id/remove)
>> would cause random crashes in user apps. The random crashes in apps were mostly due to the app having mapped a device backed BO into it's adress space was still
>> trying to access the BO while the backing device was gone.
>> To answer this first problem Christian suggested to fix the handling of mapped memory in the clients when the device goes away by forcibly unmap all buffers
>> the user processes has by clearing their respective VMAs mapping the device BOs. Then when the VMAs try to fill in the page tables again we check in the fault handler
>> if the device is removed and if so, return an error. This will generate a SIGBUS to the application which can then cleanly terminate.
>> This indeed was done but this in turn created a problem of kernel OOPs were the OOPSes were due to the fact that while the app was terminating because of the SIGBUS
>> it would trigger use after free in the driver by calling to accesses device structures that were already released from the pci remove sequence.
>> This we handled by introducing a 'flush' seqence during device removal were we wait for drm file reference to drop to 0 meaning all user clients directly using this device terminated.
>> With this I was able to cleanly emulate device unplug with X and glxgears running and later emulate device plug back and restart of X and glxgears.
>>
>> But this use case is only partial and as I see it all the use cases are as follwing and the questions it raises.
>>
>> 1) Application accesses a BO by opening drm file
>> 	1.1) BO is mapped into applications address space (BO is CPU visible) - this one we have a solution for by invaldating BO's CPU mapping casuing SIGBUS
>> 	     and termination and waiting for drm file refcound to drop to 0 before releasing the device
>> 	1.2) BO is not mapped into applcation address space (BO is CPU invisible) - no solution yet because how we force the application to terminate in this case ?
>>
>> 2) Application accesses a BO by importing a DMA-BUF
>> 	2.1)  BO is mapped into applications address space (BO is CPU visible) - solution is same as 1.1 but instead of waiting for drm file release we wait for the
>> 	      imported dma-buf's file release
>> 	2.2)  BO is not mapped into applcation address space (BO is CPU invisible) - our solution is to invalidate GPUVM page tables and destroy backing storage for
>>                all exported BOs which will in turn casue VM faults in the importing device and then when the importing driver will try to re-attach the imported BO to
>> 	      update mappings we return -ENODEV in the import hook which hopeffuly will cause the user app to terminate.
>>
>> 3) Applcation opens a drm file or imports a dma-bud and holds a reference but never access any BO or does access but never more after device was unplug - how would we
>>     force this applcation to termiante before proceeding with device removal code ? Otherwise the wait in pci remove just hangs for ever.
>>
>> The attached patches adress 1.1, 2.1 and 2.2, for now only 1.1 fully tested and I am still testing the others but I will be happy for any advise on all the
>> described use cases and maybe some alternative and better (more generic) approach to this like maybe obtaining PIDs of relevant processes through some revere
>> mapping from device file and exported dma-buf files and send them SIGKILL - would this make more sense or any other method ?
>>
>> Patches 1-3 address 1.1
>> Patch 4 addresses 2.1
>> Pathces 5-6 address 2.2
>>
>> Reference: https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.freedesktop.org%2Fdrm%2Famd%2F-%2Fissues%2F1081&data=02%7C01%7Candrey.grodzovsky%40amd.com%7Cf6eec90e9da144cb772a08d7f5921ec2%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637247880251844517&sdata=QBGIbm1KLysglvRAvoiek8jBcNLE%2B4J7gVGDAbZD5Jw%3D&reserved=0
> So we've been working on this problem for a few years already (but it's
> still not solved), I think you could have saved yourselfs some typing.
>
> Bunch of things:
> - we can't wait for userspace in the hotunplug handlers, that might never
>    happen. The correct way is to untangle the lifetime of your hw driver
>    for a specific struct pci_device from the drm_device lifetime.
>    Infrastructure is all there now, see drm_dev_get/put, drm_dev_unplug and
>    drm_dev_enter/exit.

this

To be sure I understood you - do you mean that we should 
disable/shutdown any HW related stuff such as interrupts disable, any 
shutdown related device registers programming and io regions unmapping 
during pci remove sequence (in our case amdgpu_pci_remove) while keeping 
all the drm/amdgpu structures around in memory until drm_dev_put 
refocunt drop to 0 and &drm_driver.release is called thus avoiding any 
user after free oopses when last user reference is dropped ?

Is there any point in doing any HW programming to shutdown device if 
device is already removed anyway (i assume that if driver hook for pci 
remove  is called and it's a physical remove the device is already gone, 
no ?)

What happens if drm_dev_put doesn't drop to 0 before the device is 
plugged back into the system ? In this case i have duplicates of all 
device structures in the system. Do you expect this to be not a problem 
or if it is it's up to me to resolve i guess ?


> A bunch of usb/spi drivers use this 100% correctly
>    now, so there's examples. Plus kerneldoc explains stuff.


Would you say tiny drm drivers are a good example ?


>
> - for a big driver like amdgpu doing this split up is going to be
>    horrendously complex. I know, we've done it for i915, at least
>    partially.


Can you point me to relevant code/commits for i915 ?


> I strongly recommend that you're using devm_ for managing hw
>    related resources (iomap, irq, ...) as much as possible.


 From what i saw, in DRM devres implementation  amounts to using 
devm_drm_dev_init/devm_drm_dev_init_release - is that what you mean ? If 
so i see that devm_drm_dev_init_release just calls drm_dev_put, 
drm_dev_unplug ends up calling devm_drm_dev_init_release through the 
devres infrastructure - We already call drm_dev_unplug in 
amdgpu_pci_remove, we also directly call drm_dev_put there so i am not 
clear what's the added value of using devm here ?


>
>    For drm_device resources (mostly structures and everything related to
>    that) we've just merged the drmm_ managed resources framework. There's
>    some more work to be done there for various kms objects, but you can at
>    least somewhat avoid tedious handrolling for everything internal
>    already.


I can't find drmm in the code, can you point me please ?


>
>    Don't ever use devm_kzalloc and friends, I've looked at hundreds of uses
>    of this in drm, they're all wrong.
>
> - dma-buf is hilarious (and atm unfixed), dma-fence is even worse. In
>    theory they're already refcounted and all and so should work, in
>    practice I think we need to refcount the underlying drm_device with
>    drm_dev_get/put to avoid the worst fall-out.
>
> - One unfortunate thing with drm_dev_unplug is that the driver core is
>    very opinionated and doesn't tell you whether it's a hotunplug or a
>    driver unload. In the former case trying to shut down hw just wastes
>    time (and might hit driver bugs), in the latter case driver engineers
>    very much expect everything to be shut down.
>
>    Right now you can only have one or the other, so this needs a module
>    option hack or similar (default to the correct hotunplug behaviour for
>    users).
>
> - SIGBUS is better than crashing the kernel, but it's not even close for
>    users. They still lose everything because everything crashes because in
>    my experience, in practice, no one ever handles errors. There's a few
>    more things on top:
>
>    - sighandlers are global, which means only the app can use it. You can't
>      use it in e.g. mesa. They're also not composable, so if you have on
>      sighandler for gpu1 and a 2nd one for gpu2 (could be different vendor)
>      it's all sadness. Hence "usersapce will handle SIGBUS" wont work.
>
>    - worse, neither vk nor gl (to my knowledge) have a concept of events
>      for when the gpu died. The only stuff you have is things like
>      arb_robustness which says a) everything continues as if nothing
>      happened b) there's a function where you can ask whether your gl
>      context and all the textures/buffers are toast.
>
>      I think that's about the only hotunplug application model we can
>      realistically expect applications to support. That means _all_ errors
>      need to be silently eaten by either mesa or the kernel. On i915 the
>      model (also for terminally wedged gpu hangs) is that all ioctl keep
>      working, mmaps keep working, and execbuf gives you an -EIO (which mesa
>      eats silently iirc for arb_robustness).
>
>    Conclusion is that SIGBUS is imo a no-go, and the only option we have is
>    that a) mmaps fully keep working, doable for shmem or b) we put some
>    fake memory in there (for vram or whatever), maybe even only a single
>    page for all fake memory.
>
> - you probably want arb_robustness and similar stuff in userspace as a
>    first step.
>
> tldr;
> - refcounting, not waiting for userspace
> - nothing can fail because userspace wont handle it


For nothing can fail i see in tiny drm driver examples (e.g. 
ili9225_pipe_enable) that for any function which is about to do HW 
programming they check for drm_dev_enter and silently return if device 
is not present - is that what you mean, that I should pepper all of 
amdgpu code such that any function that ends up doing some HW 
programming be guarded with drm_dev_enter/exit silently returning in 
case of device is gone ?

Thanks a lot for your detailed response.

Andrey


>
> That's at least my take on this mess, and what we've been pushing for over
> the past few years. For kms-only drm_driver we should have achieved that
> by now (plus/minus maybe some issues for dma-buf/fences, but kms-only
> dma-buf/fences are simple enough that maybe we don't go boom yet).
>
> For big gpus with rendering I think best next step would be to type up a
> reasonable Gran Plan (into Documentation/gpu/todo.rst) with all the issues
> and likely solutions. And then bikeshed that, since the above is just my
> take on all this.
>
> Cheers, Daniel
>
>> Andrey Grodzovsky (6):
>>    drm/ttm: Add unampping of the entire device address space
>>    drm/amdgpu: Force unmap all user VMAs on device removal.
>>    drm/amdgpu: Wait for all user clients
>>    drm/amdgpu: Wait for all clients importing out dma-bufs.
>>    drm/ttm: Add destroy flag in TTM BO eviction interface
>>    drm/amdgpu: Use TTM MMs destroy interface
>>
>>   drivers/gpu/drm/amd/amdgpu/amdgpu.h         |  3 ++
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c |  2 +-
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_device.c  |  7 +++-
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c | 27 ++++++++++++-
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c     | 22 ++++++++--
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_job.c     |  9 +++++
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c     |  4 ++
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_object.c  | 17 +++++++-
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_object.h  |  1 +
>>   drivers/gpu/drm/nouveau/nouveau_drm.c       |  2 +-
>>   drivers/gpu/drm/qxl/qxl_object.c            |  4 +-
>>   drivers/gpu/drm/radeon/radeon_object.c      |  2 +-
>>   drivers/gpu/drm/ttm/ttm_bo.c                | 63 +++++++++++++++++++++--------
>>   drivers/gpu/drm/vmwgfx/vmwgfx_drv.c         |  6 +--
>>   include/drm/ttm/ttm_bo_api.h                |  2 +-
>>   include/drm/ttm/ttm_bo_driver.h             |  2 +
>>   16 files changed, 139 insertions(+), 34 deletions(-)
>>
>> -- 
>> 2.7.4
>>
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH 0/6] RFC Support hot device unplug in amdgpu

[Daniel Vetter]

On Wed, May 13, 2020 at 10:32:56AM -0400, Andrey Grodzovsky wrote:
> 
> On 5/11/20 5:54 AM, Daniel Vetter wrote:
> > On Sat, May 09, 2020 at 02:51:44PM -0400, Andrey Grodzovsky wrote:
> > > This RFC is a more of a proof of concept then a fully working solution as there are a few unresolved issues we are hopping to get advise on from people on the mailing list.
> > > Until now extracting a card either by physical extraction (e.g. eGPU with thunderbold connection or by emulation through syfs -> /sys/bus/pci/devices/device_id/remove)
> > > would cause random crashes in user apps. The random crashes in apps were mostly due to the app having mapped a device backed BO into it's adress space was still
> > > trying to access the BO while the backing device was gone.
> > > To answer this first problem Christian suggested to fix the handling of mapped memory in the clients when the device goes away by forcibly unmap all buffers
> > > the user processes has by clearing their respective VMAs mapping the device BOs. Then when the VMAs try to fill in the page tables again we check in the fault handler
> > > if the device is removed and if so, return an error. This will generate a SIGBUS to the application which can then cleanly terminate.
> > > This indeed was done but this in turn created a problem of kernel OOPs were the OOPSes were due to the fact that while the app was terminating because of the SIGBUS
> > > it would trigger use after free in the driver by calling to accesses device structures that were already released from the pci remove sequence.
> > > This we handled by introducing a 'flush' seqence during device removal were we wait for drm file reference to drop to 0 meaning all user clients directly using this device terminated.
> > > With this I was able to cleanly emulate device unplug with X and glxgears running and later emulate device plug back and restart of X and glxgears.
> > > 
> > > But this use case is only partial and as I see it all the use cases are as follwing and the questions it raises.
> > > 
> > > 1) Application accesses a BO by opening drm file
> > > 	1.1) BO is mapped into applications address space (BO is CPU visible) - this one we have a solution for by invaldating BO's CPU mapping casuing SIGBUS
> > > 	     and termination and waiting for drm file refcound to drop to 0 before releasing the device
> > > 	1.2) BO is not mapped into applcation address space (BO is CPU invisible) - no solution yet because how we force the application to terminate in this case ?
> > > 
> > > 2) Application accesses a BO by importing a DMA-BUF
> > > 	2.1)  BO is mapped into applications address space (BO is CPU visible) - solution is same as 1.1 but instead of waiting for drm file release we wait for the
> > > 	      imported dma-buf's file release
> > > 	2.2)  BO is not mapped into applcation address space (BO is CPU invisible) - our solution is to invalidate GPUVM page tables and destroy backing storage for
> > >                all exported BOs which will in turn casue VM faults in the importing device and then when the importing driver will try to re-attach the imported BO to
> > > 	      update mappings we return -ENODEV in the import hook which hopeffuly will cause the user app to terminate.
> > > 
> > > 3) Applcation opens a drm file or imports a dma-bud and holds a reference but never access any BO or does access but never more after device was unplug - how would we
> > >     force this applcation to termiante before proceeding with device removal code ? Otherwise the wait in pci remove just hangs for ever.
> > > 
> > > The attached patches adress 1.1, 2.1 and 2.2, for now only 1.1 fully tested and I am still testing the others but I will be happy for any advise on all the
> > > described use cases and maybe some alternative and better (more generic) approach to this like maybe obtaining PIDs of relevant processes through some revere
> > > mapping from device file and exported dma-buf files and send them SIGKILL - would this make more sense or any other method ?
> > > 
> > > Patches 1-3 address 1.1
> > > Patch 4 addresses 2.1
> > > Pathces 5-6 address 2.2
> > > 
> > > Reference: https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.freedesktop.org%2Fdrm%2Famd%2F-%2Fissues%2F1081&data=02%7C01%7Candrey.grodzovsky%40amd.com%7Cf6eec90e9da144cb772a08d7f5921ec2%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637247880251844517&sdata=QBGIbm1KLysglvRAvoiek8jBcNLE%2B4J7gVGDAbZD5Jw%3D&reserved=0
> > So we've been working on this problem for a few years already (but it's
> > still not solved), I think you could have saved yourselfs some typing.
> > 
> > Bunch of things:
> > - we can't wait for userspace in the hotunplug handlers, that might never
> >    happen. The correct way is to untangle the lifetime of your hw driver
> >    for a specific struct pci_device from the drm_device lifetime.
> >    Infrastructure is all there now, see drm_dev_get/put, drm_dev_unplug and
> >    drm_dev_enter/exit.
> 
> this
> 
> To be sure I understood you - do you mean that we should disable/shutdown
> any HW related stuff such as interrupts disable, any shutdown related device
> registers programming and io regions unmapping during pci remove sequence
> (in our case amdgpu_pci_remove) while keeping all the drm/amdgpu structures
> around in memory until drm_dev_put refocunt drop to 0 and
> &drm_driver.release is called thus avoiding any user after free oopses when
> last user reference is dropped ?

Yes.

> Is there any point in doing any HW programming to shutdown device if device
> is already removed anyway (i assume that if driver hook for pci remove  is
> called and it's a physical remove the device is already gone, no ?)

No, all that does is result in bus timeouts, which take forever. Plus
increased chances that the driver gets confused about the values it reads
(for pci all you get is 0xffffffff, no error value, usb is a lot better
here because it's explicit packets and streams where you can get explicit
errors and bail out).

The trouble is that developers still expect you to shut down hw when they
unload the driver for testing, but I think we've discussed a reasonable
solution for that for pci drivers somewhere in this thread.

> What happens if drm_dev_put doesn't drop to 0 before the device is plugged
> back into the system ? In this case i have duplicates of all device
> structures in the system. Do you expect this to be not a problem or if it is
> it's up to me to resolve i guess ?

You get another one. Same way you get duplicates if there's actually 2
devices plugged in, so if your driver supports multiple gpus already you
should be fine.

We should also not hang onto chardev minor numbers, so I think you should
be getting the same minor number again. But not 100% sure, maybe something
we might need to fix ...

> > A bunch of usb/spi drivers use this 100% correctly
> >    now, so there's examples. Plus kerneldoc explains stuff.
> 
> 
> Would you say tiny drm drivers are a good example ?

Yup. But also, they're tiny, so lots more complexity in amdgpu that they
don't even cover. But for the basic flow of using drmm and devm and the
functions I mentioned above, they should be the most bug-free drivers we
have.

> > - for a big driver like amdgpu doing this split up is going to be
> >    horrendously complex. I know, we've done it for i915, at least
> >    partially.
> 
> 
> Can you point me to relevant code/commits for i915 ?

Anything touching the i915 drm_driver.release function. Or any function
called from there.

> > I strongly recommend that you're using devm_ for managing hw
> >    related resources (iomap, irq, ...) as much as possible.
> 
> 
> From what i saw, in DRM devres implementation  amounts to using
> devm_drm_dev_init/devm_drm_dev_init_release - is that what you mean ? If so
> i see that devm_drm_dev_init_release just calls drm_dev_put, drm_dev_unplug
> ends up calling devm_drm_dev_init_release through the devres infrastructure
> - We already call drm_dev_unplug in amdgpu_pci_remove, we also directly call
> drm_dev_put there so i am not clear what's the added value of using devm
> here ?

There's a lot more to devres, but yes that's the drm_device one. For the
full list of what can all be managed with devres see

https://dri.freedesktop.org/docs/drm/driver-api/driver-model/devres.html

There's lots of example usage in drm, especially tiny drivers and anything
that runs on arm. Only caveat is that any usage of devm_kzalloc is buggy
(at least in drm, as far as I've checked them).

> >    For drm_device resources (mostly structures and everything related to
> >    that) we've just merged the drmm_ managed resources framework. There's
> >    some more work to be done there for various kms objects, but you can at
> >    least somewhat avoid tedious handrolling for everything internal
> >    already.
> 
> 
> I can't find drmm in the code, can you point me please ?

drm_managed.c, you need latest drm-next I think. Or linux-next. Docs here:

https://dri.freedesktop.org/docs/drm/gpu/drm-internals.html#managed-resources

> 
> > 
> >    Don't ever use devm_kzalloc and friends, I've looked at hundreds of uses
> >    of this in drm, they're all wrong.
> > 
> > - dma-buf is hilarious (and atm unfixed), dma-fence is even worse. In
> >    theory they're already refcounted and all and so should work, in
> >    practice I think we need to refcount the underlying drm_device with
> >    drm_dev_get/put to avoid the worst fall-out.
> > 
> > - One unfortunate thing with drm_dev_unplug is that the driver core is
> >    very opinionated and doesn't tell you whether it's a hotunplug or a
> >    driver unload. In the former case trying to shut down hw just wastes
> >    time (and might hit driver bugs), in the latter case driver engineers
> >    very much expect everything to be shut down.
> > 
> >    Right now you can only have one or the other, so this needs a module
> >    option hack or similar (default to the correct hotunplug behaviour for
> >    users).
> > 
> > - SIGBUS is better than crashing the kernel, but it's not even close for
> >    users. They still lose everything because everything crashes because in
> >    my experience, in practice, no one ever handles errors. There's a few
> >    more things on top:
> > 
> >    - sighandlers are global, which means only the app can use it. You can't
> >      use it in e.g. mesa. They're also not composable, so if you have on
> >      sighandler for gpu1 and a 2nd one for gpu2 (could be different vendor)
> >      it's all sadness. Hence "usersapce will handle SIGBUS" wont work.
> > 
> >    - worse, neither vk nor gl (to my knowledge) have a concept of events
> >      for when the gpu died. The only stuff you have is things like
> >      arb_robustness which says a) everything continues as if nothing
> >      happened b) there's a function where you can ask whether your gl
> >      context and all the textures/buffers are toast.
> > 
> >      I think that's about the only hotunplug application model we can
> >      realistically expect applications to support. That means _all_ errors
> >      need to be silently eaten by either mesa or the kernel. On i915 the
> >      model (also for terminally wedged gpu hangs) is that all ioctl keep
> >      working, mmaps keep working, and execbuf gives you an -EIO (which mesa
> >      eats silently iirc for arb_robustness).
> > 
> >    Conclusion is that SIGBUS is imo a no-go, and the only option we have is
> >    that a) mmaps fully keep working, doable for shmem or b) we put some
> >    fake memory in there (for vram or whatever), maybe even only a single
> >    page for all fake memory.
> > 
> > - you probably want arb_robustness and similar stuff in userspace as a
> >    first step.
> > 
> > tldr;
> > - refcounting, not waiting for userspace
> > - nothing can fail because userspace wont handle it
> 
> 
> For nothing can fail i see in tiny drm driver examples (e.g.
> ili9225_pipe_enable) that for any function which is about to do HW
> programming they check for drm_dev_enter and silently return if device is
> not present - is that what you mean, that I should pepper all of amdgpu code
> such that any function that ends up doing some HW programming be guarded
> with drm_dev_enter/exit silently returning in case of device is gone ?

Yup.

Note taht e.g. usb because it's a packet/stream bus gives you explicit
errors, so those drivers don't need the drm_dev_enter/exit. But for pci we
need them to make sure we're wasting time when the hw is gone on
everything first timing out.

> Thanks a lot for your detailed response.

np.

Cheers, Daniel

> 
> Andrey
> 
> 
> > 
> > That's at least my take on this mess, and what we've been pushing for over
> > the past few years. For kms-only drm_driver we should have achieved that
> > by now (plus/minus maybe some issues for dma-buf/fences, but kms-only
> > dma-buf/fences are simple enough that maybe we don't go boom yet).
> > 
> > For big gpus with rendering I think best next step would be to type up a
> > reasonable Gran Plan (into Documentation/gpu/todo.rst) with all the issues
> > and likely solutions. And then bikeshed that, since the above is just my
> > take on all this.
> > 
> > Cheers, Daniel
> > 
> > > Andrey Grodzovsky (6):
> > >    drm/ttm: Add unampping of the entire device address space
> > >    drm/amdgpu: Force unmap all user VMAs on device removal.
> > >    drm/amdgpu: Wait for all user clients
> > >    drm/amdgpu: Wait for all clients importing out dma-bufs.
> > >    drm/ttm: Add destroy flag in TTM BO eviction interface
> > >    drm/amdgpu: Use TTM MMs destroy interface
> > > 
> > >   drivers/gpu/drm/amd/amdgpu/amdgpu.h         |  3 ++
> > >   drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c |  2 +-
> > >   drivers/gpu/drm/amd/amdgpu/amdgpu_device.c  |  7 +++-
> > >   drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c | 27 ++++++++++++-
> > >   drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c     | 22 ++++++++--
> > >   drivers/gpu/drm/amd/amdgpu/amdgpu_job.c     |  9 +++++
> > >   drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c     |  4 ++
> > >   drivers/gpu/drm/amd/amdgpu/amdgpu_object.c  | 17 +++++++-
> > >   drivers/gpu/drm/amd/amdgpu/amdgpu_object.h  |  1 +
> > >   drivers/gpu/drm/nouveau/nouveau_drm.c       |  2 +-
> > >   drivers/gpu/drm/qxl/qxl_object.c            |  4 +-
> > >   drivers/gpu/drm/radeon/radeon_object.c      |  2 +-
> > >   drivers/gpu/drm/ttm/ttm_bo.c                | 63 +++++++++++++++++++++--------
> > >   drivers/gpu/drm/vmwgfx/vmwgfx_drv.c         |  6 +--
> > >   include/drm/ttm/ttm_bo_api.h                |  2 +-
> > >   include/drm/ttm/ttm_bo_driver.h             |  2 +
> > >   16 files changed, 139 insertions(+), 34 deletions(-)
> > > 
> > > -- 
> > > 2.7.4
> > > 

-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [PATCH 0/6] RFC Support hot device unplug in amdgpu

[Daniel Vetter]

On Wed, May 13, 2020 at 10:32:56AM -0400, Andrey Grodzovsky wrote:
> 
> On 5/11/20 5:54 AM, Daniel Vetter wrote:
> > On Sat, May 09, 2020 at 02:51:44PM -0400, Andrey Grodzovsky wrote:
> > > This RFC is a more of a proof of concept then a fully working solution as there are a few unresolved issues we are hopping to get advise on from people on the mailing list.
> > > Until now extracting a card either by physical extraction (e.g. eGPU with thunderbold connection or by emulation through syfs -> /sys/bus/pci/devices/device_id/remove)
> > > would cause random crashes in user apps. The random crashes in apps were mostly due to the app having mapped a device backed BO into it's adress space was still
> > > trying to access the BO while the backing device was gone.
> > > To answer this first problem Christian suggested to fix the handling of mapped memory in the clients when the device goes away by forcibly unmap all buffers
> > > the user processes has by clearing their respective VMAs mapping the device BOs. Then when the VMAs try to fill in the page tables again we check in the fault handler
> > > if the device is removed and if so, return an error. This will generate a SIGBUS to the application which can then cleanly terminate.
> > > This indeed was done but this in turn created a problem of kernel OOPs were the OOPSes were due to the fact that while the app was terminating because of the SIGBUS
> > > it would trigger use after free in the driver by calling to accesses device structures that were already released from the pci remove sequence.
> > > This we handled by introducing a 'flush' seqence during device removal were we wait for drm file reference to drop to 0 meaning all user clients directly using this device terminated.
> > > With this I was able to cleanly emulate device unplug with X and glxgears running and later emulate device plug back and restart of X and glxgears.
> > > 
> > > But this use case is only partial and as I see it all the use cases are as follwing and the questions it raises.
> > > 
> > > 1) Application accesses a BO by opening drm file
> > > 	1.1) BO is mapped into applications address space (BO is CPU visible) - this one we have a solution for by invaldating BO's CPU mapping casuing SIGBUS
> > > 	     and termination and waiting for drm file refcound to drop to 0 before releasing the device
> > > 	1.2) BO is not mapped into applcation address space (BO is CPU invisible) - no solution yet because how we force the application to terminate in this case ?
> > > 
> > > 2) Application accesses a BO by importing a DMA-BUF
> > > 	2.1)  BO is mapped into applications address space (BO is CPU visible) - solution is same as 1.1 but instead of waiting for drm file release we wait for the
> > > 	      imported dma-buf's file release
> > > 	2.2)  BO is not mapped into applcation address space (BO is CPU invisible) - our solution is to invalidate GPUVM page tables and destroy backing storage for
> > >                all exported BOs which will in turn casue VM faults in the importing device and then when the importing driver will try to re-attach the imported BO to
> > > 	      update mappings we return -ENODEV in the import hook which hopeffuly will cause the user app to terminate.
> > > 
> > > 3) Applcation opens a drm file or imports a dma-bud and holds a reference but never access any BO or does access but never more after device was unplug - how would we
> > >     force this applcation to termiante before proceeding with device removal code ? Otherwise the wait in pci remove just hangs for ever.
> > > 
> > > The attached patches adress 1.1, 2.1 and 2.2, for now only 1.1 fully tested and I am still testing the others but I will be happy for any advise on all the
> > > described use cases and maybe some alternative and better (more generic) approach to this like maybe obtaining PIDs of relevant processes through some revere
> > > mapping from device file and exported dma-buf files and send them SIGKILL - would this make more sense or any other method ?
> > > 
> > > Patches 1-3 address 1.1
> > > Patch 4 addresses 2.1
> > > Pathces 5-6 address 2.2
> > > 
> > > Reference: https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.freedesktop.org%2Fdrm%2Famd%2F-%2Fissues%2F1081&data=02%7C01%7Candrey.grodzovsky%40amd.com%7Cf6eec90e9da144cb772a08d7f5921ec2%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637247880251844517&sdata=QBGIbm1KLysglvRAvoiek8jBcNLE%2B4J7gVGDAbZD5Jw%3D&reserved=0
> > So we've been working on this problem for a few years already (but it's
> > still not solved), I think you could have saved yourselfs some typing.
> > 
> > Bunch of things:
> > - we can't wait for userspace in the hotunplug handlers, that might never
> >    happen. The correct way is to untangle the lifetime of your hw driver
> >    for a specific struct pci_device from the drm_device lifetime.
> >    Infrastructure is all there now, see drm_dev_get/put, drm_dev_unplug and
> >    drm_dev_enter/exit.
> 
> this
> 
> To be sure I understood you - do you mean that we should disable/shutdown
> any HW related stuff such as interrupts disable, any shutdown related device
> registers programming and io regions unmapping during pci remove sequence
> (in our case amdgpu_pci_remove) while keeping all the drm/amdgpu structures
> around in memory until drm_dev_put refocunt drop to 0 and
> &drm_driver.release is called thus avoiding any user after free oopses when
> last user reference is dropped ?

Yes.

> Is there any point in doing any HW programming to shutdown device if device
> is already removed anyway (i assume that if driver hook for pci remove  is
> called and it's a physical remove the device is already gone, no ?)

No, all that does is result in bus timeouts, which take forever. Plus
increased chances that the driver gets confused about the values it reads
(for pci all you get is 0xffffffff, no error value, usb is a lot better
here because it's explicit packets and streams where you can get explicit
errors and bail out).

The trouble is that developers still expect you to shut down hw when they
unload the driver for testing, but I think we've discussed a reasonable
solution for that for pci drivers somewhere in this thread.

> What happens if drm_dev_put doesn't drop to 0 before the device is plugged
> back into the system ? In this case i have duplicates of all device
> structures in the system. Do you expect this to be not a problem or if it is
> it's up to me to resolve i guess ?

You get another one. Same way you get duplicates if there's actually 2
devices plugged in, so if your driver supports multiple gpus already you
should be fine.

We should also not hang onto chardev minor numbers, so I think you should
be getting the same minor number again. But not 100% sure, maybe something
we might need to fix ...

> > A bunch of usb/spi drivers use this 100% correctly
> >    now, so there's examples. Plus kerneldoc explains stuff.
> 
> 
> Would you say tiny drm drivers are a good example ?

Yup. But also, they're tiny, so lots more complexity in amdgpu that they
don't even cover. But for the basic flow of using drmm and devm and the
functions I mentioned above, they should be the most bug-free drivers we
have.

> > - for a big driver like amdgpu doing this split up is going to be
> >    horrendously complex. I know, we've done it for i915, at least
> >    partially.
> 
> 
> Can you point me to relevant code/commits for i915 ?

Anything touching the i915 drm_driver.release function. Or any function
called from there.

> > I strongly recommend that you're using devm_ for managing hw
> >    related resources (iomap, irq, ...) as much as possible.
> 
> 
> From what i saw, in DRM devres implementation  amounts to using
> devm_drm_dev_init/devm_drm_dev_init_release - is that what you mean ? If so
> i see that devm_drm_dev_init_release just calls drm_dev_put, drm_dev_unplug
> ends up calling devm_drm_dev_init_release through the devres infrastructure
> - We already call drm_dev_unplug in amdgpu_pci_remove, we also directly call
> drm_dev_put there so i am not clear what's the added value of using devm
> here ?

There's a lot more to devres, but yes that's the drm_device one. For the
full list of what can all be managed with devres see

https://dri.freedesktop.org/docs/drm/driver-api/driver-model/devres.html

There's lots of example usage in drm, especially tiny drivers and anything
that runs on arm. Only caveat is that any usage of devm_kzalloc is buggy
(at least in drm, as far as I've checked them).

> >    For drm_device resources (mostly structures and everything related to
> >    that) we've just merged the drmm_ managed resources framework. There's
> >    some more work to be done there for various kms objects, but you can at
> >    least somewhat avoid tedious handrolling for everything internal
> >    already.
> 
> 
> I can't find drmm in the code, can you point me please ?

drm_managed.c, you need latest drm-next I think. Or linux-next. Docs here:

https://dri.freedesktop.org/docs/drm/gpu/drm-internals.html#managed-resources

> 
> > 
> >    Don't ever use devm_kzalloc and friends, I've looked at hundreds of uses
> >    of this in drm, they're all wrong.
> > 
> > - dma-buf is hilarious (and atm unfixed), dma-fence is even worse. In
> >    theory they're already refcounted and all and so should work, in
> >    practice I think we need to refcount the underlying drm_device with
> >    drm_dev_get/put to avoid the worst fall-out.
> > 
> > - One unfortunate thing with drm_dev_unplug is that the driver core is
> >    very opinionated and doesn't tell you whether it's a hotunplug or a
> >    driver unload. In the former case trying to shut down hw just wastes
> >    time (and might hit driver bugs), in the latter case driver engineers
> >    very much expect everything to be shut down.
> > 
> >    Right now you can only have one or the other, so this needs a module
> >    option hack or similar (default to the correct hotunplug behaviour for
> >    users).
> > 
> > - SIGBUS is better than crashing the kernel, but it's not even close for
> >    users. They still lose everything because everything crashes because in
> >    my experience, in practice, no one ever handles errors. There's a few
> >    more things on top:
> > 
> >    - sighandlers are global, which means only the app can use it. You can't
> >      use it in e.g. mesa. They're also not composable, so if you have on
> >      sighandler for gpu1 and a 2nd one for gpu2 (could be different vendor)
> >      it's all sadness. Hence "usersapce will handle SIGBUS" wont work.
> > 
> >    - worse, neither vk nor gl (to my knowledge) have a concept of events
> >      for when the gpu died. The only stuff you have is things like
> >      arb_robustness which says a) everything continues as if nothing
> >      happened b) there's a function where you can ask whether your gl
> >      context and all the textures/buffers are toast.
> > 
> >      I think that's about the only hotunplug application model we can
> >      realistically expect applications to support. That means _all_ errors
> >      need to be silently eaten by either mesa or the kernel. On i915 the
> >      model (also for terminally wedged gpu hangs) is that all ioctl keep
> >      working, mmaps keep working, and execbuf gives you an -EIO (which mesa
> >      eats silently iirc for arb_robustness).
> > 
> >    Conclusion is that SIGBUS is imo a no-go, and the only option we have is
> >    that a) mmaps fully keep working, doable for shmem or b) we put some
> >    fake memory in there (for vram or whatever), maybe even only a single
> >    page for all fake memory.
> > 
> > - you probably want arb_robustness and similar stuff in userspace as a
> >    first step.
> > 
> > tldr;
> > - refcounting, not waiting for userspace
> > - nothing can fail because userspace wont handle it
> 
> 
> For nothing can fail i see in tiny drm driver examples (e.g.
> ili9225_pipe_enable) that for any function which is about to do HW
> programming they check for drm_dev_enter and silently return if device is
> not present - is that what you mean, that I should pepper all of amdgpu code
> such that any function that ends up doing some HW programming be guarded
> with drm_dev_enter/exit silently returning in case of device is gone ?

Yup.

Note taht e.g. usb because it's a packet/stream bus gives you explicit
errors, so those drivers don't need the drm_dev_enter/exit. But for pci we
need them to make sure we're wasting time when the hw is gone on
everything first timing out.

> Thanks a lot for your detailed response.

np.

Cheers, Daniel

> 
> Andrey
> 
> 
> > 
> > That's at least my take on this mess, and what we've been pushing for over
> > the past few years. For kms-only drm_driver we should have achieved that
> > by now (plus/minus maybe some issues for dma-buf/fences, but kms-only
> > dma-buf/fences are simple enough that maybe we don't go boom yet).
> > 
> > For big gpus with rendering I think best next step would be to type up a
> > reasonable Gran Plan (into Documentation/gpu/todo.rst) with all the issues
> > and likely solutions. And then bikeshed that, since the above is just my
> > take on all this.
> > 
> > Cheers, Daniel
> > 
> > > Andrey Grodzovsky (6):
> > >    drm/ttm: Add unampping of the entire device address space
> > >    drm/amdgpu: Force unmap all user VMAs on device removal.
> > >    drm/amdgpu: Wait for all user clients
> > >    drm/amdgpu: Wait for all clients importing out dma-bufs.
> > >    drm/ttm: Add destroy flag in TTM BO eviction interface
> > >    drm/amdgpu: Use TTM MMs destroy interface
> > > 
> > >   drivers/gpu/drm/amd/amdgpu/amdgpu.h         |  3 ++
> > >   drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c |  2 +-
> > >   drivers/gpu/drm/amd/amdgpu/amdgpu_device.c  |  7 +++-
> > >   drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c | 27 ++++++++++++-
> > >   drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c     | 22 ++++++++--
> > >   drivers/gpu/drm/amd/amdgpu/amdgpu_job.c     |  9 +++++
> > >   drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c     |  4 ++
> > >   drivers/gpu/drm/amd/amdgpu/amdgpu_object.c  | 17 +++++++-
> > >   drivers/gpu/drm/amd/amdgpu/amdgpu_object.h  |  1 +
> > >   drivers/gpu/drm/nouveau/nouveau_drm.c       |  2 +-
> > >   drivers/gpu/drm/qxl/qxl_object.c            |  4 +-
> > >   drivers/gpu/drm/radeon/radeon_object.c      |  2 +-
> > >   drivers/gpu/drm/ttm/ttm_bo.c                | 63 +++++++++++++++++++++--------
> > >   drivers/gpu/drm/vmwgfx/vmwgfx_drv.c         |  6 +--
> > >   include/drm/ttm/ttm_bo_api.h                |  2 +-
> > >   include/drm/ttm/ttm_bo_driver.h             |  2 +
> > >   16 files changed, 139 insertions(+), 34 deletions(-)
> > > 
> > > -- 
> > > 2.7.4
> > > 

-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx