From: Felix Kuehling <felix.kuehling@amd.com> To: "Christian König" <ckoenig.leichtzumerken@gmail.com>, dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org Subject: Re: [PATCH 1/2] drm/ttm: Don't evict SG BOs Date: Wed, 28 Apr 2021 03:49:17 -0400 [thread overview] Message-ID: <83a52589-ce81-35b1-e85d-e9485f9bc233@amd.com> (raw) In-Reply-To: <af825fa4-abb3-69a6-48cd-0a6252f72503@gmail.com> Am 2021-04-28 um 3:04 a.m. schrieb Christian König: > Am 28.04.21 um 07:33 schrieb Felix Kuehling: >> SG BOs do not occupy space that is managed by TTM. So do not evict them. >> >> This fixes unexpected evictions of KFD's userptr BOs. KFD only expects >> userptr "evictions" in the form of MMU notifiers. > > NAK, SG BOs also account for the memory the GPU can currently access. > > We can ignore them for the allocated memory, but not for the GTT domain. Hmm, the only reason I found this problem is, that I am now testing with IOMMU enabled. Evicting the userptr BO destroys the DMA mapping. Without IOMMU-enforced device isolation I was blissfully unaware that the userptr BOs were being evicted. The GPUVM mappings were unaffected and just worked without problems. Having to evict these BOs is crippling KFD's ability to map system memory for GPU access, once again. I think this affects not only userptr BOs but also DMABuf imports for BOs shared between multiple GPUs. The GTT size limitation is entirely artificial. And the only reason I know of for keeping it limited to the VRAM size is to work around some OOM issues with GTT BOs. Applying this to userptrs and DMABuf imports makes no sense. But I understand that the way TTM manages the GTT domain there is no easy fix for this. Maybe we'd have to create a new domain for validating SG BOs that's separate from GTT, so that TTM would not try to allocate GTT space for them. Failing that, I'd probably have to abandon userptr BOs altogether and switch system memory mappings over to using the new SVM API on systems where it is avaliable. Regards, Felix > > Christian. > >> >> Signed-off-by: Felix Kuehling <Felix.Kuehling@amd.com> >> --- >> drivers/gpu/drm/ttm/ttm_bo.c | 4 ++++ >> 1 file changed, 4 insertions(+) >> >> diff --git a/drivers/gpu/drm/ttm/ttm_bo.c b/drivers/gpu/drm/ttm/ttm_bo.c >> index de1ec838cf8b..0b953654fdbf 100644 >> --- a/drivers/gpu/drm/ttm/ttm_bo.c >> +++ b/drivers/gpu/drm/ttm/ttm_bo.c >> @@ -655,6 +655,10 @@ int ttm_mem_evict_first(struct ttm_device *bdev, >> list_for_each_entry(bo, &man->lru[i], lru) { >> bool busy; >> + /* Don't evict SG BOs */ >> + if (bo->ttm && bo->ttm->sg) >> + continue; >> + >> if (!ttm_bo_evict_swapout_allowable(bo, ctx, &locked, >> &busy)) { >> if (busy && !busy_bo && ticket != > _______________________________________________ dri-devel mailing list dri-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/dri-devel
WARNING: multiple messages have this Message-ID (diff)
From: Felix Kuehling <felix.kuehling@amd.com> To: "Christian König" <ckoenig.leichtzumerken@gmail.com>, dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org Subject: Re: [PATCH 1/2] drm/ttm: Don't evict SG BOs Date: Wed, 28 Apr 2021 03:49:17 -0400 [thread overview] Message-ID: <83a52589-ce81-35b1-e85d-e9485f9bc233@amd.com> (raw) In-Reply-To: <af825fa4-abb3-69a6-48cd-0a6252f72503@gmail.com> Am 2021-04-28 um 3:04 a.m. schrieb Christian König: > Am 28.04.21 um 07:33 schrieb Felix Kuehling: >> SG BOs do not occupy space that is managed by TTM. So do not evict them. >> >> This fixes unexpected evictions of KFD's userptr BOs. KFD only expects >> userptr "evictions" in the form of MMU notifiers. > > NAK, SG BOs also account for the memory the GPU can currently access. > > We can ignore them for the allocated memory, but not for the GTT domain. Hmm, the only reason I found this problem is, that I am now testing with IOMMU enabled. Evicting the userptr BO destroys the DMA mapping. Without IOMMU-enforced device isolation I was blissfully unaware that the userptr BOs were being evicted. The GPUVM mappings were unaffected and just worked without problems. Having to evict these BOs is crippling KFD's ability to map system memory for GPU access, once again. I think this affects not only userptr BOs but also DMABuf imports for BOs shared between multiple GPUs. The GTT size limitation is entirely artificial. And the only reason I know of for keeping it limited to the VRAM size is to work around some OOM issues with GTT BOs. Applying this to userptrs and DMABuf imports makes no sense. But I understand that the way TTM manages the GTT domain there is no easy fix for this. Maybe we'd have to create a new domain for validating SG BOs that's separate from GTT, so that TTM would not try to allocate GTT space for them. Failing that, I'd probably have to abandon userptr BOs altogether and switch system memory mappings over to using the new SVM API on systems where it is avaliable. Regards, Felix > > Christian. > >> >> Signed-off-by: Felix Kuehling <Felix.Kuehling@amd.com> >> --- >> drivers/gpu/drm/ttm/ttm_bo.c | 4 ++++ >> 1 file changed, 4 insertions(+) >> >> diff --git a/drivers/gpu/drm/ttm/ttm_bo.c b/drivers/gpu/drm/ttm/ttm_bo.c >> index de1ec838cf8b..0b953654fdbf 100644 >> --- a/drivers/gpu/drm/ttm/ttm_bo.c >> +++ b/drivers/gpu/drm/ttm/ttm_bo.c >> @@ -655,6 +655,10 @@ int ttm_mem_evict_first(struct ttm_device *bdev, >> list_for_each_entry(bo, &man->lru[i], lru) { >> bool busy; >> + /* Don't evict SG BOs */ >> + if (bo->ttm && bo->ttm->sg) >> + continue; >> + >> if (!ttm_bo_evict_swapout_allowable(bo, ctx, &locked, >> &busy)) { >> if (busy && !busy_bo && ticket != > _______________________________________________ amd-gfx mailing list amd-gfx@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/amd-gfx
next prev parent reply other threads:[~2021-04-28 7:49 UTC|newest] Thread overview: 22+ messages / expand[flat|nested] mbox.gz Atom feed top 2021-04-28 5:33 [PATCH 1/2] drm/ttm: Don't evict SG BOs Felix Kuehling 2021-04-28 5:33 ` Felix Kuehling 2021-04-28 5:33 ` [PATCH 2/2] drm/ttm: Fix swapout in ttm_tt_populate Felix Kuehling 2021-04-28 5:33 ` Felix Kuehling 2021-04-28 7:03 ` Christian König 2021-04-28 7:03 ` Christian König 2021-04-28 7:04 ` [PATCH 1/2] drm/ttm: Don't evict SG BOs Christian König 2021-04-28 7:04 ` Christian König 2021-04-28 7:49 ` Felix Kuehling [this message] 2021-04-28 7:49 ` Felix Kuehling 2021-04-28 9:05 ` Christian König 2021-04-28 9:05 ` Christian König 2021-04-28 15:19 ` Felix Kuehling 2021-04-28 15:19 ` Felix Kuehling 2021-04-28 16:33 ` Christian König 2021-04-28 16:33 ` Christian König 2021-04-28 16:49 ` Felix Kuehling 2021-04-28 16:49 ` Felix Kuehling 2021-04-28 16:58 ` Christian König 2021-04-28 16:58 ` Christian König 2021-04-28 17:02 ` Felix Kuehling 2021-04-28 17:02 ` Felix Kuehling
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=83a52589-ce81-35b1-e85d-e9485f9bc233@amd.com \ --to=felix.kuehling@amd.com \ --cc=amd-gfx@lists.freedesktop.org \ --cc=ckoenig.leichtzumerken@gmail.com \ --cc=dri-devel@lists.freedesktop.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.