From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: ** X-Spam-Status: No, score=2.5 required=3.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,HTML_MESSAGE,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 13BFCC433ED for ; Tue, 4 May 2021 17:16:37 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id BA934610A2 for ; Tue, 4 May 2021 17:16:36 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org BA934610A2 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 12B3789D00; Tue, 4 May 2021 17:16:36 +0000 (UTC) Received: from mail-pg1-x52e.google.com (mail-pg1-x52e.google.com [IPv6:2607:f8b0:4864:20::52e]) by gabe.freedesktop.org (Postfix) with ESMTPS id E2B6E89D00; Tue, 4 May 2021 17:16:34 +0000 (UTC) Received: by mail-pg1-x52e.google.com with SMTP id p12so7821825pgj.10; Tue, 04 May 2021 10:16:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=Gyv6ANs6xU1RsqUeM9SGO9pt7eeeRFtkcUQN+obiBTc=; b=Of/zb37H9SO5pmjm/reQeAt/2jlRBBHQqnsUM67o0PIAR8CWfB7ZacYpL9bm3yqSxo BoTflIxOiqW2hgHmMiEf0LSdebptENA2MozSxssM3kgEvB5Q1sV7QYHbR+DQl2ykHOf6 cMh6AMzY2miVxjYsEwmlgJRQnQR/rimclJSNW+4+DoqU4Jj3CN1LXzGB1iKoR7JzJvzW TBZE7Auch2w47QThclQkXUb8JV0RCwi8JkGomTR+r36SpoOyFhb4i3PMiEXWgj8qhvb1 pdibAoR1sW5gMX1osUplEWP4xEe8oNno2/bascVj2x7kLJEdC8QfI/TyZr0+1G1ozbRg enKw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=Gyv6ANs6xU1RsqUeM9SGO9pt7eeeRFtkcUQN+obiBTc=; b=rC1wytMlOtGqmhVixg8jxkI/tcpkWoNlAfUJTYPQEh7p9DUOCHWRB82SCSPZRygSXa xUFDjpTYqV/3LOPg01hU3eLllmh2ObndVzQqiL3EOJaLMEgqmWa0U292hF0/KhCVlyNc h8b6p0/AgxyrLAu8NmzGYu8aHiHe2ACk9Dmsdc8ntgAn2Lh1aoEqzkGGlm1mM4nE+pdL 6+1RkvoxPrNqDZ7darDf+glDo+lU8fg1JVU7ODQT/CtztnmOZ2qkwkVhEJb3ADdrSD3s KzEIME3+IKGMsg1VhUPddhplS5mD1JGlHmZn4UnMFUc/bxW5m+oUWoVJUpjtwrp1t94f Umow== X-Gm-Message-State: AOAM5330HacvyGw1e+AhQoif98m4cXzkHg+rPrrtJgL0HU/nLkpf0EAH l8ZDB0xQ8ak2ESzNLUeAR2JJhqLByptjylHLewE= X-Google-Smtp-Source: ABdhPJwe0jzW6yjP0WLGKmNdizn1Iyta2iFib61sSA6NKRTxbMq0N4hvHTKz1fRY7WbW6+ZUGOdxGASMf7nAz7CUPLE= X-Received: by 2002:a63:8f17:: with SMTP id n23mr16435375pgd.82.1620148594380; Tue, 04 May 2021 10:16:34 -0700 (PDT) MIME-Version: 1.0 References: <1bd8105b-4a2a-2ad9-0b3c-a81590282f2e@gmail.com> <9ccfe4b2-91f0-b8e5-6327-bf3c8b6d1a24@gmail.com> <7227e6fb-1108-1096-ab2c-017d6422e90b@gmail.com> In-Reply-To: From: =?UTF-8?B?TWFyZWsgT2zFocOhaw==?= Date: Tue, 4 May 2021 13:16:23 -0400 Message-ID: Subject: Re: [Mesa-dev] [RFC] Linux Graphics Next: Explicit fences everywhere and no BO fences - initial proposal To: =?UTF-8?Q?Christian_K=C3=B6nig?= X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: dri-devel , ML Mesa-dev , =?UTF-8?Q?Michel_D=C3=A4nzer?= , Jason Ekstrand Content-Type: multipart/mixed; boundary="===============1718242824==" Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" --===============1718242824== Content-Type: multipart/alternative; boundary="0000000000007394eb05c1843d99" --0000000000007394eb05c1843d99 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable I see some mentions of XNACK and recoverable page faults. Note that all gaming AMD hw that has userspace queues doesn't have XNACK, so there is no overhead in compute units. My understanding is that recoverable page faults are still supported without XNACK, but instead of the compute unit replaying the faulting instruction, the L1 cache does that. Anyway, the point is that XNACK is totally irrelevant here. Marek On Tue., May 4, 2021, 08:48 Christian K=C3=B6nig, < ckoenig.leichtzumerken@gmail.com> wrote: > Am 04.05.21 um 13:13 schrieb Daniel Vetter: > > On Tue, May 4, 2021 at 12:53 PM Christian K=C3=B6nig > > wrote: > >> Am 04.05.21 um 11:47 schrieb Daniel Vetter: > >>> [SNIP] > >>>> Yeah, it just takes to long for the preemption to complete to be > really > >>>> useful for the feature we are discussing here. > >>>> > >>>> As I said when the kernel requests to preempt a queue we can easily > expect a > >>>> timeout of ~100ms until that comes back. For compute that is even in > the > >>>> multiple seconds range. > >>> 100ms for preempting an idle request sounds like broken hw to me. Of > >>> course preemting something that actually runs takes a while, that's > >>> nothing new. But it's also not the thing we're talking about here. Is > this > >>> 100ms actual numbers from hw for an actual idle ringbuffer? > >> Well 100ms is just an example of the scheduler granularity. Let me > >> explain in a wider context. > >> > >> The hardware can have X queues mapped at the same time and every Y tim= e > >> interval the hardware scheduler checks if those queues have changed an= d > >> only if they have changed the necessary steps to reload them are > started. > >> > >> Multiple queues can be rendering at the same time, so you can have X a= s > >> a high priority queue active and just waiting for a signal to start an= d > >> the client rendering one frame after another and a third background > >> compute task mining bitcoins for you. > >> > >> As long as everything is static this is perfectly performant. Adding a > >> queue to the list of active queues is also relatively simple, but taki= ng > >> one down requires you to wait until we are sure the hardware has seen > >> the change and reloaded the queues. > >> > >> Think of it as an RCU grace period. This is simply not something which > >> is made to be used constantly, but rather just at process termination. > > Uh ... that indeed sounds rather broken. > > Well I wouldn't call it broken. It's just not made for the use case we > are trying to abuse it for. > > > Otoh it's just a dma_fence that'd we'd inject as this unload-fence. > > Yeah, exactly that's why it isn't much of a problem for process > termination or freeing memory. > > > So by and large everyone should already be able to cope with it taking = a > > bit longer. So from a design pov I don't see a huge problem, but I > > guess you guys wont be happy since it means on amd hw there will be > > random unsightly stalls in desktop linux usage. > > > >>>> The "preemption" feature is really called suspend and made just for > the case > >>>> when we want to put a process to sleep or need to forcefully kill it > for > >>>> misbehavior or stuff like that. It is not meant to be used in normal > >>>> operation. > >>>> > >>>> If we only attach it on ->move then yeah maybe a last resort > possibility to > >>>> do it this way, but I think in that case we could rather stick with > kernel > >>>> submissions. > >>> Well this is a hybrid userspace ring + kernel augmeted submit mode, s= o > you > >>> can keep dma-fences working. Because the dma-fence stuff wont work wi= th > >>> pure userspace submit, I think that conclusion is rather solid. Once > more > >>> even after this long thread here. > >> When assisted with unload fences, then yes. Problem is that I can't se= e > >> how we could implement those performant currently. > > Is there really no way to fix fw here? Like if process start/teardown > > takes 100ms, that's going to suck no matter what. > > As I said adding the queue is unproblematic and teardown just results in > a bit more waiting to free things up. > > Problematic is more overcommit swapping and OOM situations which need to > wait for the hw scheduler to come back and tell us that the queue is now > unmapped. > > >>>>> Also, if userspace lies to us and keeps pushing crap into the ring > >>>>> after it's supposed to be idle: Userspace is already allowed to was= te > >>>>> gpu time. If you're too worried about this set a fairly aggressive > >>>>> preempt timeout on the unload fence, and kill the context if it tak= es > >>>>> longer than what preempting an idle ring should take (because that > >>>>> would indicate broken/evil userspace). > >>>> I think you have the wrong expectation here. It is perfectly valid a= nd > >>>> expected for userspace to keep writing commands into the ring buffer= . > >>>> > >>>> After all when one frame is completed they want to immediately start > >>>> rendering the next one. > >>> Sure, for the true userspace direct submit model. But with that you > don't > >>> get dma-fence, which means this gpu will not work for 3d accel on any > >>> current linux desktop. > >> I'm not sure of that. I've looked a bit into how we could add user > >> fences to dma_resv objects and that isn't that hard after all. > > I think as a proof of concept it's fine, but as an actual solution ... > > pls no. Two reasons: > > - implicit sync is bad > > Well can't disagree with that :) But I think we can't avoid supporting it= . > > > - this doesn't fix anything for explicit sync using dma_fence in terms > > of sync_file or drm_syncobj. > > Exactly. > > If we do implicit sync or explicit sync is orthogonal to the problems > that sync must be made reliable somehow. > > So when we sync and timeout the waiter should just continue, but whoever > failed to signal will be punished. > > But since this isn't solved on Windows I don't see how we can solve it > on Linux either. > > > So if we go with the route of papering over this in the kernel, then > > it'll be a ton more work than just hacking something into dma_resv. > > I'm just now prototyping that and at least for the driver parts it > doesn't look that hard after all. > > >>> Which sucks, hence some hybrid model of using the userspace ring and > >>> kernel augmented submit is needed. Which was my idea. > >> Yeah, I think when our firmware folks would really remove the kernel > >> queue and we still don't have > > Yeah I think kernel queue can be removed. But the price is that you > > need reasonable fast preempt of idle contexts. > > > > I really can't understand how this can take multiple ms, something > > feels very broken in the design of the fw (since obviously the hw can > > preempt an idle context to another one pretty fast, or you'd render > > any multi-client desktop as a slideshow at best). > > Well the hardware doesn't preempt and idle context. See you can have a > number of active ("mapped" in the fw terminology) contexts and idle > contexts are usually kept active even when they are idle. > > So when multi-client desktop switches between context then that is > rather fast, but when the kernel asks for a context to be unmapped that > can take rather long. > > > > > >>>> [SNIP] > >>>> Can't find that of hand either, but see the amdgpu_noretry module > option. > >>>> > >>>> It basically tells the hardware if retry page faults should be > supported or > >>>> not because this whole TLB shutdown thing when they are supported is > >>>> extremely costly. > >>> Hm so synchronous tlb shootdown is a lot more costly when you allow > >>> retrying of page faults? > >> Partially correct, yes. > >> > >> See when you have retry page faults enabled and unmap something you ne= ed > >> to make sure that everybody which could have potentially translated th= at > >> page and has a TLB is either invalidated or waited until the access is > >> completed. > >> > >> Since every CU could be using a memory location that takes ages to > >> completed compared to the normal invalidation where you just invalidat= e > >> the L1/L2 and are done. > >> > >> Additional to that the recovery adds some extra overhead to every memo= ry > >> access, so even without a fault you are quite a bit slower if this is > >> enabled. > > Well yes it's complicated, and it's even more fun when the tlb > > invalidate comes in through the IOMMU through ATS. > > > > But also if you don't your hw is just broken from a security pov, no > > page fault handling for you. So it's really not optional. > > Yeah, but that is also a known issue. You either have retry faults and > live with the extra overhead or you disable them and go with the kernel > based submission approach. > > > > >>> That sounds bad, because for full hmm mode you need to be able to ret= ry > >>> pagefaults. Well at least the PASID/ATS/IOMMU side will do that, and > might just > >>> hang your gpu for a long time while it's waiting for the va->pa looku= p > >>> response to return. So retrying lookups shouldn't be any different > really. > >>> > >>> And you also need fairly fast synchronous tlb shootdown for hmm. So i= f > >>> your hw has a problem with both together that sounds bad. > >> Completely agree. And since it was my job to validate the implementati= on > >> on Vega10 I was also the first one to realize that. > >> > >> Felix, a couple of others and me are trying to work around those > >> restrictions ever since. > >> > >>> I was more thinking about handling it all in the kernel. > >>> Yeah can do, just means that you also have to copy the ringbuffer stu= ff > >>> over from userspace to the kernel. > >> That is my least worry. The IBs are just addr+length., so no more than > >> 16 bytes for each IB. > > Ah ok, maybe I'm biased from drm/i915 where an ib launch + seqno is > > rather long, because the hw folks keep piling more workarounds and > > additional flushes on top. Like on some hw the recommended w/a was to > > just issue 32 gpu cache flushes or something like that (otherwise the > > seqno write could arrive before the gpu actually finished flushing) > > :-/ > > Well I once had a conversation with a hw engineer which wanted to split > up the TLB in validations into 1Gib chunks :) > > That would have mean we would need to emit 2^17 different invalidation > requests on the kernel ring buffer.... > > Christian. > > > > > > Cheers, Daniel > > > >>> It also means that there's more differences in how your userspace wor= ks > >>> between full userspace mode (necessary for compute) and legacy > dma-fence > >>> mode (necessary for desktop 3d). Which is especially big fun for > vulkan, > >>> since that will have to do both. > >> That is the bigger problem. > >> > >> Christian. > >> > >>> But then amd is still hanging onto the amdgpu vs amdkfd split, so > you're > >>> going for max pain in this area anyway :-P > >>> -Daniel > > > > --0000000000007394eb05c1843d99 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
I see some mentions of XNACK and recoverable page faults.= Note that all gaming AMD hw that has userspace queues doesn't have XNA= CK, so there is no overhead in compute units. My understanding is that reco= verable page faults are still supported without XNACK, but instead of the c= ompute unit replaying the faulting instruction, the L1 cache does that. Any= way, the point is that XNACK is totally irrelevant here.
<= br>
Marek

=
On Tue., May 4, 2021, 08:48 Christian= K=C3=B6nig, <ckoeni= g.leichtzumerken@gmail.com> wrote:
Am 04.05.21 um 13:13 schrieb Daniel Vetter:
> On Tue, May 4, 2021 at 12:53 PM Christian K=C3=B6nig
> <ckoenig.leichtzumerken@gmail.com> wrote:
>> Am 04.05.21 um 11:47 schrieb Daniel Vetter:
>>> [SNIP]
>>>> Yeah, it just takes to long for the preemption to complete= to be really
>>>> useful for the feature we are discussing here.
>>>>
>>>> As I said when the kernel requests to preempt a queue we c= an easily expect a
>>>> timeout of ~100ms until that comes back. For compute that = is even in the
>>>> multiple seconds range.
>>> 100ms for preempting an idle request sounds like broken hw to = me. Of
>>> course preemting something that actually runs takes a while, t= hat's
>>> nothing new. But it's also not the thing we're talking= about here. Is this
>>> 100ms actual numbers from hw for an actual idle ringbuffer? >> Well 100ms is just an example of the scheduler granularity. Let me=
>> explain in a wider context.
>>
>> The hardware can have X queues mapped at the same time and every Y= time
>> interval the hardware scheduler checks if those queues have change= d and
>> only if they have changed the necessary steps to reload them are s= tarted.
>>
>> Multiple queues can be rendering at the same time, so you can have= X as
>> a high priority queue active and just waiting for a signal to star= t and
>> the client rendering one frame after another and a third backgroun= d
>> compute task mining bitcoins for you.
>>
>> As long as everything is static this is perfectly performant. Addi= ng a
>> queue to the list of active queues is also relatively simple, but = taking
>> one down requires you to wait until we are sure the hardware has s= een
>> the change and reloaded the queues.
>>
>> Think of it as an RCU grace period. This is simply not something w= hich
>> is made to be used constantly, but rather just at process terminat= ion.
> Uh ... that indeed sounds rather broken.

Well I wouldn't call it broken. It's just not made for the use case= we
are trying to abuse it for.

> Otoh it's just a dma_fence that'd we'd inject as this unlo= ad-fence.

Yeah, exactly that's why it isn't much of a problem for process termination or freeing memory.

> So by and large everyone should already be able to cope with it taking= a
> bit longer. So from a design pov I don't see a huge problem, but I=
> guess you guys wont be happy since it means on amd hw there will be > random unsightly stalls in desktop linux usage.
>
>>>> The "preemption" feature is really called suspen= d and made just for the case
>>>> when we want to put a process to sleep or need to forceful= ly kill it for
>>>> misbehavior or stuff like that. It is not meant to be used= in normal
>>>> operation.
>>>>
>>>> If we only attach it on ->move then yeah maybe a last r= esort possibility to
>>>> do it this way, but I think in that case we could rather s= tick with kernel
>>>> submissions.
>>> Well this is a hybrid userspace ring + kernel augmeted submit = mode, so you
>>> can keep dma-fences working. Because the dma-fence stuff wont = work with
>>> pure userspace submit, I think that conclusion is rather solid= . Once more
>>> even after this long thread here.
>> When assisted with unload fences, then yes. Problem is that I can&= #39;t see
>> how we could implement those performant currently.
> Is there really no way to fix fw here? Like if process start/teardown<= br> > takes 100ms, that's going to suck no matter what.

As I said adding the queue is unproblematic and teardown just results in a bit more waiting to free things up.

Problematic is more overcommit swapping and OOM situations which need to wait for the hw scheduler to come back and tell us that the queue is now unmapped.

>>>>> Also, if userspace lies to us and keeps pushing crap i= nto the ring
>>>>> after it's supposed to be idle: Userspace is alrea= dy allowed to waste
>>>>> gpu time. If you're too worried about this set a f= airly aggressive
>>>>> preempt timeout on the unload fence, and kill the cont= ext if it takes
>>>>> longer than what preempting an idle ring should take (= because that
>>>>> would indicate broken/evil userspace).
>>>> I think you have the wrong expectation here. It is perfect= ly valid and
>>>> expected for userspace to keep writing commands into the r= ing buffer.
>>>>
>>>> After all when one frame is completed they want to immedia= tely start
>>>> rendering the next one.
>>> Sure, for the true userspace direct submit model. But with tha= t you don't
>>> get dma-fence, which means this gpu will not work for 3d accel= on any
>>> current linux desktop.
>> I'm not sure of that. I've looked a bit into how we could = add user
>> fences to dma_resv objects and that isn't that hard after all.=
> I think as a proof of concept it's fine, but as an actual solution= ...
> pls no. Two reasons:
> - implicit sync is bad

Well can't disagree with that :) But I think we can't avoid support= ing it.

> - this doesn't fix anything for explicit sync using dma_fence in t= erms
> of sync_file or drm_syncobj.

Exactly.

If we do implicit sync or explicit sync is orthogonal to the problems
that sync must be made reliable somehow.

So when we sync and timeout the waiter should just continue, but whoever failed to signal will be punished.

But since this isn't solved on Windows I don't see how we can solve= it
on Linux either.

> So if we go with the route of papering over this in the kernel, then > it'll be a ton more work than just hacking something into dma_resv= .

I'm just now prototyping that and at least for the driver parts it
doesn't look that hard after all.

>>> Which sucks, hence some hybrid model of using the userspace ri= ng and
>>> kernel augmented submit is needed. Which was my idea.
>> Yeah, I think when our firmware folks would really remove the kern= el
>> queue and we still don't have
> Yeah I think kernel queue can be removed. But the price is that you > need reasonable fast preempt of idle contexts.
>
> I really can't understand how this can take multiple ms, something=
> feels very broken in the design of the fw (since obviously the hw can<= br> > preempt an idle context to another one pretty fast, or you'd rende= r
> any multi-client desktop as a slideshow at best).

Well the hardware doesn't preempt and idle context. See you can have a =
number of active ("mapped" in the fw terminology) contexts and id= le
contexts are usually kept active even when they are idle.

So when multi-client desktop switches between context then that is
rather fast, but when the kernel asks for a context to be unmapped that can take rather long.


>
>>>> [SNIP]
>>>> Can't find that of hand either, but see the amdgpu_nor= etry module option.
>>>>
>>>> It basically tells the hardware if retry page faults shoul= d be supported or
>>>> not because this whole TLB shutdown thing when they are su= pported is
>>>> extremely costly.
>>> Hm so synchronous tlb shootdown is a lot more costly when you = allow
>>> retrying of page faults?
>> Partially correct, yes.
>>
>> See when you have retry page faults enabled and unmap something yo= u need
>> to make sure that everybody which could have potentially translate= d that
>> page and has a TLB is either invalidated or waited until the acces= s is
>> completed.
>>
>> Since every CU could be using a memory location that takes ages to=
>> completed compared to the normal invalidation where you just inval= idate
>> the L1/L2 and are done.
>>
>> Additional to that the recovery adds some extra overhead to every = memory
>> access, so even without a fault you are quite a bit slower if this= is
>> enabled.
> Well yes it's complicated, and it's even more fun when the tlb=
> invalidate comes in through the IOMMU through ATS.
>
> But also if you don't your hw is just broken from a security pov, = no
> page fault handling for you. So it's really not optional.

Yeah, but that is also a known issue. You either have retry faults and
live with the extra overhead or you disable them and go with the kernel based submission approach.

>
>>> That sounds bad, because for full hmm mode you need to be able= to retry
>>> pagefaults. Well at least the PASID/ATS/IOMMU side will do tha= t, and might just
>>> hang your gpu for a long time while it's waiting for the v= a->pa lookup
>>> response to return. So retrying lookups shouldn't be any d= ifferent really.
>>>
>>> And you also need fairly fast synchronous tlb shootdown for hm= m. So if
>>> your hw has a problem with both together that sounds bad.
>> Completely agree. And since it was my job to validate the implemen= tation
>> on Vega10 I was also the first one to realize that.
>>
>> Felix, a couple of others and me are trying to work around those >> restrictions ever since.
>>
>>> I was more thinking about handling it all in the kernel.
>>> Yeah can do, just means that you also have to copy the ringbuf= fer stuff
>>> over from userspace to the kernel.
>> That is my least worry. The IBs are just addr+length., so no more = than
>> 16 bytes for each IB.
> Ah ok, maybe I'm biased from drm/i915 where an ib launch + seqno i= s
> rather long, because the hw folks keep piling more workarounds and
> additional flushes on top. Like on some hw the recommended w/a was to<= br> > just issue 32 gpu cache flushes or something like that (otherwise the<= br> > seqno write could arrive before the gpu actually finished flushing) > :-/

Well I once had a conversation with a hw engineer which wanted to split up the TLB in validations into 1Gib chunks :)

That would have mean we would need to emit 2^17 different invalidation
requests on the kernel ring buffer....

Christian.


>
> Cheers, Daniel
>
>>> It also means that there's more differences in how your us= erspace works
>>> between full userspace mode (necessary for compute) and legacy= dma-fence
>>> mode (necessary for desktop 3d). Which is especially big fun f= or vulkan,
>>> since that will have to do both.
>> That is the bigger problem.
>>
>> Christian.
>>
>>> But then amd is still hanging onto the amdgpu vs amdkfd split,= so you're
>>> going for max pain in this area anyway :-P
>>> -Daniel
>

--0000000000007394eb05c1843d99-- --===============1718242824== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ dri-devel mailing list dri-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/dri-devel --===============1718242824==--