From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: ** X-Spam-Status: No, score=2.5 required=3.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,HTML_MESSAGE,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9AE16C433ED for ; Mon, 26 Apr 2021 21:00:08 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 57FC161107 for ; Mon, 26 Apr 2021 21:00:08 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 57FC161107 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 054506E120; Mon, 26 Apr 2021 21:00:07 +0000 (UTC) Received: from mail-pf1-x42b.google.com (mail-pf1-x42b.google.com [IPv6:2607:f8b0:4864:20::42b]) by gabe.freedesktop.org (Postfix) with ESMTPS id CB3856E10C; Mon, 26 Apr 2021 21:00:05 +0000 (UTC) Received: by mail-pf1-x42b.google.com with SMTP id y62so7209560pfg.4; Mon, 26 Apr 2021 14:00:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=Y1CoAE5T8Ac6HW3BxkKVJY2WL629ZtNx5MreZ/ZYIrQ=; b=ADkFtaMzJ3sr3PGlzeyNwOWtoN/8qmXa0WNMs4i/6Ck0Pla7IpAXNhnppKIhWAbXac 1l+JQXarFydcvNI/cKdtUCc9h6eY+qbDn08EM4G3Q7HMiVfcELV2/XpJ6SEcue/Yg+Go CoxEW2YM2EfSH3BQRshmmyTkTs3yXbj6/xRhYVhaO5RQ4Q0sLxTkrWEYz/nHpuk5DoXK D5pS4p+TOWDI+MuGol7N3dIm54AsmUW88/qQkNuLhrSYo2xuz8DWbc7squOSj+BSUWiC xpUZ5DMCA5k9a8CjvVDVxoGYUi7wSk0/NEjhcTOoi7+ZVKmTGEty2AGqGUarT3h61cfz yt9Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=Y1CoAE5T8Ac6HW3BxkKVJY2WL629ZtNx5MreZ/ZYIrQ=; b=SNiQHPyWYMkKSpZicdoTj6jj8SsEY4V26EigOk4r9qJ3GDRz4iY3yvC7lCurJrdPRe TmdkD57xvFdYzg07moJ+X1xJ4Si3TUgz/+aNTRlzb93/5YKbax6Xmwd3ja+M6I3BUSIU 7K40VvUjzWAJ6445EDXsCRxoq72TGjYTa5mJghMAPTH/EiDo3J/vwKmKXwBujIWkgEa5 pZ4kgd8vEvkcDW6AKd/U7nZ8cYyYxzkOhV4Wv82RapAysRFFy7c3+6q/xLNdLlUxoEkU iVYLg0w8OVD6rfgrIcanOqqelzsnTeqCvN74z0DNv+lDZKl79+SeJ4BwbeQRIXcc9j3r XyKw== X-Gm-Message-State: AOAM532S+TvljyQvKYQpUaMAQziVaUIHOaDuFo8ZadNFvK+TgUTwxeX/ iBxbyn3N2ulW053dPpuaeuaZWLCGnx/kyIzP7ULKVE76Mek= X-Google-Smtp-Source: ABdhPJztWc45xsh2GUJndTmMhkHWttSEAR1EuEhe+wNs2YOeH47Iy3eLCYJuK2h/PcVpKkJWM/lcd0cjdjc8nRQbLXw= X-Received: by 2002:a05:6a00:1c54:b029:261:fc0f:15f7 with SMTP id s20-20020a056a001c54b0290261fc0f15f7mr19597482pfw.30.1619470805361; Mon, 26 Apr 2021 14:00:05 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: =?UTF-8?B?TWFyZWsgT2zFocOhaw==?= Date: Mon, 26 Apr 2021 16:59:28 -0400 Message-ID: Subject: Re: [Mesa-dev] [RFC] Linux Graphics Next: Explicit fences everywhere and no BO fences - initial proposal To: Daniel Stone X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: ML Mesa-dev , dri-devel Content-Type: multipart/mixed; boundary="===============1465562808==" Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" --===============1465562808== Content-Type: multipart/alternative; boundary="00000000000013e6d605c0e66e9a" --00000000000013e6d605c0e66e9a Content-Type: text/plain; charset="UTF-8" Thanks everybody. The initial proposal is dead. Here are some thoughts on how to do it differently. I think we can have direct command submission from userspace via memory-mapped queues ("user queues") without changing window systems. The memory management doesn't have to use GPU page faults like HMM. Instead, it can wait for user queues of a specific process to go idle and then unmap the queues, so that userspace can't submit anything. Buffer evictions, pinning, etc. can be executed when all queues are unmapped (suspended). Thus, no BO fences and page faults are needed. Inter-process synchronization can use timeline semaphores. Userspace will query the wait and signal value for a shared buffer from the kernel. The kernel will keep a history of those queries to know which process is responsible for signalling which buffer. There is only the wait-timeout issue and how to identify the culprit. One of the solutions is to have the GPU send all GPU signal commands and all timed out wait commands via an interrupt to the kernel driver to monitor and validate userspace behavior. With that, it can be identified whether the culprit is the waiting process or the signalling process and which one. Invalid signal/wait parameters can also be detected. The kernel can force-signal only the semaphores that time out, and punish the processes which caused the timeout or used invalid signal/wait parameters. The question is whether this synchronization solution is robust enough for dma_fence and whatever the kernel and window systems need. Marek On Tue, Apr 20, 2021 at 4:34 PM Daniel Stone wrote: > Hi, > > On Tue, 20 Apr 2021 at 20:30, Daniel Vetter wrote: > >> The thing is, you can't do this in drm/scheduler. At least not without >> splitting up the dma_fence in the kernel into separate memory fences >> and sync fences > > > I'm starting to think this thread needs its own glossary ... > > I propose we use 'residency fence' for execution fences which enact > memory-residency operations, e.g. faulting in a page ultimately depending > on GPU work retiring. > > And 'value fence' for the pure-userspace model suggested by timeline > semaphores, i.e. fences being (*addr == val) rather than being able to look > at ctx seqno. > > Cheers, > Daniel > _______________________________________________ > mesa-dev mailing list > mesa-dev@lists.freedesktop.org > https://lists.freedesktop.org/mailman/listinfo/mesa-dev > --00000000000013e6d605c0e66e9a Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Thanks everybody. The initial proposal is dead. Here = are some thoughts on how to do it differently.

I think we can have direct command submission from userspace via memory-ma= pped queues ("user queues") without changing window systems.

The memory management doesn't have to use GPU page= faults like HMM. Instead, it can wait for user queues of a specific proces= s to go idle and then unmap the queues, so that userspace can't submit = anything. Buffer evictions, pinning, etc. can be executed when all queues a= re unmapped (suspended). Thus, no BO fences and page faults are needed.
=

Inter-process synchronization can use timeline se= maphores. Userspace will query the wait and signal value for a shared buffe= r from the kernel. The kernel will keep a history of those queries to know = which process is responsible for signalling which buffer. There is only the= wait-timeout issue and how to identify the culprit. One of the solutions i= s to have the GPU send all GPU signal commands and all timed out wait comma= nds via an interrupt to the kernel driver to monitor and validate userspace= behavior. With that, it can be identified whether the culprit is the waiti= ng process or the signalling process and which one. Invalid signal/wait par= ameters can also be detected. The kernel can force-signal only the semaphor= es that time out, and punish the processes which caused the timeout or used= invalid signal/wait parameters.

The question is w= hether this synchronization solution is robust enough for dma_fence and wha= tever the kernel and window systems need.

Mare= k

On Tue, Apr 20, 2021 at 4:34 PM Daniel Stone <daniel@fooishbar.org> wro= te:
Hi,

On Tue, 20 Apr 2021 at 20:30, Daniel Vetter <daniel@ffwll.ch> = wrote:
The thing= is, you can't do this in drm/scheduler. At least not without
splitting up the dma_fence in the kernel into separate memory fences
and sync fences
=C2=A0
I'm starting to= think this thread needs its own glossary ...

I propose we use 'residency fence' for execut= ion fences which enact memory-residency operations, e.g. faulting in a page= ultimately depending on GPU work retiring.

And 'value fence' for the pure-userspace model = suggested by timeline semaphores, i.e. fences being (*addr =3D=3D val) rath= er than being able to look at ctx seqno.

=
Cheers,
Daniel
_______________________________________________
mesa-dev mailing list
mesa-de= v@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinf= o/mesa-dev
--00000000000013e6d605c0e66e9a-- --===============1465562808== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ dri-devel mailing list dri-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/dri-devel --===============1465562808==--