From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.8 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 759A3C433DF for ; Thu, 9 Jul 2020 12:31:52 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 5093C2074B for ; Thu, 9 Jul 2020 12:31:52 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=ffwll.ch header.i=@ffwll.ch header.b="hvfEgjek" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726932AbgGIMbv (ORCPT ); Thu, 9 Jul 2020 08:31:51 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56626 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726921AbgGIMbv (ORCPT ); Thu, 9 Jul 2020 08:31:51 -0400 Received: from mail-oi1-x244.google.com (mail-oi1-x244.google.com [IPv6:2607:f8b0:4864:20::244]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2771FC061A0B for ; Thu, 9 Jul 2020 05:31:51 -0700 (PDT) Received: by mail-oi1-x244.google.com with SMTP id r8so1727834oij.5 for ; Thu, 09 Jul 2020 05:31:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ffwll.ch; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=KiDJy6GOzF4hs3a4674ttNieH4VUalBdDZ/NwtwQZfA=; b=hvfEgjektFUBLSyqNgVq58lvaqS8oXwNqH7adCVDeFOG1FihOfDKIk82d6Al7lPH+z FU6TNlodJuCQP8ZaqS8KwOZ/4iZAYz0ym2tgiLvm14hq3iDakbejFKFdAsWHxPEM+3mz ReXmbWjUtPlLLIQhYjyUgm8Ql7hNnMAWBp+Ps= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=KiDJy6GOzF4hs3a4674ttNieH4VUalBdDZ/NwtwQZfA=; b=iLW7AtYq2K8Bs+LARLDAIC1boQx4ib6Y8K657w2+sZOIskRhroPBePdNcnJOxgPIU/ +Lvryn23g74HdKjkXNdeGIeRUc2k86RYosNpGpPbzR7ngaRGWm3nphJdmPfYvQm2kRCG CbZXL3QU/4xyYD5MaglhyWoqr6U0BZ5k9yCMuBB2OPj7smtcPr91JeyLJPJmJqFKjtyj A7qc0EDyUF8l6Xc28C6CzzXs9z/mkqfIBSCgs5hJyQwk50JmwkBuHYj9x2u0p0a0FU+2 NTGSknX9awawYdJFlLOpmt6NPNebAOk7+Kmzdmb6x9dhDXEgS/KweOH173Qc0FmWq7LV 183g== X-Gm-Message-State: AOAM530OzNaQYTMHlDScImWz+w/kdXeumaMF3wNADoG704cqJS3bDmOK EPdwuJyfFfHA/cVNDivYfQ7EvGP27MiHmkrkXn986A== X-Google-Smtp-Source: ABdhPJyI8E36+f5KGCTBz5yAx5csIil50BdgW5wTbo9ROaoMlgEYYqX4Zh0y84Cd5mvEYHzda2RKPPkUVIvSpgCpuaU= X-Received: by 2002:aca:da03:: with SMTP id r3mr11062166oig.14.1594297910510; Thu, 09 Jul 2020 05:31:50 -0700 (PDT) MIME-Version: 1.0 References: <20200707201229.472834-1-daniel.vetter@ffwll.ch> <20200707201229.472834-4-daniel.vetter@ffwll.ch> <20200709080458.GO3278063@phenom.ffwll.local> In-Reply-To: From: Daniel Vetter Date: Thu, 9 Jul 2020 14:31:39 +0200 Message-ID: Subject: Re: [Intel-gfx] [PATCH 03/25] dma-buf.rst: Document why idenfinite fences are a bad idea To: Daniel Stone Cc: DRI Development , =?UTF-8?Q?Christian_K=C3=B6nig?= , linux-rdma , Intel Graphics Development , amd-gfx mailing list , Chris Wilson , "moderated list:DMA BUFFER SHARING FRAMEWORK" , Steve Pronovost , Jesse Natalie , Daniel Vetter , Thomas Hellstrom , "open list:DMA BUFFER SHARING FRAMEWORK" , Felix Kuehling , Mika Kuoppala Content-Type: text/plain; charset="UTF-8" Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org On Thu, Jul 9, 2020 at 2:11 PM Daniel Stone wrote: > > On Thu, 9 Jul 2020 at 09:05, Daniel Vetter wrote: > > On Thu, Jul 09, 2020 at 08:36:43AM +0100, Daniel Stone wrote: > > > On Tue, 7 Jul 2020 at 21:13, Daniel Vetter wrote: > > > > Comes up every few years, gets somewhat tedious to discuss, let's > > > > write this down once and for all. > > > > > > Thanks for writing this up! I wonder if any of the notes from my reply > > > to the previous-version thread would be helpful to more explicitly > > > encode the carrot of dma-fence's positive guarantees, rather than just > > > the stick of 'don't do this'. ;) Either way, this is: > > > > I think the carrot should go into the intro section for dma-fence, this > > section here is very much just the "don't do this" part. The previous > > patches have an attempt at encoding this a bit, maybe see whether there's > > a place for your reply (or parts of it) to fit? > > Sounds good to me. > > > > Acked-by: Daniel Stone > > > > > > > What I'm not sure about is whether the text should be more explicit in > > > > flat out mandating the amdkfd eviction fences for long running compute > > > > workloads or workloads where userspace fencing is allowed. > > > > > > ... or whether we just say that you can never use dma-fence in > > > conjunction with userptr. > > > > Uh userptr is entirely different thing. That one is ok. It's userpsace > > fences or gpu futexes or future fences or whatever we want to call them. > > Or is there some other confusion here?. > > I mean generating a dma_fence from a batch which will try to page in > userptr. Given that userptr could be backed by absolutely anything at > all, it doesn't seem smart to allow fences to rely on a pointer to an > mmap'ed NFS file. So it seems like batches should be mutually > exclusive between arbitrary SVM userptr and generating a dma-fence? Locking is Tricky (tm) but essentially what at least amdgpu does is pull in the backing storage before we publish any dma-fence. And then some serious locking magic to make sure that doesn't race with a core mm invalidation event. So for your case here the cs ioctl just blocks until the nfs pages are pulled in. Once we've committed for the dma-fence it's only the other way round, i.e. core mm will stall on the dma-fence if it wants to throw out these pages again. More or less at least. That way we never have a dma-fence depending upon any core mm operations. The only pain here is that this severely limits what you can do in the critical path towards signalling a dma-fence, because the tldr is "no interacting with core mm at all allowed". > Speaking of entirely different things ... the virtio-gpu bit really > doesn't belong in this patch. Oops, dunno where I lost that as a sparate patch. Will split out again :-( -Daniel > > Cheers, > Daniel -- Daniel Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.5 required=3.0 tests=DKIM_INVALID,DKIM_SIGNED, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C86ACC433E1 for ; Thu, 9 Jul 2020 12:31:54 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 9E09320708 for ; Thu, 9 Jul 2020 12:31:54 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=ffwll.ch header.i=@ffwll.ch header.b="hvfEgjek" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9E09320708 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=ffwll.ch Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id D844E6EA5B; Thu, 9 Jul 2020 12:31:51 +0000 (UTC) Received: from mail-oi1-x242.google.com (mail-oi1-x242.google.com [IPv6:2607:f8b0:4864:20::242]) by gabe.freedesktop.org (Postfix) with ESMTPS id 4F89B6E42F for ; Thu, 9 Jul 2020 12:31:51 +0000 (UTC) Received: by mail-oi1-x242.google.com with SMTP id w17so1724011oie.6 for ; Thu, 09 Jul 2020 05:31:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ffwll.ch; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=KiDJy6GOzF4hs3a4674ttNieH4VUalBdDZ/NwtwQZfA=; b=hvfEgjektFUBLSyqNgVq58lvaqS8oXwNqH7adCVDeFOG1FihOfDKIk82d6Al7lPH+z FU6TNlodJuCQP8ZaqS8KwOZ/4iZAYz0ym2tgiLvm14hq3iDakbejFKFdAsWHxPEM+3mz ReXmbWjUtPlLLIQhYjyUgm8Ql7hNnMAWBp+Ps= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=KiDJy6GOzF4hs3a4674ttNieH4VUalBdDZ/NwtwQZfA=; b=gtFl8QnAuQTdyJxIggOU0YTdgmlYJTxsX97lpm9+uOP+Dyr7PsWmgufkmGSZN265FB vNZ8CsJyc1dQ4/XdtFXypMu20aJMEUc3MYVyQxt4cyfXgkmULSNYd9DYHrMNSGeVq2FP 16W+FzNovGf42sGYxlr5t+ewg2wSUc8pYfBkPrJoxi2aGtQl2he0yqv1ofX2bo9ZI1JG w0118Kkqo6fZjFruD2pt4WSY04yNGqEJ1YDBDesldJjyH9nOJ35Y3SH5hYtNI7k3nEfE iOBhXELU5bzqWfQnr8HCTn9rWvI65n9Ql3rfhQ8MXuccUP6YjpMqjCpROC1WQlib4yRb Ng1w== X-Gm-Message-State: AOAM530XALISRqiOND9i4UJxVFMmvNPWMN8/bHOF0EdpIKC0ThPTd+8G H/csbqtKmbuFKGzoiJOK0y7CnFnRNin+3NlcfQOq7A== X-Google-Smtp-Source: ABdhPJyI8E36+f5KGCTBz5yAx5csIil50BdgW5wTbo9ROaoMlgEYYqX4Zh0y84Cd5mvEYHzda2RKPPkUVIvSpgCpuaU= X-Received: by 2002:aca:da03:: with SMTP id r3mr11062166oig.14.1594297910510; Thu, 09 Jul 2020 05:31:50 -0700 (PDT) MIME-Version: 1.0 References: <20200707201229.472834-1-daniel.vetter@ffwll.ch> <20200707201229.472834-4-daniel.vetter@ffwll.ch> <20200709080458.GO3278063@phenom.ffwll.local> In-Reply-To: From: Daniel Vetter Date: Thu, 9 Jul 2020 14:31:39 +0200 Message-ID: Subject: Re: [Intel-gfx] [PATCH 03/25] dma-buf.rst: Document why idenfinite fences are a bad idea To: Daniel Stone X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Felix Kuehling , linux-rdma , Intel Graphics Development , amd-gfx mailing list , Chris Wilson , "moderated list:DMA BUFFER SHARING FRAMEWORK" , Steve Pronovost , DRI Development , Jesse Natalie , Daniel Vetter , Thomas Hellstrom , Mika Kuoppala , =?UTF-8?Q?Christian_K=C3=B6nig?= , "open list:DMA BUFFER SHARING FRAMEWORK" Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" On Thu, Jul 9, 2020 at 2:11 PM Daniel Stone wrote: > > On Thu, 9 Jul 2020 at 09:05, Daniel Vetter wrote: > > On Thu, Jul 09, 2020 at 08:36:43AM +0100, Daniel Stone wrote: > > > On Tue, 7 Jul 2020 at 21:13, Daniel Vetter wrote: > > > > Comes up every few years, gets somewhat tedious to discuss, let's > > > > write this down once and for all. > > > > > > Thanks for writing this up! I wonder if any of the notes from my reply > > > to the previous-version thread would be helpful to more explicitly > > > encode the carrot of dma-fence's positive guarantees, rather than just > > > the stick of 'don't do this'. ;) Either way, this is: > > > > I think the carrot should go into the intro section for dma-fence, this > > section here is very much just the "don't do this" part. The previous > > patches have an attempt at encoding this a bit, maybe see whether there's > > a place for your reply (or parts of it) to fit? > > Sounds good to me. > > > > Acked-by: Daniel Stone > > > > > > > What I'm not sure about is whether the text should be more explicit in > > > > flat out mandating the amdkfd eviction fences for long running compute > > > > workloads or workloads where userspace fencing is allowed. > > > > > > ... or whether we just say that you can never use dma-fence in > > > conjunction with userptr. > > > > Uh userptr is entirely different thing. That one is ok. It's userpsace > > fences or gpu futexes or future fences or whatever we want to call them. > > Or is there some other confusion here?. > > I mean generating a dma_fence from a batch which will try to page in > userptr. Given that userptr could be backed by absolutely anything at > all, it doesn't seem smart to allow fences to rely on a pointer to an > mmap'ed NFS file. So it seems like batches should be mutually > exclusive between arbitrary SVM userptr and generating a dma-fence? Locking is Tricky (tm) but essentially what at least amdgpu does is pull in the backing storage before we publish any dma-fence. And then some serious locking magic to make sure that doesn't race with a core mm invalidation event. So for your case here the cs ioctl just blocks until the nfs pages are pulled in. Once we've committed for the dma-fence it's only the other way round, i.e. core mm will stall on the dma-fence if it wants to throw out these pages again. More or less at least. That way we never have a dma-fence depending upon any core mm operations. The only pain here is that this severely limits what you can do in the critical path towards signalling a dma-fence, because the tldr is "no interacting with core mm at all allowed". > Speaking of entirely different things ... the virtio-gpu bit really > doesn't belong in this patch. Oops, dunno where I lost that as a sparate patch. Will split out again :-( -Daniel > > Cheers, > Daniel -- Daniel Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch _______________________________________________ dri-devel mailing list dri-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/dri-devel From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.5 required=3.0 tests=DKIM_INVALID,DKIM_SIGNED, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 348E7C433E0 for ; Thu, 9 Jul 2020 12:31:58 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 059B620708 for ; Thu, 9 Jul 2020 12:31:58 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=ffwll.ch header.i=@ffwll.ch header.b="hvfEgjek" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 059B620708 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=ffwll.ch Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=intel-gfx-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 5C1D36EA5D; Thu, 9 Jul 2020 12:31:52 +0000 (UTC) Received: from mail-oi1-x241.google.com (mail-oi1-x241.google.com [IPv6:2607:f8b0:4864:20::241]) by gabe.freedesktop.org (Postfix) with ESMTPS id 5D0D96EA5B for ; Thu, 9 Jul 2020 12:31:51 +0000 (UTC) Received: by mail-oi1-x241.google.com with SMTP id h17so1738756oie.3 for ; Thu, 09 Jul 2020 05:31:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ffwll.ch; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=KiDJy6GOzF4hs3a4674ttNieH4VUalBdDZ/NwtwQZfA=; b=hvfEgjektFUBLSyqNgVq58lvaqS8oXwNqH7adCVDeFOG1FihOfDKIk82d6Al7lPH+z FU6TNlodJuCQP8ZaqS8KwOZ/4iZAYz0ym2tgiLvm14hq3iDakbejFKFdAsWHxPEM+3mz ReXmbWjUtPlLLIQhYjyUgm8Ql7hNnMAWBp+Ps= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=KiDJy6GOzF4hs3a4674ttNieH4VUalBdDZ/NwtwQZfA=; b=tDP+UUXWlEBUnJlABjcGZHKCWXB3QXS4n3NGPQluSnX+fcscN+bOoFW8xEO8exVtoT sR1Sg5eHwkDwYQK7Xaz6bBkmmxSg2MCpQeE86dTP8mGNjbZI6UDh8FVITlPn6Ad8XuRD tHIjMVKBJCJoA2gujjRn9sNgZTAaHCOsvFEvzxmH8rdCp0HWrmCni68AT2Cb2GUP0C+q Hyni6AvmhsjFBeCWV3r7RhXwxIAdQxugGJGE3okzkzJecQkNdZ4Ot/ILfI/An9cuuktS +1sxaOHEgwRgaFHlW8155y0Fcn1MZ+VadNrc1fJHEMd2RRAEQZtrj/fGWytfDmKunpTh 3mNA== X-Gm-Message-State: AOAM532VLsmHwKaRU3L9tswvShaNetS0dAqDjbQZnJecH7toIVq9mfSo 3ez/3hfvFh0hLH9y/Oq41QpxaiQMLM5GwG0sDId06A== X-Google-Smtp-Source: ABdhPJyI8E36+f5KGCTBz5yAx5csIil50BdgW5wTbo9ROaoMlgEYYqX4Zh0y84Cd5mvEYHzda2RKPPkUVIvSpgCpuaU= X-Received: by 2002:aca:da03:: with SMTP id r3mr11062166oig.14.1594297910510; Thu, 09 Jul 2020 05:31:50 -0700 (PDT) MIME-Version: 1.0 References: <20200707201229.472834-1-daniel.vetter@ffwll.ch> <20200707201229.472834-4-daniel.vetter@ffwll.ch> <20200709080458.GO3278063@phenom.ffwll.local> In-Reply-To: From: Daniel Vetter Date: Thu, 9 Jul 2020 14:31:39 +0200 Message-ID: To: Daniel Stone Subject: Re: [Intel-gfx] [PATCH 03/25] dma-buf.rst: Document why idenfinite fences are a bad idea X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Felix Kuehling , linux-rdma , Intel Graphics Development , amd-gfx mailing list , Chris Wilson , "moderated list:DMA BUFFER SHARING FRAMEWORK" , Steve Pronovost , DRI Development , Jesse Natalie , Daniel Vetter , Thomas Hellstrom , Mika Kuoppala , =?UTF-8?Q?Christian_K=C3=B6nig?= , "open list:DMA BUFFER SHARING FRAMEWORK" Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" On Thu, Jul 9, 2020 at 2:11 PM Daniel Stone wrote: > > On Thu, 9 Jul 2020 at 09:05, Daniel Vetter wrote: > > On Thu, Jul 09, 2020 at 08:36:43AM +0100, Daniel Stone wrote: > > > On Tue, 7 Jul 2020 at 21:13, Daniel Vetter wrote: > > > > Comes up every few years, gets somewhat tedious to discuss, let's > > > > write this down once and for all. > > > > > > Thanks for writing this up! I wonder if any of the notes from my reply > > > to the previous-version thread would be helpful to more explicitly > > > encode the carrot of dma-fence's positive guarantees, rather than just > > > the stick of 'don't do this'. ;) Either way, this is: > > > > I think the carrot should go into the intro section for dma-fence, this > > section here is very much just the "don't do this" part. The previous > > patches have an attempt at encoding this a bit, maybe see whether there's > > a place for your reply (or parts of it) to fit? > > Sounds good to me. > > > > Acked-by: Daniel Stone > > > > > > > What I'm not sure about is whether the text should be more explicit in > > > > flat out mandating the amdkfd eviction fences for long running compute > > > > workloads or workloads where userspace fencing is allowed. > > > > > > ... or whether we just say that you can never use dma-fence in > > > conjunction with userptr. > > > > Uh userptr is entirely different thing. That one is ok. It's userpsace > > fences or gpu futexes or future fences or whatever we want to call them. > > Or is there some other confusion here?. > > I mean generating a dma_fence from a batch which will try to page in > userptr. Given that userptr could be backed by absolutely anything at > all, it doesn't seem smart to allow fences to rely on a pointer to an > mmap'ed NFS file. So it seems like batches should be mutually > exclusive between arbitrary SVM userptr and generating a dma-fence? Locking is Tricky (tm) but essentially what at least amdgpu does is pull in the backing storage before we publish any dma-fence. And then some serious locking magic to make sure that doesn't race with a core mm invalidation event. So for your case here the cs ioctl just blocks until the nfs pages are pulled in. Once we've committed for the dma-fence it's only the other way round, i.e. core mm will stall on the dma-fence if it wants to throw out these pages again. More or less at least. That way we never have a dma-fence depending upon any core mm operations. The only pain here is that this severely limits what you can do in the critical path towards signalling a dma-fence, because the tldr is "no interacting with core mm at all allowed". > Speaking of entirely different things ... the virtio-gpu bit really > doesn't belong in this patch. Oops, dunno where I lost that as a sparate patch. Will split out again :-( -Daniel > > Cheers, > Daniel -- Daniel Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch _______________________________________________ Intel-gfx mailing list Intel-gfx@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/intel-gfx From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.5 required=3.0 tests=DKIM_INVALID,DKIM_SIGNED, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 62748C433E0 for ; Thu, 9 Jul 2020 12:31:52 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 35B3D20708 for ; Thu, 9 Jul 2020 12:31:52 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=ffwll.ch header.i=@ffwll.ch header.b="hvfEgjek" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 35B3D20708 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=ffwll.ch Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=amd-gfx-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 9D4346E42F; Thu, 9 Jul 2020 12:31:51 +0000 (UTC) Received: from mail-oi1-x241.google.com (mail-oi1-x241.google.com [IPv6:2607:f8b0:4864:20::241]) by gabe.freedesktop.org (Postfix) with ESMTPS id 5D9786EA5C for ; Thu, 9 Jul 2020 12:31:51 +0000 (UTC) Received: by mail-oi1-x241.google.com with SMTP id r8so1727837oij.5 for ; Thu, 09 Jul 2020 05:31:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ffwll.ch; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=KiDJy6GOzF4hs3a4674ttNieH4VUalBdDZ/NwtwQZfA=; b=hvfEgjektFUBLSyqNgVq58lvaqS8oXwNqH7adCVDeFOG1FihOfDKIk82d6Al7lPH+z FU6TNlodJuCQP8ZaqS8KwOZ/4iZAYz0ym2tgiLvm14hq3iDakbejFKFdAsWHxPEM+3mz ReXmbWjUtPlLLIQhYjyUgm8Ql7hNnMAWBp+Ps= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=KiDJy6GOzF4hs3a4674ttNieH4VUalBdDZ/NwtwQZfA=; b=o7KDia+MzXX6DabkZ31w5ZU9Q717lLUINHnS25feFS8WhQbhGDt11LNVYzpsxr3Wm0 RBZghJv9PHflxyMrMfgmCg4lVdJ1Rl7zqrh/qia4mjPEFM7ECKHGk52Zwmeb6tbDH2UE i0z8UOl2VrM4J95UtV9qhCMIicKztCzt+qKVqbt2T1e2H+18IHvPvOnvV5Q+ccfYFfnG z0Rz6D46aefaHC34kgsc//ZjEtG+y8E5MLpsJwxz8zMXkT1alTGt3uYLfKhTkh0UjSLw WexzRBGdPehjS4GCaxB3CPLyE6Ai6KqjKK/jJFI5F7r1pKMdALwp1T2t0gHWTtv/Uixp nWvg== X-Gm-Message-State: AOAM532SDcKGjn6OuV/OARqaBW4l91c8gOyUgefvpLpDnPi7bxH+CGLL sBsFmynkpr8pLSvrNkuVNFDC9ujAr/29O3REAX2ZHA== X-Google-Smtp-Source: ABdhPJyI8E36+f5KGCTBz5yAx5csIil50BdgW5wTbo9ROaoMlgEYYqX4Zh0y84Cd5mvEYHzda2RKPPkUVIvSpgCpuaU= X-Received: by 2002:aca:da03:: with SMTP id r3mr11062166oig.14.1594297910510; Thu, 09 Jul 2020 05:31:50 -0700 (PDT) MIME-Version: 1.0 References: <20200707201229.472834-1-daniel.vetter@ffwll.ch> <20200707201229.472834-4-daniel.vetter@ffwll.ch> <20200709080458.GO3278063@phenom.ffwll.local> In-Reply-To: From: Daniel Vetter Date: Thu, 9 Jul 2020 14:31:39 +0200 Message-ID: Subject: Re: [Intel-gfx] [PATCH 03/25] dma-buf.rst: Document why idenfinite fences are a bad idea To: Daniel Stone X-BeenThere: amd-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion list for AMD gfx List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Felix Kuehling , linux-rdma , Intel Graphics Development , amd-gfx mailing list , Chris Wilson , "moderated list:DMA BUFFER SHARING FRAMEWORK" , Steve Pronovost , DRI Development , Jesse Natalie , Daniel Vetter , Thomas Hellstrom , Mika Kuoppala , =?UTF-8?Q?Christian_K=C3=B6nig?= , "open list:DMA BUFFER SHARING FRAMEWORK" Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: amd-gfx-bounces@lists.freedesktop.org Sender: "amd-gfx" On Thu, Jul 9, 2020 at 2:11 PM Daniel Stone wrote: > > On Thu, 9 Jul 2020 at 09:05, Daniel Vetter wrote: > > On Thu, Jul 09, 2020 at 08:36:43AM +0100, Daniel Stone wrote: > > > On Tue, 7 Jul 2020 at 21:13, Daniel Vetter wrote: > > > > Comes up every few years, gets somewhat tedious to discuss, let's > > > > write this down once and for all. > > > > > > Thanks for writing this up! I wonder if any of the notes from my reply > > > to the previous-version thread would be helpful to more explicitly > > > encode the carrot of dma-fence's positive guarantees, rather than just > > > the stick of 'don't do this'. ;) Either way, this is: > > > > I think the carrot should go into the intro section for dma-fence, this > > section here is very much just the "don't do this" part. The previous > > patches have an attempt at encoding this a bit, maybe see whether there's > > a place for your reply (or parts of it) to fit? > > Sounds good to me. > > > > Acked-by: Daniel Stone > > > > > > > What I'm not sure about is whether the text should be more explicit in > > > > flat out mandating the amdkfd eviction fences for long running compute > > > > workloads or workloads where userspace fencing is allowed. > > > > > > ... or whether we just say that you can never use dma-fence in > > > conjunction with userptr. > > > > Uh userptr is entirely different thing. That one is ok. It's userpsace > > fences or gpu futexes or future fences or whatever we want to call them. > > Or is there some other confusion here?. > > I mean generating a dma_fence from a batch which will try to page in > userptr. Given that userptr could be backed by absolutely anything at > all, it doesn't seem smart to allow fences to rely on a pointer to an > mmap'ed NFS file. So it seems like batches should be mutually > exclusive between arbitrary SVM userptr and generating a dma-fence? Locking is Tricky (tm) but essentially what at least amdgpu does is pull in the backing storage before we publish any dma-fence. And then some serious locking magic to make sure that doesn't race with a core mm invalidation event. So for your case here the cs ioctl just blocks until the nfs pages are pulled in. Once we've committed for the dma-fence it's only the other way round, i.e. core mm will stall on the dma-fence if it wants to throw out these pages again. More or less at least. That way we never have a dma-fence depending upon any core mm operations. The only pain here is that this severely limits what you can do in the critical path towards signalling a dma-fence, because the tldr is "no interacting with core mm at all allowed". > Speaking of entirely different things ... the virtio-gpu bit really > doesn't belong in this patch. Oops, dunno where I lost that as a sparate patch. Will split out again :-( -Daniel > > Cheers, > Daniel -- Daniel Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch _______________________________________________ amd-gfx mailing list amd-gfx@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/amd-gfx