From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.3 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,NICE_REPLY_A, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 809E6C433E2 for ; Tue, 21 Jul 2020 10:47:42 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 5A1072077D for ; Tue, 21 Jul 2020 10:47:42 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=shipmail.org header.i=@shipmail.org header.b="SWKK5JDY" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726127AbgGUKrl (ORCPT ); Tue, 21 Jul 2020 06:47:41 -0400 Received: from ste-pvt-msa1.bahnhof.se ([213.80.101.70]:3426 "EHLO ste-pvt-msa1.bahnhof.se" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726089AbgGUKrl (ORCPT ); Tue, 21 Jul 2020 06:47:41 -0400 Received: from localhost (localhost [127.0.0.1]) by ste-pvt-msa1.bahnhof.se (Postfix) with ESMTP id 634B53FAA9; Tue, 21 Jul 2020 12:47:38 +0200 (CEST) Authentication-Results: ste-pvt-msa1.bahnhof.se; dkim=pass (1024-bit key; unprotected) header.d=shipmail.org header.i=@shipmail.org header.b=SWKK5JDY; dkim-atps=neutral X-Virus-Scanned: Debian amavisd-new at bahnhof.se Received: from ste-pvt-msa1.bahnhof.se ([127.0.0.1]) by localhost (ste-pvt-msa1.bahnhof.se [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Lnu7aDB-FEDZ; Tue, 21 Jul 2020 12:47:37 +0200 (CEST) Received: from mail1.shipmail.org (h-205-35.A357.priv.bahnhof.se [155.4.205.35]) (Authenticated sender: mb878879) by ste-pvt-msa1.bahnhof.se (Postfix) with ESMTPA id B619E3FA36; Tue, 21 Jul 2020 12:47:33 +0200 (CEST) Received: from localhost.localdomain (h-205-35.A357.priv.bahnhof.se [155.4.205.35]) by mail1.shipmail.org (Postfix) with ESMTPSA id C976C36014B; Tue, 21 Jul 2020 12:47:32 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=shipmail.org; s=mail; t=1595328452; bh=SgSXLo8jlp/fUxGq5/reKrY6eohj3hTAWwVyZVHKpuw=; h=Subject:To:Cc:References:From:Date:In-Reply-To:From; b=SWKK5JDYlv6qtQrYz/vvU9mKdIqWPE5+OPb/yzw09qAZIG2pjw+OHeMsgSa4jHpuS uZ9fdDlzD4wuNqhhrbfgV1ASbwsDxz6PZs1TKxFzA/6WXfA+IuNOgIOyKzdvMZWhVC PNVEsDJjINjdKaof0Wvjc1rMMD4+T1qlUEMbp7y4= Subject: Re: [Linaro-mm-sig] [PATCH 1/2] dma-buf.rst: Document why indefinite fences are a bad idea To: Daniel Vetter Cc: =?UTF-8?Q?Christian_K=c3=b6nig?= , DRI Development , Daniel Stone , linux-rdma , Intel Graphics Development , Maarten Lankhorst , amd-gfx list , "moderated list:DMA BUFFER SHARING FRAMEWORK" , Steve Pronovost , Daniel Vetter , Jason Ekstrand , Jesse Natalie , Felix Kuehling , Thomas Hellstrom , "open list:DMA BUFFER SHARING FRAMEWORK" , Mika Kuoppala References: <20200707201229.472834-4-daniel.vetter@ffwll.ch> <20200709123339.547390-1-daniel.vetter@ffwll.ch> <93b673b7-bb48-96eb-dc2c-bd4f9304000e@shipmail.org> <20200721074157.GB3278063@phenom.ffwll.local> <3603bb71-318b-eb53-0532-9daab62dce86@amd.com> <57a5eb9d-b74f-8ce4-7199-94e911d9b68b@shipmail.org> <2ca2c004-1e11-87f5-4bd8-761e1b44d21f@amd.com> <74727f17-b3a5-ca12-6db6-e47543797b72@shipmail.org> From: =?UTF-8?Q?Thomas_Hellstr=c3=b6m_=28Intel=29?= Message-ID: Date: Tue, 21 Jul 2020 12:47:32 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.10.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org On 7/21/20 11:50 AM, Daniel Vetter wrote: > On Tue, Jul 21, 2020 at 11:38 AM Thomas Hellström (Intel) > wrote: >> >> On 7/21/20 10:55 AM, Christian König wrote: >>> Am 21.07.20 um 10:47 schrieb Thomas Hellström (Intel): >>>> On 7/21/20 9:45 AM, Christian König wrote: >>>>> Am 21.07.20 um 09:41 schrieb Daniel Vetter: >>>>>> On Mon, Jul 20, 2020 at 01:15:17PM +0200, Thomas Hellström (Intel) >>>>>> wrote: >>>>>>> Hi, >>>>>>> >>>>>>> On 7/9/20 2:33 PM, Daniel Vetter wrote: >>>>>>>> Comes up every few years, gets somewhat tedious to discuss, let's >>>>>>>> write this down once and for all. >>>>>>>> >>>>>>>> What I'm not sure about is whether the text should be more >>>>>>>> explicit in >>>>>>>> flat out mandating the amdkfd eviction fences for long running >>>>>>>> compute >>>>>>>> workloads or workloads where userspace fencing is allowed. >>>>>>> Although (in my humble opinion) it might be possible to completely >>>>>>> untangle >>>>>>> kernel-introduced fences for resource management and dma-fences >>>>>>> used for >>>>>>> completion- and dependency tracking and lift a lot of restrictions >>>>>>> for the >>>>>>> dma-fences, including prohibiting infinite ones, I think this >>>>>>> makes sense >>>>>>> describing the current state. >>>>>> Yeah I think a future patch needs to type up how we want to make that >>>>>> happen (for some cross driver consistency) and what needs to be >>>>>> considered. Some of the necessary parts are already there (with >>>>>> like the >>>>>> preemption fences amdkfd has as an example), but I think some clear >>>>>> docs >>>>>> on what's required from both hw, drivers and userspace would be really >>>>>> good. >>>>> I'm currently writing that up, but probably still need a few days >>>>> for this. >>>> Great! I put down some (very) initial thoughts a couple of weeks ago >>>> building on eviction fences for various hardware complexity levels here: >>>> >>>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.freedesktop.org%2Fthomash%2Fdocs%2F-%2Fblob%2Fmaster%2FUntangling%2520dma-fence%2520and%2520memory%2520allocation.odt&data=02%7C01%7Cchristian.koenig%40amd.com%7C8978bbd7823e4b41663708d82d52add3%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637309180424312390&sdata=tTxx2vfzfwLM1IBJSqqAZRw1604R%2F0bI3MwN1%2FBf2VQ%3D&reserved=0 >>>> >>> I don't think that this will ever be possible. >>> >>> See that Daniel describes in his text is that indefinite fences are a >>> bad idea for memory management, and I think that this is a fixed fact. >>> >>> In other words the whole concept of submitting work to the kernel >>> which depends on some user space interaction doesn't work and never will. >> Well the idea here is that memory management will *never* depend on >> indefinite fences: As soon as someone waits on a memory manager fence >> (be it eviction, shrinker or mmu notifier) it breaks out of any >> dma-fence dependencies and /or user-space interaction. The text tries to >> describe what's required to be able to do that (save for non-preemptible >> gpus where someone submits a forever-running shader). > Yeah I think that part of your text is good to describe how to > untangle memory fences from synchronization fences given how much the > hw can do. > >> So while I think this is possible (until someone comes up with a case >> where it wouldn't work of course), I guess Daniel has a point in that it >> won't happen because of inertia and there might be better options. > Yeah it's just I don't see much chance for splitting dma-fence itself. > That's also why I'm not positive on the "no hw preemption, only > scheduler" case: You still have a dma_fence for the batch itself, > which means still no userspace controlled synchronization or other > form of indefinite batches allowed. So not getting us any closer to > enabling the compute use cases people want. Yes, we can't do magic. As soon as an indefinite batch makes it to such hardware we've lost. But since we can break out while the batch is stuck in the scheduler waiting, what I believe we *can* do with this approach is to avoid deadlocks due to locally unknown dependencies, which has some bearing on this documentation patch, and also to allow memory allocation in dma-fence (not memory-fence) critical sections, like gpu fault- and error handlers without resorting to using memory pools. But again. I'm not saying we should actually implement this. Better to consider it and reject it than not consider it at all. /Thomas