From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3BDC9C433EF for ; Fri, 24 Jun 2022 20:33:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231423AbiFXUdC (ORCPT ); Fri, 24 Jun 2022 16:33:02 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42946 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231232AbiFXUdA (ORCPT ); Fri, 24 Jun 2022 16:33:00 -0400 Received: from mail-wr1-x429.google.com (mail-wr1-x429.google.com [IPv6:2a00:1450:4864:20::429]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0A7C9E53 for ; Fri, 24 Jun 2022 13:32:58 -0700 (PDT) Received: by mail-wr1-x429.google.com with SMTP id e7so4539986wrc.13 for ; Fri, 24 Jun 2022 13:32:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=JOawYYsMmXVXjmJlEQmmHixFMSNq4rwAUlbsY6fSBgM=; b=tg7SyNOOcpVv7QNLGjiD5KPi52/+81+/GdcF4vMbyPBIUjBBrCNv+tyegVY20uye/I 8Vyz/JGoeTKB1w5QEdFMlq0X9w98WbqAaklpSLN4EHbesVPHavlkpHILt1ZfuYLKpZ8W QWm+Maw4GPgy2ApKPUsawPdD52CVDNryzSvuJBVG/RGE1Y030n422Z6wuG4tFFvBcR27 LjOjXsrh4klnn1DcCwwhoI0t8TTxE908V6t7zZjTqlwnAGdcuAOVwj1XL6eJ5KD7NEWR SUf/6y2OeRLz1/Liz1zYe+r9C1qmHOr6YxgkPxXUsGyjo51BPVTKh4OJcFsWrx7rdii2 KEPw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=JOawYYsMmXVXjmJlEQmmHixFMSNq4rwAUlbsY6fSBgM=; b=3sEpT1H7T3WOesORQahu0PlNheo8R65BT4CDPpwropnlzA9CXJKsx10NofDlKXF9p+ +kHkyjnpG2K9zJeBccYpEdwmtwvaWphmwnPUnULu/tR8oe/DDqvaZtxm2cS+dRj6UGcU xJ/bZtr4u5yPsL0uO4xSqtrl5Y95iuhw2LVAaUzFqos2fdTKKxjpRZB3KKbFMl6jp02E /zag7Lc4WbP24vbGcuqOIyMEzyyS198Tr7Ovr3tam1kERZqNlpi5U77vLCWJpbiqznN+ Qpj5vXVKbiapBT5pgOa/2Gx+drxqklHd40E+QwwLhRDIDpeWs3dX787vWn8MPEN0oKWE OdFg== X-Gm-Message-State: AJIora9GBPzJTDKpeA9Z1k+ee1w6UMBQKWx1AKMqoufAm1yy4Mpo51GQ reE6aqhUmFMj/X2V2cLZoJh0cMVrWF0F/KjuS3XF X-Google-Smtp-Source: AGRyM1sLPS9VvP6ADwx2J+StFzJT3dvaf4MBsF0swSWZC7Xkp/3biNCnrEVvva/rSqr0ebVtvt4hYokyoFIu+/3DmQ4= X-Received: by 2002:adf:efd2:0:b0:21b:91ae:68ca with SMTP id i18-20020adfefd2000000b0021b91ae68camr833768wrp.514.1656102776455; Fri, 24 Jun 2022 13:32:56 -0700 (PDT) MIME-Version: 1.0 References: <20220510235653.933868-1-tjmercier@google.com> <3365cd1d750e84fedc8e75d646a77ffd85619d35.camel@ndufresne.ca> <81026ef07c1ce20f8673b75b17bab79a2b39c548.camel@ndufresne.ca> In-Reply-To: From: John Stultz Date: Fri, 24 Jun 2022 13:32:45 -0700 Message-ID: Subject: Re: [PATCH v7 0/6] Proposal for a GPU cgroup controller To: "T.J. Mercier" , Tejun Heo , Nicolas Dufresne , Zefan Li , Johannes Weiner , Jonathan Corbet , Greg Kroah-Hartman , =?UTF-8?B?QXJ2ZSBIasO4bm5ldsOlZw==?= , Todd Kjos , Martijn Coenen , Joel Fernandes , Christian Brauner , Hridya Valsaraju , Suren Baghdasaryan , Sumit Semwal , =?UTF-8?Q?Christian_K=C3=B6nig?= , Benjamin Gaignard , Liam Mark , Laura Abbott , Brian Starkey , John Stultz , Shuah Khan , John Stultz , Carlos Llamas , Kalesh Singh , Kenny.Ho@amd.com, =?UTF-8?Q?Michal_Koutn=C3=BD?= , Shuah Khan , kernel-team@android.com, cgroups@vger.kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org, linaro-mm-sig@lists.linaro.org, linux-kselftest@vger.kernel.org Cc: Daniel Vetter Content-Type: text/plain; charset="UTF-8" Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Fri, Jun 24, 2022 at 1:17 PM Daniel Vetter wrote: > > On Wed, Jun 15, 2022 at 10:31:21AM -0700, T.J. Mercier wrote: > > On Fri, May 20, 2022 at 9:25 AM T.J. Mercier wrote: > > > > > > On Fri, May 20, 2022 at 12:47 AM Tejun Heo wrote: > > > > > > > > Hello, > > > > > > > > On Tue, May 17, 2022 at 04:30:29PM -0700, T.J. Mercier wrote: > > > > > Thanks for your suggestion. This almost works. "dmabuf" as a key could > > > > > work, but I'd actually like to account for each heap. Since heaps can > > > > > be dynamically added, I can't accommodate every potential heap name by > > > > > hardcoding registrations in the misc controller. > > > > > > > > On its own, that's a pretty weak reason to be adding a separate gpu > > > > controller especially given that it doesn't really seem to be one with > > > > proper abstractions for gpu resources. We don't want to keep adding random > > > > keys to misc controller but can definitely add limited flexibility. What > > > > kind of keys do you need? > > > > > > > Well the dmabuf-from-heaps component of this is the initial use case. > > > I was envisioning we'd have additional keys as discussed here: > > > https://lore.kernel.org/lkml/20220328035951.1817417-1-tjmercier@google.com/T/#m82e5fe9d8674bb60160701e52dae4356fea2ddfa > > > So we'd end up with a well-defined core set of keys like "system", and > > > then drivers would be free to use their own keys for their own unique > > > purposes which could be complementary or orthogonal to the core set. > > > Yesterday I was talking with someone who is interested in limiting gpu > > > cores and bus IDs in addition to gpu memory. How to define core keys > > > is the part where it looks like there's trouble. > > > > > > For my use case it would be sufficient to have current and maximum > > > values for an arbitrary number of keys - one per heap. So the only > > > part missing from the misc controller (for my use case) is the ability > > > to register a new key at runtime as heaps are added. Instead of > > > keeping track of resources with enum misc_res_type, requesting a > > > resource handle/ID from the misc controller at runtime is what I think > > > would be required instead. > > > > > Quick update: I'm going to make an attempt to modify the misc > > controller to support a limited amount of dynamic resource > > registration/tracking in place of the new controller in this series. > > > > Thanks everyone for the feedback. > > Somehow I missed this entire chain here. > > I'm not a fan, because I'm kinda hoping we could finally unify gpu memory > account. Atm everyone just adds their one-off solution in a random corner: > - total tracking in misc cgroup controller > - dma-buf sysfs files (except apparently too slow so it'll get deleted > again) > - random other stuff on open device files os OOM killer can see it > > This doesn't look good. But I also think one could see it as "gpu memory" is the drm subsystem doing the same thing (in that it's artificially narrow to gpus). It seems we need something to account for buffers allocated by drivers, no matter which subsystem it was in (drm, v4l2, or networking or whatever). thanks -john From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E240CC43334 for ; Sun, 26 Jun 2022 17:48:05 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id A6C8310EB4B; Sun, 26 Jun 2022 17:47:59 +0000 (UTC) Received: from mail-wr1-x42f.google.com (mail-wr1-x42f.google.com [IPv6:2a00:1450:4864:20::42f]) by gabe.freedesktop.org (Postfix) with ESMTPS id 2100D10F2BA for ; Fri, 24 Jun 2022 20:32:58 +0000 (UTC) Received: by mail-wr1-x42f.google.com with SMTP id i10so4651778wrc.0 for ; Fri, 24 Jun 2022 13:32:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=JOawYYsMmXVXjmJlEQmmHixFMSNq4rwAUlbsY6fSBgM=; b=tg7SyNOOcpVv7QNLGjiD5KPi52/+81+/GdcF4vMbyPBIUjBBrCNv+tyegVY20uye/I 8Vyz/JGoeTKB1w5QEdFMlq0X9w98WbqAaklpSLN4EHbesVPHavlkpHILt1ZfuYLKpZ8W QWm+Maw4GPgy2ApKPUsawPdD52CVDNryzSvuJBVG/RGE1Y030n422Z6wuG4tFFvBcR27 LjOjXsrh4klnn1DcCwwhoI0t8TTxE908V6t7zZjTqlwnAGdcuAOVwj1XL6eJ5KD7NEWR SUf/6y2OeRLz1/Liz1zYe+r9C1qmHOr6YxgkPxXUsGyjo51BPVTKh4OJcFsWrx7rdii2 KEPw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=JOawYYsMmXVXjmJlEQmmHixFMSNq4rwAUlbsY6fSBgM=; b=lqY5j1v3XnuocWEee3gK2FM2M9og3ds78kWcQlXVF5S0irUNa1M3F2yRbo9lbDn2EE iaGzokXI/ZPlQROM4/JE9pY4+nqtdRvOzVaJewK3dssFmYUyMuFye/7+GJYHf+JsfohK DfblbII8Qb+GxWRqR2VGcEwN1eJVuvaQl7F1n2gPXHz4Fpx+Nh2Dn9F29kmUVtuMPEOX ML0u23wNd3lbZEl4Rz+1p7nGEykxAiQQmBvMdSef8ssh9UkWzbxAEHRCfuGK2C9O4Yzn i7usR4VK5nB4ib34iNcKo+jUj8+e6naJRePOrOmV+ozIv/VciL70V5lHhw7O+l1FMXm0 SViw== X-Gm-Message-State: AJIora+Y0aZtx1I0vfAASP2HGo7AClMmDxtDt2tOEfe8BCcA+BAeSCWq qsxo+pCy7mgR7BU4kZkksqR6aYpfmO5dfUrMxa6F X-Google-Smtp-Source: AGRyM1sLPS9VvP6ADwx2J+StFzJT3dvaf4MBsF0swSWZC7Xkp/3biNCnrEVvva/rSqr0ebVtvt4hYokyoFIu+/3DmQ4= X-Received: by 2002:adf:efd2:0:b0:21b:91ae:68ca with SMTP id i18-20020adfefd2000000b0021b91ae68camr833768wrp.514.1656102776455; Fri, 24 Jun 2022 13:32:56 -0700 (PDT) MIME-Version: 1.0 References: <20220510235653.933868-1-tjmercier@google.com> <3365cd1d750e84fedc8e75d646a77ffd85619d35.camel@ndufresne.ca> <81026ef07c1ce20f8673b75b17bab79a2b39c548.camel@ndufresne.ca> In-Reply-To: From: John Stultz Date: Fri, 24 Jun 2022 13:32:45 -0700 Message-ID: Subject: Re: [PATCH v7 0/6] Proposal for a GPU cgroup controller To: "T.J. Mercier" , Tejun Heo , Nicolas Dufresne , Zefan Li , Johannes Weiner , Jonathan Corbet , Greg Kroah-Hartman , =?UTF-8?B?QXJ2ZSBIasO4bm5ldsOlZw==?= , Todd Kjos , Martijn Coenen , Joel Fernandes , Christian Brauner , Hridya Valsaraju , Suren Baghdasaryan , Sumit Semwal , =?UTF-8?Q?Christian_K=C3=B6nig?= , Benjamin Gaignard , Liam Mark , Laura Abbott , Brian Starkey , John Stultz , Shuah Khan , John Stultz , Carlos Llamas , Kalesh Singh , Kenny.Ho@amd.com, =?UTF-8?Q?Michal_Koutn=C3=BD?= , Shuah Khan , kernel-team@android.com, cgroups@vger.kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org, linaro-mm-sig@lists.linaro.org, linux-kselftest@vger.kernel.org Content-Type: text/plain; charset="UTF-8" X-Mailman-Approved-At: Sun, 26 Jun 2022 17:47:58 +0000 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" On Fri, Jun 24, 2022 at 1:17 PM Daniel Vetter wrote: > > On Wed, Jun 15, 2022 at 10:31:21AM -0700, T.J. Mercier wrote: > > On Fri, May 20, 2022 at 9:25 AM T.J. Mercier wrote: > > > > > > On Fri, May 20, 2022 at 12:47 AM Tejun Heo wrote: > > > > > > > > Hello, > > > > > > > > On Tue, May 17, 2022 at 04:30:29PM -0700, T.J. Mercier wrote: > > > > > Thanks for your suggestion. This almost works. "dmabuf" as a key could > > > > > work, but I'd actually like to account for each heap. Since heaps can > > > > > be dynamically added, I can't accommodate every potential heap name by > > > > > hardcoding registrations in the misc controller. > > > > > > > > On its own, that's a pretty weak reason to be adding a separate gpu > > > > controller especially given that it doesn't really seem to be one with > > > > proper abstractions for gpu resources. We don't want to keep adding random > > > > keys to misc controller but can definitely add limited flexibility. What > > > > kind of keys do you need? > > > > > > > Well the dmabuf-from-heaps component of this is the initial use case. > > > I was envisioning we'd have additional keys as discussed here: > > > https://lore.kernel.org/lkml/20220328035951.1817417-1-tjmercier@google.com/T/#m82e5fe9d8674bb60160701e52dae4356fea2ddfa > > > So we'd end up with a well-defined core set of keys like "system", and > > > then drivers would be free to use their own keys for their own unique > > > purposes which could be complementary or orthogonal to the core set. > > > Yesterday I was talking with someone who is interested in limiting gpu > > > cores and bus IDs in addition to gpu memory. How to define core keys > > > is the part where it looks like there's trouble. > > > > > > For my use case it would be sufficient to have current and maximum > > > values for an arbitrary number of keys - one per heap. So the only > > > part missing from the misc controller (for my use case) is the ability > > > to register a new key at runtime as heaps are added. Instead of > > > keeping track of resources with enum misc_res_type, requesting a > > > resource handle/ID from the misc controller at runtime is what I think > > > would be required instead. > > > > > Quick update: I'm going to make an attempt to modify the misc > > controller to support a limited amount of dynamic resource > > registration/tracking in place of the new controller in this series. > > > > Thanks everyone for the feedback. > > Somehow I missed this entire chain here. > > I'm not a fan, because I'm kinda hoping we could finally unify gpu memory > account. Atm everyone just adds their one-off solution in a random corner: > - total tracking in misc cgroup controller > - dma-buf sysfs files (except apparently too slow so it'll get deleted > again) > - random other stuff on open device files os OOM killer can see it > > This doesn't look good. But I also think one could see it as "gpu memory" is the drm subsystem doing the same thing (in that it's artificially narrow to gpus). It seems we need something to account for buffers allocated by drivers, no matter which subsystem it was in (drm, v4l2, or networking or whatever). thanks -john