From mboxrd@z Thu Jan 1 00:00:00 1970 From: Yiwei Zhang Subject: Re: Proposal to report GPU private memory allocations with sysfs nodes [plain text version] Date: Thu, 31 Oct 2019 09:59:18 -0700 Message-ID: References: <20191028152602.GA5057@redhat.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============1547981709==" Return-path: Received: from mail-lj1-x230.google.com (mail-lj1-x230.google.com [IPv6:2a00:1450:4864:20::230]) by gabe.freedesktop.org (Postfix) with ESMTPS id 089B36F3B2 for ; Thu, 31 Oct 2019 16:59:32 +0000 (UTC) Received: by mail-lj1-x230.google.com with SMTP id x9so2587112lji.3 for ; Thu, 31 Oct 2019 09:59:31 -0700 (PDT) In-Reply-To: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" To: Kenny Ho Cc: Alistair Delva , Prahlad Kilambi , dri-devel , Jerome Glisse , Sean Paul , kraxel@redhat.com, Chris Forbes , kernel-team@android.com List-Id: dri-devel@lists.freedesktop.org --===============1547981709== Content-Type: multipart/alternative; boundary="000000000000d30311059637c528" --000000000000d30311059637c528 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi Kenny, Thanks for the info. Do you mind forwarding the existing discussion to me or have me cc'ed in that thread? Best, Yiwei On Wed, Oct 30, 2019 at 10:23 PM Kenny Ho wrote: > Hi Yiwei, > > I am not sure if you are aware, there is an ongoing RFC on adding drm > support in cgroup for the purpose of resource tracking. One of the > resource is GPU memory. It's not exactly the same as what you are > proposing (it doesn't track API usage, but it tracks the type of GPU > memory from kmd perspective) but perhaps it would be of interest to > you. There are no consensus on it at this point. > > (sorry for being late to the discussion. I only noticed this thread > when one of the email got lucky and escape the spam folder.) > > Regards, > Kenny > > On Wed, Oct 30, 2019 at 4:14 AM Yiwei Zhang wrote: > > > > Hi Jerome and all folks, > > > > In addition to my last reply, I just wanna get some more information > regarding this on the upstream side. > > > > 1. Do you think this(standardize a way to report GPU private > allocations) is going to be a useful thing on the upstream as well? It > grants a lot benefits for Android, but I'd like to get an idea for the > non-Android world. > > > > 2. There might be some worries that upstream kernel driver has no idea > regarding the API. However, to achieve good fidelity around memory > reporting, we'd have to pass down certain metadata which is known only by > the userland. Consider this use case: on the upstream side, freedreno for > example, some memory buffer object(BO) during its own lifecycle could > represent totally different things, and kmd is not aware of that. When we= 'd > like to take memory snapshots at certain granularity, we have to know wha= t > that buffer represents so that the snapshot can be meaningful and useful. > > > > If we just keep this Android specific, I'd worry some day the upstream > has standardized a way to report this and Android vendors have to take > extra efforts to migrate over. This is one of the main reasons we'd like = to > do this on the upstream side. > > > > Timeline wise, Android has explicit deadlines for the next release and > we have to push hard towards those. Any prompt responses are very much > appreciated! > > > > Best regards, > > Yiwei > > > > On Mon, Oct 28, 2019 at 11:33 AM Yiwei Zhang wrote= : > >> > >> On Mon, Oct 28, 2019 at 8:26 AM Jerome Glisse > wrote: > >>> > >>> On Fri, Oct 25, 2019 at 11:35:32AM -0700, Yiwei Zhang wrote: > >>> > Hi folks, > >>> > > >>> > This is the plain text version of the previous email in case that w= as > >>> > considered as spam. > >>> > > >>> > --- Background --- > >>> > On the downstream Android, vendors used to report GPU private memor= y > >>> > allocations with debugfs nodes in their own formats. However, > debugfs nodes > >>> > are getting deprecated in the next Android release. > >>> > >>> Maybe explain why it is useful first ? > >> > >> > >> Memory is precious on Android mobile platforms. Apps using a large > amount of > >> memory, games, tend to maintain a table for the memory on different > devices with > >> different prediction models. Private gpu memory allocations is > currently semi-blind > >> to the apps and the platform as well. > >> > >> By having the data, the platform can do: > >> (1) GPU memory profiling as part of the huge Android profiler in > progress. > >> (2) Android system health team can enrich the performance test coverag= e. > >> (3) We can collect filed metrics to detect any regression on the gpu > private memory > >> allocations in the production population. > >> (4) Shell user can easily dump the allocations in a uniform way across > vendors. > >> (5) Platform can feed the data to the apps so that apps can do memory > allocations > >> in a more predictable way. > >> > >>> > >>> > > >>> > --- Proposal --- > >>> > We are taking the chance to unify all the vendors to migrate their > existing > >>> > debugfs nodes into a standardized sysfs node structure. Then the > platform > >>> > is able to do a bunch of useful things: memory profiling, system > health > >>> > coverage, field metrics, local shell dump, in-app api, etc. This > proposal > >>> > is better served upstream as all GPU vendors can standardize a gpu > memory > >>> > structure and reduce fragmentation across Android and Linux that > clients > >>> > can rely on. > >>> > > >>> > --- Detailed design --- > >>> > The sysfs node structure looks like below: > >>> > /sys/devices/// > >>> > e.g. "/sys/devices/mali0/gpu_mem/606/gl_buffer" and the gl_buffer i= s > a node > >>> > having the comma separated size values: "4096,81920,...,4096". > >>> > >>> How does kernel knows what API the allocation is use for ? With the > >>> open source driver you never specify what API is creating a gem objec= t > >>> (opengl, vulkan, ...) nor what purpose (transient, shader, ...). > >> > >> > >> Oh, is this a hard requirement for the open source drivers to not > bookkeep any > >> data from userland? I think the API is just some additional metadata > passed down. > >> > >>> > >>> > >>> > For the top level root, vendors can choose their own names based on > the > >>> > value of ro.gfx.sysfs.0 the vendors set. (1) For the multiple gpu > driver > >>> > cases, we can use ro.gfx.sysfs.1, ro.gfx.sysfs.2 for the 2nd and 3r= d > KMDs. > >>> > (2) It's also allowed to put some sub-dir for example "kgsl/gpu_mem= " > or > >>> > "mali0/gpu_mem" in the ro.gfx.sysfs. property if the root > name > >>> > under /sys/devices/ is already created and used for other purposes. > >>> > >>> On one side you want to standardize on the other you want to give > >>> complete freedom on the top level naming scheme. I would rather see a > >>> consistent naming scheme (ie something more restraint and with little > >>> place for interpration by individual driver) > >> > >> > >> Thanks for commenting on this. We definitely need some suggestions on > the root > >> directory. In the multi-gpu case on desktop, is there some existing > consumer to > >> query "some data" from all the GPUs? How does the tool find all GPUs a= nd > >> differentiate between them? Is this already standardized? > >> > >>> > For the 2nd level "pid", there are usually just a couple of them pe= r > >>> > snapshot, since we only takes snapshot for the active ones. > >>> > >>> ? Do not understand here, you can have any number of applications wit= h > >>> GPU objects ? And thus there is no bound on the number of PID. Please > >>> consider desktop too, i do not know what kind of limitation android > >>> impose. > >> > >> > >> We are only interested in tracking *active* GPU private allocations. S= o > yes, any > >> application currently holding an active GPU context will probably has = a > node here. > >> Since we want to do profiling for specific apps, the data has to be pe= r > application > >> based. I don't get your concerns here. If it's about the tracking > overhead, it's rare > >> to see tons of application doing private gpu allocations at the same > time. Could > >> you help elaborate a bit? > >> > >>> > For the 3rd level "type_name", the type name will be one of the GPU > memory > >>> > object types in lower case, and the value will be a comma separated > >>> > sequence of size values for all the allocations under that specific > type. > >>> > > >>> > We especially would like some comments on this part. For the GPU > memory > >>> > object types, we defined 9 different types for Android: > >>> > (1) UNKNOWN // not accounted for in any other category > >>> > (2) SHADER // shader binaries > >>> > (3) COMMAND // allocations which have a lifetime similar to a > >>> > VkCommandBuffer > >>> > (4) VULKAN // backing for VkDeviceMemory > >>> > (5) GL_TEXTURE // GL Texture and RenderBuffer > >>> > (6) GL_BUFFER // GL Buffer > >>> > (7) QUERY // backing for query > >>> > (8) DESCRIPTOR // allocations which have a lifetime similar to a > >>> > VkDescriptorSet > >>> > (9) TRANSIENT // random transient things that the driver needs > >>> > > >>> > We are wondering if those type enumerations make sense to the > upstream side > >>> > as well, or maybe we just deal with our own different type sets. Cu= z > on the > >>> > Android side, we'll just read those nodes named after the types we > defined > >>> > in the sysfs node structure. > >>> > >>> See my above point of open source driver and kernel being unaware > >>> of the allocation purpose and use. > >>> > >>> Cheers, > >>> J=C3=A9r=C3=B4me > >>> > >> > >> Many thanks for the reply! > >> Yiwei > > > > _______________________________________________ > > dri-devel mailing list > > dri-devel@lists.freedesktop.org > > https://lists.freedesktop.org/mailman/listinfo/dri-devel > --000000000000d30311059637c528 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi=C2=A0Kenny,

Thanks for the info. Do = you mind forwarding the existing discussion to me or have me cc'ed in t= hat thread?

Best,
Yiwei

<= div class=3D"gmail_quote">
On Wed, Oct= 30, 2019 at 10:23 PM Kenny Ho <y2k= enny@gmail.com> wrote:
Hi Yiwei,

I am not sure if you are aware, there is an ongoing RFC on adding drm
support in cgroup for the purpose of resource tracking.=C2=A0 One of the resource is GPU memory.=C2=A0 It's not exactly the same as what you are=
proposing (it doesn't track API usage, but it tracks the type of GPU memory from kmd perspective) but perhaps it would be of interest to
you.=C2=A0 There are no consensus on it at this point.

(sorry for being late to the discussion.=C2=A0 I only noticed this thread when one of the email got lucky and escape the spam folder.)

Regards,
Kenny

On Wed, Oct 30, 2019 at 4:14 AM Yiwei Zhang <zzyiwei@google.com> wrote:
>
> Hi Jerome and all folks,
>
> In addition to my last reply, I just wanna get some more information r= egarding this on the upstream side.
>
> 1. Do you think this(standardize a way to report GPU private allocatio= ns) is going to be a useful thing on the upstream as well? It grants a lot = benefits for Android, but I'd like to get an idea for the non-Android w= orld.
>
> 2. There might be some worries that upstream kernel driver has no idea= regarding the API. However, to achieve good fidelity around memory reporti= ng, we'd have to pass down certain metadata which is known only by the = userland. Consider this use case: on the upstream side, freedreno for examp= le, some memory buffer object(BO) during its own lifecycle could represent = totally different things, and kmd is not aware of that. When we'd like = to take memory snapshots at certain granularity, we have to know what that = buffer represents so that the snapshot can be meaningful and useful.
>
> If we just keep this Android specific, I'd worry some day the upst= ream has standardized a way to report this and Android vendors have to take= extra efforts to migrate over. This is one of the main reasons we'd li= ke to do this on the upstream side.
>
> Timeline wise, Android has explicit deadlines for the next release and= we have to push hard towards those. Any prompt responses are very much app= reciated!
>
> Best regards,
> Yiwei
>
> On Mon, Oct 28, 2019 at 11:33 AM Yiwei Zhang <zzyiwei@google.com> wrote:
>>
>> On Mon, Oct 28, 2019 at 8:26 AM Jerome Glisse <jglisse@redhat.com> wrote: >>>
>>> On Fri, Oct 25, 2019 at 11:35:32AM -0700, Yiwei Zhang wrote: >>> > Hi folks,
>>> >
>>> > This is the plain text version of the previous email in c= ase that was
>>> > considered as spam.
>>> >
>>> > --- Background ---
>>> > On the downstream Android, vendors used to report GPU pri= vate memory
>>> > allocations with debugfs nodes in their own formats. Howe= ver, debugfs nodes
>>> > are getting deprecated in the next Android release.
>>>
>>> Maybe explain why it is useful first ?
>>
>>
>> Memory is precious on Android mobile platforms. Apps using a large= amount of
>> memory, games, tend to maintain a table for the memory on differen= t devices with
>> different prediction models. Private gpu memory allocations is cur= rently semi-blind
>> to the apps and the platform as well.
>>
>> By having the data, the platform can do:
>> (1) GPU memory profiling as part of the huge Android profiler in p= rogress.
>> (2) Android system health team can enrich the performance test cov= erage.
>> (3) We can collect filed metrics to detect any regression on the g= pu private memory
>> allocations in the production population.
>> (4) Shell user can easily dump the allocations in a uniform way ac= ross vendors.
>> (5) Platform can feed the data to the apps so that apps can do mem= ory allocations
>> in a more predictable way.
>>
>>>
>>> >
>>> > --- Proposal ---
>>> > We are taking the chance to unify all the vendors to migr= ate their existing
>>> > debugfs nodes into a standardized sysfs node structure. T= hen the platform
>>> > is able to do a bunch of useful things: memory profiling,= system health
>>> > coverage, field metrics, local shell dump, in-app api, et= c. This proposal
>>> > is better served upstream as all GPU vendors can standard= ize a gpu memory
>>> > structure and reduce fragmentation across Android and Lin= ux that clients
>>> > can rely on.
>>> >
>>> > --- Detailed design ---
>>> > The sysfs node structure looks like below:
>>> > /sys/devices/<ro.gfx.sysfs.0>/<pid>/<type_= name>
>>> > e.g. "/sys/devices/mali0/gpu_mem/606/gl_buffer"= and the gl_buffer is a node
>>> > having the comma separated size values: "4096,81920,= ...,4096".
>>>
>>> How does kernel knows what API the allocation is use for ? Wit= h the
>>> open source driver you never specify what API is creating a ge= m object
>>> (opengl, vulkan, ...) nor what purpose (transient, shader, ...= ).
>>
>>
>> Oh, is this a hard requirement for the open source drivers to not = bookkeep any
>> data from userland? I think the API is just some additional metada= ta passed down.
>>
>>>
>>>
>>> > For the top level root, vendors can choose their own name= s based on the
>>> > value of ro.gfx.sysfs.0 the vendors set. (1) For the mult= iple gpu driver
>>> > cases, we can use ro.gfx.sysfs.1, ro.gfx.sysfs.2 for the = 2nd and 3rd KMDs.
>>> > (2) It's also allowed to put some sub-dir for example= "kgsl/gpu_mem" or
>>> > "mali0/gpu_mem" in the ro.gfx.sysfs.<channel= > property if the root name
>>> > under /sys/devices/ is already created and used for other= purposes.
>>>
>>> On one side you want to standardize on the other you want to g= ive
>>> complete freedom on the top level naming scheme. I would rathe= r see a
>>> consistent naming scheme (ie something more restraint and with= little
>>> place for interpration by individual driver)
>>
>>
>> Thanks for commenting on this. We definitely need some suggestions= on the root
>> directory. In the multi-gpu case on desktop, is there some existin= g consumer to
>> query "some data" from all the GPUs? How does the tool f= ind all GPUs and
>> differentiate between them? Is this already standardized?
>>
>>> > For the 2nd level "pid", there are usually just= a couple of them per
>>> > snapshot, since we only takes snapshot for the active one= s.
>>>
>>> ? Do not understand here, you can have any number of applicati= ons with
>>> GPU objects ? And thus there is no bound on the number of PID.= Please
>>> consider desktop too, i do not know what kind of limitation an= droid
>>> impose.
>>
>>
>> We are only interested in tracking *active* GPU private allocation= s. So yes, any
>> application currently holding an active GPU context will probably = has a node here.
>> Since we want to do profiling for specific apps, the data has to b= e per application
>> based. I don't get your concerns here. If it's about the t= racking overhead, it's rare
>> to see tons of application doing private gpu allocations at the sa= me time. Could
>> you help elaborate a bit?
>>
>>> > For the 3rd level "type_name", the type name wi= ll be one of the GPU memory
>>> > object types in lower case, and the value will be a comma= separated
>>> > sequence of size values for all the allocations under tha= t specific type.
>>> >
>>> > We especially would like some comments on this part. For = the GPU memory
>>> > object types, we defined 9 different types for Android: >>> > (1) UNKNOWN // not accounted for in any other category >>> > (2) SHADER // shader binaries
>>> > (3) COMMAND // allocations which have a lifetime similar = to a
>>> > VkCommandBuffer
>>> > (4) VULKAN // backing for VkDeviceMemory
>>> > (5) GL_TEXTURE // GL Texture and RenderBuffer
>>> > (6) GL_BUFFER // GL Buffer
>>> > (7) QUERY // backing for query
>>> > (8) DESCRIPTOR // allocations which have a lifetime simil= ar to a
>>> > VkDescriptorSet
>>> > (9) TRANSIENT // random transient things that the driver = needs
>>> >
>>> > We are wondering if those type enumerations make sense to= the upstream side
>>> > as well, or maybe we just deal with our own different typ= e sets. Cuz on the
>>> > Android side, we'll just read those nodes named after= the types we defined
>>> > in the sysfs node structure.
>>>
>>> See my above point of open source driver and kernel being unaw= are
>>> of the allocation purpose and use.
>>>
>>> Cheers,
>>> J=C3=A9r=C3=B4me
>>>
>>
>> Many thanks for the reply!
>> Yiwei
>
> _______________________________________________
> dri-devel mailing list
> d= ri-devel@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/l= istinfo/dri-devel
--000000000000d30311059637c528-- --===============1547981709== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: inline X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KZHJpLWRldmVs IG1haWxpbmcgbGlzdApkcmktZGV2ZWxAbGlzdHMuZnJlZWRlc2t0b3Aub3JnCmh0dHBzOi8vbGlz dHMuZnJlZWRlc2t0b3Aub3JnL21haWxtYW4vbGlzdGluZm8vZHJpLWRldmVs --===============1547981709==--