Well in my opinion exposing it through fdinfo turned out to be a really clean approach. It describes exactly the per file descriptor information we need. Making that device driver independent is potentially useful as well. Regards, Christian. Am 14.05.21 um 09:22 schrieb Nieto, David M: > > [AMD Official Use Only - Internal Distribution Only] > > > We had entertained the idea of exposing the processes as sysfs nodes > as you proposed, but we had concerns about exposing process info in > there, especially since /proc already exists for that purpose. > > I think if you were to follow that approach, we could have tools like > top that support exposing GPU engine usage. > ------------------------------------------------------------------------ > *From:* Alex Deucher > *Sent:* Thursday, May 13, 2021 10:58 PM > *To:* Tvrtko Ursulin ; Nieto, David M > ; Koenig, Christian > *Cc:* Intel Graphics Development ; > Maling list - DRI developers ; Daniel > Vetter > *Subject:* Re: [PATCH 0/7] Per client engine busyness > + David, Christian > > On Thu, May 13, 2021 at 12:41 PM Tvrtko Ursulin > wrote: > > > > > > Hi, > > > > On 13/05/2021 16:48, Alex Deucher wrote: > > > On Thu, May 13, 2021 at 7:00 AM Tvrtko Ursulin > > > wrote: > > >> > > >> From: Tvrtko Ursulin > > >> > > >> Resurrect of the previosuly merged per client engine busyness > patches. In a > > >> nutshell it enables intel_gpu_top to be more top(1) like useful > and show not > > >> only physical GPU engine usage but per process view as well. > > >> > > >> Example screen capture: > > >> > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > > >> intel-gpu-top -  906/ 955 MHz;    0% RC6; 5.30 Watts;      933 irqs/s > > >> > > >>        IMC reads:     4414 MiB/s > > >>       IMC writes:     3805 MiB/s > > >> > > >>            ENGINE BUSY                                      > MI_SEMA MI_WAIT > > >>       Render/3D/0   93.46% |████████████████████████████████▋  > |      0%      0% > > >>         Blitter/0    0.00% |                                   > |      0%      0% > > >>           Video/0    0.00% |                                   > |      0%      0% > > >>    VideoEnhance/0    0.00% |                                   > |      0%      0% > > >> > > >>    PID            NAME  Render/3D Blitter        Video      > VideoEnhance > > >>   2733       neverball |██████▌ ||            ||            > ||            | > > >>   2047            Xorg |███▊ ||            ||            > ||            | > > >>   2737        glxgears |█▍ ||            ||            > ||            | > > >>   2128           xfwm4 | ||            ||            ||            | > > >>   2047            Xorg | ||            ||            ||            | > > >> > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > > >> > > >> Internally we track time spent on engines for each struct > intel_context, both > > >> for current and past contexts belonging to each open DRM file. > > >> > > >> This can serve as a building block for several features from the > wanted list: > > >> smarter scheduler decisions, getrusage(2)-like per-GEM-context > functionality > > >> wanted by some customers, setrlimit(2) like controls, cgroups > controller, > > >> dynamic SSEU tuning, ... > > >> > > >> To enable userspace access to the tracked data, we expose time > spent on GPU per > > >> client and per engine class in sysfs with a hierarchy like the below: > > >> > > >>          # cd /sys/class/drm/card0/clients/ > > >>          # tree > > >>          . > > >>          ├── 7 > > >>          │   ├── busy > > >>          │   │   ├── 0 > > >>          │   │   ├── 1 > > >>          │   │   ├── 2 > > >>          │   │   └── 3 > > >>          │   ├── name > > >>          │   └── pid > > >>          ├── 8 > > >>          │   ├── busy > > >>          │   │   ├── 0 > > >>          │   │   ├── 1 > > >>          │   │   ├── 2 > > >>          │   │   └── 3 > > >>          │   ├── name > > >>          │   └── pid > > >>          └── 9 > > >>              ├── busy > > >>              │   ├── 0 > > >>              │   ├── 1 > > >>              │   ├── 2 > > >>              │   └── 3 > > >>              ├── name > > >>              └── pid > > >> > > >> Files in 'busy' directories are numbered using the engine class > ABI values and > > >> they contain accumulated nanoseconds each client spent on engines > of a > > >> respective class. > > > > > > We did something similar in amdgpu using the gpu scheduler.  We then > > > expose the data via fdinfo.  See > > > > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcgit.freedesktop.org%2Fdrm%2Fdrm-misc%2Fcommit%2F%3Fid%3D1774baa64f9395fa884ea9ed494bcb043f3b83f5&data=04%7C01%7CDavid.Nieto%40amd.com%7C5e3c05578ef14be3692508d9169d55bf%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637565687273144615%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=mt1EIL%2Fc9pHCXR%2FYSd%2BTr1e64XHoeYcdQ2cYufJ%2FcYQ%3D&reserved=0 > > > > > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcgit.freedesktop.org%2Fdrm%2Fdrm-misc%2Fcommit%2F%3Fid%3D874442541133f78c78b6880b8cc495bab5c61704&data=04%7C01%7CDavid.Nieto%40amd.com%7C5e3c05578ef14be3692508d9169d55bf%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637565687273144615%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=%2F3zMGw0LPTC1kG4NebTwUPTx7QCtEyw%2B4JToXDK5QXI%3D&reserved=0 > > > > > Interesting! > > > > Is yours wall time or actual GPU time taking preemption and such into > > account? Do you have some userspace tools parsing this data and how to > > do you client discovery? Presumably there has to be a better way that > > going through all open file descriptors? > > Wall time.  It uses the fences in the scheduler to calculate engine > time.  We have some python scripts to make it look pretty, but mainly > just reading the files directly.  If you know the process, you can > look it up in procfs. > > > > > Our implementation was merged in January but Daniel took it out recently > > because he wanted to have discussion about a common vendor framework for > > this whole story on dri-devel. I think. +Daniel to comment. > > > > I couldn't find the patch you pasted on the mailing list to see if there > > was any such discussion around your version. > > It was on the amd-gfx mailing list. > > Alex > > > > > Regards, > > > > Tvrtko > > > > > > > > Alex > > > > > > > > >> > > >> Tvrtko Ursulin (7): > > >>    drm/i915: Expose list of clients in sysfs > > >>    drm/i915: Update client name on context create > > >>    drm/i915: Make GEM contexts track DRM clients > > >>    drm/i915: Track runtime spent in closed and unreachable GEM > contexts > > >>    drm/i915: Track all user contexts per client > > >>    drm/i915: Track context current active time > > >>    drm/i915: Expose per-engine client busyness > > >> > > >> drivers/gpu/drm/i915/Makefile                 |   5 +- > > >> drivers/gpu/drm/i915/gem/i915_gem_context.c   |  61 ++- > > >> .../gpu/drm/i915/gem/i915_gem_context_types.h |  16 +- > > >> drivers/gpu/drm/i915/gt/intel_context.c       |  27 +- > > >> drivers/gpu/drm/i915/gt/intel_context.h       |  15 +- > > >> drivers/gpu/drm/i915/gt/intel_context_types.h |  24 +- > > >> .../drm/i915/gt/intel_execlists_submission.c  |  23 +- > > >> .../gpu/drm/i915/gt/intel_gt_clock_utils.c    |   4 + > > >> drivers/gpu/drm/i915/gt/intel_lrc.c           |  27 +- > > >> drivers/gpu/drm/i915/gt/intel_lrc.h           |  24 ++ > > >> drivers/gpu/drm/i915/gt/selftest_lrc.c        |  10 +- > > >> drivers/gpu/drm/i915/i915_drm_client.c        | 365 > ++++++++++++++++++ > > >> drivers/gpu/drm/i915/i915_drm_client.h        | 123 ++++++ > > >> drivers/gpu/drm/i915/i915_drv.c               |   6 + > > >> drivers/gpu/drm/i915/i915_drv.h               |   5 + > > >> drivers/gpu/drm/i915/i915_gem.c               |  21 +- > > >> drivers/gpu/drm/i915/i915_gpu_error.c         |  31 +- > > >> drivers/gpu/drm/i915/i915_gpu_error.h         |   2 +- > > >> drivers/gpu/drm/i915/i915_sysfs.c             |   8 + > > >>   19 files changed, 716 insertions(+), 81 deletions(-) > > >>   create mode 100644 drivers/gpu/drm/i915/i915_drm_client.c > > >>   create mode 100644 drivers/gpu/drm/i915/i915_drm_client.h > > >> > > >> -- > > >> 2.30.2 > > >>