Multiple DRI card detection in compositor systemd units

Multiple DRI card detection in compositor systemd units

[Hoosier, Matt]

[-- Attachment #1: Type: text/plain, Size: 2928 bytes --]

I'm trying to find a robust way to handle the phrasing of UDev rules and systemd dependencies about /dev/dri/cardXXX nodes to ensure that several compositor services running on different graphics cards of a device all can reliably start up.



First, some basic observations:



  *   The DRM subsystem in the kernel will automatically acquire DRM master status on an fd opened against a card, if no other fd open against that card is already a DRM master.
  *   DRI cards do have a human-readable identifier -- the "name" field reported by drmGetVersion() -- which can be used to reliably identify them. But this name is not exposed as a sysfs attribute on the device. So it's invisible to UDev rules.
  *   DRI cards initialize in parallel to each other, so the basic kernel names such as 'card0', 'card1', and similar are not stable.



On a system with multiple cards with each one used by a separate compositor application, this makes it a bit difficult to start each compositor only when its specific card is enumerated in UDev a bit tricky.



  *   The lack of a sysfs property exposing the drmGetVersion()->name field, makes phrasing a UDev rule directly such as this impossible:

SUBSYSTEM=="drm", ATTRS{name}=="amd", SYMLINK+="card-amd", TAG+="systemd"

  *   You could imagine writing a helper program invoked by the UDev rule:

SUBSYSTEM=="drm", PROGRAM="print-dri-symlink-name $env{DEVNAME}" SYMLINK+="%c", TAG+="systemd"

The "print-dri-symlink-name" program would be responsible for using the libdrm API to fetch out the card's "name" property.

  *   But it is not guaranteed that UDev rules fire only once for a given sysfs device. Which means that the next bullet point applies.

  *   The UDev helper program temporarily acquires DRM master status because of that built-in behavior from the DRI subsystem. If you're unlucky, this will conflict with the compositor's legitimate attempt to become DRM master.



If the systemd .device unit (e.g. dev-dri-card-amd.device) activated by the initial UDev rule firing is already up and running, and a second UDev rule instance executes at the same time that a compositor application is just starting up, there is a small but feasible chance that the UDev helper program will acquire DRM master rights during exactly the duration that the compositor is attempting its own initializer. The compositor will therefore be denied DRM master status, and fail.



I'm not really sure how the race in the final bullet point can be prevented. It seems to me that the only ironclad way to avoid it would be to perform the UDev rule matches strictly based on sysfs attributes. Has there ever been any talk about exporting the DRI card name through sysfs?



On the other hand, maybe I'm approach this problem completely wrongly. Is it a losing game to try to phrase systemd ordering constraints on /dev/dri cards?



-Matt



[-- Attachment #2: Type: text/html, Size: 13413 bytes --]

Re: Multiple DRI card detection in compositor systemd units

[Simon Ser]

Maybe try creating multiple physical seats with logind, and start each
compositor on its own seat? A physical seat is a collection of devices like
DRM nodes and evdev device files.

Also udev creates files in /dev/dri/by-path/, these should be stable across
reboots. `udevadm settle` before a compositor start-up can wait for udev to
finish its job.

Out of curiosity, can you explain your use-case? Why do you need to start
multiple compositors, each on its own GPU?

Re: Multiple DRI card detection in compositor systemd units

[Hoosier, Matt]

I get the intuition behind the suggestion to aggregate using logind seats, as far as it goes. But it still seems to me that this just pushes the question back: how do you identify which card is which in order to assign it to a seat? Normally this is done using udev rules, right? Additionally, I'm working with some constraints that not all of the compositors are logind-aware. I realize that's not really your problem, and it doesn't really influence the merits of your suggestion.

The /dev/dri/by-path idea works, I suppose, if you have different physical graphics cards. In my case, that's not true. These are virtualized cards that the silicon vendor's DRM drivers use to expose different subsets of DRM resources as different cards. So there's only one /dev/dri/by-path card here. Think: DRM leases, but with the lessees popping out as card nodes rather than arranged dynamically using the drm ioctl()'s to manufature leases.

The use-case here is to allow separate DRM domains for each of several containers. It's not really desirable to try to funnel everybody's graphics through a common compositor that runs all the connectors.

Thanks for the thoughts.

-Matt

On 9/22/21, 10:29 AM, "Simon Ser" <contact@emersion.fr> wrote:

    CAUTION - EXTERNAL EMAIL: Do not click any links or open any attachments unless you trust the sender and know the content is safe.


    Maybe try creating multiple physical seats with logind, and start each
    compositor on its own seat? A physical seat is a collection of devices like
    DRM nodes and evdev device files.

    Also udev creates files in /dev/dri/by-path/, these should be stable across
    reboots. `udevadm settle` before a compositor start-up can wait for udev to
    finish its job.

    Out of curiosity, can you explain your use-case? Why do you need to start
    multiple compositors, each on its own GPU?


________________________________

CONFIDENTIALITY NOTICE: This email and any attachments are for the sole use of the intended recipient(s) and contain information that may be Garmin confidential and/or Garmin legally privileged. If you have received this email in error, please notify the sender by reply email and delete the message. Any disclosure, copying, distribution or use of this communication (including attachments) by someone other than the intended recipient is prohibited. Thank you.

Re: Multiple DRI card detection in compositor systemd units

[Pekka Paalanen]

[-- Attachment #1: Type: text/plain, Size: 1364 bytes --]

On Wed, 22 Sep 2021 16:16:48 +0000
"Hoosier, Matt" <Matt.Hoosier@garmin.com> wrote:

> 
> The /dev/dri/by-path idea works, I suppose, if you have different
> physical graphics cards. In my case, that's not true. These are
> virtualized cards that the silicon vendor's DRM drivers use to expose
> different subsets of DRM resources as different cards. So there's
> only one /dev/dri/by-path card here. Think: DRM leases, but with the
> lessees popping out as card nodes rather than arranged dynamically
> using the drm ioctl()'s to manufature leases.

That's the standard solution though, I believe: use devpath for
matching the device, because the device doesn't randomly jump from a
physical connector (e.g. PCIe slot) to another.

But since you have virtual cards, that obviously doesn't work. I'm
afraid you need to solve this with your virtual card provider. Maybe
there could be some sort of virtual bus with persistent addresses which
would make devpath reliable?

I wouldn't expect drmGetVersion()->name to differ between the (virtual)
devices since they are all using the same driver, right?


Sorry,
pq


> The use-case here is to allow separate DRM domains for each of
> several containers. It's not really desirable to try to funnel
> everybody's graphics through a common compositor that runs all the
> connectors.
> 

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Multiple DRI card detection in compositor systemd units

[Hoosier, Matt]

Hi Pekka,

Those are fair answers. In my case, it does happen that the configuration available on the virtual cards includes a way to set distinct names reported by drmGetVersion()->name.

It looks like the consensus is that UDev rules probably shouldn't be consulting the name as a distinguishing/durable identifier of cards, and that the physical path is the standard thing used to disambiguate. Thanks.

-Matt

On 9/23/21, 8:39 AM, "Pekka Paalanen" <ppaalanen@gmail.com> wrote:

    On Wed, 22 Sep 2021 16:16:48 +0000
    "Hoosier, Matt" <Matt.Hoosier@garmin.com> wrote:

    >
    > The /dev/dri/by-path idea works, I suppose, if you have different
    > physical graphics cards. In my case, that's not true. These are
    > virtualized cards that the silicon vendor's DRM drivers use to expose
    > different subsets of DRM resources as different cards. So there's
    > only one /dev/dri/by-path card here. Think: DRM leases, but with the
    > lessees popping out as card nodes rather than arranged dynamically
    > using the drm ioctl()'s to manufature leases.

    That's the standard solution though, I believe: use devpath for
    matching the device, because the device doesn't randomly jump from a
    physical connector (e.g. PCIe slot) to another.

    But since you have virtual cards, that obviously doesn't work. I'm
    afraid you need to solve this with your virtual card provider. Maybe
    there could be some sort of virtual bus with persistent addresses which
    would make devpath reliable?

    I wouldn't expect drmGetVersion()->name to differ between the (virtual)
    devices since they are all using the same driver, right?


    Sorry,
    pq


    > The use-case here is to allow separate DRM domains for each of
    > several containers. It's not really desirable to try to funnel
    > everybody's graphics through a common compositor that runs all the
    > connectors.
    >


________________________________

CONFIDENTIALITY NOTICE: This email and any attachments are for the sole use of the intended recipient(s) and contain information that may be Garmin confidential and/or Garmin legally privileged. If you have received this email in error, please notify the sender by reply email and delete the message. Any disclosure, copying, distribution or use of this communication (including attachments) by someone other than the intended recipient is prohibited. Thank you.