Re: [PATCH v2 rdma-core] irdma: Add ice and irdma to kernel-boot rules

[Jason Gunthorpe <jgg@nvidia.com>]

On Mon, Sep 20, 2021 at 07:41:21PM +0000, Nikolova, Tatyana E wrote:
> 
> 
> > From: Jason Gunthorpe <jgg@nvidia.com>
> > Sent: Thursday, September 2, 2021 10:40 AM
> > To: Nikolova, Tatyana E <tatyana.e.nikolova@intel.com>
> > Cc: dledford@redhat.com; leon@kernel.org; linux-rdma@vger.kernel.org
> > Subject: Re: [PATCH v2 rdma-core] irdma: Add ice and irdma to kernel-boot
> > rules
> > 
> > On Thu, Sep 02, 2021 at 03:29:43PM +0000, Nikolova, Tatyana E wrote:
> > > > Given that ice is both iwarp and roce, is there some better way to
> > > > detect this? Doesn't the aux device encode it?
> > >
> > > Hi Jason,
> > >
> > > We tried a few experiments without success. The auxiliary devices
> > > alias with our driver and not ice, so maybe this is the reason?
> > >
> > > Here is an example of what we tried.
> > >
> > > udevadm info
> > > /sys/devices/pci0000:2e/0000:2e:00.0/0000:2f:00.0/ice.roce.0
> > > P: /devices/pci0000:2e/0000:2e:00.0/0000:2f:00.0/ice.roce.0
> > > E: DEVPATH=/devices/pci0000:2e/0000:2e:00.0/0000:2f:00.0/ice.roce.0
> > > E: DRIVER=irdma
> > > E: MODALIAS=auxiliary:ice.roce
> > > E: SUBSYSTEM=auxiliary
> > >
> > > udevadm info /sys/bus/auxiliary/devices/ice.roce.0
> > > P: /devices/pci0000:2e/0000:2e:00.0/0000:2f:00.0/ice.roce.0
> > > E: DEVPATH=/devices/pci0000:2e/0000:2e:00.0/0000:2f:00.0/ice.roce.0
> > > E: DRIVER=irdma
> > > E: MODALIAS=auxiliary:ice.roce
> > > E: SUBSYSTEM=auxiliary
> > >
> > > Given the udevadm output, we put the following line in the udev rdma-
> > description.rules:
> > >
> > > SUBSYSTEMS=="auxiliary",
> > DEVPATH=="*/devices/pci0000:2e/0000:2e:00.0/0000:2f:00.0/ice.roce.0/*",
> > ENV{ID_RDMA_ROCE}="1"
> > 
> > What is the SUBSYSTEM=="infiniband" device like?
> > 
> > This seems like the right direction, you need to wrangle udev though..
> > 
> 
> Hi Jason,
> 
> After more research and given the udevadm output, we revised the irdma udev rule to make it work. Could you please review the patch bellow?
> 
> diff --git a/kernel-boot/rdma-description.rules b/kernel-boot/rdma-description.rules
> index 48a7cede..09deb451 100644
> +++ b/kernel-boot/rdma-description.rules
> @@ -1,7 +1,7 @@
>  # This is a version of net-description.rules for /sys/class/infiniband devices
>  
>  ACTION=="remove", GOTO="rdma_description_end"
> -SUBSYSTEM!="infiniband", GOTO="rdma_description_end"
> +SUBSYSTEM!="infiniband", GOTO="rdma_infiniband_end"
>  
>  # NOTE: DRIVERS searches up the sysfs path to find the driver that is bound to  # the PCI/etc device that the RDMA device is linked to. This is not the kernel @@ -40,4 +40,9 @@ DEVPATH=="*/infiniband/rxe*", ATTR{parent}=="*", ENV{ID_RDMA_ROCE}="1"
>  SUBSYSTEMS=="pci", ENV{ID_BUS}="pci", ENV{ID_VENDOR_ID}="$attr{vendor}", ENV{ID_MODEL_ID}="$attr{device}"
>  SUBSYSTEMS=="pci", IMPORT{builtin}="hwdb --subsystem=pci"
>  
> +LABEL="rdma_infiniband_end"
> +
> +SUBSYSTEM!="auxiliary", GOTO="rdma_description_end"
> +KERNEL=="ice.iwarp.?", ENV{ID_RDMA_IWARP}="1" 
> +KERNEL=="ice.roce.?", ENV{ID_RDMA_ROCE}="1"
>  LABEL="rdma_description_end"

This doesn't seem right, the ID_* must be applied to an infiniband
device or the other stuff doesn't that consumes this won't work right.

What does the udev debugging say about these ID tags?

The SUBSYSTEMS=="" is the right approach, as shown above for the other
metadata. If you are having trobule I'm wondering if there is some
kind of kernel problem creating the wrong sysfs?

Jason