Re: [PATCH v2 rdma-core] irdma: Add ice and irdma to kernel-boot rules

From: Jason Gunthorpe <jgg@nvidia.com>
To: "Nikolova, Tatyana E" <tatyana.e.nikolova@intel.com>
Cc: "dledford@redhat.com" <dledford@redhat.com>,
	"leon@kernel.org" <leon@kernel.org>,
	"linux-rdma@vger.kernel.org" <linux-rdma@vger.kernel.org>
Subject: Re: [PATCH v2 rdma-core] irdma: Add ice and irdma to kernel-boot rules
Date: Thu, 14 Oct 2021 20:36:44 -0300	[thread overview]
Message-ID: <20211014233644.GA2744544@nvidia.com> (raw)
In-Reply-To: <DM6PR11MB4692B56B4C7D1E790B50888DCBB89@DM6PR11MB4692.namprd11.prod.outlook.com>

On Thu, Oct 14, 2021 at 08:11:25PM +0000, Nikolova, Tatyana E wrote:
> Based on the following output, it seems that some systemd services
> won't work. I just tested with the port mapper which worked.
> 
> udevadm info -q all  /sys/devices/pci0000:2e/0000:2e:00.0/0000:2f:00.0/ice.iwarp.0
> P: /devices/pci0000:2e/0000:2e:00.0/0000:2f:00.0/ice.iwarp.0
> E: DEVPATH=/devices/pci0000:2e/0000:2e:00.0/0000:2f:00.0/ice.iwarp.0
> E: DRIVER=irdma
> E: ID_RDMA_IWARP=1
> E: MODALIAS=auxiliary:ice.iwarp
> E: SUBSYSTEM=auxiliary
> E: SYSTEMD_WANTS=iwpmd.service
> E: TAGS=:systemd:
> E: USEC_INITIALIZED=33683420

Right, this is not an RDMA device, so it should not have ID_RDMA_* set
on it at all.

> If we need to use the aux device name in the udev rules, then I am
> not aware how to get to the aux device through the infiniband or the
> pci subsystem.
> 
> > What does the udev debugging say about these ID tags?
> > 
> > The SUBSYSTEMS=="" is the right approach, as shown above for the other
> > metadata. If you are having trobule I'm wondering if there is some kind of
> > kernel problem creating the wrong sysfs?
> >
> 
> Previously I was using an RC1 kernel and seeing issues with sysfs. After switching to a GA kernel, it works better. 
> 
> udevadm info --attribute-walk /sys/class/infiniband/rocep47s0f0
>
>   looking at device '/devices/pci0000:2e/0000:2e:00.0/0000:2f:00.0/infiniband/rocep47s0f0':

This looks like the problem. For any of this to work the infiniband
device needs to be parented to the aux device, not the PCI device.

mlx5 did not due this for backwards compat reasons, but this is a new
driver so it could do it properly.

Then you can use SUBSYSTEMS and so forth as I first suggested.

> diff --git a/kernel-boot/rdma-description.rules b/kernel-boot/rdma-description.rules
> index 48a7ced..9a18b67 100644
> +++ b/kernel-boot/rdma-description.rules
> @@ -33,6 +33,8 @@ DRIVERS=="mlx4_core", ENV{ID_RDMA_ROCE}="1"
>  DRIVERS=="mlx5_core", ENV{ID_RDMA_ROCE}="1"
>  DRIVERS=="qede", ENV{ID_RDMA_ROCE}="1"
>  DRIVERS=="vmw_pvrdma", ENV{ID_RDMA_ROCE}="1"
> +KERNEL=="iw*", ENV{ID_RDMA_IWARP}="1"
> +KERNEL=="roce*", ENV{ID_RDMA_ROCE}="1"

No, this is matching against the RDMA device name.

The other alternative is to replace this entire file with a C program
that is invoked by udev

The C program would use netlink to query the rdma device properties
and return the ID_xx strings for setting.

Jason