[PATCH rdma-core 0/5] Common systemd/udev based boot support

[PATCH rdma-core 0/5] Common systemd/udev based boot support

[Jason Gunthorpe]

This is my attempt to migrate the module loading functions from RedHat's
rdma.service into common code suitable for all distros.

The approach is completly hotplug safe and relies heavily on udev and systemd
to do the actual decision making and work, there are no shell scripts involved.

I do not have nearly enough hardware to exhaustively test this, I have checked
that the basic elements are working, but there is lots of room for little
errors, particularly in module names and udev matching. It would be great of
someone could try this on iwarp hardware and confirm all the autoloading works
as expected.

As should the active roce vendors. I expect that all roce hardware will
autoload the RDMA modules with this approach.

Compared to redhat/rdma.kernel-init:

 - Config of what ULP modules to load is split into files in
   /etc/rdma/modules/*.conf, each conf reflects a certain configuration. Eg
   infiniband.conf is loaded when IB hardware is present, iwarp.conf is loaded
   for iWarp, etc.

   This allows each techology to load appropriate modules automatically, ie
   roce hardware will not autoload ipoib out of the box

   Users can configure these files to turn things on/off as appropriate, or just
   use the usual /etc/modules-load.d/ mechanism. systemd-modules-load is used
   to  load the modules lists so behavior is identical to the standard loader.
   eg blacklists work, etc.

   This obsoletes the /etc/rdma/rdma.conf file

 - Load of HW support drivers (eg the RDMA component of an ethernet NIC) is
   done completely inside udev using the udev built in module load, based on
   the hotplug event of the ethernet nic.

   This means RDMA support modules are only loaded if the rdma-core package is
   installed and only if the actual base ethernet hardware is present. (rather
   than triggering on loaded modules)

 - More roce hardware is supported now, I filled in entries for all of our
   new providers (eg qedr).

 - srp_daemon and iwpmd support kernel modules autoload automatically as part of
   the .service file, using systemd dependencies. If those packages are not
   installed/running then those modules do not load.

 - iwpmd now automatically activates if iwarp hardware is present, systemd does
   not run it automatically.

 - None of the obsolete mtrr or pci config meddling is included

 - Nothing to do with sriov

 - udev has more information now about rdma devices:

$ udevadm info /sys/class/infiniband/mlx4_0
P: /devices/pci0000:00/0000:00:04.0/infiniband/mlx4_0
E: DEVPATH=/devices/pci0000:00/0000:00:04.0/infiniband/mlx4_0
E: ID_BUS=pci
E: ID_MODEL_FROM_DATABASE=MT27500 Family [ConnectX-3]
E: ID_MODEL_ID=0x1003
E: ID_PCI_CLASS_FROM_DATABASE=Network controller
E: ID_PCI_SUBCLASS_FROM_DATABASE=Network controller
E: ID_RDMA_INFINIBAND=1
E: ID_RDMA_ROCE=1
E: ID_VENDOR_FROM_DATABASE=Mellanox Technologies
E: ID_VENDOR_ID=0x15b3
E: NAME=mlx4_0
E: SUBSYSTEM=infiniband
E: SYSTEMD_WANTS=rdma-ndd.service rdma-load-modules-XSrBNAzizW1TDjBF/Jpztg@public.gmane.org rdma-load-modules-WW8hglTQ5Ylgot6vXr2xLYq0XWbe/Iga@public.gmane.org rdma-load-modules-eW91h9tu8Y5TDjBF/Jpztg@public.gmane.org
E: TAGS=:systemd:
E: USEC_INITIALIZED=7954413272

    A successful test should show appropriate ID_RDMA_* entries.

I did not go ahead and remove now duplicate functionality from the
redhat/scripts, I think that Jarod should be involved in that. This approach
is not incompatible with the scripts in redhat/, but modules loads will be
attempted twice if both systems are present.

https://github.com/jgunthorpe/rdma-plumbing/tree/systemd

Jason Gunthorpe (5):
  Common infrastructure for auto loading rdma modules
  srp: Autoload the SRP kernel module if required
  iwpmd: Autoload iwpmd if required
  redhat: Remove cxgb3/4.modprobe files
  Move umad udev rules to rdma-core

 CMakeLists.txt                                     |  1 +
 Documentation/udev.md                              | 83 ++++++++++++++++++++++
 debian/control                                     |  2 +-
 debian/iwpmd.install                               |  2 +
 debian/libibumad3.install                          |  1 -
 debian/rdma-core.install                           | 10 +++
 debian/srptools.install                            |  1 +
 iwpmd/CMakeLists.txt                               |  8 +++
 iwpmd/iwpmd.rules                                  |  1 +
 iwpmd/iwpmd.service.in                             |  6 +-
 iwpmd/modules-iwpmd.conf                           |  2 +
 kernel-boot/CMakeLists.txt                         | 28 ++++++++
 kernel-boot/modules/infiniband.conf                | 12 ++++
 kernel-boot/modules/iwarp.conf                     |  1 +
 kernel-boot/modules/opa.conf                       | 10 +++
 kernel-boot/modules/rdma.conf                      | 21 ++++++
 kernel-boot/modules/roce.conf                      |  2 +
 kernel-boot/rdma-description.rules                 | 41 +++++++++++
 kernel-boot/rdma-hw-modules.rules                  | 39 ++++++++++
 kernel-boot/rdma-load-modules@.service.in          | 15 ++++
 kernel-boot/rdma-ulp-modules.rules                 | 11 +++
 .../rdma-umad.rules                                |  0
 libibumad/CMakeLists.txt                           |  5 --
 redhat/rdma-core.spec                              | 20 ++++--
 redhat/rdma.cxgb3.sys.modprobe                     |  1 -
 redhat/rdma.cxgb4.sys.modprobe                     |  1 -
 srp_daemon/CMakeLists.txt                          |  4 ++
 srp_daemon/modules-srp_daemon.conf                 |  2 +
 srp_daemon/srp_daemon.service.in                   |  2 +-
 srp_daemon/srp_daemon_port@.service.in             |  3 +-
 30 files changed, 315 insertions(+), 20 deletions(-)
 create mode 100644 Documentation/udev.md
 create mode 100644 iwpmd/iwpmd.rules
 create mode 100644 iwpmd/modules-iwpmd.conf
 create mode 100644 kernel-boot/CMakeLists.txt
 create mode 100644 kernel-boot/modules/infiniband.conf
 create mode 100644 kernel-boot/modules/iwarp.conf
 create mode 100644 kernel-boot/modules/opa.conf
 create mode 100644 kernel-boot/modules/rdma.conf
 create mode 100644 kernel-boot/modules/roce.conf
 create mode 100644 kernel-boot/rdma-description.rules
 create mode 100644 kernel-boot/rdma-hw-modules.rules
 create mode 100644 kernel-boot/rdma-load-modules@.service.in
 create mode 100644 kernel-boot/rdma-ulp-modules.rules
 rename libibumad/libibumad.udev-rules => kernel-boot/rdma-umad.rules (100%)
 delete mode 100644 redhat/rdma.cxgb3.sys.modprobe
 delete mode 100644 redhat/rdma.cxgb4.sys.modprobe
 create mode 100644 srp_daemon/modules-srp_daemon.conf

-- 
2.7.4

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH rdma-core 1/5] Common infrastructure for auto loading rdma modules

[Jason Gunthorpe]

This is inspired by the similar approach in the redhat directory but
takes a more general approach relying on udev and systemd to do the
actual work fully dynamically instead of a oneshot shell script.

Loading is split into two cases
 1) Loading RDMA support modules when RDMA capable hardware is installed.
    This is only needed for ethernet devices which do not load their RDMA
    support modules via request_module in the kernel.

    udev is used to detect when an ethernet device controlled by a specific
    module is hot plugged and then udev directly loads the RDMA module

 2) Loading RDMA ULP support when RDMA hardware is installed
    This is done by having udev detect when RDMA hardware is installed and
    udev causes systemd to load a list of modules from config files in
    /etc/rdma/modules/

    The user can customize these files to select which ULP modules should be
    loaded.

This broadly replaces the redhat/rdma.conf scheme.

In all cases the users can prevent a module from being auto-loaded on their
system by blacking listing it in a file in /etc/modprobe.d/

Signed-off-by: Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
---
 CMakeLists.txt                            |  1 +
 Documentation/udev.md                     | 83 +++++++++++++++++++++++++++++++
 debian/rdma-core.install                  |  9 ++++
 kernel-boot/CMakeLists.txt                | 24 +++++++++
 kernel-boot/modules/infiniband.conf       | 12 +++++
 kernel-boot/modules/iwarp.conf            |  2 +
 kernel-boot/modules/opa.conf              | 10 ++++
 kernel-boot/modules/rdma.conf             | 21 ++++++++
 kernel-boot/modules/roce.conf             |  2 +
 kernel-boot/rdma-description.rules        | 41 +++++++++++++++
 kernel-boot/rdma-hw-modules.rules         | 39 +++++++++++++++
 kernel-boot/rdma-load-modules@.service.in | 15 ++++++
 kernel-boot/rdma-ulp-modules.rules        | 11 ++++
 redhat/rdma-core.spec                     | 11 +++-
 14 files changed, 280 insertions(+), 1 deletion(-)
 create mode 100644 Documentation/udev.md
 create mode 100644 kernel-boot/CMakeLists.txt
 create mode 100644 kernel-boot/modules/infiniband.conf
 create mode 100644 kernel-boot/modules/iwarp.conf
 create mode 100644 kernel-boot/modules/opa.conf
 create mode 100644 kernel-boot/modules/rdma.conf
 create mode 100644 kernel-boot/modules/roce.conf
 create mode 100644 kernel-boot/rdma-description.rules
 create mode 100644 kernel-boot/rdma-hw-modules.rules
 create mode 100644 kernel-boot/rdma-load-modules@.service.in
 create mode 100644 kernel-boot/rdma-ulp-modules.rules

diff --git a/CMakeLists.txt b/CMakeLists.txt
index 16196205035f61..a03d8da31cbc5d 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -398,6 +398,7 @@ configure_file("${BUILDLIB}/config.h.in" "${BUILD_INCLUDE}/config.h" ESCAPE_QUOT
 add_subdirectory(ccan)
 add_subdirectory(util)
 add_subdirectory(Documentation)
+add_subdirectory(kernel-boot)
 # Libraries
 add_subdirectory(libibumad)
 add_subdirectory(libibumad/man)
diff --git a/Documentation/udev.md b/Documentation/udev.md
new file mode 100644
index 00000000000000..e30e6007344fba
--- /dev/null
+++ b/Documentation/udev.md
@@ -0,0 +1,83 @@
+# Kernel Module Loading
+
+The RDMA subsystem relies on the kernel, udev and systemd to load modules on
+demand when RDMA hardware is present. The RDMA subsystem is unique in that we
+do not load the optional RDMA hardware modules unless the system has the
+rdma-core package installed.
+
+This is to avoid exposing systems not using RDMA from having RDMA enabled, for
+instance if a system has a multi-protocol ethernet adaptor, but is only using
+the net stack interface.
+
+## Boot ordering with systemd
+
+systemd assumes everything is hot pluggable and runs in an event driven
+manner.  When working with RDMA devices we are firstly concerned with when the
+physical hardware its module loaded into the kernel.
+
+This can happen in several spots along the bootup:
+
+ - From the initrd or built into the kernel. If hardware modules are present
+   inthe initrd then they are loaded into the kernel before booting the
+   system. This is done largely synchronously with the boot process.
+
+ - From udev when it auto detects PCI hardware or otherwise.
+   This happens asynchronously in the boot process, systemd does not wait for
+   udev to finish loading modules before it continues on.
+
+   This path makes it very likely the system will experience a RDMA 'hot plug'
+   scenario.
+
+ - From systemd's fixed module loader systemd-modules-load.service, eg from
+   the list in /etc/modules-load.d/. In this case the modules load happens
+   synchronously within systemd and it will hold off sysinit.target until
+   modules are loaded
+
+Once the hardware module is loaded it may be necessary to load a protocol
+module, eg to enable RDMA support on an ethernet device.
+
+This is triggered automatically by udev rules that match the master devices
+and load the protocol module with udev's module loader. This happens
+asynchronously to the rest of the systemd startup.
+
+Once a RDMA device is created by the kernel then udev will cause systemd to
+schedule ULP module loading services (eg rdma-load-modules@.service) specific
+to the plugged hardware. If sysinit.target has not yet been passed then these
+loaders will defer sysinit.target until they complete, otherwise this is a hot
+plug event and things will load asynchronously to the boot up process.
+
+Finally udev will cause systemd to start RDMA specific daemons like
+srp_deamon, rdma-ndd and iwpmd. These starts are linked to the detection of
+the first RDMA hardware, and the daemons internally handle hot plug events for
+other hardware.
+
+## Hot Plug compatible services
+
+RDMA using services need to have device specific systemd dependencies in their
+unit files, either created by hand by the admin or by using udev rules.
+
+For instance, a service that uses /dev/infiniband/umad0 requires:
+
+```
+After=dev-infiniband-umad0.device
+BindsTo=dev-infiniband-umad0.device
+```
+
+Which will ensure the service will not run until the required umad device
+appears.
+
+This is similar to how systemd handles mounting filesystems and configuring
+ethernet devices.
+
+## Interaction with legacy non-hotplug services
+
+Services that cannot handle hot plug must be ordered after
+systemd-udev-settle.service, which will wait for udev to complete loading
+modules and scheduling systemd services. This ensures that all RDMA hardware
+present at boot is setup before proceeding to run the legacy service.
+
+Admins using legacy services can also place their RDMA hardware modules (eg
+mlx4_ib) directly in /etc/modules-load.d/ or in their initrd which will cause
+systemd to defer passing to sysinit.target until all RDMA hardware is setup,
+this is usually sufficient for legacy services. This is probably the default
+behavior in many configurations.
diff --git a/debian/rdma-core.install b/debian/rdma-core.install
index c19d766fc0f48e..0bda539494189a 100644
--- a/debian/rdma-core.install
+++ b/debian/rdma-core.install
@@ -1,7 +1,16 @@
 etc/modprobe.d/mlx4.conf
 etc/modprobe.d/truescale.conf
+etc/rdma/modules/infiniband.conf
+etc/rdma/modules/iwarp.conf
+etc/rdma/modules/opa.conf
+etc/rdma/modules/rdma.conf
+etc/rdma/modules/roce.conf
+lib/systemd/system/rdma-load-modules@.service
 lib/systemd/system/rdma-ndd.service
 lib/udev/rules.d/60-rdma-ndd.rules
+lib/udev/rules.d/75-rdma-description.rules
+lib/udev/rules.d/90-rdma-hw-modules.rules
+lib/udev/rules.d/90-rdma-ulp-modules.rules
 usr/bin/rxe_cfg
 usr/lib/truescale-serdes.cmds
 usr/sbin/rdma-ndd
diff --git a/kernel-boot/CMakeLists.txt b/kernel-boot/CMakeLists.txt
new file mode 100644
index 00000000000000..0d4a2aec1c6a94
--- /dev/null
+++ b/kernel-boot/CMakeLists.txt
@@ -0,0 +1,24 @@
+rdma_subst_install(FILES rdma-load-modules@.service.in
+  DESTINATION "${CMAKE_INSTALL_SYSTEMD_SERVICEDIR}"
+  RENAME rdma-load-modules@.service
+  PERMISSIONS OWNER_WRITE OWNER_READ GROUP_READ WORLD_READ)
+
+install(FILES
+  modules/infiniband.conf
+  modules/iwarp.conf
+  modules/opa.conf
+  modules/rdma.conf
+  modules/roce.conf
+  DESTINATION "${CMAKE_INSTALL_SYSCONFDIR}/rdma/modules")
+
+install(FILES "rdma-description.rules"
+  RENAME "75-rdma-description.rules"
+  DESTINATION "${CMAKE_INSTALL_UDEV_RULESDIR}")
+
+install(FILES "rdma-hw-modules.rules"
+  RENAME "90-rdma-hw-modules.rules"
+  DESTINATION "${CMAKE_INSTALL_UDEV_RULESDIR}")
+
+install(FILES "rdma-ulp-modules.rules"
+  RENAME "90-rdma-ulp-modules.rules"
+  DESTINATION "${CMAKE_INSTALL_UDEV_RULESDIR}")
diff --git a/kernel-boot/modules/infiniband.conf b/kernel-boot/modules/infiniband.conf
new file mode 100644
index 00000000000000..f247d840b7d1bd
--- /dev/null
+++ b/kernel-boot/modules/infiniband.conf
@@ -0,0 +1,12 @@
+# These modules are loaded by the system if any InfiniBand device is installed
+# Infiniband over IP netdevice
+ib_ipoib
+
+# Access to fabric management SMPs and GMPs from userspace.
+ib_umad
+
+# SCSI Remote Protocol target support
+# ib_srpt
+
+# ib_ucm provides the obsolete /dev/infiniband/ucm0
+# ib_ucm
diff --git a/kernel-boot/modules/iwarp.conf b/kernel-boot/modules/iwarp.conf
new file mode 100644
index 00000000000000..882146e41ee2ba
--- /dev/null
+++ b/kernel-boot/modules/iwarp.conf
@@ -0,0 +1,2 @@
+# These modules are loaded by the system if any iWarp device is installed
+iw_cm
diff --git a/kernel-boot/modules/opa.conf b/kernel-boot/modules/opa.conf
new file mode 100644
index 00000000000000..b9bc9f1f0146af
--- /dev/null
+++ b/kernel-boot/modules/opa.conf
@@ -0,0 +1,10 @@
+# These modules are loaded by the system if any OmniPath Architecture device
+# is installed
+# Infiniband over IP netdevice
+ib_ipoib
+
+# Access to fabric management SMPs and GMPs from userspace.
+ib_umad
+
+# Omnipath Ethernet Virtual NIC netdevice
+opa_vnic
diff --git a/kernel-boot/modules/rdma.conf b/kernel-boot/modules/rdma.conf
new file mode 100644
index 00000000000000..2d342dd82f7db0
--- /dev/null
+++ b/kernel-boot/modules/rdma.conf
@@ -0,0 +1,21 @@
+# These modules are loaded by the system if any RDMA devices is installed
+# iSCSI over RDMA client support
+ib_iser
+
+# iSCSI over RDMA target support
+# ib_isert
+
+# User access to RDMA verbs (supports libibverbs)
+ib_uverbs
+
+# User access to RDMA connection management (supports librdmacm)
+rdma_ucm
+
+# RDS over RDMA support
+# rds_rdma
+
+# NFS over RDMA client support
+xprtrdma
+
+# NFS over RDMA server support
+svcrdma
diff --git a/kernel-boot/modules/roce.conf b/kernel-boot/modules/roce.conf
new file mode 100644
index 00000000000000..8e4927ce26f043
--- /dev/null
+++ b/kernel-boot/modules/roce.conf
@@ -0,0 +1,2 @@
+# These modules are loaded by the system if any RDMA over Converged Ethernet
+# device is installed
diff --git a/kernel-boot/rdma-description.rules b/kernel-boot/rdma-description.rules
new file mode 100644
index 00000000000000..dc25db07219cac
--- /dev/null
+++ b/kernel-boot/rdma-description.rules
@@ -0,0 +1,41 @@
+# This is a version of net-description.rules for /sys/class/infiniband devices
+
+ACTION=="remove", GOTO="rdma_description_end"
+SUBSYSTEM!="infiniband", GOTO="rdma_description_end"
+
+# NOTE: DRIVERS searches up the sysfs path to find the driver that is bound to
+# the PCI/etc device that the RDMA device is linked to. This is not the kernel
+# driver that is supplying the RDMA device (eg as seen in ID_NET_DRIVER)
+
+# FIXME: with kernel support we could actually detect the protocols the RDMA
+# driver itself supports, this is a work around for lack of that support.
+# In future we could do this with a udev IMPORT{program} helper program
+# that extracted the ID information from the RDMA netlink.
+
+# Hardware that supports InfiniBand
+DRIVERS=="mlx4_core", ENV{ID_RDMA_INFINIBAND}="1"
+DRIVERS=="mlx5_core", ENV{ID_RDMA_INFINIBAND}="1"
+DRIVERS=="qib", ENV{ID_RDMA_INFINIBAND}="1"
+
+# Hardware that supports OPA
+DRIVERS=="hfi1verbs", ENV{ID_RDMA_IWARP}="1"
+
+# Hardware that supports iWarp
+DRIVERS=="cxgb3", ENV{ID_RDMA_IWARP}="1"
+DRIVERS=="cxgb4", ENV{ID_RDMA_IWARP}="1"
+
+# Hardware that supports RoCE
+DRIVERS=="be2net", ENV{ID_RDMA_ROCE}="1"
+DRIVERS=="bnxt_en", ENV{ID_RDMA_ROCE}="1"
+DRIVERS=="hns", ENV{ID_RDMA_ROCE}="1"
+DRIVERS=="i40e", ENV{ID_RDMA_ROCE}="1"
+DRIVERS=="mlx4_core", ENV{ID_RDMA_ROCE}="1"
+DRIVERS=="mlx5_core", ENV{ID_RDMA_ROCE}="1"
+DRIVERS=="qede", ENV{ID_RDMA_ROCE}="1"
+
+# Setup the usual ID information so that systemd will display a sane name for
+# the RDMA device units.
+SUBSYSTEMS=="pci", ENV{ID_BUS}="pci", ENV{ID_VENDOR_ID}="$attr{vendor}", ENV{ID_MODEL_ID}="$attr{device}"
+SUBSYSTEMS=="pci", IMPORT{builtin}="hwdb --subsystem=pci"
+
+LABEL="rdma_description_end"
diff --git a/kernel-boot/rdma-hw-modules.rules b/kernel-boot/rdma-hw-modules.rules
new file mode 100644
index 00000000000000..dde0ab8dacacab
--- /dev/null
+++ b/kernel-boot/rdma-hw-modules.rules
@@ -0,0 +1,39 @@
+ACTION=="remove", GOTO="rdma_hw_modules_end"
+SUBSYSTEM!="net", GOTO="rdma_hw_modules_end"
+
+# Automatically load RDMA specific kernel modules when a multi-function device is installed
+
+# These drivers autoload an ethernet driver based on hardware detection and
+# need userspace to load the module that has their RDMA component to turn on
+# RDMA.
+ENV{ID_NET_DRIVER}=="be2net", RUN{builtin}+="kmod load ocrdma"
+ENV{ID_NET_DRIVER}=="bnxt_en", RUN{builtin}+="kmod load bnxt_re"
+ENV{ID_NET_DRIVER}=="cxgb3", RUN{builtin}+="kmod load iw_cxgb3"
+ENV{ID_NET_DRIVER}=="cxgb4", RUN{builtin}+="kmod load iw_cxgb4"
+ENV{ID_NET_DRIVER}=="hns", RUN{builtin}+="kmod load hns_roce"
+ENV{ID_NET_DRIVER}=="i40e", RUN{builtin}+="kmod load i40iw"
+ENV{ID_NET_DRIVER}=="mlx4_en", RUN{builtin}+="kmod load mlx4_ib"
+ENV{ID_NET_DRIVER}=="mlx5_core", RUN{builtin}+="kmod load mlx5_ib"
+ENV{ID_NET_DRIVER}=="qede", RUN{builtin}+="kmod load qedr"
+
+# The user must explicitly load these modules via /etc/modules-load.d/ or otherwise
+# rxe
+
+# When in IB mode the kernel PCI core module autoloads the protocol modules
+# for these providers
+# mlx4
+# mlx5
+
+# enic no longer has a userspace verbs driver, this rule should probably be
+# owned by libfabric
+ENV{ID_NET_DRIVER}=="enic", RUN{builtin}+="kmod load usnic_verbs"
+
+# These providers are single function and autoload RDMA automatically based on
+# PCI probing
+# hfi1verbs
+# ipathverbs
+# mthca
+# vmw_pvrdma
+# nes
+
+LABEL="rdma_hw_modules_end"
diff --git a/kernel-boot/rdma-load-modules@.service.in b/kernel-boot/rdma-load-modules@.service.in
new file mode 100644
index 00000000000000..b35a493ebf230b
--- /dev/null
+++ b/kernel-boot/rdma-load-modules@.service.in
@@ -0,0 +1,15 @@
+[Unit]
+Description=Load RDMA modules from @CMAKE_INSTALL_SYSCONFDIR@/rdma/modules/%I.conf
+DefaultDependencies=no
+Conflicts=shutdown.target
+# network-pre.target is to support distro network setup scripts that run after
+# systemd-modules-load.service but before sysinit.target, eg a classic network
+# setup script.
+Before=sysinit.target shutdown.target network-pre.target
+ConditionCapability=CAP_SYS_MODULE
+
+[Service]
+Type=oneshot
+RemainAfterExit=yes
+ExecStart=/lib/systemd/systemd-modules-load @CMAKE_INSTALL_SYSCONFDIR@/rdma/modules/%I.conf
+TimeoutSec=90s
diff --git a/kernel-boot/rdma-ulp-modules.rules b/kernel-boot/rdma-ulp-modules.rules
new file mode 100644
index 00000000000000..c090700c754b19
--- /dev/null
+++ b/kernel-boot/rdma-ulp-modules.rules
@@ -0,0 +1,11 @@
+ACTION=="remove", GOTO="rdma_ulp_modules_end"
+SUBSYSTEM!="infiniband", GOTO="rdma_ulp_modules_end"
+
+# Automatically load general RDMA ULP modules when RDMA hardware is installed
+TAG+="systemd", ENV{SYSTEMD_WANTS}+="rdma-load-modules-XSrBNAzizW1TDjBF/Jpztg@public.gmane.org"
+TAG+="systemd", ENV{ID_RDMA_INFINIBAND}=="1", ENV{SYSTEMD_WANTS}+="rdma-load-modules-WW8hglTQ5Ylgot6vXr2xLYq0XWbe/Iga@public.gmane.org"
+TAG+="systemd", ENV{ID_RDMA_IWARP}=="1", ENV{SYSTEMD_WANTS}+="rdma-load-modules-mPXOxrv7gkYc5CkaFG5UGg@public.gmane.org"
+TAG+="systemd", ENV{ID_RDMA_OPA}=="1", ENV{SYSTEMD_WANTS}+="rdma-load-modules-EhpGOaQAOk9DcWalurthmA@public.gmane.org"
+TAG+="systemd", ENV{ID_RDMA_ROCE}=="1", ENV{SYSTEMD_WANTS}+="rdma-load-modules-eW91h9tu8Y5TDjBF/Jpztg@public.gmane.org"
+
+LABEL="rdma_ulp_modules_end"
diff --git a/redhat/rdma-core.spec b/redhat/rdma-core.spec
index 2565290e72c04e..612cf5c808091c 100644
--- a/redhat/rdma-core.spec
+++ b/redhat/rdma-core.spec
@@ -320,17 +320,26 @@ rm -rf %{buildroot}/%{_initrddir}/
 %doc %{_docdir}/%{name}-%{version}/README.md
 %doc %{_docdir}/%{name}-%{version}/rxe.md
 %config(noreplace) %{_sysconfdir}/rdma/mlx4.conf
+%config(noreplace) %{_sysconfdir}/rdma/modules/infiniband.conf
+%config(noreplace) %{_sysconfdir}/rdma/modules/iwarp.conf
+%config(noreplace) %{_sysconfdir}/rdma/modules/opa.conf
+%config(noreplace) %{_sysconfdir}/rdma/modules/rdma.conf
+%config(noreplace) %{_sysconfdir}/rdma/modules/roce.conf
 %config(noreplace) %{_sysconfdir}/rdma/rdma.conf
 %config(noreplace) %{_sysconfdir}/rdma/sriov-vfs
 %config(noreplace) %{_sysconfdir}/udev/rules.d/*
 %config(noreplace) %{_sysconfdir}/modprobe.d/mlx4.conf
 %config(noreplace) %{_sysconfdir}/modprobe.d/truescale.conf
 %{_sysconfdir}/sysconfig/network-scripts/*
+%{_unitdir}/rdma-load-modules@.service
 %{_unitdir}/rdma.service
 %dir %{dracutlibdir}/modules.d/05rdma
 %{dracutlibdir}/modules.d/05rdma/module-setup.sh
-%{_udevrulesdir}/98-rdma.rules
 %{_udevrulesdir}/60-rdma-ndd.rules
+%{_udevrulesdir}/75-rdma-description.rules
+%{_udevrulesdir}/90-rdma-hw-modules.rules
+%{_udevrulesdir}/90-rdma-ulp-modules.rules
+%{_udevrulesdir}/98-rdma.rules
 %{sysmodprobedir}/libmlx4.conf
 %{sysmodprobedir}/cxgb3.conf
 %{sysmodprobedir}/cxgb4.conf
-- 
2.7.4

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH rdma-core 2/5] srp: Autoload the SRP kernel module if required

[Jason Gunthorpe]

We use the rdma-core module loading infrastructure to load a srp_daemon
specific module list before allowing srp_daemon to start.

This is in lieu of kernel module auto loading, which is hard to trigger
for sysfs based interfaces.

Signed-off-by: Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
---
 debian/control                         | 2 +-
 debian/srptools.install                | 1 +
 redhat/rdma-core.spec                  | 1 +
 srp_daemon/CMakeLists.txt              | 4 ++++
 srp_daemon/modules-srp_daemon.conf     | 2 ++
 srp_daemon/srp_daemon.service.in       | 2 +-
 srp_daemon/srp_daemon_port@.service.in | 3 ++-
 7 files changed, 12 insertions(+), 3 deletions(-)
 create mode 100644 srp_daemon/modules-srp_daemon.conf

diff --git a/debian/control b/debian/control
index 9d80e38bb07864..d7e20e4879a19c 100644
--- a/debian/control
+++ b/debian/control
@@ -341,7 +341,7 @@ Description: Examples for the librdmacm library
 
 Package: srptools
 Architecture: linux-any
-Depends: lsb-base (>= 3.2-14~), ${misc:Depends}, ${shlibs:Depends}
+Depends: lsb-base (>= 3.2-14~), rdma-core, ${misc:Depends}, ${shlibs:Depends}
 Description: Tools for Infiniband attached storage (SRP)
  In conjunction with the kernel ib_srp driver, srptools allows you to
  discover and use Infiniband attached storage devices which use the
diff --git a/debian/srptools.install b/debian/srptools.install
index f780d663c9113b..a30798cf088a03 100644
--- a/debian/srptools.install
+++ b/debian/srptools.install
@@ -1,3 +1,4 @@
+etc/rdma/modules/srp_daemon.conf
 etc/srp_daemon.conf
 lib/systemd/system/srp_daemon.service
 lib/systemd/system/srp_daemon_port@.service
diff --git a/redhat/rdma-core.spec b/redhat/rdma-core.spec
index 612cf5c808091c..124aec402997c5 100644
--- a/redhat/rdma-core.spec
+++ b/redhat/rdma-core.spec
@@ -452,6 +452,7 @@ rm -rf %{buildroot}/%{_initrddir}/
 
 %files -n srp_daemon
 %config(noreplace) %{_sysconfdir}/srp_daemon.conf
+%config(noreplace) %{_sysconfdir}/rdma/modules/srp_daemon.conf
 %{_libexecdir}/srp_daemon/start_on_all_ports
 %{_unitdir}/srp_daemon.service
 %{_unitdir}/srp_daemon_port@.service
diff --git a/srp_daemon/CMakeLists.txt b/srp_daemon/CMakeLists.txt
index 92a1d709364b0b..0e7de9594b6375 100644
--- a/srp_daemon/CMakeLists.txt
+++ b/srp_daemon/CMakeLists.txt
@@ -47,6 +47,10 @@ install(FILES "srp_daemon.rules"
   RENAME "60-srp_daemon.rules"
   DESTINATION "${CMAKE_INSTALL_UDEV_RULESDIR}")
 
+install(FILES modules-srp_daemon.conf
+  RENAME "srp_daemon.conf"
+  DESTINATION "${CMAKE_INSTALL_SYSCONFDIR}/rdma/modules")
+
 # FIXME: The ib init.d file should really be included in rdma-core as well.
 set(RDMA_SERVICE "openibd" CACHE STRING "init.d file service name to order srpd after")
 # NOTE: These defaults are for CentOS, packagers should override.
diff --git a/srp_daemon/modules-srp_daemon.conf b/srp_daemon/modules-srp_daemon.conf
new file mode 100644
index 00000000000000..dbe434362f10a3
--- /dev/null
+++ b/srp_daemon/modules-srp_daemon.conf
@@ -0,0 +1,2 @@
+# These modules are loaded by the system if srp_daemon is to be run
+ib_srp
diff --git a/srp_daemon/srp_daemon.service.in b/srp_daemon/srp_daemon.service.in
index 33dddd5cb46fef..cca1fce9c99283 100644
--- a/srp_daemon/srp_daemon.service.in
+++ b/srp_daemon/srp_daemon.service.in
@@ -1,6 +1,6 @@
 [Unit]
 Description=Daemon that discovers and logs in to SRP target systems
-Documentation=man:srp_daemon file:/etc/rdma/rdma.conf file:/etc/srp_daemon.conf
+Documentation=man:srp_daemon file:/etc/srp_daemon.conf
 DefaultDependencies=false
 Conflicts=emergency.target emergency.service
 Before=remote-fs-pre.target
diff --git a/srp_daemon/srp_daemon_port@.service.in b/srp_daemon/srp_daemon_port@.service.in
index 03b9cb9a70119a..69d3075ae77a1b 100644
--- a/srp_daemon/srp_daemon_port@.service.in
+++ b/srp_daemon/srp_daemon_port@.service.in
@@ -3,7 +3,8 @@ Description=SRP daemon that monitors port %i
 Documentation=man:srp_daemon file:/etc/rdma/rdma.conf file:/etc/srp_daemon.conf
 DefaultDependencies=false
 Conflicts=emergency.target emergency.service
-After=srp_daemon.service sys-subsystem-rdma-devices-%i-umad.device network.target
+Requires=rdma-load-modules@srp_daemon.service
+After=srp_daemon.service rdma-load-modules@srp_daemon.service sys-subsystem-rdma-devices-%i-umad.device network.target
 BindsTo=srp_daemon.service sys-subsystem-rdma-devices-%i-umad.device
 Before=remote-fs-pre.target
 
-- 
2.7.4

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH rdma-core 3/5] iwpmd: Autoload iwpmd if required

[Jason Gunthorpe]

Use udev to determine if an iWarp device is present, and if so then start
iwpmd and autoload the required module. iwpmd no longer starts if it is
simply installed.

Signed-off-by: Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
---
 debian/iwpmd.install           | 2 ++
 iwpmd/CMakeLists.txt           | 8 ++++++++
 iwpmd/iwpmd.rules              | 1 +
 iwpmd/iwpmd.service.in         | 6 +++---
 iwpmd/modules-iwpmd.conf       | 2 ++
 kernel-boot/modules/iwarp.conf | 1 -
 redhat/rdma-core.spec          | 2 ++
 7 files changed, 18 insertions(+), 4 deletions(-)
 create mode 100644 iwpmd/iwpmd.rules
 create mode 100644 iwpmd/modules-iwpmd.conf

diff --git a/debian/iwpmd.install b/debian/iwpmd.install
index cd9f7adbdd287d..51ac19ad382e9e 100644
--- a/debian/iwpmd.install
+++ b/debian/iwpmd.install
@@ -1,6 +1,8 @@
 etc/init.d/iwpmd
 etc/iwpmd.conf
+etc/rdma/modules/iwpmd.conf
 lib/systemd/system/iwpmd.service
+lib/udev/rules.d/90-iwpmd.rules
 usr/sbin/iwpmd
 usr/share/man/man1/iwpmd.1
 usr/share/man/man5/iwpmd.conf.5
diff --git a/iwpmd/CMakeLists.txt b/iwpmd/CMakeLists.txt
index 200b14c1786422..f500d196acf649 100644
--- a/iwpmd/CMakeLists.txt
+++ b/iwpmd/CMakeLists.txt
@@ -21,3 +21,11 @@ rdma_subst_install(FILES "iwpmd_init.in"
   RENAME "iwpmd"
   PERMISSIONS OWNER_WRITE OWNER_READ GROUP_READ WORLD_READ OWNER_EXECUTE GROUP_EXECUTE WORLD_EXECUTE)
 install(FILES "iwpmd.conf" DESTINATION "${CMAKE_INSTALL_SYSCONFDIR}")
+
+install(FILES "iwpmd.rules"
+  RENAME "90-iwpmd.rules"
+  DESTINATION "${CMAKE_INSTALL_UDEV_RULESDIR}")
+
+install(FILES modules-iwpmd.conf
+  RENAME "iwpmd.conf"
+  DESTINATION "${CMAKE_INSTALL_SYSCONFDIR}/rdma/modules")
diff --git a/iwpmd/iwpmd.rules b/iwpmd/iwpmd.rules
new file mode 100644
index 00000000000000..5b22cceaf29f8c
--- /dev/null
+++ b/iwpmd/iwpmd.rules
@@ -0,0 +1 @@
+TAG+="systemd", ENV{ID_RDMA_IWARP}=="1", ENV{SYSTEMD_WANTS}+="iwpmd.service"
diff --git a/iwpmd/iwpmd.service.in b/iwpmd/iwpmd.service.in
index 47ca6518ee127c..83c1da98558a9f 100644
--- a/iwpmd/iwpmd.service.in
+++ b/iwpmd/iwpmd.service.in
@@ -1,12 +1,12 @@
 [Unit]
 Description=iWarp Port Mapper
 Documentation=man:iwpmd file:/etc/iwpmd.conf
-After=network.target
+Requires=rdma-load-modules-r79FGtN6zq8c5CkaFG5UGg@public.gmane.org
+After=network.target rdma-load-modules-r79FGtN6zq8c5CkaFG5UGg@public.gmane.org
 
 [Service]
 ExecStart=@CMAKE_INSTALL_FULL_SBINDIR@/iwpmd
 LimitNOFILE=102400
 KillMode=process
 
-[Install]
-WantedBy=multi-user.target
+# iwpmd is automatically started by udev when an iWarp RDMA device is present
diff --git a/iwpmd/modules-iwpmd.conf b/iwpmd/modules-iwpmd.conf
new file mode 100644
index 00000000000000..5544b359c08814
--- /dev/null
+++ b/iwpmd/modules-iwpmd.conf
@@ -0,0 +1,2 @@
+# These modules are loaded by the system if iwpmd is to be run
+iw_cm
diff --git a/kernel-boot/modules/iwarp.conf b/kernel-boot/modules/iwarp.conf
index 882146e41ee2ba..0cb831da5bb557 100644
--- a/kernel-boot/modules/iwarp.conf
+++ b/kernel-boot/modules/iwarp.conf
@@ -1,2 +1 @@
 # These modules are loaded by the system if any iWarp device is installed
-iw_cm
diff --git a/redhat/rdma-core.spec b/redhat/rdma-core.spec
index 124aec402997c5..901c726037fdfe 100644
--- a/redhat/rdma-core.spec
+++ b/redhat/rdma-core.spec
@@ -403,7 +403,9 @@ rm -rf %{buildroot}/%{_initrddir}/
 %files -n iwpmd
 %{_sbindir}/iwpmd
 %{_unitdir}/iwpmd.service
+%config(noreplace) %{_sysconfdir}/rdma/modules/iwpmd.conf
 %config(noreplace) %{_sysconfdir}/iwpmd.conf
+%{_udevrulesdir}/90-iwpmd.rules
 %{_mandir}/man1/iwpmd.*
 %{_mandir}/man5/iwpmd.*
 
-- 
2.7.4

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH rdma-core 4/5] redhat: Remove cxgb3/4.modprobe files

[Jason Gunthorpe]

This is now handled by the udev rules in kernel-boot

Signed-off-by: Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
---
 redhat/rdma-core.spec          | 4 ----
 redhat/rdma.cxgb3.sys.modprobe | 1 -
 redhat/rdma.cxgb4.sys.modprobe | 1 -
 3 files changed, 6 deletions(-)
 delete mode 100644 redhat/rdma.cxgb3.sys.modprobe
 delete mode 100644 redhat/rdma.cxgb4.sys.modprobe

diff --git a/redhat/rdma-core.spec b/redhat/rdma-core.spec
index 901c726037fdfe..1e4aee6a211f33 100644
--- a/redhat/rdma-core.spec
+++ b/redhat/rdma-core.spec
@@ -260,8 +260,6 @@ install -D -m0644 redhat/rdma.udev-ipoib-naming.rules %{buildroot}%{_sysconfdir}
 install -D -m0755 redhat/rdma.modules-setup.sh %{buildroot}%{dracutlibdir}/modules.d/05rdma/module-setup.sh
 install -D -m0644 redhat/rdma.udev-rules %{buildroot}%{_udevrulesdir}/98-rdma.rules
 install -D -m0644 redhat/rdma.mlx4.sys.modprobe %{buildroot}%{sysmodprobedir}/libmlx4.conf
-install -D -m0644 redhat/rdma.cxgb3.sys.modprobe %{buildroot}%{sysmodprobedir}/cxgb3.conf
-install -D -m0644 redhat/rdma.cxgb4.sys.modprobe %{buildroot}%{sysmodprobedir}/cxgb4.conf
 install -D -m0755 redhat/rdma.kernel-init %{buildroot}%{_libexecdir}/rdma-init-kernel
 install -D -m0755 redhat/rdma.sriov-init %{buildroot}%{_libexecdir}/rdma-set-sriov-vf
 install -D -m0644 redhat/rdma.fixup-mtrr.awk %{buildroot}%{_libexecdir}/rdma-fixup-mtrr.awk
@@ -341,8 +339,6 @@ rm -rf %{buildroot}/%{_initrddir}/
 %{_udevrulesdir}/90-rdma-ulp-modules.rules
 %{_udevrulesdir}/98-rdma.rules
 %{sysmodprobedir}/libmlx4.conf
-%{sysmodprobedir}/cxgb3.conf
-%{sysmodprobedir}/cxgb4.conf
 %{_libexecdir}/rdma-init-kernel
 %{_libexecdir}/rdma-set-sriov-vf
 %{_libexecdir}/rdma-fixup-mtrr.awk
diff --git a/redhat/rdma.cxgb3.sys.modprobe b/redhat/rdma.cxgb3.sys.modprobe
deleted file mode 100644
index d5925a715ccee1..00000000000000
diff --git a/redhat/rdma.cxgb4.sys.modprobe b/redhat/rdma.cxgb4.sys.modprobe
deleted file mode 100644
index 44163ab43cf649..00000000000000
-- 
2.7.4

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH rdma-core 5/5] Move umad udev rules to rdma-core

[Jason Gunthorpe]

This works better from a packaging perspective, eg rdma-core
is already a matched dependency for srp_daemon in RedHat, and provides
one place to get all the udev meta data rules.

Also update debian's dependencies to rely on a new RDMA core
for srptools, as it now requires the rules files and
rdma-load-modules@.service.in from rdma-core.

Signed-off-by: Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
---
 debian/control                                                | 2 +-
 debian/libibumad3.install                                     | 1 -
 debian/rdma-core.install                                      | 1 +
 kernel-boot/CMakeLists.txt                                    | 4 ++++
 libibumad/libibumad.udev-rules => kernel-boot/rdma-umad.rules | 0
 libibumad/CMakeLists.txt                                      | 5 -----
 redhat/rdma-core.spec                                         | 2 +-
 7 files changed, 7 insertions(+), 8 deletions(-)
 rename libibumad/libibumad.udev-rules => kernel-boot/rdma-umad.rules (100%)

diff --git a/debian/control b/debian/control
index d7e20e4879a19c..9a40ef17d60833 100644
--- a/debian/control
+++ b/debian/control
@@ -341,7 +341,7 @@ Description: Examples for the librdmacm library
 
 Package: srptools
 Architecture: linux-any
-Depends: lsb-base (>= 3.2-14~), rdma-core, ${misc:Depends}, ${shlibs:Depends}
+Depends: lsb-base (>= 3.2-14~), rdma-core (>= 15), ${misc:Depends}, ${shlibs:Depends}
 Description: Tools for Infiniband attached storage (SRP)
  In conjunction with the kernel ib_srp driver, srptools allows you to
  discover and use Infiniband attached storage devices which use the
diff --git a/debian/libibumad3.install b/debian/libibumad3.install
index 41339ebfa2f229..f6c298db52eb34 100644
--- a/debian/libibumad3.install
+++ b/debian/libibumad3.install
@@ -1,2 +1 @@
-lib/udev/rules.d/90-libibumad.rules
 usr/lib/*/libibumad*.so.*
diff --git a/debian/rdma-core.install b/debian/rdma-core.install
index 0bda539494189a..df33b8a6070d22 100644
--- a/debian/rdma-core.install
+++ b/debian/rdma-core.install
@@ -11,6 +11,7 @@ lib/udev/rules.d/60-rdma-ndd.rules
 lib/udev/rules.d/75-rdma-description.rules
 lib/udev/rules.d/90-rdma-hw-modules.rules
 lib/udev/rules.d/90-rdma-ulp-modules.rules
+lib/udev/rules.d/90-rdma-umad.rules
 usr/bin/rxe_cfg
 usr/lib/truescale-serdes.cmds
 usr/sbin/rdma-ndd
diff --git a/kernel-boot/CMakeLists.txt b/kernel-boot/CMakeLists.txt
index 0d4a2aec1c6a94..fdb70117f5899c 100644
--- a/kernel-boot/CMakeLists.txt
+++ b/kernel-boot/CMakeLists.txt
@@ -22,3 +22,7 @@ install(FILES "rdma-hw-modules.rules"
 install(FILES "rdma-ulp-modules.rules"
   RENAME "90-rdma-ulp-modules.rules"
   DESTINATION "${CMAKE_INSTALL_UDEV_RULESDIR}")
+
+install(FILES "rdma-umad.rules"
+  RENAME "90-rdma-umad.rules"
+  DESTINATION "${CMAKE_INSTALL_UDEV_RULESDIR}")
diff --git a/libibumad/libibumad.udev-rules b/kernel-boot/rdma-umad.rules
similarity index 100%
rename from libibumad/libibumad.udev-rules
rename to kernel-boot/rdma-umad.rules
diff --git a/libibumad/CMakeLists.txt b/libibumad/CMakeLists.txt
index 84673ee01271cb..ba2463b47ed82a 100644
--- a/libibumad/CMakeLists.txt
+++ b/libibumad/CMakeLists.txt
@@ -14,8 +14,3 @@ rdma_library(ibumad libibumad.map
   umad.c
   umad_str.c
   )
-
-install(FILES libibumad.udev-rules
-  RENAME 90-libibumad.rules
-  DESTINATION "${CMAKE_INSTALL_UDEV_RULESDIR}")
-
diff --git a/redhat/rdma-core.spec b/redhat/rdma-core.spec
index 1e4aee6a211f33..7ed419394d22d6 100644
--- a/redhat/rdma-core.spec
+++ b/redhat/rdma-core.spec
@@ -337,6 +337,7 @@ rm -rf %{buildroot}/%{_initrddir}/
 %{_udevrulesdir}/75-rdma-description.rules
 %{_udevrulesdir}/90-rdma-hw-modules.rules
 %{_udevrulesdir}/90-rdma-ulp-modules.rules
+%{_udevrulesdir}/90-rdma-umad.rules
 %{_udevrulesdir}/98-rdma.rules
 %{sysmodprobedir}/libmlx4.conf
 %{_libexecdir}/rdma-init-kernel
@@ -411,7 +412,6 @@ rm -rf %{buildroot}/%{_initrddir}/
 
 %files -n libibumad
 %{_libdir}/libibumad*.so.*
-%{_udevrulesdir}/90-libibumad.rules
 
 %files -n librdmacm
 %{_libdir}/librdmacm*.so.*
-- 
2.7.4

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH rdma-core 0/5] Common systemd/udev based boot support

[Steve Wise]

> I do not have nearly enough hardware to exhaustively test this, I have checked
> that the basic elements are working, but there is lots of room for little
> errors, particularly in module names and udev matching. It would be great of
> someone could try this on iwarp hardware and confirm all the autoloading works
> as expected.

Hey Jason,

Thanks for doing all this!  I'm testing with cxgb4 today.

I get these errors building the rpms using the spec file in your tree:

Checking for unpackaged file(s): /usr/lib/rpm/check-files
/root/rpmbuild/BUILDROOT/rdma-core-15-1.el7.x86_64
error: Installed (but unpackaged) file(s) found:
   /etc/rdma/modules/infiniband.conf
   /etc/rdma/modules/iwarp.conf
   /etc/rdma/modules/iwpmd.conf
   /etc/rdma/modules/opa.conf
   /etc/rdma/modules/rdma.conf
   /etc/rdma/modules/roce.conf
   /etc/rdma/modules/srp_daemon.conf


RPM build errors:
    Installed (but unpackaged) file(s) found:
   /etc/rdma/modules/infiniband.conf
   /etc/rdma/modules/iwarp.conf
   /etc/rdma/modules/iwpmd.conf
   /etc/rdma/modules/opa.conf
   /etc/rdma/modules/rdma.conf
   /etc/rdma/modules/roce.conf
   /etc/rdma/modules/srp_daemon.conf

Are these expected?


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH rdma-core 0/5] Common systemd/udev based boot support

[Jason Gunthorpe]

On Tue, Jul 25, 2017 at 11:27:50AM -0500, Steve Wise wrote:
> > I do not have nearly enough hardware to exhaustively test this, I have checked
> > that the basic elements are working, but there is lots of room for little
> > errors, particularly in module names and udev matching. It would be great of
> > someone could try this on iwarp hardware and confirm all the autoloading works
> > as expected.
> 
> Hey Jason,
> 
> Thanks for doing all this!  I'm testing with cxgb4 today.
> 
> I get these errors building the rpms using the spec file in your tree:

I think you are using ./rdma-core.spec? I forgot about that one and
didn't make it work, I will fix it.

You probably want to use redhat/rdma-core.spec anyhow?

When testing, you want to start with a system that doesn't load any
modules, then install the new rpms and see what happens. (rdma-core, iwpmd)

For iwarp I expect that systemd will auto start these units:

  rdma-load-modules-r79FGtN6zq8c5CkaFG5UGg@public.gmane.org  loaded active     exited    Load RDMA modules from /etc/rdma/modules/iwpmd.conf
  rdma-load-modules-XSrBNAzizW1TDjBF/Jpztg@public.gmane.org   loaded active     exited    Load RDMA modules from /etc/rdma/modules/rdma.conf
  rdma-load-modules-mPXOxrv7gkYc5CkaFG5UGg@public.gmane.org  loaded active     exited    Load RDMA modules from /etc/rdma/modules/iwarp.conf
  iwpmd.service                    loaded active     running   iWarp Port Mapper

And you should have a reasonable set of modules loaded. Check the
journal for interesting errors also..

For cxgb4 these are the two critical udev lines:

ENV{ID_NET_DRIVER}=="cxgb4", RUN{builtin}+="kmod load iw_cxgb4"
DRIVERS=="cxgb4", ENV{ID_RDMA_IWARP}="1"

The first should load iw_cxgb4 if an ethernet device with the cxgb4
driver is found (the driver name being matched is as reported by
ethtool -i)

Both need to trigger for things to work properly, you can check with:

$ udevadm info /sys/class/net/ethX
$ udevadm info /sys/class/infiniband/cxgb4_0 # ??

Thanks,
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH rdma-core 0/5] Common systemd/udev based boot support

[Steve Wise]

> >
> > Hey Jason,
> >
> > Thanks for doing all this!  I'm testing with cxgb4 today.
> >
> > I get these errors building the rpms using the spec file in your tree:
> 
> I think you are using ./rdma-core.spec? I forgot about that one and
> didn't make it work, I will fix it.
> 
> You probably want to use redhat/rdma-core.spec anyhow?
>

Ok.  I haven't been paying attention to the distro-specific stuff. I had a
process I have been following to build an rpm from ./rdma-core.spec.  I'll try
the redhat spec file since I'm installing on a RHEL7.2 system.
 
> When testing, you want to start with a system that doesn't load any
> modules, then install the new rpms and see what happens. (rdma-core, iwpmd)
> 

I've uninstalled all the pertinent packages, I think, with this:

yum autoremove -y libibverbs librdmacm libibumad libiwpm

> For iwarp I expect that systemd will auto start these units:
> 
>   rdma-load-modules-r79FGtN6zq8c5CkaFG5UGg@public.gmane.org  loaded active     exited    Load RDMA
> modules from /etc/rdma/modules/iwpmd.conf
>   rdma-load-modules-XSrBNAzizW1TDjBF/Jpztg@public.gmane.org   loaded active     exited    Load RDMA
modules
> from /etc/rdma/modules/rdma.conf
>   rdma-load-modules-mPXOxrv7gkYc5CkaFG5UGg@public.gmane.org  loaded active     exited    Load RDMA
modules
> from /etc/rdma/modules/iwarp.conf
>   iwpmd.service                    loaded active     running   iWarp Port
Mapper
> 
> And you should have a reasonable set of modules loaded. Check the
> journal for interesting errors also..
> 

Will do.

> For cxgb4 these are the two critical udev lines:
> 
> ENV{ID_NET_DRIVER}=="cxgb4", RUN{builtin}+="kmod load iw_cxgb4"
> DRIVERS=="cxgb4", ENV{ID_RDMA_IWARP}="1"
> 
> The first should load iw_cxgb4 if an ethernet device with the cxgb4
> driver is found (the driver name being matched is as reported by
> ethtool -i)
> 
> Both need to trigger for things to work properly, you can check with:
> 
> $ udevadm info /sys/class/net/ethX
> $ udevadm info /sys/class/infiniband/cxgb4_0 # ??
>

Thanks!  I'll let you know how it goes.

Steve.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH rdma-core 0/5] Common systemd/udev based boot support

[Jason Gunthorpe]

On Tue, Jul 25, 2017 at 11:43:49AM -0500, Steve Wise wrote:
> 
> > >
> > > Hey Jason,
> > >
> > > Thanks for doing all this!  I'm testing with cxgb4 today.
> > >
> > > I get these errors building the rpms using the spec file in your tree:
> > 
> > I think you are using ./rdma-core.spec? I forgot about that one and
> > didn't make it work, I will fix it.
> > 
> > You probably want to use redhat/rdma-core.spec anyhow?
> >
> 
> Ok.  I haven't been paying attention to the distro-specific stuff. I had a
> process I have been following to build an rpm from ./rdma-core.spec.  I'll try
> the redhat spec file since I'm installing on a RHEL7.2 system.

Actually.. Humm, I've been testing the debian build. If you use the
redhat/rdma-core.spec it will include the old redhat init scheme and
that kind of invalidates the testing of the new scheme ...

I just forced pushed the needed fix to my branch so you should be able
to keep doing what you were doing.

> I've uninstalled all the pertinent packages, I think, with this:
> 
> yum autoremove -y libibverbs librdmacm libibumad libiwpm

Also the rdma package needs to go (eg rdma.service and
/lib/udev/rules.d/98-rdma.rules must not exist)

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH rdma-core 0/5] Common systemd/udev based boot support

[Steve Wise]

> 
> 
> > >
> > > Hey Jason,
> > >
> > > Thanks for doing all this!  I'm testing with cxgb4 today.
> > >
> > > I get these errors building the rpms using the spec file in your tree:
> >
> > I think you are using ./rdma-core.spec? I forgot about that one and
> > didn't make it work, I will fix it.
> >
> > You probably want to use redhat/rdma-core.spec anyhow?
> >
> 
> Ok.  I haven't been paying attention to the distro-specific stuff. I had a
process I
> have been following to build an rpm from ./rdma-core.spec.  I'll try the
redhat spec
> file since I'm installing on a RHEL7.2 system.

Hmm, the rdma-core rpm built ok, but there are no libs or commands in it. ?  I
think I'm in the weeds. here :)  

What I did was:

[root@stevo3 rdma-core-15]# git clone
https://github.com/jgunthorpe/rdma-plumbing rdma-core-15
[root@stevo3 rdma-core-15]# tar czf ~/rpmbuild/SOURCES/rdma-core-15.tgz
rdma-core-15
[root@stevo3 rdma-core-15]# rpmbuild -bb rdma-core-15/redhat/rdma-core.spec
[root@stevo3 rdma-core-15]# rpm --install
~/rpmbuild/RPMS/x86_64/rdma-core-devel-15-1.el7.x86_64.rpm
[root@stevo3 rdma-core-15]# rpm -ql rdma-core
/etc/modprobe.d/mlx4.conf
/etc/modprobe.d/truescale.conf
/etc/rdma
/etc/rdma/mlx4.conf
/etc/rdma/modules/infiniband.conf
/etc/rdma/modules/iwarp.conf
/etc/rdma/modules/opa.conf
/etc/rdma/modules/rdma.conf
/etc/rdma/modules/roce.conf
/etc/rdma/rdma.conf
/etc/rdma/sriov-vfs
/etc/sysconfig/network-scripts/ifdown-ib
/etc/sysconfig/network-scripts/ifup-ib
/etc/udev/rules.d/70-persistent-ipoib.rules
/usr/bin/rxe_cfg
/usr/lib/dracut/modules.d/05rdma
/usr/lib/dracut/modules.d/05rdma/module-setup.sh
/usr/lib/modprobe.d/libmlx4.conf
/usr/lib/systemd/system/rdma-load-modules@.service
/usr/lib/systemd/system/rdma-ndd.service
/usr/lib/systemd/system/rdma.service
/usr/lib/udev/rules.d/60-rdma-ndd.rules
/usr/lib/udev/rules.d/75-rdma-description.rules
/usr/lib/udev/rules.d/90-rdma-hw-modules.rules
/usr/lib/udev/rules.d/90-rdma-ulp-modules.rules
/usr/lib/udev/rules.d/90-rdma-umad.rules
/usr/lib/udev/rules.d/98-rdma.rules
/usr/libexec/mlx4-setup.sh
/usr/libexec/rdma-fixup-mtrr.awk
/usr/libexec/rdma-init-kernel
/usr/libexec/rdma-set-sriov-vf
/usr/libexec/truescale-serdes.cmds
/usr/sbin/rdma-ndd
/usr/share/doc/rdma-core-15
/usr/share/doc/rdma-core-15/README.md
/usr/share/doc/rdma-core-15/rxe.md
/usr/share/licenses/rdma-core-15
/usr/share/licenses/rdma-core-15/COPYING.BSD_FB
/usr/share/licenses/rdma-core-15/COPYING.BSD_MIT
/usr/share/licenses/rdma-core-15/COPYING.GPL2
/usr/share/licenses/rdma-core-15/COPYING.md
/usr/share/man/man7/rxe.7.gz
/usr/share/man/man8/rdma-ndd.8.gz
/usr/share/man/man8/rxe_cfg.8.gz

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH rdma-core 0/5] Common systemd/udev based boot support

[Jason Gunthorpe]

On Tue, Jul 25, 2017 at 12:01:43PM -0500, Steve Wise wrote:

> Hmm, the rdma-core rpm built ok, but there are no libs or commands in it. ?  I
> think I'm in the weeds. here :)  

The redhat/rdma-core.spec produces 12 rpms:

ibacm-15-1.el7.centos.x86_64.rpm
iwpmd-15-1.el7.centos.x86_64.rpm
libibcm-15-1.el7.centos.x86_64.rpm
libibumad-15-1.el7.centos.x86_64.rpm
libibverbs-15-1.el7.centos.x86_64.rpm
libibverbs-utils-15-1.el7.centos.x86_64.rpm
librdmacm-15-1.el7.centos.x86_64.rpm
librdmacm-utils-15-1.el7.centos.x86_64.rpm
rdma-core-15-1.el7.centos.x86_64.rpm
rdma-core-debuginfo-15-1.el7.centos.x86_64.rpm
rdma-core-devel-15-1.el7.centos.x86_64.rpm
srp_daemon-15-1.el7.centos.x86_64.rpm

The rdma-core package in this system only contains the kernel boot
support.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH rdma-core 0/5] Common systemd/udev based boot support

[Steve Wise]

> 
> The rdma-core package in this system only contains the kernel boot
> support.
> 

Oops.  Thanks.


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH rdma-core 1/5] Common infrastructure for auto loading rdma modules

[Bart Van Assche]

On Mon, 2017-07-24 at 14:44 -0600, Jason Gunthorpe wrote:
> +This is to avoid exposing systems not using RDMA from having RDMA enabled, for
> +instance if a system has a multi-protocol ethernet adaptor, but is only using
> +the net stack interface.

adaptor -> adapter ?

> +Finally udev will cause systemd to start RDMA specific daemons like
> +srp_deamon, rdma-ndd and iwpmd. These starts are linked to the detection of
> +the first RDMA hardware, and the daemons internally handle hot plug events for
> +other hardware.

Please change srp_deamon into srp_daemon such that the spelling matches the name
of the executable.

> +
> +## Hot Plug compatible services
> +
> +RDMA using services need to have device specific systemd dependencies in their
> +unit files, either created by hand by the admin or by using udev rules.

"RDMA using services" -> "Services using RDMA" ?

> +++ b/kernel-boot/modules/infiniband.conf
> @@ -0,0 +1,12 @@
> +# These modules are loaded by the system if any InfiniBand device is installed
> +# Infiniband over IP netdevice

Please spell "InfiniBand" consistently in the above comment.

> +ib_ipoib
> +
> +# Access to fabric management SMPs and GMPs from userspace.
> +ib_umad
> +
> +# SCSI Remote Protocol target support
> +# ib_srpt
> +
> +# ib_ucm provides the obsolete /dev/infiniband/ucm0
> +# ib_ucm

If ib_iser is loaded by default, should ib_srp also be loaded by default if the
appropriate hardware is present? I don't think that there are fewer SRP users
than iSER users.

> diff --git a/kernel-boot/rdma-description.rules b/kernel-boot/rdma-description.rules
> [ ... ]
> +# Hardware that supports RoCE
> +DRIVERS=="be2net", ENV{ID_RDMA_ROCE}="1"
> +DRIVERS=="bnxt_en", ENV{ID_RDMA_ROCE}="1"
> +DRIVERS=="hns", ENV{ID_RDMA_ROCE}="1"
> +DRIVERS=="i40e", ENV{ID_RDMA_ROCE}="1"
> +DRIVERS=="mlx4_core", ENV{ID_RDMA_ROCE}="1"
> +DRIVERS=="mlx5_core", ENV{ID_RDMA_ROCE}="1"
> +DRIVERS=="qede", ENV{ID_RDMA_ROCE}="1"

Should the "rdma_rxe" driver be added to this list?

> +ENV{ID_NET_DRIVER}=="mlx4_en", RUN{builtin}+="kmod load mlx4_ib"
> +ENV{ID_NET_DRIVER}=="mlx5_core", RUN{builtin}+="kmod load mlx5_ib"

Why this inconsistency between mlx4 and mlx5? Additionally, if these rules are
added, shouldn't the request_module() calls be removed from the mlx4 and ml5 core
drivers?

Anyway, nice work!

Bart.--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH rdma-core 1/5] Common infrastructure for auto loading rdma modules

[Jason Gunthorpe]

On Tue, Jul 25, 2017 at 05:15:18PM +0000, Bart Van Assche wrote:

Got all the spelling errors, thanks

> > +ib_ipoib
> > +
> > +# Access to fabric management SMPs and GMPs from userspace.
> > +ib_umad
> > +
> > +# SCSI Remote Protocol target support
> > +# ib_srpt
> > +
> > +# ib_ucm provides the obsolete /dev/infiniband/ucm0
> > +# ib_ucm
> 
> If ib_iser is loaded by default, should ib_srp also be loaded by default if the
> appropriate hardware is present? I don't think that there are fewer SRP users
> than iSER users.

ib_srp now autoloads when srp_daemon is started, which AFAIK, is a
requirement to use srp? Since it autoloads it does not need to be
specified in this file any more.

Generally as things learn to autoload they should be removed from
these files, and people should be really thinking hard about getting
things to autoload as it solves hard boot time ordering problems and
races.

iser is left uncommented only because that is historically what RH has
done by default. 

> > diff --git a/kernel-boot/rdma-description.rules b/kernel-boot/rdma-description.rules
> > [ ... ]
> > +# Hardware that supports RoCE
> > +DRIVERS=="be2net", ENV{ID_RDMA_ROCE}="1"
> > +DRIVERS=="bnxt_en", ENV{ID_RDMA_ROCE}="1"
> > +DRIVERS=="hns", ENV{ID_RDMA_ROCE}="1"
> > +DRIVERS=="i40e", ENV{ID_RDMA_ROCE}="1"
> > +DRIVERS=="mlx4_core", ENV{ID_RDMA_ROCE}="1"
> > +DRIVERS=="mlx5_core", ENV{ID_RDMA_ROCE}="1"
> > +DRIVERS=="qede", ENV{ID_RDMA_ROCE}="1"
> 
> Should the "rdma_rxe" driver be added to this list?

No.. I need to figure out something else for rxe. We have no way in
RDMA to determine the driver binding to the RDMA device (eg like
ethtool -i), the above is detecting which driver is bound to the PCI
device that the RDMA device is a child of.

Since rxe doesn't work that way, it cannot use this detection method.

What I'd like is proper kernel support to determine ID_RDMA_*, which I
think we can ultimately implement in Leon's rdma netlink.

> > +ENV{ID_NET_DRIVER}=="mlx4_en", RUN{builtin}+="kmod load mlx4_ib"
> > +ENV{ID_NET_DRIVER}=="mlx5_core", RUN{builtin}+="kmod load mlx5_ib"
> 
> Why this inconsistency between mlx4 and mlx5?

This is matching the net driver name (eg struct ethtool_drvinfo ->
driver) used by each thing.. As far as I can tell these are the names
Mellanox choose to report here. I think it reflects that in mlx5 the
mlx5_core module provides the netdevice support, while in mlx4 it is
in the mlx4_en module.

> Additionally, if these rules are added, shouldn't the
> request_module() calls be removed from the mlx4 and ml5 core
> drivers?

This approach still relies on those request_modules to support pure-IB
versions of Mellanox cards. I would think the rocee related
request_modules should ultimately be removed though.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH rdma-core 0/5] Common systemd/udev based boot support

[Steve Wise]

> 
> For iwarp I expect that systemd will auto start these units:
> 
>   rdma-load-modules-r79FGtN6zq8c5CkaFG5UGg@public.gmane.org  loaded active     exited    Load RDMA
> modules from /etc/rdma/modules/iwpmd.conf
>   rdma-load-modules-XSrBNAzizW1TDjBF/Jpztg@public.gmane.org   loaded active     exited    Load RDMA
modules
> from /etc/rdma/modules/rdma.conf
>   rdma-load-modules-mPXOxrv7gkYc5CkaFG5UGg@public.gmane.org  loaded active     exited    Load RDMA
modules
> from /etc/rdma/modules/iwarp.conf
>   iwpmd.service                    loaded active     running   iWarp Port
Mapper
> 

After I rebooted, cxgb4 and iw_cxgb4 were loaded, but the services were not
started.  I think that is probably due to the fact that iw_cxgb4 doesn't
register with the rdma core until the cxgb4 interfaces are brought up.    I then
brought up port 0 of the cxgb4 card:

[root@stevo3 ~]# ip link set enp4s0f4 up
[root@stevo3 ~]# ip addr add 172.16.3.3/24 dev enp4s0f4

And then the services were started:

[root@stevo3 ~]# systemctl status rdma-load-modules-mPXOxrv7gkYc5CkaFG5UGg@public.gmane.org
â rdma-load-modules-mPXOxrv7gkYc5CkaFG5UGg@public.gmane.org - Load RDMA modules from
/etc/rdma/modules/iwarp.conf
   Loaded: loaded (/usr/lib/systemd/system/rdma-load-modules@.service; static;
vendor preset: disabled)
   Active: active (exited) since Tue 2017-07-25 11:43:21 PDT; 5s ago
  Process: 2571 ExecStart=/lib/systemd/systemd-modules-load
/etc/rdma/modules/%I.conf (code=exited, status=0/SUCCESS)
 Main PID: 2571 (code=exited, status=0/SUCCESS)

Jul 25 11:43:21 stevo3.asicdesigners.com systemd[1]: Starting Load RDMA modules
from /etc/rdma/modules/iwarp.conf...
Jul 25 11:43:21 stevo3.asicdesigners.com systemd[1]: Started Load RDMA modules
from /etc/rdma/modules/iwarp.conf.
[root@stevo3 ~]# systemctl status rdma-load-modules-XSrBNAzizW1TDjBF/Jpztg@public.gmane.org
â rdma-load-modules-XSrBNAzizW1TDjBF/Jpztg@public.gmane.org - Load RDMA modules from
/etc/rdma/modules/rdma.conf
   Loaded: loaded (/usr/lib/systemd/system/rdma-load-modules@.service; static;
vendor preset: disabled)
   Active: active (exited) since Tue 2017-07-25 11:43:21 PDT; 10s ago
  Process: 2572 ExecStart=/lib/systemd/systemd-modules-load
/etc/rdma/modules/%I.conf (code=exited, status=0/SUCCESS)
 Main PID: 2572 (code=exited, status=0/SUCCESS)

Jul 25 11:43:21 stevo3.asicdesigners.com systemd[1]: Starting Load RDMA modules
from /etc/rdma/modules/rdma.conf...
Jul 25 11:43:21 stevo3.asicdesigners.com systemd-modules-load[2572]: Inserted
module 'ib_iser'
Jul 25 11:43:21 stevo3.asicdesigners.com systemd-modules-load[2572]: Inserted
module 'ib_uverbs'
Jul 25 11:43:21 stevo3.asicdesigners.com systemd-modules-load[2572]: Inserted
module 'rdma_ucm'
Jul 25 11:43:21 stevo3.asicdesigners.com systemd[1]: Started Load RDMA modules
from /etc/rdma/modules/rdma.conf.
[root@stevo3 ~]# systemctl status rdma-load-modules-r79FGtN6zq8c5CkaFG5UGg@public.gmane.org
â rdma-load-modules-r79FGtN6zq8c5CkaFG5UGg@public.gmane.org - Load RDMA modules from
/etc/rdma/modules/iwpmd.conf
   Loaded: loaded (/usr/lib/systemd/system/rdma-load-modules@.service; static;
vendor preset: disabled)
   Active: active (exited) since Tue 2017-07-25 11:43:21 PDT; 17s ago
  Process: 2573 ExecStart=/lib/systemd/systemd-modules-load
/etc/rdma/modules/%I.conf (code=exited, status=0/SUCCESS)
 Main PID: 2573 (code=exited, status=0/SUCCESS)

Jul 25 11:43:21 stevo3.asicdesigners.com systemd[1]: Starting Load RDMA modules
from /etc/rdma/modules/iwpmd.conf...
Jul 25 11:43:21 stevo3.asicdesigners.com systemd[1]: Started Load RDMA modules
from /etc/rdma/modules/iwpmd.conf.

And iwpmd is running:

[root@stevo3 ~]# pgrep iwpmd
2583

> And you should have a reasonable set of modules loaded. Check the
> journal for interesting errors also..
> 

[root@stevo3 ~]# lsmod|egrep "ib_|rdma"
rpcrdma                94208  0
rdma_ucm               24576  0
ib_uverbs              61440  1 rdma_ucm
ib_iser                49152  0
rdma_cm                53248  3 ib_iser,rpcrdma,rdma_ucm
ib_cm                  45056  1 rdma_cm
libiscsi               57344  1 ib_iser
iw_cm                  45056  1 rdma_cm
scsi_transport_iscsi    98304  2 ib_iser,libiscsi
ib_core               208896  9
ib_iser,ib_cm,rdma_cm,iw_cxgb4,ib_uverbs,rpcrdma,iw_cm,rdma_ucm
sunrpc                348160  8 auth_rpcgss,nfsd,rpcrdma,nfs_acl,lockd

> For cxgb4 these are the two critical udev lines:
> 
> ENV{ID_NET_DRIVER}=="cxgb4", RUN{builtin}+="kmod load iw_cxgb4"
> DRIVERS=="cxgb4", ENV{ID_RDMA_IWARP}="1"
> 
> The first should load iw_cxgb4 if an ethernet device with the cxgb4
> driver is found (the driver name being matched is as reported by
> ethtool -i)

I assume since iw_cxgb4 was loaded, that they worked.  Because before I
installed rdma-core, after a reboot only cxgb4 was loaded.

> 
> Both need to trigger for things to work properly, you can check with:
> 
> $ udevadm info /sys/class/net/ethX
> $ udevadm info /sys/class/infiniband/cxgb4_0 # ??
> 

How does this look?

[root@stevo3 ~]# udevadm info /sys/class/net/enp4s0f4
P: /devices/pci0000:00/0000:00:02.0/0000:04:00.4/net/enp4s0f4
E: DEVPATH=/devices/pci0000:00/0000:00:02.0/0000:04:00.4/net/enp4s0f4
E: ID_BUS=pci
E: ID_MM_CANDIDATE=1
E: ID_MODEL_FROM_DATABASE=T520-LL-CR Unified Wire Ethernet Controller
E: ID_MODEL_ID=0x5411
E: ID_NET_DRIVER=cxgb4
E: ID_NET_NAME_MAC=enx000743292f20
E: ID_NET_NAME_PATH=enp4s0f4
E: ID_OUI_FROM_DATABASE=Chelsio Communications
E: ID_PATH=pci-0000:04:00.4
E: ID_PATH_TAG=pci-0000_04_00_4
E: ID_PCI_CLASS_FROM_DATABASE=Network controller
E: ID_PCI_SUBCLASS_FROM_DATABASE=Ethernet controller
E: ID_VENDOR_FROM_DATABASE=Chelsio Communications Inc
E: ID_VENDOR_ID=0x1425
E: IFINDEX=6
E: INTERFACE=enp4s0f4
E: MAJOR=0
E: MINOR=0
E: SUBSYSTEM=net
E: SYSTEMD_ALIAS=/sys/subsystem/net/devices/enp4s0f4
/sys/subsystem/net/devices/enp4s0f4
E: TAGS=:systemd:
E: USEC_INITIALIZED=68132

[root@stevo3 ~]# udevadm info /sys/class/infiniband/cxgb4_0 #??
P: /devices/pci0000:00/0000:00:02.0/0000:04:00.4/infiniband/cxgb4_0
E: DEVPATH=/devices/pci0000:00/0000:00:02.0/0000:04:00.4/infiniband/cxgb4_0
E: ID_BUS=pci
E: ID_MODEL_FROM_DATABASE=T520-LL-CR Unified Wire Ethernet Controller
E: ID_MODEL_ID=0x5411
E: ID_PCI_CLASS_FROM_DATABASE=Network controller
E: ID_PCI_SUBCLASS_FROM_DATABASE=Ethernet controller
E: ID_RDMA_IWARP=1
E: ID_VENDOR_FROM_DATABASE=Chelsio Communications Inc
E: ID_VENDOR_ID=0x1425
E: NAME=cxgb4_0
E: SUBSYSTEM=infiniband
E: SYSTEMD_WANTS=rdma-ndd.service iwpmd.service rdma-load-modules-XSrBNAzizW1TDjBF/Jpztg@public.gmane.org
rdma-load-modules-mPXOxrv7gkYc5CkaFG5UGg@public.gmane.org
E: TAGS=:systemd:
E: USEC_INITIALIZED=428610768


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH rdma-core 0/5] Common systemd/udev based boot support

[Steve Wise]

> > And you should have a reasonable set of modules loaded. Check the
> > journal for interesting errors also..

Oh and here's the journal after bringing up the cxgb4 interface.  I think it
looks good.


Jul 25 11:43:21 stevo3.asicdesigners.com kernel: iw_cxgb4:0000:04:00.4: Up
Jul 25 11:43:21 stevo3.asicdesigners.com kernel: 0000:04:00.4: On-Chip Queues
not supported on this device.
Jul 25 11:43:21 stevo3.asicdesigners.com systemd[1]: Started RDMA Node
Description Daemon.
Jul 25 11:43:21 stevo3.asicdesigners.com systemd[1]: Starting RDMA Node
Description Daemon...
Jul 25 11:43:21 stevo3.asicdesigners.com systemd[1]: Created slice
system-rdma\x2dload\x2dmodules.slice.
Jul 25 11:43:21 stevo3.asicdesigners.com systemd[1]: Starting
system-rdma\x2dload\x2dmodules.slice.
Jul 25 11:43:21 stevo3.asicdesigners.com systemd[1]: Starting Load RDMA modules
from /etc/rdma/modules/iwarp.conf...
Jul 25 11:43:21 stevo3.asicdesigners.com systemd[1]: Starting Load RDMA modules
from /etc/rdma/modules/rdma.conf...
Jul 25 11:43:21 stevo3.asicdesigners.com systemd[1]: Starting Load RDMA modules
from /etc/rdma/modules/iwpmd.conf...
Jul 25 11:43:21 stevo3.asicdesigners.com systemd[1]: Started Load RDMA modules
from /etc/rdma/modules/iwarp.conf.
Jul 25 11:43:21 stevo3.asicdesigners.com kernel: IPv6: ADDRCONF(NETDEV_UP):
enp4s0f4: link is not ready
Jul 25 11:43:21 stevo3.asicdesigners.com kernel: cxgb4 0000:04:00.4 enp4s0f4:
passive DA module inserted
Jul 25 11:43:21 stevo3.asicdesigners.com kernel: Loading iSCSI transport class
v2.0-870.
Jul 25 11:43:21 stevo3.asicdesigners.com avahi-daemon[808]: Joining mDNS
multicast group on interface enp4s0f4.IPv4 with address 172.16.3.3.
Jul 25 11:43:21 stevo3.asicdesigners.com systemd-modules-load[2573]: Inserted
module 'iw_cm'
Jul 25 11:43:21 stevo3.asicdesigners.com systemd[1]: Started Load RDMA modules
from /etc/rdma/modules/iwpmd.conf.
Jul 25 11:43:21 stevo3.asicdesigners.com avahi-daemon[808]: New relevant
interface enp4s0f4.IPv4 for mDNS.
Jul 25 11:43:21 stevo3.asicdesigners.com avahi-daemon[808]: Registering new
address record for 172.16.3.3 on enp4s0f4.IPv4.
Jul 25 11:43:21 stevo3.asicdesigners.com systemd[1]: Started iWarp Port Mapper.
Jul 25 11:43:21 stevo3.asicdesigners.com systemd[1]: Starting iWarp Port
Mapper...
Jul 25 11:43:21 stevo3.asicdesigners.com iWarpPortMapper[2583]: get_iwpm_param:
Got param (name = nl_sock_rbuf_size val = 419430400)
Jul 25 11:43:21 stevo3.asicdesigners.com kernel: ib_core:ibnl_rcv_msg: Index 3
wasn't found in client list
Jul 25 11:43:21 stevo3.asicdesigners.com systemd-modules-load[2572]: Inserted
module 'ib_iser'
Jul 25 11:43:21 stevo3.asicdesigners.com kernel: iscsi: registered transport
(iser)
Jul 25 11:43:21 stevo3.asicdesigners.com systemd-modules-load[2572]: Inserted
module 'ib_uverbs'
Jul 25 11:43:21 stevo3.asicdesigners.com systemd-modules-load[2572]: Inserted
module 'rdma_ucm'
Jul 25 11:43:21 stevo3.asicdesigners.com systemd-modules-load[2572]: Inserted
module 'rpcrdma'
Jul 25 11:43:21 stevo3.asicdesigners.com systemd[1]: Started Load RDMA modules
from /etc/rdma/modules/rdma.conf.
Jul 25 11:43:21 stevo3.asicdesigners.com kernel: RPC: Registered rdma transport
module.
Jul 25 11:43:21 stevo3.asicdesigners.com kernel: RPC: Registered rdma
backchannel transport module.
Jul 25 11:43:22 stevo3.asicdesigners.com kernel: cxgb4 0000:04:00.4 enp4s0f4:
link up, 10Gbps, full-duplex, Tx/Rx PAUSE
Jul 25 11:43:22 stevo3.asicdesigners.com kernel: IPv6: ADDRCONF(NETDEV_CHANGE):
enp4s0f4: link becomes ready
Jul 25 11:43:22 stevo3.asicdesigners.com NetworkManager[781]: <info>
(enp4s0f4): link connected
Jul 25 11:43:23 stevo3.asicdesigners.com avahi-daemon[808]: Registering new
address record for fe80::207:43ff:fe29:2f20 on enp4s0f4.*.
Jul 25 11:50:01 stevo3.asicdesigners.com systemd[1]: Started Session 4 of user
root.


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH rdma-core 0/5] Common systemd/udev based boot support

[Jason Gunthorpe]

On Tue, Jul 25, 2017 at 01:49:37PM -0500, Steve Wise wrote:

> After I rebooted, cxgb4 and iw_cxgb4 were loaded, but the services were not
> started.  I think that is probably due to the fact that iw_cxgb4 doesn't
> register with the rdma core until the cxgb4 interfaces are brought
> up.

Yes, it is done on demand when the kernel creates its side of the
interface.

> I then brought up port 0 of the cxgb4 card:
> 
> [root@stevo3 ~]# ip link set enp4s0f4 up
> [root@stevo3 ~]# ip addr add 172.16.3.3/24 dev enp4s0f4
> 
> And then the services were started:

Yep, everything you sent me looks fine, thanks for taking a look at
this.

This sort of hotplug that cxbg4 does is quite strange, what happens
when 'ip link set X down' is done? Does it remove the RDMA device?
Does 'ip link set down' block until all users go away?

This is going to make it harder for cxgb users to get a reliably
bootup at this time, we need more kernel autoloading for things to be
reliable, and I'm sure iwpmd.service needs some dependency adjusting,
I just don't know enough about it to do it right. :\

> I assume since iw_cxgb4 was loaded, that they worked.  Because before I
> installed rdma-core, after a reboot only cxgb4 was loaded.

Yep

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH rdma-core 0/5] Common systemd/udev based boot support

[Steve Wise]

> -----Original Message-----
> From: Jason Gunthorpe [mailto:jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org]
> Sent: Tuesday, July 25, 2017 4:34 PM
> To: Steve Wise
> Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; 'Doug Ledford'; 'Ram Amrani'; 'Ira Weiny';
> 'Benjamin Drung'; 'Jarod Wilson'
> Subject: Re: [PATCH rdma-core 0/5] Common systemd/udev based boot support
> 
> On Tue, Jul 25, 2017 at 01:49:37PM -0500, Steve Wise wrote:
> 
> > After I rebooted, cxgb4 and iw_cxgb4 were loaded, but the services were not
> > started.  I think that is probably due to the fact that iw_cxgb4 doesn't
> > register with the rdma core until the cxgb4 interfaces are brought
> > up.
> 
> Yes, it is done on demand when the kernel creates its side of the
> interface.
> 
> > I then brought up port 0 of the cxgb4 card:
> >
> > [root@stevo3 ~]# ip link set enp4s0f4 up
> > [root@stevo3 ~]# ip addr add 172.16.3.3/24 dev enp4s0f4
> >
> > And then the services were started:
> 
> Yep, everything you sent me looks fine, thanks for taking a look at
> this.
> 

Great!  You can add a Tested-by: line for me.

> This sort of hotplug that cxbg4 does is quite strange, what happens
> when 'ip link set X down' is done? Does it remove the RDMA device?
> Does 'ip link set down' block until all users go away?
> 

No.  iw_cxgb4 just triggers on the first 'up', to add the rdma provider instance
for that device.  The Low Level Driver (LLD), cxgb4, passes the CXGB4_STATE_UP
event to all registered upper level drivers (ULDs) when the first port is
enabled (see cxgb_up).  Any rdma connections that are active when a link goes
down still function, as any TCP connection would function if the interface was
brought down;  eg: tcp retransmits if there is pending data until it gives up
and aborts the connection.  So Netdev link down/up transitions are hidden from
the rdma application.   

> This is going to make it harder for cxgb users to get a reliably
> bootup at this time, we need more kernel autoloading for things to be
> reliable, and I'm sure iwpmd.service needs some dependency adjusting,
> I just don't know enough about it to do it right. :\

I don't understand?

Stevo

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH rdma-core 0/5] Common systemd/udev based boot support

[Jason Gunthorpe]

On Tue, Jul 25, 2017 at 04:52:01PM -0500, Steve Wise wrote:

> > This sort of hotplug that cxbg4 does is quite strange, what happens
> > when 'ip link set X down' is done? Does it remove the RDMA device?
> > Does 'ip link set down' block until all users go away?
> 
> No.  iw_cxgb4 just triggers on the first 'up', to add the rdma provider instance
> for that device.  The Low Level Driver (LLD), cxgb4, passes the CXGB4_STATE_UP
> event to all registered upper level drivers (ULDs) when the first port is
> enabled (see cxgb_up).  Any rdma connections that are active when a link goes
> down still function, as any TCP connection would function if the interface was
> brought down;  eg: tcp retransmits if there is pending data until it gives up
> and aborts the connection.  So Netdev link down/up transitions are hidden from
> the rdma application.   

I think you should change this to create the RDMA device when the
module is installed and the hardware is present..

> > This is going to make it harder for cxgb users to get a reliably
> > bootup at this time, we need more kernel autoloading for things to be
> > reliable, and I'm sure iwpmd.service needs some dependency adjusting,
> > I just don't know enough about it to do it right. :\
> 
> I don't understand?

At the present moment udev will start running rules at the link up
time, which happens sometime around 'network.target'

However, systemd will continue processing unknowing what udev is
doing.

So, if you have a RDMA enabled daemon, and you make it start after the
RDMA device is plugged we have some races..

- udev is creating /dev/ nodes and telling systemd to start module loading
  units, and run iwpmd
- systemd may have already started loading the RDMA daemon before udev
  gets to any of this (racy) eg the /dev/ nodes may not exist yet, or
  the modules may still in process to be loaded
- systemd may have started iwpmd, but it is not yet ready and then
  starts the RDMA daemon (racy differently, this is helped with
  sd_notify)
- The RDMA daemon now needs explicit dependencies on the RDMA device
  to order properly, something simple like sysinit.target isn't going to work

Basically, it is very hard to start a RDMA daemon and not have it race
with something and randomly fail to start properly the more hotpluggy
things are.

The existing RDMA stuff largely relies on some sequentiality, eg
loading the RDMA module is enough to create the RDMA device, and that
more reliably happens before sysinit.target, so we can create some
predictable ordering in the system.

This is also why I have been so insistent that the only way to make
all of this work properly and reliably is to have robust kernel auto
loading.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH rdma-core 1/5] Common infrastructure for auto loading rdma modules

[Dennis Dalessandro]

On 7/24/2017 4:44 PM, Jason Gunthorpe wrote:
> diff --git a/kernel-boot/modules/opa.conf b/kernel-boot/modules/opa.conf
> new file mode 100644
> index 00000000000000..b9bc9f1f0146af
> --- /dev/null
> +++ b/kernel-boot/modules/opa.conf
> @@ -0,0 +1,10 @@
> +# These modules are loaded by the system if any OmniPath Architecture device
> +# is installed
> +# Infiniband over IP netdevice
> +ib_ipoib

Do we need to have rdmavt listed here too?

> +# Hardware that supports InfiniBand
> +DRIVERS=="mlx4_core", ENV{ID_RDMA_INFINIBAND}="1"
> +DRIVERS=="mlx5_core", ENV{ID_RDMA_INFINIBAND}="1"
> +DRIVERS=="qib", ENV{ID_RDMA_INFINIBAND}="1"
> +
> +# Hardware that supports OPA
> +DRIVERS=="hfi1verbs", ENV{ID_RDMA_IWARP}="1"

Why calling this IWARP?

-Denny
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH rdma-core 0/5] Common systemd/udev based boot support

[Steve Wise]

> 
> On Tue, Jul 25, 2017 at 04:52:01PM -0500, Steve Wise wrote:
> 
> > > This sort of hotplug that cxbg4 does is quite strange, what happens
> > > when 'ip link set X down' is done? Does it remove the RDMA device?
> > > Does 'ip link set down' block until all users go away?
> >
> > No.  iw_cxgb4 just triggers on the first 'up', to add the rdma provider
instance
> > for that device.  The Low Level Driver (LLD), cxgb4, passes the
CXGB4_STATE_UP
> > event to all registered upper level drivers (ULDs) when the first port is
> > enabled (see cxgb_up).  Any rdma connections that are active when a link
goes
> > down still function, as any TCP connection would function if the interface
was
> > brought down;  eg: tcp retransmits if there is pending data until it gives
up
> > and aborts the connection.  So Netdev link down/up transitions are hidden
from
> > the rdma application.
> 
> I think you should change this to create the RDMA device when the
> module is installed and the hardware is present..
>

Not gonna happen.  cxgb4 doesn't setup the queues, rss, irq mappings, etc, until
an interface is brought up.  So iw_cxgb4 cannot initialize and register with the
rdma core until that happens. 
 
> > > This is going to make it harder for cxgb users to get a reliably
> > > bootup at this time, we need more kernel autoloading for things to be
> > > reliable, and I'm sure iwpmd.service needs some dependency adjusting,
> > > I just don't know enough about it to do it right. :\
> >
> > I don't understand?
> 
> At the present moment udev will start running rules at the link up
> time, which happens sometime around 'network.target'
> 
> However, systemd will continue processing unknowing what udev is
> doing.
> 
> So, if you have a RDMA enabled daemon, and you make it start after the
> RDMA device is plugged we have some races..
> 
> - udev is creating /dev/ nodes and telling systemd to start module loading
>   units, and run iwpmd
> - systemd may have already started loading the RDMA daemon before udev
>   gets to any of this (racy) eg the /dev/ nodes may not exist yet, or
>   the modules may still in process to be loaded
> - systemd may have started iwpmd, but it is not yet ready and then
>   starts the RDMA daemon (racy differently, this is helped with
>   sd_notify)
> - The RDMA daemon now needs explicit dependencies on the RDMA device
>   to order properly, something simple like sysinit.target isn't going to work
> 
> Basically, it is very hard to start a RDMA daemon and not have it race
> with something and randomly fail to start properly the more hotpluggy
> things are.
> 

I think these races exist today, no?  Or is this patch series introducing the
races?   The iwpmd does not need the providers registered at the time it starts.
It will discover new iwarp providers as they initialize. 

> The existing RDMA stuff largely relies on some sequentiality, eg
> loading the RDMA module is enough to create the RDMA device, and that
> more reliably happens before sysinit.target, so we can create some
> predictable ordering in the system.
> 
> This is also why I have been so insistent that the only way to make
> all of this work properly and reliably is to have robust kernel auto
> loading.
> 

Can you think of other ways to address your concerns?

Steve.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH rdma-core 1/5] Common infrastructure for auto loading rdma modules

[Jason Gunthorpe]

On Wed, Jul 26, 2017 at 09:48:04AM -0400, Dennis Dalessandro wrote:
> On 7/24/2017 4:44 PM, Jason Gunthorpe wrote:
> >diff --git a/kernel-boot/modules/opa.conf b/kernel-boot/modules/opa.conf
> >new file mode 100644
> >index 00000000000000..b9bc9f1f0146af
> >+++ b/kernel-boot/modules/opa.conf
> >@@ -0,0 +1,10 @@
> >+# These modules are loaded by the system if any OmniPath Architecture device
> >+# is installed
> >+# Infiniband over IP netdevice
> >+ib_ipoib
> 
> Do we need to have rdmavt listed here too?

Don't think so, isn't it an internal library? It will autoload when
hfi1 autoloads.


> >+# Hardware that supports InfiniBand
> >+DRIVERS=="mlx4_core", ENV{ID_RDMA_INFINIBAND}="1"
> >+DRIVERS=="mlx5_core", ENV{ID_RDMA_INFINIBAND}="1"
> >+DRIVERS=="qib", ENV{ID_RDMA_INFINIBAND}="1"
> >+
> >+# Hardware that supports OPA
> >+DRIVERS=="hfi1verbs", ENV{ID_RDMA_IWARP}="1"
> 
> Why calling this IWARP?

Oops, thanks, copy&paste error

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH rdma-core 0/5] Common systemd/udev based boot support

[Jason Gunthorpe]

On Wed, Jul 26, 2017 at 09:05:03AM -0500, Steve Wise wrote:

> > I think you should change this to create the RDMA device when the
> > module is installed and the hardware is present..
> >
> 
> Not gonna happen.  cxgb4 doesn't setup the queues, rss, irq mappings, etc, until
> an interface is brought up.  So iw_cxgb4 cannot initialize and register with the
> rdma core until that happens. 

But it doesn't tear all that stuff down on ifdown?

> > Basically, it is very hard to start a RDMA daemon and not have it race
> > with something and randomly fail to start properly the more hotpluggy
> > things are.
> 
> I think these races exist today, no?  Or is this patch series introducing the
> races?   The iwpmd does not need the providers registered at the time it starts.
> It will discover new iwarp providers as they initialize. 

There are races today, particularly with systemd, but I suspect they
are slightly different.. That doesn't mean we should ignore them :)

In particular the rdma.target approach loaded all the modules, even if
there wasn't hardware support, so when the cxgb4 RDMA device is
created everything is already to go. We still have races, but they are
probably smaller

Now, we don't even plug the modules until the RDMA device appears, so
we have broader possibilities for racing..

Overall it makes it hard to fit into the ideal system configuration
where RDMA sets up before network-pre.target when a RDMA device
doesn't even plug in until after network.target.

> Can you think of other ways to address your concerns?

Maybe we should have udev trigger loading iwarp services on the cxgb4
ethernet device? At least the races then become more like what we had
before. Are other roce/iwarp devices like this?

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH rdma-core 0/5] Common systemd/udev based boot support

[Steve Wise]

> On Wed, Jul 26, 2017 at 09:05:03AM -0500, Steve Wise wrote:
> 
> > > I think you should change this to create the RDMA device when the
> > > module is installed and the hardware is present..
> > >
> >
> > Not gonna happen.  cxgb4 doesn't setup the queues, rss, irq mappings, etc,
until
> > an interface is brought up.  So iw_cxgb4 cannot initialize and register with
the
> > rdma core until that happens.
> 
> But it doesn't tear all that stuff down on ifdown?

hmm.  I need to relook at this.  I'm very sure the intent was offload
connections do not get torn down if the link goes down.

> 
> > > Basically, it is very hard to start a RDMA daemon and not have it race
> > > with something and randomly fail to start properly the more hotpluggy
> > > things are.
> >
> > I think these races exist today, no?  Or is this patch series introducing
the
> > races?   The iwpmd does not need the providers registered at the time it
starts.
> > It will discover new iwarp providers as they initialize.
> 
> There are races today, particularly with systemd, but I suspect they
> are slightly different.. That doesn't mean we should ignore them :)
> 
> In particular the rdma.target approach loaded all the modules, even if
> there wasn't hardware support, so when the cxgb4 RDMA device is
> created everything is already to go. We still have races, but they are
> probably smaller
> 
> Now, we don't even plug the modules until the RDMA device appears, so
> we have broader possibilities for racing..
> 
> Overall it makes it hard to fit into the ideal system configuration
> where RDMA sets up before network-pre.target when a RDMA device
> doesn't even plug in until after network.target.
> 
> > Can you think of other ways to address your concerns?
> 
> Maybe we should have udev trigger loading iwarp services on the cxgb4
> ethernet device? At least the races then become more like what we had
> before. Are other roce/iwarp devices like this?
> 

I'm not sure.  I would be interested to hear what Intel does. +Chien + Mustafa

Steve.



--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH rdma-core 0/5] Common systemd/udev based boot support

[Amrani, Ram]

Hi Jason,

> I do not have nearly enough hardware to exhaustively test this, I have checked
> that the basic elements are working, but there is lots of room for little
> errors, particularly in module names and udev matching. It would be great of
> someone could try this on iwarp hardware and confirm all the autoloading works
> as expected.
> 
> As should the active roce vendors. I expect that all roce hardware will
> autoload the RDMA modules with this approach.

It works for qedr.

I have followed Steve's steps and built the rpms on Red Hat 7.3 from your repo/systemd.
I've removed all the previous libraries and 98-rdma.rules and then installed the newly built rpms 
After that I rebooted and qedr was loaded.
I didn't have to configure the interfaces as Steve did (ifconfig <device> up).

I've notice that 98-rdma.rules got recreated, but the new file doesn't contain any qed entry.
Why is it recreated? If it is in use, why isn't qed listed?

> +# Hardware that supports iWarp
> +DRIVERS=="cxgb3", ENV{ID_RDMA_IWARP}="1"
> +DRIVERS=="cxgb4", ENV{ID_RDMA_IWARP}="1"
> +
> +# Hardware that supports RoCE
> +DRIVERS=="be2net", ENV{ID_RDMA_ROCE}="1"
> +DRIVERS=="bnxt_en", ENV{ID_RDMA_ROCE}="1"
> +DRIVERS=="hns", ENV{ID_RDMA_ROCE}="1"
> +DRIVERS=="i40e", ENV{ID_RDMA_ROCE}="1"
> +DRIVERS=="mlx4_core", ENV{ID_RDMA_ROCE}="1"
> +DRIVERS=="mlx5_core", ENV{ID_RDMA_ROCE}="1"
> +DRIVERS=="qede", ENV{ID_RDMA_ROCE}="1"

We've recently sent iWARP patches this means qede will be used by either RoCE or iWARP.
In any case the RDMA driver is the same - qedr.

Can you explain what effect does this ENV property has on actual operation?
How do you think it should look in the light of the upcoming patch?

Thanks for coding this,
Ram

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH rdma-core 0/5] Common systemd/udev based boot support

[Jason Gunthorpe]

On Thu, Jul 27, 2017 at 07:47:03AM +0000, Amrani, Ram wrote:
> Hi Jason,
> 
> > I do not have nearly enough hardware to exhaustively test this, I have checked
> > that the basic elements are working, but there is lots of room for little
> > errors, particularly in module names and udev matching. It would be great of
> > someone could try this on iwarp hardware and confirm all the autoloading works
> > as expected.
> > 
> > As should the active roce vendors. I expect that all roce hardware will
> > autoload the RDMA modules with this approach.
> 
> It works for qedr.
> 
> I have followed Steve's steps and built the rpms on Red Hat 7.3 from
> your repo/systemd.  I've removed all the previous libraries and
> 98-rdma.rules and then installed the newly built rpms After that I
> rebooted and qedr was loaded.  I didn't have to configure the
> interfaces as Steve did (ifconfig <device> up).

Okay, that is good.. I thin kthe ifconfig up thing is unique to cxgb4

> I've notice that 98-rdma.rules got recreated, but the new file
> doesn't contain any qed entry.

Because no-one sent a patch for the RedHat boot stuff when QED was
merged..

> Why is it recreated? If it is in use, why isn't qed listed?

The redhat/rdma-core.spec packaging (used by cbuild/etc) still
includes this file, you will have to remove it manually.

> > +# Hardware that supports iWarp
> > +DRIVERS=="cxgb3", ENV{ID_RDMA_IWARP}="1"
> > +DRIVERS=="cxgb4", ENV{ID_RDMA_IWARP}="1"
> > +
> > +# Hardware that supports RoCE
> > +DRIVERS=="be2net", ENV{ID_RDMA_ROCE}="1"
> > +DRIVERS=="bnxt_en", ENV{ID_RDMA_ROCE}="1"
> > +DRIVERS=="hns", ENV{ID_RDMA_ROCE}="1"
> > +DRIVERS=="i40e", ENV{ID_RDMA_ROCE}="1"
> > +DRIVERS=="mlx4_core", ENV{ID_RDMA_ROCE}="1"
> > +DRIVERS=="mlx5_core", ENV{ID_RDMA_ROCE}="1"
> > +DRIVERS=="qede", ENV{ID_RDMA_ROCE}="1"
> 
> We've recently sent iWARP patches this means qede will be used by
> either RoCE or iWARP.  In any case the RDMA driver is the same -
> qedr.

For the interm, you should research this and find a way to tell what
mode the RDMA device is in from sysfs, eg you may be able to match the
node_type sysfile or something else appropraite for your device.

You want set the ID_ tags properly if possible.

If you cannot, then you must set both tags to get correct behavior in
all situations (as the mlx drivers do)

My hope is to eventually have this work automatically, but that would
be based on the rdma netlink Leon is working on.

> Can you explain what effect does this ENV property has on actual
> operation?  How do you think it should look in the light of the
> upcoming patch?

Within rdma-core ID_RDMA_IWARP will cause /etc/rdma/modules/iwarp.conf
to be processed and iwpmd to be started.

ID_RDMA_ROCE will cause /etc/rdma/modules/roce.conf to be processed.

Due to iwpmd it is best to get this right.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH rdma-core 1/5] Common infrastructure for auto loading rdma modules

[Jason Gunthorpe]

On Tue, Jul 25, 2017 at 11:39:47AM -0600, Jason Gunthorpe wrote:
> > > +# Hardware that supports RoCE
> > > +DRIVERS=="be2net", ENV{ID_RDMA_ROCE}="1"
> > > +DRIVERS=="bnxt_en", ENV{ID_RDMA_ROCE}="1"
> > > +DRIVERS=="hns", ENV{ID_RDMA_ROCE}="1"
> > > +DRIVERS=="i40e", ENV{ID_RDMA_ROCE}="1"
> > > +DRIVERS=="mlx4_core", ENV{ID_RDMA_ROCE}="1"
> > > +DRIVERS=="mlx5_core", ENV{ID_RDMA_ROCE}="1"
> > > +DRIVERS=="qede", ENV{ID_RDMA_ROCE}="1"
> > 
> > Should the "rdma_rxe" driver be added to this list?
> 
> No.. I need to figure out something else for rxe. We have no way in
> RDMA to determine the driver binding to the RDMA device (eg like

This looks like a good bet for now:

DEVPATH=="*/infiniband/rxe*", ATTR{parent}=="*", ENV{ID_RDMA_ROCE}="1"

Now that I'm looking at rxe I notice that srp_daemon starts
automatically for roce ports.

I assume this is not desired?

This is going to be a bit harder to fix..

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH rdma-core 1/5] Common infrastructure for auto loading rdma modules

[Bart Van Assche]

On Thu, 2017-07-27 at 16:18 -0600, Jason Gunthorpe wrote:
> On Tue, Jul 25, 2017 at 11:39:47AM -0600, Jason Gunthorpe wrote:
> > > > +# Hardware that supports RoCE
> > > > +DRIVERS=="be2net", ENV{ID_RDMA_ROCE}="1"
> > > > +DRIVERS=="bnxt_en", ENV{ID_RDMA_ROCE}="1"
> > > > +DRIVERS=="hns", ENV{ID_RDMA_ROCE}="1"
> > > > +DRIVERS=="i40e", ENV{ID_RDMA_ROCE}="1"
> > > > +DRIVERS=="mlx4_core", ENV{ID_RDMA_ROCE}="1"
> > > > +DRIVERS=="mlx5_core", ENV{ID_RDMA_ROCE}="1"
> > > > +DRIVERS=="qede", ENV{ID_RDMA_ROCE}="1"
> > > 
> > > Should the "rdma_rxe" driver be added to this list?
> > 
> > No.. I need to figure out something else for rxe. We have no way in
> > RDMA to determine the driver binding to the RDMA device (eg like
> 
> This looks like a good bet for now:
> 
> DEVPATH=="*/infiniband/rxe*", ATTR{parent}=="*", ENV{ID_RDMA_ROCE}="1"
> 
> Now that I'm looking at rxe I notice that srp_daemon starts
> automatically for roce ports.
> 
> I assume this is not desired?

Hello Jason,

The srp_daemon communicates with the IB SA to discover SRP target ports. As
you know there is no SA in a RoCE network. I think it is harmless to start
srp_daemon against a RoCE port but I don't think this makes sense.

Bart.--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH rdma-core 1/5] Common infrastructure for auto loading rdma modules

[Jason Gunthorpe]

On Thu, Jul 27, 2017 at 10:28:59PM +0000, Bart Van Assche wrote:

> The srp_daemon communicates with the IB SA to discover SRP target ports. As
> you know there is no SA in a RoCE network. I think it is harmless to start
> srp_daemon against a RoCE port but I don't think this makes sense.

I agree, it looks like it stops here:

 Jul 27 22:14:34 ib9 srp_daemon[258]: SM LID is 0, maybe no opensm is running

and seems harmless. Let us leave it like this for now.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH rdma-core 0/5] Common systemd/udev based boot support

[Steve Wise]

> > On Wed, Jul 26, 2017 at 09:05:03AM -0500, Steve Wise wrote:
> >
> > > > I think you should change this to create the RDMA device when the
> > > > module is installed and the hardware is present..
> > > >
> > >
> > > Not gonna happen.  cxgb4 doesn't setup the queues, rss, irq mappings, etc,
until
> > > an interface is brought up.  So iw_cxgb4 cannot initialize and register
with the
> > > rdma core until that happens.
> >
> > But it doesn't tear all that stuff down on ifdown?
> 
> hmm.  I need to relook at this.  I'm very sure the intent was offload
connections do
> not get torn down if the link goes down.
> 

Hey Jason,

Bringing down the NIC interfaces doesn't totally stop the adapter nor destroy
the queues needed for RDMA.  It does bring the links down on the ethernet
interfaces though. So it behaves like I thought.   I tested this by running an
rping, and while it was pushing rdma traffic, I did 'ifconfig ethX 0.0.0.0 down'
on the interface in use.  The data flow stopped (and the iWARP/TCP stack started
retransmitting).  I then brought the link back up, and after a few seconds the
rping traffic continued on. 

Steve

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH rdma-core 0/5] Common systemd/udev based boot support

[Jason Gunthorpe]

On Fri, Jul 28, 2017 at 08:51:28AM -0500, Steve Wise wrote:

> Bringing down the NIC interfaces doesn't totally stop the adapter nor destroy
> the queues needed for RDMA.  It does bring the links down on the ethernet
> interfaces though. So it behaves like I thought.   I tested this by running an
> rping, and while it was pushing rdma traffic, I did 'ifconfig ethX 0.0.0.0 down'
> on the interface in use.  The data flow stopped (and the iWARP/TCP stack started
> retransmitting).  I then brought the link back up, and after a few seconds the
> rping traffic continued on. 

So, why defer creating queues until the first ifup?

I can understand a nic driver that frees all memory when down (that is
fairly common), but this arrangement seems really strange..

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH rdma-core 0/5] Common systemd/udev based boot support

[Steve Wise]

> 
> On Fri, Jul 28, 2017 at 08:51:28AM -0500, Steve Wise wrote:
> 
> > Bringing down the NIC interfaces doesn't totally stop the adapter nor
destroy
> > the queues needed for RDMA.  It does bring the links down on the ethernet
> > interfaces though. So it behaves like I thought.   I tested this by running
an
> > rping, and while it was pushing rdma traffic, I did 'ifconfig ethX 0.0.0.0
down'
> > on the interface in use.  The data flow stopped (and the iWARP/TCP stack
started
> > retransmitting).  I then brought the link back up, and after a few seconds
the
> > rping traffic continued on.
> 
> So, why defer creating queues until the first ifup?
> 
> I can understand a nic driver that frees all memory when down (that is
> fairly common), but this arrangement seems really strange..

Maybe I'm mistaken about the need to wait for ifup to register the ULDs.  I do
see the same queues are actually available after cxgb4 loads and before an ifup.
So maybe I can change the uld activation to earlier.



--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html