* [PATCH rdma-core 1/5] Common infrastructure for auto loading rdma modules
[not found] ` <1500929067-1583-1-git-send-email-jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
@ 2017-07-24 20:44 ` Jason Gunthorpe
[not found] ` <1500929067-1583-2-git-send-email-jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
2017-07-24 20:44 ` [PATCH rdma-core 2/5] srp: Autoload the SRP kernel module if required Jason Gunthorpe
` (5 subsequent siblings)
6 siblings, 1 reply; 33+ messages in thread
From: Jason Gunthorpe @ 2017-07-24 20:44 UTC (permalink / raw)
To: linux-rdma-u79uwXL29TY76Z2rM5mHXA, Leon Romanovsky, Doug Ledford,
Ram Amrani, Steve Wise, Ira Weiny, Benjamin Drung, Jarod Wilson
This is inspired by the similar approach in the redhat directory but
takes a more general approach relying on udev and systemd to do the
actual work fully dynamically instead of a oneshot shell script.
Loading is split into two cases
1) Loading RDMA support modules when RDMA capable hardware is installed.
This is only needed for ethernet devices which do not load their RDMA
support modules via request_module in the kernel.
udev is used to detect when an ethernet device controlled by a specific
module is hot plugged and then udev directly loads the RDMA module
2) Loading RDMA ULP support when RDMA hardware is installed
This is done by having udev detect when RDMA hardware is installed and
udev causes systemd to load a list of modules from config files in
/etc/rdma/modules/
The user can customize these files to select which ULP modules should be
loaded.
This broadly replaces the redhat/rdma.conf scheme.
In all cases the users can prevent a module from being auto-loaded on their
system by blacking listing it in a file in /etc/modprobe.d/
Signed-off-by: Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
---
CMakeLists.txt | 1 +
Documentation/udev.md | 83 +++++++++++++++++++++++++++++++
debian/rdma-core.install | 9 ++++
kernel-boot/CMakeLists.txt | 24 +++++++++
kernel-boot/modules/infiniband.conf | 12 +++++
kernel-boot/modules/iwarp.conf | 2 +
kernel-boot/modules/opa.conf | 10 ++++
kernel-boot/modules/rdma.conf | 21 ++++++++
kernel-boot/modules/roce.conf | 2 +
kernel-boot/rdma-description.rules | 41 +++++++++++++++
kernel-boot/rdma-hw-modules.rules | 39 +++++++++++++++
kernel-boot/rdma-load-modules@.service.in | 15 ++++++
kernel-boot/rdma-ulp-modules.rules | 11 ++++
redhat/rdma-core.spec | 11 +++-
14 files changed, 280 insertions(+), 1 deletion(-)
create mode 100644 Documentation/udev.md
create mode 100644 kernel-boot/CMakeLists.txt
create mode 100644 kernel-boot/modules/infiniband.conf
create mode 100644 kernel-boot/modules/iwarp.conf
create mode 100644 kernel-boot/modules/opa.conf
create mode 100644 kernel-boot/modules/rdma.conf
create mode 100644 kernel-boot/modules/roce.conf
create mode 100644 kernel-boot/rdma-description.rules
create mode 100644 kernel-boot/rdma-hw-modules.rules
create mode 100644 kernel-boot/rdma-load-modules@.service.in
create mode 100644 kernel-boot/rdma-ulp-modules.rules
diff --git a/CMakeLists.txt b/CMakeLists.txt
index 16196205035f61..a03d8da31cbc5d 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -398,6 +398,7 @@ configure_file("${BUILDLIB}/config.h.in" "${BUILD_INCLUDE}/config.h" ESCAPE_QUOT
add_subdirectory(ccan)
add_subdirectory(util)
add_subdirectory(Documentation)
+add_subdirectory(kernel-boot)
# Libraries
add_subdirectory(libibumad)
add_subdirectory(libibumad/man)
diff --git a/Documentation/udev.md b/Documentation/udev.md
new file mode 100644
index 00000000000000..e30e6007344fba
--- /dev/null
+++ b/Documentation/udev.md
@@ -0,0 +1,83 @@
+# Kernel Module Loading
+
+The RDMA subsystem relies on the kernel, udev and systemd to load modules on
+demand when RDMA hardware is present. The RDMA subsystem is unique in that we
+do not load the optional RDMA hardware modules unless the system has the
+rdma-core package installed.
+
+This is to avoid exposing systems not using RDMA from having RDMA enabled, for
+instance if a system has a multi-protocol ethernet adaptor, but is only using
+the net stack interface.
+
+## Boot ordering with systemd
+
+systemd assumes everything is hot pluggable and runs in an event driven
+manner. When working with RDMA devices we are firstly concerned with when the
+physical hardware its module loaded into the kernel.
+
+This can happen in several spots along the bootup:
+
+ - From the initrd or built into the kernel. If hardware modules are present
+ inthe initrd then they are loaded into the kernel before booting the
+ system. This is done largely synchronously with the boot process.
+
+ - From udev when it auto detects PCI hardware or otherwise.
+ This happens asynchronously in the boot process, systemd does not wait for
+ udev to finish loading modules before it continues on.
+
+ This path makes it very likely the system will experience a RDMA 'hot plug'
+ scenario.
+
+ - From systemd's fixed module loader systemd-modules-load.service, eg from
+ the list in /etc/modules-load.d/. In this case the modules load happens
+ synchronously within systemd and it will hold off sysinit.target until
+ modules are loaded
+
+Once the hardware module is loaded it may be necessary to load a protocol
+module, eg to enable RDMA support on an ethernet device.
+
+This is triggered automatically by udev rules that match the master devices
+and load the protocol module with udev's module loader. This happens
+asynchronously to the rest of the systemd startup.
+
+Once a RDMA device is created by the kernel then udev will cause systemd to
+schedule ULP module loading services (eg rdma-load-modules@.service) specific
+to the plugged hardware. If sysinit.target has not yet been passed then these
+loaders will defer sysinit.target until they complete, otherwise this is a hot
+plug event and things will load asynchronously to the boot up process.
+
+Finally udev will cause systemd to start RDMA specific daemons like
+srp_deamon, rdma-ndd and iwpmd. These starts are linked to the detection of
+the first RDMA hardware, and the daemons internally handle hot plug events for
+other hardware.
+
+## Hot Plug compatible services
+
+RDMA using services need to have device specific systemd dependencies in their
+unit files, either created by hand by the admin or by using udev rules.
+
+For instance, a service that uses /dev/infiniband/umad0 requires:
+
+```
+After=dev-infiniband-umad0.device
+BindsTo=dev-infiniband-umad0.device
+```
+
+Which will ensure the service will not run until the required umad device
+appears.
+
+This is similar to how systemd handles mounting filesystems and configuring
+ethernet devices.
+
+## Interaction with legacy non-hotplug services
+
+Services that cannot handle hot plug must be ordered after
+systemd-udev-settle.service, which will wait for udev to complete loading
+modules and scheduling systemd services. This ensures that all RDMA hardware
+present at boot is setup before proceeding to run the legacy service.
+
+Admins using legacy services can also place their RDMA hardware modules (eg
+mlx4_ib) directly in /etc/modules-load.d/ or in their initrd which will cause
+systemd to defer passing to sysinit.target until all RDMA hardware is setup,
+this is usually sufficient for legacy services. This is probably the default
+behavior in many configurations.
diff --git a/debian/rdma-core.install b/debian/rdma-core.install
index c19d766fc0f48e..0bda539494189a 100644
--- a/debian/rdma-core.install
+++ b/debian/rdma-core.install
@@ -1,7 +1,16 @@
etc/modprobe.d/mlx4.conf
etc/modprobe.d/truescale.conf
+etc/rdma/modules/infiniband.conf
+etc/rdma/modules/iwarp.conf
+etc/rdma/modules/opa.conf
+etc/rdma/modules/rdma.conf
+etc/rdma/modules/roce.conf
+lib/systemd/system/rdma-load-modules@.service
lib/systemd/system/rdma-ndd.service
lib/udev/rules.d/60-rdma-ndd.rules
+lib/udev/rules.d/75-rdma-description.rules
+lib/udev/rules.d/90-rdma-hw-modules.rules
+lib/udev/rules.d/90-rdma-ulp-modules.rules
usr/bin/rxe_cfg
usr/lib/truescale-serdes.cmds
usr/sbin/rdma-ndd
diff --git a/kernel-boot/CMakeLists.txt b/kernel-boot/CMakeLists.txt
new file mode 100644
index 00000000000000..0d4a2aec1c6a94
--- /dev/null
+++ b/kernel-boot/CMakeLists.txt
@@ -0,0 +1,24 @@
+rdma_subst_install(FILES rdma-load-modules@.service.in
+ DESTINATION "${CMAKE_INSTALL_SYSTEMD_SERVICEDIR}"
+ RENAME rdma-load-modules@.service
+ PERMISSIONS OWNER_WRITE OWNER_READ GROUP_READ WORLD_READ)
+
+install(FILES
+ modules/infiniband.conf
+ modules/iwarp.conf
+ modules/opa.conf
+ modules/rdma.conf
+ modules/roce.conf
+ DESTINATION "${CMAKE_INSTALL_SYSCONFDIR}/rdma/modules")
+
+install(FILES "rdma-description.rules"
+ RENAME "75-rdma-description.rules"
+ DESTINATION "${CMAKE_INSTALL_UDEV_RULESDIR}")
+
+install(FILES "rdma-hw-modules.rules"
+ RENAME "90-rdma-hw-modules.rules"
+ DESTINATION "${CMAKE_INSTALL_UDEV_RULESDIR}")
+
+install(FILES "rdma-ulp-modules.rules"
+ RENAME "90-rdma-ulp-modules.rules"
+ DESTINATION "${CMAKE_INSTALL_UDEV_RULESDIR}")
diff --git a/kernel-boot/modules/infiniband.conf b/kernel-boot/modules/infiniband.conf
new file mode 100644
index 00000000000000..f247d840b7d1bd
--- /dev/null
+++ b/kernel-boot/modules/infiniband.conf
@@ -0,0 +1,12 @@
+# These modules are loaded by the system if any InfiniBand device is installed
+# Infiniband over IP netdevice
+ib_ipoib
+
+# Access to fabric management SMPs and GMPs from userspace.
+ib_umad
+
+# SCSI Remote Protocol target support
+# ib_srpt
+
+# ib_ucm provides the obsolete /dev/infiniband/ucm0
+# ib_ucm
diff --git a/kernel-boot/modules/iwarp.conf b/kernel-boot/modules/iwarp.conf
new file mode 100644
index 00000000000000..882146e41ee2ba
--- /dev/null
+++ b/kernel-boot/modules/iwarp.conf
@@ -0,0 +1,2 @@
+# These modules are loaded by the system if any iWarp device is installed
+iw_cm
diff --git a/kernel-boot/modules/opa.conf b/kernel-boot/modules/opa.conf
new file mode 100644
index 00000000000000..b9bc9f1f0146af
--- /dev/null
+++ b/kernel-boot/modules/opa.conf
@@ -0,0 +1,10 @@
+# These modules are loaded by the system if any OmniPath Architecture device
+# is installed
+# Infiniband over IP netdevice
+ib_ipoib
+
+# Access to fabric management SMPs and GMPs from userspace.
+ib_umad
+
+# Omnipath Ethernet Virtual NIC netdevice
+opa_vnic
diff --git a/kernel-boot/modules/rdma.conf b/kernel-boot/modules/rdma.conf
new file mode 100644
index 00000000000000..2d342dd82f7db0
--- /dev/null
+++ b/kernel-boot/modules/rdma.conf
@@ -0,0 +1,21 @@
+# These modules are loaded by the system if any RDMA devices is installed
+# iSCSI over RDMA client support
+ib_iser
+
+# iSCSI over RDMA target support
+# ib_isert
+
+# User access to RDMA verbs (supports libibverbs)
+ib_uverbs
+
+# User access to RDMA connection management (supports librdmacm)
+rdma_ucm
+
+# RDS over RDMA support
+# rds_rdma
+
+# NFS over RDMA client support
+xprtrdma
+
+# NFS over RDMA server support
+svcrdma
diff --git a/kernel-boot/modules/roce.conf b/kernel-boot/modules/roce.conf
new file mode 100644
index 00000000000000..8e4927ce26f043
--- /dev/null
+++ b/kernel-boot/modules/roce.conf
@@ -0,0 +1,2 @@
+# These modules are loaded by the system if any RDMA over Converged Ethernet
+# device is installed
diff --git a/kernel-boot/rdma-description.rules b/kernel-boot/rdma-description.rules
new file mode 100644
index 00000000000000..dc25db07219cac
--- /dev/null
+++ b/kernel-boot/rdma-description.rules
@@ -0,0 +1,41 @@
+# This is a version of net-description.rules for /sys/class/infiniband devices
+
+ACTION=="remove", GOTO="rdma_description_end"
+SUBSYSTEM!="infiniband", GOTO="rdma_description_end"
+
+# NOTE: DRIVERS searches up the sysfs path to find the driver that is bound to
+# the PCI/etc device that the RDMA device is linked to. This is not the kernel
+# driver that is supplying the RDMA device (eg as seen in ID_NET_DRIVER)
+
+# FIXME: with kernel support we could actually detect the protocols the RDMA
+# driver itself supports, this is a work around for lack of that support.
+# In future we could do this with a udev IMPORT{program} helper program
+# that extracted the ID information from the RDMA netlink.
+
+# Hardware that supports InfiniBand
+DRIVERS=="mlx4_core", ENV{ID_RDMA_INFINIBAND}="1"
+DRIVERS=="mlx5_core", ENV{ID_RDMA_INFINIBAND}="1"
+DRIVERS=="qib", ENV{ID_RDMA_INFINIBAND}="1"
+
+# Hardware that supports OPA
+DRIVERS=="hfi1verbs", ENV{ID_RDMA_IWARP}="1"
+
+# Hardware that supports iWarp
+DRIVERS=="cxgb3", ENV{ID_RDMA_IWARP}="1"
+DRIVERS=="cxgb4", ENV{ID_RDMA_IWARP}="1"
+
+# Hardware that supports RoCE
+DRIVERS=="be2net", ENV{ID_RDMA_ROCE}="1"
+DRIVERS=="bnxt_en", ENV{ID_RDMA_ROCE}="1"
+DRIVERS=="hns", ENV{ID_RDMA_ROCE}="1"
+DRIVERS=="i40e", ENV{ID_RDMA_ROCE}="1"
+DRIVERS=="mlx4_core", ENV{ID_RDMA_ROCE}="1"
+DRIVERS=="mlx5_core", ENV{ID_RDMA_ROCE}="1"
+DRIVERS=="qede", ENV{ID_RDMA_ROCE}="1"
+
+# Setup the usual ID information so that systemd will display a sane name for
+# the RDMA device units.
+SUBSYSTEMS=="pci", ENV{ID_BUS}="pci", ENV{ID_VENDOR_ID}="$attr{vendor}", ENV{ID_MODEL_ID}="$attr{device}"
+SUBSYSTEMS=="pci", IMPORT{builtin}="hwdb --subsystem=pci"
+
+LABEL="rdma_description_end"
diff --git a/kernel-boot/rdma-hw-modules.rules b/kernel-boot/rdma-hw-modules.rules
new file mode 100644
index 00000000000000..dde0ab8dacacab
--- /dev/null
+++ b/kernel-boot/rdma-hw-modules.rules
@@ -0,0 +1,39 @@
+ACTION=="remove", GOTO="rdma_hw_modules_end"
+SUBSYSTEM!="net", GOTO="rdma_hw_modules_end"
+
+# Automatically load RDMA specific kernel modules when a multi-function device is installed
+
+# These drivers autoload an ethernet driver based on hardware detection and
+# need userspace to load the module that has their RDMA component to turn on
+# RDMA.
+ENV{ID_NET_DRIVER}=="be2net", RUN{builtin}+="kmod load ocrdma"
+ENV{ID_NET_DRIVER}=="bnxt_en", RUN{builtin}+="kmod load bnxt_re"
+ENV{ID_NET_DRIVER}=="cxgb3", RUN{builtin}+="kmod load iw_cxgb3"
+ENV{ID_NET_DRIVER}=="cxgb4", RUN{builtin}+="kmod load iw_cxgb4"
+ENV{ID_NET_DRIVER}=="hns", RUN{builtin}+="kmod load hns_roce"
+ENV{ID_NET_DRIVER}=="i40e", RUN{builtin}+="kmod load i40iw"
+ENV{ID_NET_DRIVER}=="mlx4_en", RUN{builtin}+="kmod load mlx4_ib"
+ENV{ID_NET_DRIVER}=="mlx5_core", RUN{builtin}+="kmod load mlx5_ib"
+ENV{ID_NET_DRIVER}=="qede", RUN{builtin}+="kmod load qedr"
+
+# The user must explicitly load these modules via /etc/modules-load.d/ or otherwise
+# rxe
+
+# When in IB mode the kernel PCI core module autoloads the protocol modules
+# for these providers
+# mlx4
+# mlx5
+
+# enic no longer has a userspace verbs driver, this rule should probably be
+# owned by libfabric
+ENV{ID_NET_DRIVER}=="enic", RUN{builtin}+="kmod load usnic_verbs"
+
+# These providers are single function and autoload RDMA automatically based on
+# PCI probing
+# hfi1verbs
+# ipathverbs
+# mthca
+# vmw_pvrdma
+# nes
+
+LABEL="rdma_hw_modules_end"
diff --git a/kernel-boot/rdma-load-modules@.service.in b/kernel-boot/rdma-load-modules@.service.in
new file mode 100644
index 00000000000000..b35a493ebf230b
--- /dev/null
+++ b/kernel-boot/rdma-load-modules@.service.in
@@ -0,0 +1,15 @@
+[Unit]
+Description=Load RDMA modules from @CMAKE_INSTALL_SYSCONFDIR@/rdma/modules/%I.conf
+DefaultDependencies=no
+Conflicts=shutdown.target
+# network-pre.target is to support distro network setup scripts that run after
+# systemd-modules-load.service but before sysinit.target, eg a classic network
+# setup script.
+Before=sysinit.target shutdown.target network-pre.target
+ConditionCapability=CAP_SYS_MODULE
+
+[Service]
+Type=oneshot
+RemainAfterExit=yes
+ExecStart=/lib/systemd/systemd-modules-load @CMAKE_INSTALL_SYSCONFDIR@/rdma/modules/%I.conf
+TimeoutSec=90s
diff --git a/kernel-boot/rdma-ulp-modules.rules b/kernel-boot/rdma-ulp-modules.rules
new file mode 100644
index 00000000000000..c090700c754b19
--- /dev/null
+++ b/kernel-boot/rdma-ulp-modules.rules
@@ -0,0 +1,11 @@
+ACTION=="remove", GOTO="rdma_ulp_modules_end"
+SUBSYSTEM!="infiniband", GOTO="rdma_ulp_modules_end"
+
+# Automatically load general RDMA ULP modules when RDMA hardware is installed
+TAG+="systemd", ENV{SYSTEMD_WANTS}+="rdma-load-modules-XSrBNAzizW1TDjBF/Jpztg@public.gmane.org"
+TAG+="systemd", ENV{ID_RDMA_INFINIBAND}=="1", ENV{SYSTEMD_WANTS}+="rdma-load-modules-WW8hglTQ5Ylgot6vXr2xLYq0XWbe/Iga@public.gmane.org"
+TAG+="systemd", ENV{ID_RDMA_IWARP}=="1", ENV{SYSTEMD_WANTS}+="rdma-load-modules-mPXOxrv7gkYc5CkaFG5UGg@public.gmane.org"
+TAG+="systemd", ENV{ID_RDMA_OPA}=="1", ENV{SYSTEMD_WANTS}+="rdma-load-modules-EhpGOaQAOk9DcWalurthmA@public.gmane.org"
+TAG+="systemd", ENV{ID_RDMA_ROCE}=="1", ENV{SYSTEMD_WANTS}+="rdma-load-modules-eW91h9tu8Y5TDjBF/Jpztg@public.gmane.org"
+
+LABEL="rdma_ulp_modules_end"
diff --git a/redhat/rdma-core.spec b/redhat/rdma-core.spec
index 2565290e72c04e..612cf5c808091c 100644
--- a/redhat/rdma-core.spec
+++ b/redhat/rdma-core.spec
@@ -320,17 +320,26 @@ rm -rf %{buildroot}/%{_initrddir}/
%doc %{_docdir}/%{name}-%{version}/README.md
%doc %{_docdir}/%{name}-%{version}/rxe.md
%config(noreplace) %{_sysconfdir}/rdma/mlx4.conf
+%config(noreplace) %{_sysconfdir}/rdma/modules/infiniband.conf
+%config(noreplace) %{_sysconfdir}/rdma/modules/iwarp.conf
+%config(noreplace) %{_sysconfdir}/rdma/modules/opa.conf
+%config(noreplace) %{_sysconfdir}/rdma/modules/rdma.conf
+%config(noreplace) %{_sysconfdir}/rdma/modules/roce.conf
%config(noreplace) %{_sysconfdir}/rdma/rdma.conf
%config(noreplace) %{_sysconfdir}/rdma/sriov-vfs
%config(noreplace) %{_sysconfdir}/udev/rules.d/*
%config(noreplace) %{_sysconfdir}/modprobe.d/mlx4.conf
%config(noreplace) %{_sysconfdir}/modprobe.d/truescale.conf
%{_sysconfdir}/sysconfig/network-scripts/*
+%{_unitdir}/rdma-load-modules@.service
%{_unitdir}/rdma.service
%dir %{dracutlibdir}/modules.d/05rdma
%{dracutlibdir}/modules.d/05rdma/module-setup.sh
-%{_udevrulesdir}/98-rdma.rules
%{_udevrulesdir}/60-rdma-ndd.rules
+%{_udevrulesdir}/75-rdma-description.rules
+%{_udevrulesdir}/90-rdma-hw-modules.rules
+%{_udevrulesdir}/90-rdma-ulp-modules.rules
+%{_udevrulesdir}/98-rdma.rules
%{sysmodprobedir}/libmlx4.conf
%{sysmodprobedir}/cxgb3.conf
%{sysmodprobedir}/cxgb4.conf
--
2.7.4
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 33+ messages in thread* [PATCH rdma-core 2/5] srp: Autoload the SRP kernel module if required
[not found] ` <1500929067-1583-1-git-send-email-jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
2017-07-24 20:44 ` [PATCH rdma-core 1/5] Common infrastructure for auto loading rdma modules Jason Gunthorpe
@ 2017-07-24 20:44 ` Jason Gunthorpe
2017-07-24 20:44 ` [PATCH rdma-core 3/5] iwpmd: Autoload iwpmd " Jason Gunthorpe
` (4 subsequent siblings)
6 siblings, 0 replies; 33+ messages in thread
From: Jason Gunthorpe @ 2017-07-24 20:44 UTC (permalink / raw)
To: linux-rdma-u79uwXL29TY76Z2rM5mHXA, Leon Romanovsky, Doug Ledford,
Ram Amrani, Steve Wise, Ira Weiny, Benjamin Drung, Jarod Wilson
We use the rdma-core module loading infrastructure to load a srp_daemon
specific module list before allowing srp_daemon to start.
This is in lieu of kernel module auto loading, which is hard to trigger
for sysfs based interfaces.
Signed-off-by: Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
---
debian/control | 2 +-
debian/srptools.install | 1 +
redhat/rdma-core.spec | 1 +
srp_daemon/CMakeLists.txt | 4 ++++
srp_daemon/modules-srp_daemon.conf | 2 ++
srp_daemon/srp_daemon.service.in | 2 +-
srp_daemon/srp_daemon_port@.service.in | 3 ++-
7 files changed, 12 insertions(+), 3 deletions(-)
create mode 100644 srp_daemon/modules-srp_daemon.conf
diff --git a/debian/control b/debian/control
index 9d80e38bb07864..d7e20e4879a19c 100644
--- a/debian/control
+++ b/debian/control
@@ -341,7 +341,7 @@ Description: Examples for the librdmacm library
Package: srptools
Architecture: linux-any
-Depends: lsb-base (>= 3.2-14~), ${misc:Depends}, ${shlibs:Depends}
+Depends: lsb-base (>= 3.2-14~), rdma-core, ${misc:Depends}, ${shlibs:Depends}
Description: Tools for Infiniband attached storage (SRP)
In conjunction with the kernel ib_srp driver, srptools allows you to
discover and use Infiniband attached storage devices which use the
diff --git a/debian/srptools.install b/debian/srptools.install
index f780d663c9113b..a30798cf088a03 100644
--- a/debian/srptools.install
+++ b/debian/srptools.install
@@ -1,3 +1,4 @@
+etc/rdma/modules/srp_daemon.conf
etc/srp_daemon.conf
lib/systemd/system/srp_daemon.service
lib/systemd/system/srp_daemon_port@.service
diff --git a/redhat/rdma-core.spec b/redhat/rdma-core.spec
index 612cf5c808091c..124aec402997c5 100644
--- a/redhat/rdma-core.spec
+++ b/redhat/rdma-core.spec
@@ -452,6 +452,7 @@ rm -rf %{buildroot}/%{_initrddir}/
%files -n srp_daemon
%config(noreplace) %{_sysconfdir}/srp_daemon.conf
+%config(noreplace) %{_sysconfdir}/rdma/modules/srp_daemon.conf
%{_libexecdir}/srp_daemon/start_on_all_ports
%{_unitdir}/srp_daemon.service
%{_unitdir}/srp_daemon_port@.service
diff --git a/srp_daemon/CMakeLists.txt b/srp_daemon/CMakeLists.txt
index 92a1d709364b0b..0e7de9594b6375 100644
--- a/srp_daemon/CMakeLists.txt
+++ b/srp_daemon/CMakeLists.txt
@@ -47,6 +47,10 @@ install(FILES "srp_daemon.rules"
RENAME "60-srp_daemon.rules"
DESTINATION "${CMAKE_INSTALL_UDEV_RULESDIR}")
+install(FILES modules-srp_daemon.conf
+ RENAME "srp_daemon.conf"
+ DESTINATION "${CMAKE_INSTALL_SYSCONFDIR}/rdma/modules")
+
# FIXME: The ib init.d file should really be included in rdma-core as well.
set(RDMA_SERVICE "openibd" CACHE STRING "init.d file service name to order srpd after")
# NOTE: These defaults are for CentOS, packagers should override.
diff --git a/srp_daemon/modules-srp_daemon.conf b/srp_daemon/modules-srp_daemon.conf
new file mode 100644
index 00000000000000..dbe434362f10a3
--- /dev/null
+++ b/srp_daemon/modules-srp_daemon.conf
@@ -0,0 +1,2 @@
+# These modules are loaded by the system if srp_daemon is to be run
+ib_srp
diff --git a/srp_daemon/srp_daemon.service.in b/srp_daemon/srp_daemon.service.in
index 33dddd5cb46fef..cca1fce9c99283 100644
--- a/srp_daemon/srp_daemon.service.in
+++ b/srp_daemon/srp_daemon.service.in
@@ -1,6 +1,6 @@
[Unit]
Description=Daemon that discovers and logs in to SRP target systems
-Documentation=man:srp_daemon file:/etc/rdma/rdma.conf file:/etc/srp_daemon.conf
+Documentation=man:srp_daemon file:/etc/srp_daemon.conf
DefaultDependencies=false
Conflicts=emergency.target emergency.service
Before=remote-fs-pre.target
diff --git a/srp_daemon/srp_daemon_port@.service.in b/srp_daemon/srp_daemon_port@.service.in
index 03b9cb9a70119a..69d3075ae77a1b 100644
--- a/srp_daemon/srp_daemon_port@.service.in
+++ b/srp_daemon/srp_daemon_port@.service.in
@@ -3,7 +3,8 @@ Description=SRP daemon that monitors port %i
Documentation=man:srp_daemon file:/etc/rdma/rdma.conf file:/etc/srp_daemon.conf
DefaultDependencies=false
Conflicts=emergency.target emergency.service
-After=srp_daemon.service sys-subsystem-rdma-devices-%i-umad.device network.target
+Requires=rdma-load-modules@srp_daemon.service
+After=srp_daemon.service rdma-load-modules@srp_daemon.service sys-subsystem-rdma-devices-%i-umad.device network.target
BindsTo=srp_daemon.service sys-subsystem-rdma-devices-%i-umad.device
Before=remote-fs-pre.target
--
2.7.4
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 33+ messages in thread* [PATCH rdma-core 3/5] iwpmd: Autoload iwpmd if required
[not found] ` <1500929067-1583-1-git-send-email-jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
2017-07-24 20:44 ` [PATCH rdma-core 1/5] Common infrastructure for auto loading rdma modules Jason Gunthorpe
2017-07-24 20:44 ` [PATCH rdma-core 2/5] srp: Autoload the SRP kernel module if required Jason Gunthorpe
@ 2017-07-24 20:44 ` Jason Gunthorpe
2017-07-24 20:44 ` [PATCH rdma-core 4/5] redhat: Remove cxgb3/4.modprobe files Jason Gunthorpe
` (3 subsequent siblings)
6 siblings, 0 replies; 33+ messages in thread
From: Jason Gunthorpe @ 2017-07-24 20:44 UTC (permalink / raw)
To: linux-rdma-u79uwXL29TY76Z2rM5mHXA, Leon Romanovsky, Doug Ledford,
Ram Amrani, Steve Wise, Ira Weiny, Benjamin Drung, Jarod Wilson
Use udev to determine if an iWarp device is present, and if so then start
iwpmd and autoload the required module. iwpmd no longer starts if it is
simply installed.
Signed-off-by: Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
---
debian/iwpmd.install | 2 ++
iwpmd/CMakeLists.txt | 8 ++++++++
iwpmd/iwpmd.rules | 1 +
iwpmd/iwpmd.service.in | 6 +++---
iwpmd/modules-iwpmd.conf | 2 ++
kernel-boot/modules/iwarp.conf | 1 -
redhat/rdma-core.spec | 2 ++
7 files changed, 18 insertions(+), 4 deletions(-)
create mode 100644 iwpmd/iwpmd.rules
create mode 100644 iwpmd/modules-iwpmd.conf
diff --git a/debian/iwpmd.install b/debian/iwpmd.install
index cd9f7adbdd287d..51ac19ad382e9e 100644
--- a/debian/iwpmd.install
+++ b/debian/iwpmd.install
@@ -1,6 +1,8 @@
etc/init.d/iwpmd
etc/iwpmd.conf
+etc/rdma/modules/iwpmd.conf
lib/systemd/system/iwpmd.service
+lib/udev/rules.d/90-iwpmd.rules
usr/sbin/iwpmd
usr/share/man/man1/iwpmd.1
usr/share/man/man5/iwpmd.conf.5
diff --git a/iwpmd/CMakeLists.txt b/iwpmd/CMakeLists.txt
index 200b14c1786422..f500d196acf649 100644
--- a/iwpmd/CMakeLists.txt
+++ b/iwpmd/CMakeLists.txt
@@ -21,3 +21,11 @@ rdma_subst_install(FILES "iwpmd_init.in"
RENAME "iwpmd"
PERMISSIONS OWNER_WRITE OWNER_READ GROUP_READ WORLD_READ OWNER_EXECUTE GROUP_EXECUTE WORLD_EXECUTE)
install(FILES "iwpmd.conf" DESTINATION "${CMAKE_INSTALL_SYSCONFDIR}")
+
+install(FILES "iwpmd.rules"
+ RENAME "90-iwpmd.rules"
+ DESTINATION "${CMAKE_INSTALL_UDEV_RULESDIR}")
+
+install(FILES modules-iwpmd.conf
+ RENAME "iwpmd.conf"
+ DESTINATION "${CMAKE_INSTALL_SYSCONFDIR}/rdma/modules")
diff --git a/iwpmd/iwpmd.rules b/iwpmd/iwpmd.rules
new file mode 100644
index 00000000000000..5b22cceaf29f8c
--- /dev/null
+++ b/iwpmd/iwpmd.rules
@@ -0,0 +1 @@
+TAG+="systemd", ENV{ID_RDMA_IWARP}=="1", ENV{SYSTEMD_WANTS}+="iwpmd.service"
diff --git a/iwpmd/iwpmd.service.in b/iwpmd/iwpmd.service.in
index 47ca6518ee127c..83c1da98558a9f 100644
--- a/iwpmd/iwpmd.service.in
+++ b/iwpmd/iwpmd.service.in
@@ -1,12 +1,12 @@
[Unit]
Description=iWarp Port Mapper
Documentation=man:iwpmd file:/etc/iwpmd.conf
-After=network.target
+Requires=rdma-load-modules-r79FGtN6zq8c5CkaFG5UGg@public.gmane.org
+After=network.target rdma-load-modules-r79FGtN6zq8c5CkaFG5UGg@public.gmane.org
[Service]
ExecStart=@CMAKE_INSTALL_FULL_SBINDIR@/iwpmd
LimitNOFILE=102400
KillMode=process
-[Install]
-WantedBy=multi-user.target
+# iwpmd is automatically started by udev when an iWarp RDMA device is present
diff --git a/iwpmd/modules-iwpmd.conf b/iwpmd/modules-iwpmd.conf
new file mode 100644
index 00000000000000..5544b359c08814
--- /dev/null
+++ b/iwpmd/modules-iwpmd.conf
@@ -0,0 +1,2 @@
+# These modules are loaded by the system if iwpmd is to be run
+iw_cm
diff --git a/kernel-boot/modules/iwarp.conf b/kernel-boot/modules/iwarp.conf
index 882146e41ee2ba..0cb831da5bb557 100644
--- a/kernel-boot/modules/iwarp.conf
+++ b/kernel-boot/modules/iwarp.conf
@@ -1,2 +1 @@
# These modules are loaded by the system if any iWarp device is installed
-iw_cm
diff --git a/redhat/rdma-core.spec b/redhat/rdma-core.spec
index 124aec402997c5..901c726037fdfe 100644
--- a/redhat/rdma-core.spec
+++ b/redhat/rdma-core.spec
@@ -403,7 +403,9 @@ rm -rf %{buildroot}/%{_initrddir}/
%files -n iwpmd
%{_sbindir}/iwpmd
%{_unitdir}/iwpmd.service
+%config(noreplace) %{_sysconfdir}/rdma/modules/iwpmd.conf
%config(noreplace) %{_sysconfdir}/iwpmd.conf
+%{_udevrulesdir}/90-iwpmd.rules
%{_mandir}/man1/iwpmd.*
%{_mandir}/man5/iwpmd.*
--
2.7.4
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 33+ messages in thread* [PATCH rdma-core 4/5] redhat: Remove cxgb3/4.modprobe files
[not found] ` <1500929067-1583-1-git-send-email-jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
` (2 preceding siblings ...)
2017-07-24 20:44 ` [PATCH rdma-core 3/5] iwpmd: Autoload iwpmd " Jason Gunthorpe
@ 2017-07-24 20:44 ` Jason Gunthorpe
2017-07-24 20:44 ` [PATCH rdma-core 5/5] Move umad udev rules to rdma-core Jason Gunthorpe
` (2 subsequent siblings)
6 siblings, 0 replies; 33+ messages in thread
From: Jason Gunthorpe @ 2017-07-24 20:44 UTC (permalink / raw)
To: linux-rdma-u79uwXL29TY76Z2rM5mHXA, Leon Romanovsky, Doug Ledford,
Ram Amrani, Steve Wise, Ira Weiny, Benjamin Drung, Jarod Wilson
This is now handled by the udev rules in kernel-boot
Signed-off-by: Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
---
redhat/rdma-core.spec | 4 ----
redhat/rdma.cxgb3.sys.modprobe | 1 -
redhat/rdma.cxgb4.sys.modprobe | 1 -
3 files changed, 6 deletions(-)
delete mode 100644 redhat/rdma.cxgb3.sys.modprobe
delete mode 100644 redhat/rdma.cxgb4.sys.modprobe
diff --git a/redhat/rdma-core.spec b/redhat/rdma-core.spec
index 901c726037fdfe..1e4aee6a211f33 100644
--- a/redhat/rdma-core.spec
+++ b/redhat/rdma-core.spec
@@ -260,8 +260,6 @@ install -D -m0644 redhat/rdma.udev-ipoib-naming.rules %{buildroot}%{_sysconfdir}
install -D -m0755 redhat/rdma.modules-setup.sh %{buildroot}%{dracutlibdir}/modules.d/05rdma/module-setup.sh
install -D -m0644 redhat/rdma.udev-rules %{buildroot}%{_udevrulesdir}/98-rdma.rules
install -D -m0644 redhat/rdma.mlx4.sys.modprobe %{buildroot}%{sysmodprobedir}/libmlx4.conf
-install -D -m0644 redhat/rdma.cxgb3.sys.modprobe %{buildroot}%{sysmodprobedir}/cxgb3.conf
-install -D -m0644 redhat/rdma.cxgb4.sys.modprobe %{buildroot}%{sysmodprobedir}/cxgb4.conf
install -D -m0755 redhat/rdma.kernel-init %{buildroot}%{_libexecdir}/rdma-init-kernel
install -D -m0755 redhat/rdma.sriov-init %{buildroot}%{_libexecdir}/rdma-set-sriov-vf
install -D -m0644 redhat/rdma.fixup-mtrr.awk %{buildroot}%{_libexecdir}/rdma-fixup-mtrr.awk
@@ -341,8 +339,6 @@ rm -rf %{buildroot}/%{_initrddir}/
%{_udevrulesdir}/90-rdma-ulp-modules.rules
%{_udevrulesdir}/98-rdma.rules
%{sysmodprobedir}/libmlx4.conf
-%{sysmodprobedir}/cxgb3.conf
-%{sysmodprobedir}/cxgb4.conf
%{_libexecdir}/rdma-init-kernel
%{_libexecdir}/rdma-set-sriov-vf
%{_libexecdir}/rdma-fixup-mtrr.awk
diff --git a/redhat/rdma.cxgb3.sys.modprobe b/redhat/rdma.cxgb3.sys.modprobe
deleted file mode 100644
index d5925a715ccee1..00000000000000
diff --git a/redhat/rdma.cxgb4.sys.modprobe b/redhat/rdma.cxgb4.sys.modprobe
deleted file mode 100644
index 44163ab43cf649..00000000000000
--
2.7.4
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 33+ messages in thread* [PATCH rdma-core 5/5] Move umad udev rules to rdma-core
[not found] ` <1500929067-1583-1-git-send-email-jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
` (3 preceding siblings ...)
2017-07-24 20:44 ` [PATCH rdma-core 4/5] redhat: Remove cxgb3/4.modprobe files Jason Gunthorpe
@ 2017-07-24 20:44 ` Jason Gunthorpe
2017-07-25 16:27 ` [PATCH rdma-core 0/5] Common systemd/udev based boot support Steve Wise
2017-07-27 7:47 ` Amrani, Ram
6 siblings, 0 replies; 33+ messages in thread
From: Jason Gunthorpe @ 2017-07-24 20:44 UTC (permalink / raw)
To: linux-rdma-u79uwXL29TY76Z2rM5mHXA, Leon Romanovsky, Doug Ledford,
Ram Amrani, Steve Wise, Ira Weiny, Benjamin Drung, Jarod Wilson
This works better from a packaging perspective, eg rdma-core
is already a matched dependency for srp_daemon in RedHat, and provides
one place to get all the udev meta data rules.
Also update debian's dependencies to rely on a new RDMA core
for srptools, as it now requires the rules files and
rdma-load-modules@.service.in from rdma-core.
Signed-off-by: Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
---
debian/control | 2 +-
debian/libibumad3.install | 1 -
debian/rdma-core.install | 1 +
kernel-boot/CMakeLists.txt | 4 ++++
libibumad/libibumad.udev-rules => kernel-boot/rdma-umad.rules | 0
libibumad/CMakeLists.txt | 5 -----
redhat/rdma-core.spec | 2 +-
7 files changed, 7 insertions(+), 8 deletions(-)
rename libibumad/libibumad.udev-rules => kernel-boot/rdma-umad.rules (100%)
diff --git a/debian/control b/debian/control
index d7e20e4879a19c..9a40ef17d60833 100644
--- a/debian/control
+++ b/debian/control
@@ -341,7 +341,7 @@ Description: Examples for the librdmacm library
Package: srptools
Architecture: linux-any
-Depends: lsb-base (>= 3.2-14~), rdma-core, ${misc:Depends}, ${shlibs:Depends}
+Depends: lsb-base (>= 3.2-14~), rdma-core (>= 15), ${misc:Depends}, ${shlibs:Depends}
Description: Tools for Infiniband attached storage (SRP)
In conjunction with the kernel ib_srp driver, srptools allows you to
discover and use Infiniband attached storage devices which use the
diff --git a/debian/libibumad3.install b/debian/libibumad3.install
index 41339ebfa2f229..f6c298db52eb34 100644
--- a/debian/libibumad3.install
+++ b/debian/libibumad3.install
@@ -1,2 +1 @@
-lib/udev/rules.d/90-libibumad.rules
usr/lib/*/libibumad*.so.*
diff --git a/debian/rdma-core.install b/debian/rdma-core.install
index 0bda539494189a..df33b8a6070d22 100644
--- a/debian/rdma-core.install
+++ b/debian/rdma-core.install
@@ -11,6 +11,7 @@ lib/udev/rules.d/60-rdma-ndd.rules
lib/udev/rules.d/75-rdma-description.rules
lib/udev/rules.d/90-rdma-hw-modules.rules
lib/udev/rules.d/90-rdma-ulp-modules.rules
+lib/udev/rules.d/90-rdma-umad.rules
usr/bin/rxe_cfg
usr/lib/truescale-serdes.cmds
usr/sbin/rdma-ndd
diff --git a/kernel-boot/CMakeLists.txt b/kernel-boot/CMakeLists.txt
index 0d4a2aec1c6a94..fdb70117f5899c 100644
--- a/kernel-boot/CMakeLists.txt
+++ b/kernel-boot/CMakeLists.txt
@@ -22,3 +22,7 @@ install(FILES "rdma-hw-modules.rules"
install(FILES "rdma-ulp-modules.rules"
RENAME "90-rdma-ulp-modules.rules"
DESTINATION "${CMAKE_INSTALL_UDEV_RULESDIR}")
+
+install(FILES "rdma-umad.rules"
+ RENAME "90-rdma-umad.rules"
+ DESTINATION "${CMAKE_INSTALL_UDEV_RULESDIR}")
diff --git a/libibumad/libibumad.udev-rules b/kernel-boot/rdma-umad.rules
similarity index 100%
rename from libibumad/libibumad.udev-rules
rename to kernel-boot/rdma-umad.rules
diff --git a/libibumad/CMakeLists.txt b/libibumad/CMakeLists.txt
index 84673ee01271cb..ba2463b47ed82a 100644
--- a/libibumad/CMakeLists.txt
+++ b/libibumad/CMakeLists.txt
@@ -14,8 +14,3 @@ rdma_library(ibumad libibumad.map
umad.c
umad_str.c
)
-
-install(FILES libibumad.udev-rules
- RENAME 90-libibumad.rules
- DESTINATION "${CMAKE_INSTALL_UDEV_RULESDIR}")
-
diff --git a/redhat/rdma-core.spec b/redhat/rdma-core.spec
index 1e4aee6a211f33..7ed419394d22d6 100644
--- a/redhat/rdma-core.spec
+++ b/redhat/rdma-core.spec
@@ -337,6 +337,7 @@ rm -rf %{buildroot}/%{_initrddir}/
%{_udevrulesdir}/75-rdma-description.rules
%{_udevrulesdir}/90-rdma-hw-modules.rules
%{_udevrulesdir}/90-rdma-ulp-modules.rules
+%{_udevrulesdir}/90-rdma-umad.rules
%{_udevrulesdir}/98-rdma.rules
%{sysmodprobedir}/libmlx4.conf
%{_libexecdir}/rdma-init-kernel
@@ -411,7 +412,6 @@ rm -rf %{buildroot}/%{_initrddir}/
%files -n libibumad
%{_libdir}/libibumad*.so.*
-%{_udevrulesdir}/90-libibumad.rules
%files -n librdmacm
%{_libdir}/librdmacm*.so.*
--
2.7.4
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 33+ messages in thread* RE: [PATCH rdma-core 0/5] Common systemd/udev based boot support
[not found] ` <1500929067-1583-1-git-send-email-jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
` (4 preceding siblings ...)
2017-07-24 20:44 ` [PATCH rdma-core 5/5] Move umad udev rules to rdma-core Jason Gunthorpe
@ 2017-07-25 16:27 ` Steve Wise
2017-07-25 16:40 ` Jason Gunthorpe
2017-07-27 7:47 ` Amrani, Ram
6 siblings, 1 reply; 33+ messages in thread
From: Steve Wise @ 2017-07-25 16:27 UTC (permalink / raw)
To: 'Jason Gunthorpe',
linux-rdma-u79uwXL29TY76Z2rM5mHXA, 'Leon Romanovsky',
'Doug Ledford', 'Ram Amrani', 'Ira Weiny',
'Benjamin Drung', 'Jarod Wilson'
> I do not have nearly enough hardware to exhaustively test this, I have checked
> that the basic elements are working, but there is lots of room for little
> errors, particularly in module names and udev matching. It would be great of
> someone could try this on iwarp hardware and confirm all the autoloading works
> as expected.
Hey Jason,
Thanks for doing all this! I'm testing with cxgb4 today.
I get these errors building the rpms using the spec file in your tree:
Checking for unpackaged file(s): /usr/lib/rpm/check-files
/root/rpmbuild/BUILDROOT/rdma-core-15-1.el7.x86_64
error: Installed (but unpackaged) file(s) found:
/etc/rdma/modules/infiniband.conf
/etc/rdma/modules/iwarp.conf
/etc/rdma/modules/iwpmd.conf
/etc/rdma/modules/opa.conf
/etc/rdma/modules/rdma.conf
/etc/rdma/modules/roce.conf
/etc/rdma/modules/srp_daemon.conf
RPM build errors:
Installed (but unpackaged) file(s) found:
/etc/rdma/modules/infiniband.conf
/etc/rdma/modules/iwarp.conf
/etc/rdma/modules/iwpmd.conf
/etc/rdma/modules/opa.conf
/etc/rdma/modules/rdma.conf
/etc/rdma/modules/roce.conf
/etc/rdma/modules/srp_daemon.conf
Are these expected?
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 33+ messages in thread* Re: [PATCH rdma-core 0/5] Common systemd/udev based boot support
2017-07-25 16:27 ` [PATCH rdma-core 0/5] Common systemd/udev based boot support Steve Wise
@ 2017-07-25 16:40 ` Jason Gunthorpe
[not found] ` <20170725164004.GA20959-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
` (2 more replies)
0 siblings, 3 replies; 33+ messages in thread
From: Jason Gunthorpe @ 2017-07-25 16:40 UTC (permalink / raw)
To: Steve Wise
Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA, 'Leon Romanovsky',
'Doug Ledford', 'Ram Amrani', 'Ira Weiny',
'Benjamin Drung', 'Jarod Wilson'
On Tue, Jul 25, 2017 at 11:27:50AM -0500, Steve Wise wrote:
> > I do not have nearly enough hardware to exhaustively test this, I have checked
> > that the basic elements are working, but there is lots of room for little
> > errors, particularly in module names and udev matching. It would be great of
> > someone could try this on iwarp hardware and confirm all the autoloading works
> > as expected.
>
> Hey Jason,
>
> Thanks for doing all this! I'm testing with cxgb4 today.
>
> I get these errors building the rpms using the spec file in your tree:
I think you are using ./rdma-core.spec? I forgot about that one and
didn't make it work, I will fix it.
You probably want to use redhat/rdma-core.spec anyhow?
When testing, you want to start with a system that doesn't load any
modules, then install the new rpms and see what happens. (rdma-core, iwpmd)
For iwarp I expect that systemd will auto start these units:
rdma-load-modules-r79FGtN6zq8c5CkaFG5UGg@public.gmane.org loaded active exited Load RDMA modules from /etc/rdma/modules/iwpmd.conf
rdma-load-modules-XSrBNAzizW1TDjBF/Jpztg@public.gmane.org loaded active exited Load RDMA modules from /etc/rdma/modules/rdma.conf
rdma-load-modules-mPXOxrv7gkYc5CkaFG5UGg@public.gmane.org loaded active exited Load RDMA modules from /etc/rdma/modules/iwarp.conf
iwpmd.service loaded active running iWarp Port Mapper
And you should have a reasonable set of modules loaded. Check the
journal for interesting errors also..
For cxgb4 these are the two critical udev lines:
ENV{ID_NET_DRIVER}=="cxgb4", RUN{builtin}+="kmod load iw_cxgb4"
DRIVERS=="cxgb4", ENV{ID_RDMA_IWARP}="1"
The first should load iw_cxgb4 if an ethernet device with the cxgb4
driver is found (the driver name being matched is as reported by
ethtool -i)
Both need to trigger for things to work properly, you can check with:
$ udevadm info /sys/class/net/ethX
$ udevadm info /sys/class/infiniband/cxgb4_0 # ??
Thanks,
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 33+ messages in thread[parent not found: <20170725164004.GA20959-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>]
* RE: [PATCH rdma-core 0/5] Common systemd/udev based boot support
[not found] ` <20170725164004.GA20959-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
@ 2017-07-25 16:43 ` Steve Wise
2017-07-25 16:55 ` Jason Gunthorpe
2017-07-25 18:49 ` Steve Wise
1 sibling, 1 reply; 33+ messages in thread
From: Steve Wise @ 2017-07-25 16:43 UTC (permalink / raw)
To: 'Jason Gunthorpe'
Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA, 'Leon Romanovsky',
'Doug Ledford', 'Ram Amrani', 'Ira Weiny',
'Benjamin Drung', 'Jarod Wilson'
> >
> > Hey Jason,
> >
> > Thanks for doing all this! I'm testing with cxgb4 today.
> >
> > I get these errors building the rpms using the spec file in your tree:
>
> I think you are using ./rdma-core.spec? I forgot about that one and
> didn't make it work, I will fix it.
>
> You probably want to use redhat/rdma-core.spec anyhow?
>
Ok. I haven't been paying attention to the distro-specific stuff. I had a
process I have been following to build an rpm from ./rdma-core.spec. I'll try
the redhat spec file since I'm installing on a RHEL7.2 system.
> When testing, you want to start with a system that doesn't load any
> modules, then install the new rpms and see what happens. (rdma-core, iwpmd)
>
I've uninstalled all the pertinent packages, I think, with this:
yum autoremove -y libibverbs librdmacm libibumad libiwpm
> For iwarp I expect that systemd will auto start these units:
>
> rdma-load-modules-r79FGtN6zq8c5CkaFG5UGg@public.gmane.org loaded active exited Load RDMA
> modules from /etc/rdma/modules/iwpmd.conf
> rdma-load-modules-XSrBNAzizW1TDjBF/Jpztg@public.gmane.org loaded active exited Load RDMA
modules
> from /etc/rdma/modules/rdma.conf
> rdma-load-modules-mPXOxrv7gkYc5CkaFG5UGg@public.gmane.org loaded active exited Load RDMA
modules
> from /etc/rdma/modules/iwarp.conf
> iwpmd.service loaded active running iWarp Port
Mapper
>
> And you should have a reasonable set of modules loaded. Check the
> journal for interesting errors also..
>
Will do.
> For cxgb4 these are the two critical udev lines:
>
> ENV{ID_NET_DRIVER}=="cxgb4", RUN{builtin}+="kmod load iw_cxgb4"
> DRIVERS=="cxgb4", ENV{ID_RDMA_IWARP}="1"
>
> The first should load iw_cxgb4 if an ethernet device with the cxgb4
> driver is found (the driver name being matched is as reported by
> ethtool -i)
>
> Both need to trigger for things to work properly, you can check with:
>
> $ udevadm info /sys/class/net/ethX
> $ udevadm info /sys/class/infiniband/cxgb4_0 # ??
>
Thanks! I'll let you know how it goes.
Steve.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 33+ messages in thread* Re: [PATCH rdma-core 0/5] Common systemd/udev based boot support
2017-07-25 16:43 ` Steve Wise
@ 2017-07-25 16:55 ` Jason Gunthorpe
0 siblings, 0 replies; 33+ messages in thread
From: Jason Gunthorpe @ 2017-07-25 16:55 UTC (permalink / raw)
To: Steve Wise
Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA, 'Leon Romanovsky',
'Doug Ledford', 'Ram Amrani', 'Ira Weiny',
'Benjamin Drung', 'Jarod Wilson'
On Tue, Jul 25, 2017 at 11:43:49AM -0500, Steve Wise wrote:
>
> > >
> > > Hey Jason,
> > >
> > > Thanks for doing all this! I'm testing with cxgb4 today.
> > >
> > > I get these errors building the rpms using the spec file in your tree:
> >
> > I think you are using ./rdma-core.spec? I forgot about that one and
> > didn't make it work, I will fix it.
> >
> > You probably want to use redhat/rdma-core.spec anyhow?
> >
>
> Ok. I haven't been paying attention to the distro-specific stuff. I had a
> process I have been following to build an rpm from ./rdma-core.spec. I'll try
> the redhat spec file since I'm installing on a RHEL7.2 system.
Actually.. Humm, I've been testing the debian build. If you use the
redhat/rdma-core.spec it will include the old redhat init scheme and
that kind of invalidates the testing of the new scheme ...
I just forced pushed the needed fix to my branch so you should be able
to keep doing what you were doing.
> I've uninstalled all the pertinent packages, I think, with this:
>
> yum autoremove -y libibverbs librdmacm libibumad libiwpm
Also the rdma package needs to go (eg rdma.service and
/lib/udev/rules.d/98-rdma.rules must not exist)
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 33+ messages in thread
* RE: [PATCH rdma-core 0/5] Common systemd/udev based boot support
[not found] ` <20170725164004.GA20959-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
2017-07-25 16:43 ` Steve Wise
@ 2017-07-25 18:49 ` Steve Wise
2017-07-25 21:33 ` Jason Gunthorpe
1 sibling, 1 reply; 33+ messages in thread
From: Steve Wise @ 2017-07-25 18:49 UTC (permalink / raw)
To: 'Jason Gunthorpe'
Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA, 'Leon Romanovsky',
'Doug Ledford', 'Ram Amrani', 'Ira Weiny',
'Benjamin Drung', 'Jarod Wilson'
>
> For iwarp I expect that systemd will auto start these units:
>
> rdma-load-modules-r79FGtN6zq8c5CkaFG5UGg@public.gmane.org loaded active exited Load RDMA
> modules from /etc/rdma/modules/iwpmd.conf
> rdma-load-modules-XSrBNAzizW1TDjBF/Jpztg@public.gmane.org loaded active exited Load RDMA
modules
> from /etc/rdma/modules/rdma.conf
> rdma-load-modules-mPXOxrv7gkYc5CkaFG5UGg@public.gmane.org loaded active exited Load RDMA
modules
> from /etc/rdma/modules/iwarp.conf
> iwpmd.service loaded active running iWarp Port
Mapper
>
After I rebooted, cxgb4 and iw_cxgb4 were loaded, but the services were not
started. I think that is probably due to the fact that iw_cxgb4 doesn't
register with the rdma core until the cxgb4 interfaces are brought up. I then
brought up port 0 of the cxgb4 card:
[root@stevo3 ~]# ip link set enp4s0f4 up
[root@stevo3 ~]# ip addr add 172.16.3.3/24 dev enp4s0f4
And then the services were started:
[root@stevo3 ~]# systemctl status rdma-load-modules-mPXOxrv7gkYc5CkaFG5UGg@public.gmane.org
â rdma-load-modules-mPXOxrv7gkYc5CkaFG5UGg@public.gmane.org - Load RDMA modules from
/etc/rdma/modules/iwarp.conf
Loaded: loaded (/usr/lib/systemd/system/rdma-load-modules@.service; static;
vendor preset: disabled)
Active: active (exited) since Tue 2017-07-25 11:43:21 PDT; 5s ago
Process: 2571 ExecStart=/lib/systemd/systemd-modules-load
/etc/rdma/modules/%I.conf (code=exited, status=0/SUCCESS)
Main PID: 2571 (code=exited, status=0/SUCCESS)
Jul 25 11:43:21 stevo3.asicdesigners.com systemd[1]: Starting Load RDMA modules
from /etc/rdma/modules/iwarp.conf...
Jul 25 11:43:21 stevo3.asicdesigners.com systemd[1]: Started Load RDMA modules
from /etc/rdma/modules/iwarp.conf.
[root@stevo3 ~]# systemctl status rdma-load-modules-XSrBNAzizW1TDjBF/Jpztg@public.gmane.org
â rdma-load-modules-XSrBNAzizW1TDjBF/Jpztg@public.gmane.org - Load RDMA modules from
/etc/rdma/modules/rdma.conf
Loaded: loaded (/usr/lib/systemd/system/rdma-load-modules@.service; static;
vendor preset: disabled)
Active: active (exited) since Tue 2017-07-25 11:43:21 PDT; 10s ago
Process: 2572 ExecStart=/lib/systemd/systemd-modules-load
/etc/rdma/modules/%I.conf (code=exited, status=0/SUCCESS)
Main PID: 2572 (code=exited, status=0/SUCCESS)
Jul 25 11:43:21 stevo3.asicdesigners.com systemd[1]: Starting Load RDMA modules
from /etc/rdma/modules/rdma.conf...
Jul 25 11:43:21 stevo3.asicdesigners.com systemd-modules-load[2572]: Inserted
module 'ib_iser'
Jul 25 11:43:21 stevo3.asicdesigners.com systemd-modules-load[2572]: Inserted
module 'ib_uverbs'
Jul 25 11:43:21 stevo3.asicdesigners.com systemd-modules-load[2572]: Inserted
module 'rdma_ucm'
Jul 25 11:43:21 stevo3.asicdesigners.com systemd[1]: Started Load RDMA modules
from /etc/rdma/modules/rdma.conf.
[root@stevo3 ~]# systemctl status rdma-load-modules-r79FGtN6zq8c5CkaFG5UGg@public.gmane.org
â rdma-load-modules-r79FGtN6zq8c5CkaFG5UGg@public.gmane.org - Load RDMA modules from
/etc/rdma/modules/iwpmd.conf
Loaded: loaded (/usr/lib/systemd/system/rdma-load-modules@.service; static;
vendor preset: disabled)
Active: active (exited) since Tue 2017-07-25 11:43:21 PDT; 17s ago
Process: 2573 ExecStart=/lib/systemd/systemd-modules-load
/etc/rdma/modules/%I.conf (code=exited, status=0/SUCCESS)
Main PID: 2573 (code=exited, status=0/SUCCESS)
Jul 25 11:43:21 stevo3.asicdesigners.com systemd[1]: Starting Load RDMA modules
from /etc/rdma/modules/iwpmd.conf...
Jul 25 11:43:21 stevo3.asicdesigners.com systemd[1]: Started Load RDMA modules
from /etc/rdma/modules/iwpmd.conf.
And iwpmd is running:
[root@stevo3 ~]# pgrep iwpmd
2583
> And you should have a reasonable set of modules loaded. Check the
> journal for interesting errors also..
>
[root@stevo3 ~]# lsmod|egrep "ib_|rdma"
rpcrdma 94208 0
rdma_ucm 24576 0
ib_uverbs 61440 1 rdma_ucm
ib_iser 49152 0
rdma_cm 53248 3 ib_iser,rpcrdma,rdma_ucm
ib_cm 45056 1 rdma_cm
libiscsi 57344 1 ib_iser
iw_cm 45056 1 rdma_cm
scsi_transport_iscsi 98304 2 ib_iser,libiscsi
ib_core 208896 9
ib_iser,ib_cm,rdma_cm,iw_cxgb4,ib_uverbs,rpcrdma,iw_cm,rdma_ucm
sunrpc 348160 8 auth_rpcgss,nfsd,rpcrdma,nfs_acl,lockd
> For cxgb4 these are the two critical udev lines:
>
> ENV{ID_NET_DRIVER}=="cxgb4", RUN{builtin}+="kmod load iw_cxgb4"
> DRIVERS=="cxgb4", ENV{ID_RDMA_IWARP}="1"
>
> The first should load iw_cxgb4 if an ethernet device with the cxgb4
> driver is found (the driver name being matched is as reported by
> ethtool -i)
I assume since iw_cxgb4 was loaded, that they worked. Because before I
installed rdma-core, after a reboot only cxgb4 was loaded.
>
> Both need to trigger for things to work properly, you can check with:
>
> $ udevadm info /sys/class/net/ethX
> $ udevadm info /sys/class/infiniband/cxgb4_0 # ??
>
How does this look?
[root@stevo3 ~]# udevadm info /sys/class/net/enp4s0f4
P: /devices/pci0000:00/0000:00:02.0/0000:04:00.4/net/enp4s0f4
E: DEVPATH=/devices/pci0000:00/0000:00:02.0/0000:04:00.4/net/enp4s0f4
E: ID_BUS=pci
E: ID_MM_CANDIDATE=1
E: ID_MODEL_FROM_DATABASE=T520-LL-CR Unified Wire Ethernet Controller
E: ID_MODEL_ID=0x5411
E: ID_NET_DRIVER=cxgb4
E: ID_NET_NAME_MAC=enx000743292f20
E: ID_NET_NAME_PATH=enp4s0f4
E: ID_OUI_FROM_DATABASE=Chelsio Communications
E: ID_PATH=pci-0000:04:00.4
E: ID_PATH_TAG=pci-0000_04_00_4
E: ID_PCI_CLASS_FROM_DATABASE=Network controller
E: ID_PCI_SUBCLASS_FROM_DATABASE=Ethernet controller
E: ID_VENDOR_FROM_DATABASE=Chelsio Communications Inc
E: ID_VENDOR_ID=0x1425
E: IFINDEX=6
E: INTERFACE=enp4s0f4
E: MAJOR=0
E: MINOR=0
E: SUBSYSTEM=net
E: SYSTEMD_ALIAS=/sys/subsystem/net/devices/enp4s0f4
/sys/subsystem/net/devices/enp4s0f4
E: TAGS=:systemd:
E: USEC_INITIALIZED=68132
[root@stevo3 ~]# udevadm info /sys/class/infiniband/cxgb4_0 #??
P: /devices/pci0000:00/0000:00:02.0/0000:04:00.4/infiniband/cxgb4_0
E: DEVPATH=/devices/pci0000:00/0000:00:02.0/0000:04:00.4/infiniband/cxgb4_0
E: ID_BUS=pci
E: ID_MODEL_FROM_DATABASE=T520-LL-CR Unified Wire Ethernet Controller
E: ID_MODEL_ID=0x5411
E: ID_PCI_CLASS_FROM_DATABASE=Network controller
E: ID_PCI_SUBCLASS_FROM_DATABASE=Ethernet controller
E: ID_RDMA_IWARP=1
E: ID_VENDOR_FROM_DATABASE=Chelsio Communications Inc
E: ID_VENDOR_ID=0x1425
E: NAME=cxgb4_0
E: SUBSYSTEM=infiniband
E: SYSTEMD_WANTS=rdma-ndd.service iwpmd.service rdma-load-modules-XSrBNAzizW1TDjBF/Jpztg@public.gmane.org
rdma-load-modules-mPXOxrv7gkYc5CkaFG5UGg@public.gmane.org
E: TAGS=:systemd:
E: USEC_INITIALIZED=428610768
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 33+ messages in thread* Re: [PATCH rdma-core 0/5] Common systemd/udev based boot support
2017-07-25 18:49 ` Steve Wise
@ 2017-07-25 21:33 ` Jason Gunthorpe
[not found] ` <20170725213354.GE10905-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
0 siblings, 1 reply; 33+ messages in thread
From: Jason Gunthorpe @ 2017-07-25 21:33 UTC (permalink / raw)
To: Steve Wise
Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA, 'Doug Ledford',
'Ram Amrani', 'Ira Weiny',
'Benjamin Drung', 'Jarod Wilson'
On Tue, Jul 25, 2017 at 01:49:37PM -0500, Steve Wise wrote:
> After I rebooted, cxgb4 and iw_cxgb4 were loaded, but the services were not
> started. I think that is probably due to the fact that iw_cxgb4 doesn't
> register with the rdma core until the cxgb4 interfaces are brought
> up.
Yes, it is done on demand when the kernel creates its side of the
interface.
> I then brought up port 0 of the cxgb4 card:
>
> [root@stevo3 ~]# ip link set enp4s0f4 up
> [root@stevo3 ~]# ip addr add 172.16.3.3/24 dev enp4s0f4
>
> And then the services were started:
Yep, everything you sent me looks fine, thanks for taking a look at
this.
This sort of hotplug that cxbg4 does is quite strange, what happens
when 'ip link set X down' is done? Does it remove the RDMA device?
Does 'ip link set down' block until all users go away?
This is going to make it harder for cxgb users to get a reliably
bootup at this time, we need more kernel autoloading for things to be
reliable, and I'm sure iwpmd.service needs some dependency adjusting,
I just don't know enough about it to do it right. :\
> I assume since iw_cxgb4 was loaded, that they worked. Because before I
> installed rdma-core, after a reboot only cxgb4 was loaded.
Yep
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 33+ messages in thread
* RE: [PATCH rdma-core 0/5] Common systemd/udev based boot support
2017-07-25 16:40 ` Jason Gunthorpe
[not found] ` <20170725164004.GA20959-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
@ 2017-07-25 17:01 ` Steve Wise
2017-07-25 17:05 ` Jason Gunthorpe
2017-07-25 18:55 ` Steve Wise
2 siblings, 1 reply; 33+ messages in thread
From: Steve Wise @ 2017-07-25 17:01 UTC (permalink / raw)
To: 'Jason Gunthorpe'
Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA, 'Leon Romanovsky',
'Doug Ledford', 'Ram Amrani', 'Ira Weiny',
'Benjamin Drung', 'Jarod Wilson'
>
>
> > >
> > > Hey Jason,
> > >
> > > Thanks for doing all this! I'm testing with cxgb4 today.
> > >
> > > I get these errors building the rpms using the spec file in your tree:
> >
> > I think you are using ./rdma-core.spec? I forgot about that one and
> > didn't make it work, I will fix it.
> >
> > You probably want to use redhat/rdma-core.spec anyhow?
> >
>
> Ok. I haven't been paying attention to the distro-specific stuff. I had a
process I
> have been following to build an rpm from ./rdma-core.spec. I'll try the
redhat spec
> file since I'm installing on a RHEL7.2 system.
Hmm, the rdma-core rpm built ok, but there are no libs or commands in it. ? I
think I'm in the weeds. here :)
What I did was:
[root@stevo3 rdma-core-15]# git clone
https://github.com/jgunthorpe/rdma-plumbing rdma-core-15
[root@stevo3 rdma-core-15]# tar czf ~/rpmbuild/SOURCES/rdma-core-15.tgz
rdma-core-15
[root@stevo3 rdma-core-15]# rpmbuild -bb rdma-core-15/redhat/rdma-core.spec
[root@stevo3 rdma-core-15]# rpm --install
~/rpmbuild/RPMS/x86_64/rdma-core-devel-15-1.el7.x86_64.rpm
[root@stevo3 rdma-core-15]# rpm -ql rdma-core
/etc/modprobe.d/mlx4.conf
/etc/modprobe.d/truescale.conf
/etc/rdma
/etc/rdma/mlx4.conf
/etc/rdma/modules/infiniband.conf
/etc/rdma/modules/iwarp.conf
/etc/rdma/modules/opa.conf
/etc/rdma/modules/rdma.conf
/etc/rdma/modules/roce.conf
/etc/rdma/rdma.conf
/etc/rdma/sriov-vfs
/etc/sysconfig/network-scripts/ifdown-ib
/etc/sysconfig/network-scripts/ifup-ib
/etc/udev/rules.d/70-persistent-ipoib.rules
/usr/bin/rxe_cfg
/usr/lib/dracut/modules.d/05rdma
/usr/lib/dracut/modules.d/05rdma/module-setup.sh
/usr/lib/modprobe.d/libmlx4.conf
/usr/lib/systemd/system/rdma-load-modules@.service
/usr/lib/systemd/system/rdma-ndd.service
/usr/lib/systemd/system/rdma.service
/usr/lib/udev/rules.d/60-rdma-ndd.rules
/usr/lib/udev/rules.d/75-rdma-description.rules
/usr/lib/udev/rules.d/90-rdma-hw-modules.rules
/usr/lib/udev/rules.d/90-rdma-ulp-modules.rules
/usr/lib/udev/rules.d/90-rdma-umad.rules
/usr/lib/udev/rules.d/98-rdma.rules
/usr/libexec/mlx4-setup.sh
/usr/libexec/rdma-fixup-mtrr.awk
/usr/libexec/rdma-init-kernel
/usr/libexec/rdma-set-sriov-vf
/usr/libexec/truescale-serdes.cmds
/usr/sbin/rdma-ndd
/usr/share/doc/rdma-core-15
/usr/share/doc/rdma-core-15/README.md
/usr/share/doc/rdma-core-15/rxe.md
/usr/share/licenses/rdma-core-15
/usr/share/licenses/rdma-core-15/COPYING.BSD_FB
/usr/share/licenses/rdma-core-15/COPYING.BSD_MIT
/usr/share/licenses/rdma-core-15/COPYING.GPL2
/usr/share/licenses/rdma-core-15/COPYING.md
/usr/share/man/man7/rxe.7.gz
/usr/share/man/man8/rdma-ndd.8.gz
/usr/share/man/man8/rxe_cfg.8.gz
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 33+ messages in thread* Re: [PATCH rdma-core 0/5] Common systemd/udev based boot support
2017-07-25 17:01 ` Steve Wise
@ 2017-07-25 17:05 ` Jason Gunthorpe
[not found] ` <20170725170506.GB3164-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
0 siblings, 1 reply; 33+ messages in thread
From: Jason Gunthorpe @ 2017-07-25 17:05 UTC (permalink / raw)
To: Steve Wise
Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA, 'Leon Romanovsky',
'Doug Ledford', 'Ram Amrani', 'Ira Weiny',
'Benjamin Drung', 'Jarod Wilson'
On Tue, Jul 25, 2017 at 12:01:43PM -0500, Steve Wise wrote:
> Hmm, the rdma-core rpm built ok, but there are no libs or commands in it. ? I
> think I'm in the weeds. here :)
The redhat/rdma-core.spec produces 12 rpms:
ibacm-15-1.el7.centos.x86_64.rpm
iwpmd-15-1.el7.centos.x86_64.rpm
libibcm-15-1.el7.centos.x86_64.rpm
libibumad-15-1.el7.centos.x86_64.rpm
libibverbs-15-1.el7.centos.x86_64.rpm
libibverbs-utils-15-1.el7.centos.x86_64.rpm
librdmacm-15-1.el7.centos.x86_64.rpm
librdmacm-utils-15-1.el7.centos.x86_64.rpm
rdma-core-15-1.el7.centos.x86_64.rpm
rdma-core-debuginfo-15-1.el7.centos.x86_64.rpm
rdma-core-devel-15-1.el7.centos.x86_64.rpm
srp_daemon-15-1.el7.centos.x86_64.rpm
The rdma-core package in this system only contains the kernel boot
support.
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 33+ messages in thread
* RE: [PATCH rdma-core 0/5] Common systemd/udev based boot support
2017-07-25 16:40 ` Jason Gunthorpe
[not found] ` <20170725164004.GA20959-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
2017-07-25 17:01 ` Steve Wise
@ 2017-07-25 18:55 ` Steve Wise
2 siblings, 0 replies; 33+ messages in thread
From: Steve Wise @ 2017-07-25 18:55 UTC (permalink / raw)
To: 'Jason Gunthorpe'
Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA, 'Leon Romanovsky',
'Doug Ledford', 'Ram Amrani', 'Ira Weiny',
'Benjamin Drung', 'Jarod Wilson'
> > And you should have a reasonable set of modules loaded. Check the
> > journal for interesting errors also..
Oh and here's the journal after bringing up the cxgb4 interface. I think it
looks good.
Jul 25 11:43:21 stevo3.asicdesigners.com kernel: iw_cxgb4:0000:04:00.4: Up
Jul 25 11:43:21 stevo3.asicdesigners.com kernel: 0000:04:00.4: On-Chip Queues
not supported on this device.
Jul 25 11:43:21 stevo3.asicdesigners.com systemd[1]: Started RDMA Node
Description Daemon.
Jul 25 11:43:21 stevo3.asicdesigners.com systemd[1]: Starting RDMA Node
Description Daemon...
Jul 25 11:43:21 stevo3.asicdesigners.com systemd[1]: Created slice
system-rdma\x2dload\x2dmodules.slice.
Jul 25 11:43:21 stevo3.asicdesigners.com systemd[1]: Starting
system-rdma\x2dload\x2dmodules.slice.
Jul 25 11:43:21 stevo3.asicdesigners.com systemd[1]: Starting Load RDMA modules
from /etc/rdma/modules/iwarp.conf...
Jul 25 11:43:21 stevo3.asicdesigners.com systemd[1]: Starting Load RDMA modules
from /etc/rdma/modules/rdma.conf...
Jul 25 11:43:21 stevo3.asicdesigners.com systemd[1]: Starting Load RDMA modules
from /etc/rdma/modules/iwpmd.conf...
Jul 25 11:43:21 stevo3.asicdesigners.com systemd[1]: Started Load RDMA modules
from /etc/rdma/modules/iwarp.conf.
Jul 25 11:43:21 stevo3.asicdesigners.com kernel: IPv6: ADDRCONF(NETDEV_UP):
enp4s0f4: link is not ready
Jul 25 11:43:21 stevo3.asicdesigners.com kernel: cxgb4 0000:04:00.4 enp4s0f4:
passive DA module inserted
Jul 25 11:43:21 stevo3.asicdesigners.com kernel: Loading iSCSI transport class
v2.0-870.
Jul 25 11:43:21 stevo3.asicdesigners.com avahi-daemon[808]: Joining mDNS
multicast group on interface enp4s0f4.IPv4 with address 172.16.3.3.
Jul 25 11:43:21 stevo3.asicdesigners.com systemd-modules-load[2573]: Inserted
module 'iw_cm'
Jul 25 11:43:21 stevo3.asicdesigners.com systemd[1]: Started Load RDMA modules
from /etc/rdma/modules/iwpmd.conf.
Jul 25 11:43:21 stevo3.asicdesigners.com avahi-daemon[808]: New relevant
interface enp4s0f4.IPv4 for mDNS.
Jul 25 11:43:21 stevo3.asicdesigners.com avahi-daemon[808]: Registering new
address record for 172.16.3.3 on enp4s0f4.IPv4.
Jul 25 11:43:21 stevo3.asicdesigners.com systemd[1]: Started iWarp Port Mapper.
Jul 25 11:43:21 stevo3.asicdesigners.com systemd[1]: Starting iWarp Port
Mapper...
Jul 25 11:43:21 stevo3.asicdesigners.com iWarpPortMapper[2583]: get_iwpm_param:
Got param (name = nl_sock_rbuf_size val = 419430400)
Jul 25 11:43:21 stevo3.asicdesigners.com kernel: ib_core:ibnl_rcv_msg: Index 3
wasn't found in client list
Jul 25 11:43:21 stevo3.asicdesigners.com systemd-modules-load[2572]: Inserted
module 'ib_iser'
Jul 25 11:43:21 stevo3.asicdesigners.com kernel: iscsi: registered transport
(iser)
Jul 25 11:43:21 stevo3.asicdesigners.com systemd-modules-load[2572]: Inserted
module 'ib_uverbs'
Jul 25 11:43:21 stevo3.asicdesigners.com systemd-modules-load[2572]: Inserted
module 'rdma_ucm'
Jul 25 11:43:21 stevo3.asicdesigners.com systemd-modules-load[2572]: Inserted
module 'rpcrdma'
Jul 25 11:43:21 stevo3.asicdesigners.com systemd[1]: Started Load RDMA modules
from /etc/rdma/modules/rdma.conf.
Jul 25 11:43:21 stevo3.asicdesigners.com kernel: RPC: Registered rdma transport
module.
Jul 25 11:43:21 stevo3.asicdesigners.com kernel: RPC: Registered rdma
backchannel transport module.
Jul 25 11:43:22 stevo3.asicdesigners.com kernel: cxgb4 0000:04:00.4 enp4s0f4:
link up, 10Gbps, full-duplex, Tx/Rx PAUSE
Jul 25 11:43:22 stevo3.asicdesigners.com kernel: IPv6: ADDRCONF(NETDEV_CHANGE):
enp4s0f4: link becomes ready
Jul 25 11:43:22 stevo3.asicdesigners.com NetworkManager[781]: <info>
(enp4s0f4): link connected
Jul 25 11:43:23 stevo3.asicdesigners.com avahi-daemon[808]: Registering new
address record for fe80::207:43ff:fe29:2f20 on enp4s0f4.*.
Jul 25 11:50:01 stevo3.asicdesigners.com systemd[1]: Started Session 4 of user
root.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 33+ messages in thread
* RE: [PATCH rdma-core 0/5] Common systemd/udev based boot support
[not found] ` <1500929067-1583-1-git-send-email-jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
` (5 preceding siblings ...)
2017-07-25 16:27 ` [PATCH rdma-core 0/5] Common systemd/udev based boot support Steve Wise
@ 2017-07-27 7:47 ` Amrani, Ram
[not found] ` <BN3PR07MB257807A6AE85B9B5575AAD7AF8BE0-EldUQEzkDQfpW3VS/XPqkOFPX92sqiQdvxpqHgZTriW3zl9H0oFU5g@public.gmane.org>
6 siblings, 1 reply; 33+ messages in thread
From: Amrani, Ram @ 2017-07-27 7:47 UTC (permalink / raw)
To: Jason Gunthorpe, linux-rdma-u79uwXL29TY76Z2rM5mHXA,
Leon Romanovsky, Doug Ledford, Steve Wise, Ira Weiny,
Benjamin Drung, Jarod Wilson
Hi Jason,
> I do not have nearly enough hardware to exhaustively test this, I have checked
> that the basic elements are working, but there is lots of room for little
> errors, particularly in module names and udev matching. It would be great of
> someone could try this on iwarp hardware and confirm all the autoloading works
> as expected.
>
> As should the active roce vendors. I expect that all roce hardware will
> autoload the RDMA modules with this approach.
It works for qedr.
I have followed Steve's steps and built the rpms on Red Hat 7.3 from your repo/systemd.
I've removed all the previous libraries and 98-rdma.rules and then installed the newly built rpms
After that I rebooted and qedr was loaded.
I didn't have to configure the interfaces as Steve did (ifconfig <device> up).
I've notice that 98-rdma.rules got recreated, but the new file doesn't contain any qed entry.
Why is it recreated? If it is in use, why isn't qed listed?
> +# Hardware that supports iWarp
> +DRIVERS=="cxgb3", ENV{ID_RDMA_IWARP}="1"
> +DRIVERS=="cxgb4", ENV{ID_RDMA_IWARP}="1"
> +
> +# Hardware that supports RoCE
> +DRIVERS=="be2net", ENV{ID_RDMA_ROCE}="1"
> +DRIVERS=="bnxt_en", ENV{ID_RDMA_ROCE}="1"
> +DRIVERS=="hns", ENV{ID_RDMA_ROCE}="1"
> +DRIVERS=="i40e", ENV{ID_RDMA_ROCE}="1"
> +DRIVERS=="mlx4_core", ENV{ID_RDMA_ROCE}="1"
> +DRIVERS=="mlx5_core", ENV{ID_RDMA_ROCE}="1"
> +DRIVERS=="qede", ENV{ID_RDMA_ROCE}="1"
We've recently sent iWARP patches this means qede will be used by either RoCE or iWARP.
In any case the RDMA driver is the same - qedr.
Can you explain what effect does this ENV property has on actual operation?
How do you think it should look in the light of the upcoming patch?
Thanks for coding this,
Ram
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 33+ messages in thread