* [PATCH blktests 0/3] Three SRP patches @ 2021-01-26 4:45 Bart Van Assche 2021-01-26 4:45 ` [PATCH blktests 1/3] tests/block/030: Make this test less noisy Bart Van Assche ` (2 more replies) 0 siblings, 3 replies; 8+ messages in thread From: Bart Van Assche @ 2021-01-26 4:45 UTC (permalink / raw) To: Omar Sandoval; +Cc: linux-block, Yi Zhang, Bart Van Assche Hi Omar, The three patches in this series improve the SRP tests. Please consider these patches for inclusion in the upstream blktests repository. Thanks, Bart. Bart Van Assche (3): tests/block/030: Make this test less noisy tests/srp/rc: Improve reliability of stop_lio_srpt() rdma: Use rdma link instead of /sys/class/infiniband/*/parent common/multipath-over-rdma | 111 +++++++++++-------------------------- tests/block/030 | 2 +- tests/srp/rc | 52 ++++++----------- 3 files changed, 48 insertions(+), 117 deletions(-) ^ permalink raw reply [flat|nested] 8+ messages in thread
* [PATCH blktests 1/3] tests/block/030: Make this test less noisy 2021-01-26 4:45 [PATCH blktests 0/3] Three SRP patches Bart Van Assche @ 2021-01-26 4:45 ` Bart Van Assche 2021-01-26 4:45 ` [PATCH blktests 2/3] tests/srp/rc: Improve reliability of stop_lio_srpt() Bart Van Assche 2021-01-26 4:45 ` [PATCH blktests 3/3] rdma: Use rdma link instead of /sys/class/infiniband/*/parent Bart Van Assche 2 siblings, 0 replies; 8+ messages in thread From: Bart Van Assche @ 2021-01-26 4:45 UTC (permalink / raw) To: Omar Sandoval; +Cc: linux-block, Yi Zhang, Bart Van Assche Since test block/030 injects blk_mq_realloc_hw_ctxs() failures, it is expected that writes into the 'submit_queues' attribute can fail. Send the 'nproc: write error: Cannot allocate memory' failures to $FULL instead of stderr. See also commit a668c61064f2 ("Add a test that triggers the blk_mq_realloc_hw_ctxs() error path"). Signed-off-by: Bart Van Assche <bvanassche@acm.org> --- tests/block/030 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/block/030 b/tests/block/030 index 84302c109c0c..d2e528697873 100755 --- a/tests/block/030 +++ b/tests/block/030 @@ -42,7 +42,7 @@ test() { if { echo "$(<"$sq")" >$sq; } 2>/dev/null; then for ((i = 0; i < 100; i++)); do echo 1 > $sq - nproc > $sq + { nproc > $sq; } >>"$FULL" 2>&1 done else SKIP_REASON="Skipping test because $sq cannot be modified" ^ permalink raw reply related [flat|nested] 8+ messages in thread
* [PATCH blktests 2/3] tests/srp/rc: Improve reliability of stop_lio_srpt() 2021-01-26 4:45 [PATCH blktests 0/3] Three SRP patches Bart Van Assche 2021-01-26 4:45 ` [PATCH blktests 1/3] tests/block/030: Make this test less noisy Bart Van Assche @ 2021-01-26 4:45 ` Bart Van Assche 2021-01-26 4:45 ` [PATCH blktests 3/3] rdma: Use rdma link instead of /sys/class/infiniband/*/parent Bart Van Assche 2 siblings, 0 replies; 8+ messages in thread From: Bart Van Assche @ 2021-01-26 4:45 UTC (permalink / raw) To: Omar Sandoval; +Cc: linux-block, Yi Zhang, Bart Van Assche Remove the 'np' directory if it exists. Unload the iscsi_target_mod kernel module if it has been loaded. Signed-off-by: Bart Van Assche <bvanassche@acm.org> --- tests/srp/rc | 43 +++++++++++++++---------------------------- 1 file changed, 15 insertions(+), 28 deletions(-) diff --git a/tests/srp/rc b/tests/srp/rc index 1f665a28db66..700cd71ea155 100755 --- a/tests/srp/rc +++ b/tests/srp/rc @@ -545,34 +545,21 @@ stop_lio_srpt() { fi done - if [ -e /sys/kernel/config/target/srpt ]; then - ( - cd /sys/kernel/config/target/srpt && ( - for d in */*/acls/*/*/lun*; do [ -L "$d" ] && rm "$d"; done - for d in */*/acls/*/lun*; do [ -d "$d" ] && rmdir "$d"; done - for d in */*/acls/*; do [ -d "$d" ] && rmdir "$d"; done - for d in */*/lun/lun*/*; do [ -L "$d" ] && rm "$d"; done - for d in */*/lun/lun*; do [ -d "$d" ] && rmdir "$d"; done - for d in */*; do [ -e "$d/lun" ] && rmdir "$d"; done - for d in *; do [ -e "$d/fabric_statistics" ] && rmdir "$d"; done - true - ) && - cd .. && - for ((i=0;i<10;i++)); do - rmdir srpt - [ -e srpt ] || break - sleep .1 - done && - [ ! -e srpt ] && - unload_module ib_srpt 10 - ) || return $? - fi - - rmdir /sys/kernel/config/target/core/*/* >&/dev/null - rmdir /sys/kernel/config/target/core/* >&/dev/null - - for m in ib_srpt target_core_pscsi target_core_iblock target_core_file \ - target_core_stgt target_core_user target_core_mod + rmdir /sys/kernel/config/target/*/*/*/np/* >&/dev/null + rmdir /sys/kernel/config/target/*/*/*/np >&/dev/null + rm -f /sys/kernel/config/target/*/*/*/acls/*/*/* >&/dev/null + rmdir /sys/kernel/config/target/*/*/*/acls/*/* >&/dev/null + rmdir /sys/kernel/config/target/*/*/*/acls/* >&/dev/null + rm -f /sys/kernel/config/target/*/*/*/lun/*/* >&/dev/null + rmdir /sys/kernel/config/target/*/*/*/lun/* >&/dev/null + rmdir /sys/kernel/config/target/*/*/*/*/* >&/dev/null + rmdir /sys/kernel/config/target/*/*/* >&/dev/null + rmdir /sys/kernel/config/target/*/* >&/dev/null + rmdir /sys/kernel/config/target/* >&/dev/null + + for m in ib_srpt iscsi_target_mod target_core_pscsi target_core_iblock \ + target_core_file target_core_stgt target_core_user \ + target_core_mod do unload_module $m 10 || return $? done ^ permalink raw reply related [flat|nested] 8+ messages in thread
* [PATCH blktests 3/3] rdma: Use rdma link instead of /sys/class/infiniband/*/parent 2021-01-26 4:45 [PATCH blktests 0/3] Three SRP patches Bart Van Assche 2021-01-26 4:45 ` [PATCH blktests 1/3] tests/block/030: Make this test less noisy Bart Van Assche 2021-01-26 4:45 ` [PATCH blktests 2/3] tests/srp/rc: Improve reliability of stop_lio_srpt() Bart Van Assche @ 2021-01-26 4:45 ` Bart Van Assche 2021-02-03 16:40 ` Omar Sandoval 2 siblings, 1 reply; 8+ messages in thread From: Bart Van Assche @ 2021-01-26 4:45 UTC (permalink / raw) To: Omar Sandoval; +Cc: linux-block, Yi Zhang, Bart Van Assche, Jason Gunthorpe The approach of verifying whether or not an RDMA interface is associated with the rdma_rxe interface by looking up its parent device is deprecated and will be removed soon from the Linux kernel. Hence this patch that uses the rdma link command instead. Cc: Jason Gunthorpe <jgg@nvidia.com> Cc: Yi Zhang <yi.zhang@redhat.com> Signed-off-by: Bart Van Assche <bvanassche@acm.org> --- common/multipath-over-rdma | 111 +++++++++++-------------------------- tests/srp/rc | 9 +-- 2 files changed, 32 insertions(+), 88 deletions(-) diff --git a/common/multipath-over-rdma b/common/multipath-over-rdma index 9d9d2b27af83..9e68189b56fd 100644 --- a/common/multipath-over-rdma +++ b/common/multipath-over-rdma @@ -57,6 +57,9 @@ _multipathd_version_ge() { } get_ipv4_addr() { + if [ ! -e "/sys/class/net/$1" ]; then + echo "get_ipv4_addr(): $1 is not a network interface" 1>&2 + fi ip -4 -o addr show dev "$1" | sed -n 's/.*[[:blank:]]inet[[:blank:]]*\([^[:blank:]/]*\).*/\1/p' } @@ -76,41 +79,11 @@ is_number() { [ "$1" -eq "0$1" ] 2>/dev/null } -# Check whether a device is an RDMA device. An example argument: -# /sys/devices/pci0000:00/0000:00:03.0/0000:04:00.0 -is_rdma_device() { - local d i inode1 inode2 - - inode1=$(stat -c %i "$1") - # echo "inode1 = $inode1" - for i in /sys/class/infiniband/*; do - d=/sys/class/infiniband/"$(readlink "$i")" - d=$(dirname "$(dirname "$d")") - inode2=$(stat -c %i "$d") - # echo "inode2 = $inode2" - if [ "$inode1" = "$inode2" ]; then - return - fi - done - false -} - # Lists RDMA capable network interface names, e.g. ib0 ib1. rdma_network_interfaces() { - ( - cd /sys/class/net && - for i in *; do - [ -e "$i" ] || continue - # Skip IPoIB (ARPHRD_INFINIBAND) network - # interfaces. - [ "$(<"$i"/type)" = 32 ] && continue - [ -L "$i/device" ] || continue - d=$(readlink "$i/device" 2>/dev/null) - if [ -n "$d" ] && is_rdma_device "$i/$d"; then - echo "$i" - fi - done - ) + rdma link show | + sed -n 's/^.*[[:blank:]]netdev[[:blank:]]\+\([^[:blank:]]*\)[[:blank:]]*/\1/p' | + sort -u } # Check whether any stacked block device holds block device $1. If so, echo @@ -411,47 +384,36 @@ all_primary_gids() { done } -# Check whether or not an rdma_rxe instance has been associated with network -# interface $1. -has_rdma_rxe() { - local f - - for f in /sys/class/infiniband/*/parent; do - if [ -e "$f" ] && [ "$(<"$f")" = "$1" ]; then - return 0 - fi - done - - return 1 +# Check whether or not an rdma_rxe or siw instance has been associated with +# network interface $1. +has_soft_rdma() { + rdma link | grep -q " netdev $1[[:blank:]]*\$" } # Load the rdma_rxe or siw kernel module and associate it with all network # interfaces. start_soft_rdma() { + local type + { if [ -n "$use_siw" ]; then modprobe siw || return $? - ( - cd /sys/class/net && - for i in *; do - [ -e "$i" ] || continue - [ -e "/sys/class/infiniband/${i}_siw" ] && continue - rdma link add "${i}_siw" type siw netdev "$i" || - echo "Failed to bind the siw driver to $i" - done - ) + type=siw else modprobe rdma_rxe || return $? - ( - cd /sys/class/net && - for i in *; do - if [ -e "$i" ] && ! has_rdma_rxe "$i"; then - echo "$i" > /sys/module/rdma_rxe/parameters/add || - echo "Failed to bind the rdma_rxe driver to $i" - fi - done - ) + type=rxe fi + ( + cd /sys/class/net && + for i in *; do + [ -e "$i" ] || continue + [ "$i" = "lo" ] && continue + [ "$(<"$i/addr_len")" = 6 ] || continue + has_soft_rdma "$i" && continue + rdma link add "${i}_$type" type $type netdev "$i" || + echo "Failed to bind the $type driver to $i" + done + ) } >>"$FULL" } @@ -459,27 +421,16 @@ start_soft_rdma() { # unload the rdma_rxe kernel module. stop_soft_rdma() { { - ( - cd /sys/class/net && - for i in *; do - if [ -e "$i" ] && has_rdma_rxe "$i"; then - { echo "$i" > /sys/module/rdma_rxe/parameters/remove; } \ - 2>/dev/null - fi - done - ) + rdma link | + sed -n 's,^link[[:blank:]]*\([^/]*\)/.* netdev .*,\1,p' | + while read -r i; do + echo "$i ..." + rdma link del "${i}" || echo "Failed to remove ${i}" + done if ! unload_module rdma_rxe 10; then echo "Unloading rdma_rxe failed" return 1 fi - ( - cd /sys/class/net && - for i in *_siw; do - [ -e "$i" ] || continue - rdma link del "${i}" || - echo "Failed to unbind the siw driver from ${i%_siw}" - done - ) if ! unload_module siw 10; then echo "Unloading siw failed" return 1 diff --git a/tests/srp/rc b/tests/srp/rc index 700cd71ea155..07378fab2f2c 100755 --- a/tests/srp/rc +++ b/tests/srp/rc @@ -142,14 +142,7 @@ do_ib_cm_login() { } rdma_dev_to_net_dev() { - local b d rdma_dev=$1 - - b=/sys/class/infiniband/$rdma_dev/parent - if [ -e "$b" ]; then - echo "$(<"$b")" - else - echo "${rdma_dev%_siw}" - fi + rdma link show "$1/1" | sed 's/.* netdev //;s/[[:blank:]]*$//' } # Tell the SRP initiator to log in to an SRP target using the RDMA/CM. ^ permalink raw reply related [flat|nested] 8+ messages in thread
* Re: [PATCH blktests 3/3] rdma: Use rdma link instead of /sys/class/infiniband/*/parent 2021-01-26 4:45 ` [PATCH blktests 3/3] rdma: Use rdma link instead of /sys/class/infiniband/*/parent Bart Van Assche @ 2021-02-03 16:40 ` Omar Sandoval 2021-02-04 3:58 ` Bart Van Assche 0 siblings, 1 reply; 8+ messages in thread From: Omar Sandoval @ 2021-02-03 16:40 UTC (permalink / raw) To: Bart Van Assche; +Cc: Omar Sandoval, linux-block, Yi Zhang, Jason Gunthorpe On Mon, Jan 25, 2021 at 08:45:19PM -0800, Bart Van Assche wrote: > The approach of verifying whether or not an RDMA interface is associated > with the rdma_rxe interface by looking up its parent device is deprecated > and will be removed soon from the Linux kernel. Hence this patch that uses > the rdma link command instead. > > Cc: Jason Gunthorpe <jgg@nvidia.com> > Cc: Yi Zhang <yi.zhang@redhat.com> > Signed-off-by: Bart Van Assche <bvanassche@acm.org> > --- > common/multipath-over-rdma | 111 +++++++++++-------------------------- > tests/srp/rc | 9 +-- > 2 files changed, 32 insertions(+), 88 deletions(-) I think we need to add _have_program rdma checks to srp and nvmeof-mp, right? The first two patches look fine, I'll merge those. ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH blktests 3/3] rdma: Use rdma link instead of /sys/class/infiniband/*/parent 2021-02-03 16:40 ` Omar Sandoval @ 2021-02-04 3:58 ` Bart Van Assche 2021-02-28 14:45 ` Yi Zhang 0 siblings, 1 reply; 8+ messages in thread From: Bart Van Assche @ 2021-02-04 3:58 UTC (permalink / raw) To: Omar Sandoval; +Cc: Omar Sandoval, linux-block, Yi Zhang, Jason Gunthorpe On 2/3/21 8:40 AM, Omar Sandoval wrote: > On Mon, Jan 25, 2021 at 08:45:19PM -0800, Bart Van Assche wrote: >> The approach of verifying whether or not an RDMA interface is associated >> with the rdma_rxe interface by looking up its parent device is deprecated >> and will be removed soon from the Linux kernel. Hence this patch that uses >> the rdma link command instead. >> >> Cc: Jason Gunthorpe <jgg@nvidia.com> >> Cc: Yi Zhang <yi.zhang@redhat.com> >> Signed-off-by: Bart Van Assche <bvanassche@acm.org> >> --- >> common/multipath-over-rdma | 111 +++++++++++-------------------------- >> tests/srp/rc | 9 +-- >> 2 files changed, 32 insertions(+), 88 deletions(-) > > I think we need to add _have_program rdma checks to srp and nvmeof-mp, > right? The first two patches look fine, I'll merge those. Sure, I will add a _have_program rdma check and repost this patch. Thanks, Bart. ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH blktests 3/3] rdma: Use rdma link instead of /sys/class/infiniband/*/parent 2021-02-04 3:58 ` Bart Van Assche @ 2021-02-28 14:45 ` Yi Zhang 2021-02-28 22:34 ` Bart Van Assche 0 siblings, 1 reply; 8+ messages in thread From: Yi Zhang @ 2021-02-28 14:45 UTC (permalink / raw) To: Bart Van Assche, Omar Sandoval Cc: Omar Sandoval, linux-block, Jason Gunthorpe Hello Bart Currently blktests was broken on latest kernel, could you resend this updated patch. :) Thanks Yi On 2/4/21 11:58 AM, Bart Van Assche wrote: > On 2/3/21 8:40 AM, Omar Sandoval wrote: >> On Mon, Jan 25, 2021 at 08:45:19PM -0800, Bart Van Assche wrote: >>> The approach of verifying whether or not an RDMA interface is associated >>> with the rdma_rxe interface by looking up its parent device is deprecated >>> and will be removed soon from the Linux kernel. Hence this patch that uses >>> the rdma link command instead. >>> >>> Cc: Jason Gunthorpe <jgg@nvidia.com> >>> Cc: Yi Zhang <yi.zhang@redhat.com> >>> Signed-off-by: Bart Van Assche <bvanassche@acm.org> >>> --- >>> common/multipath-over-rdma | 111 +++++++++++-------------------------- >>> tests/srp/rc | 9 +-- >>> 2 files changed, 32 insertions(+), 88 deletions(-) >> I think we need to add _have_program rdma checks to srp and nvmeof-mp, >> right? The first two patches look fine, I'll merge those. > Sure, I will add a _have_program rdma check and repost this patch. > > Thanks, > > Bart. > > ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH blktests 3/3] rdma: Use rdma link instead of /sys/class/infiniband/*/parent 2021-02-28 14:45 ` Yi Zhang @ 2021-02-28 22:34 ` Bart Van Assche 0 siblings, 0 replies; 8+ messages in thread From: Bart Van Assche @ 2021-02-28 22:34 UTC (permalink / raw) To: Yi Zhang, Omar Sandoval; +Cc: Omar Sandoval, linux-block, Jason Gunthorpe On 2/28/21 6:45 AM, Yi Zhang wrote: > Currently blktests was broken on latest kernel, could you resend this > updated patch. :) Thanks for the reminder. A new version of this patch has just been posted. Bart. ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2021-02-28 22:35 UTC | newest] Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-01-26 4:45 [PATCH blktests 0/3] Three SRP patches Bart Van Assche 2021-01-26 4:45 ` [PATCH blktests 1/3] tests/block/030: Make this test less noisy Bart Van Assche 2021-01-26 4:45 ` [PATCH blktests 2/3] tests/srp/rc: Improve reliability of stop_lio_srpt() Bart Van Assche 2021-01-26 4:45 ` [PATCH blktests 3/3] rdma: Use rdma link instead of /sys/class/infiniband/*/parent Bart Van Assche 2021-02-03 16:40 ` Omar Sandoval 2021-02-04 3:58 ` Bart Van Assche 2021-02-28 14:45 ` Yi Zhang 2021-02-28 22:34 ` Bart Van Assche
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.