All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH ] tcm_qla2xxx  Add SCSI command jammer/discard capabilty to the tcm_qla2xxx module
       [not found] <588344732.13485469.1425769450275.JavaMail.zimbra@redhat.com>
@ 2015-03-08  3:26 ` Laurence Oberman
  2015-03-08  8:10   ` Bart Van Assche
  2015-03-12 22:07   ` [PATCH ] tcm_qla2xxx Add SCSI command jammer/discard capabilty to the tcm_qla2xxx module Quinn Tran
  0 siblings, 2 replies; 9+ messages in thread
From: Laurence Oberman @ 2015-03-08  3:26 UTC (permalink / raw)
  To: Andy Grover, linux-scsi, nab; +Cc: Laurence Oberman

Hello

I use target LIO for all my storage array test targets and customer problem resolution here at Red Hat.
This patch resulted from a requirement to mimic behaviour of an expensive hardware jammer for a customer.
I have used this for some time with good success to simulate and reproduce latency and slow drain fabric issues and
for testing and validating error handling behaviour in the Emulex, Qlogic and other F/C drivers.

Works by checking jammer_flag==1 and host # and discards SCSI command, controlled using echo to sys parameter.

I decided to share the patch, in the hope it may be useful for others but I do understand this is a special use case.
If this is useful and Nab wants to include it I will create a proper documentation patch as well.

filename:       /lib/modules/3.17.7-200.jammer.fc20.x86_64/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko
license:        GPL
description:    TCM QLA2XXX series NPIV enabled fabric driver
depends:        target_core_mod,qla2xxx,scsi_transport_fc
intree:         Y
vermagic:       3.17.7-200.jammer.fc20.x86_64 SMP mod_unload 
parm:           jammer_flag:Set to 1: Enable jammer (int)
parm:           host_flag:host number to match on (int)


Enable host 6 to be jammed
echo 6 > /sys/module/tcm_qla2xxx/parameters/host_flag

Usage example script:

#!/bin/bash
host=`cat /sys/module/tcm_qla2xxx/parameters/host_flag`
sleep_time=120  ### Time to jam for
echo "We start to discard commands on SCSI host $host"
logger "Jammer started"
echo 1 >  /sys/module/tcm_qla2xxx/parameters/jammer_flag
sleep $sleep_time
echo 0 >  /sys/module/tcm_qla2xxx/parameters/jammer_flag
echo "We stopped the jammer"
logger "Jammer stopped"

This Patch diff against 3.19.1

Tested by: Laurence Oberman <loberman@redhat.com>
Signed-off-by: Laurence Oberman <loberman@redhat.com>

diff -Nurp a/drivers/scsi/qla2xxx/tcm_qla2xxx.c b/drivers/scsi/qla2xxx/tcm_qla2xxx.c
--- a/drivers/scsi/qla2xxx/tcm_qla2xxx.c	2015-03-07 18:35:15.246737589 -0500
+++ b/drivers/scsi/qla2xxx/tcm_qla2xxx.c	2015-03-07 18:35:40.168599630 -0500
@@ -50,6 +50,14 @@
 #include "qla_target.h"
 #include "tcm_qla2xxx.h"
 
+int message_flag=0;
+int jammer_flag = 0;
+module_param(jammer_flag, int,0644);
+MODULE_PARM_DESC(jammer_flag, "If set to 1: Enable jammer");
+int host_flag=0;
+module_param(host_flag, int,0644);
+MODULE_PARM_DESC(host_flag, "host number to match on");
+
 static struct workqueue_struct *tcm_qla2xxx_free_wq;
 static struct workqueue_struct *tcm_qla2xxx_cmd_wq;
 
@@ -570,6 +578,22 @@ static int tcm_qla2xxx_handle_cmd(scsi_q
 		pr_err("Unable to locate active struct se_session\n");
 		return -EINVAL;
 	}
+	
+ 	// Control messaging here
+	message_flag += jammer_flag;
+	if(message_flag == 1)
+		printk("tcm_qla2xx:SCSI Jammer enabled on host %d\n",host_flag);
+	if((jammer_flag == 0) && (message_flag >=0)) {
+		printk("tcm_qla2xx:SCSI Jammer stopped, %d SCSI commands discarded for host %d\n",message_flag,host_flag);
+		message_flag=-1;
+	}
+		
+	if ((vha->host_no == host_flag) && (jammer_flag == 1))
+	{
+		// return, and don't run target_submit_cmd, effectively discarding command
+		return 0;
+	}
+
 
 	return target_submit_cmd(se_cmd, se_sess, cdb, &cmd->sense_buffer[0],
 				cmd->unpacked_lun, data_length, fcp_task_attr,
@@ -2165,6 +2189,7 @@ static void tcm_qla2xxx_deregister_confi
 static int __init tcm_qla2xxx_init(void)
 {
 	int ret;
+	jammer_flag = 0;
 
 	ret = tcm_qla2xxx_register_configfs();
 	if (ret < 0)



^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH ] tcm_qla2xxx  Add SCSI command jammer/discard capabilty to the tcm_qla2xxx module
  2015-03-08  3:26 ` [PATCH ] tcm_qla2xxx Add SCSI command jammer/discard capabilty to the tcm_qla2xxx module Laurence Oberman
@ 2015-03-08  8:10   ` Bart Van Assche
  2015-03-08 15:38     ` [PATCH ] tcm_qla2xxx Add SCSI command jammer/discard capabilty to the tcm_qla2xxx module - revision2 Laurence Oberman
  2015-03-12 22:07   ` [PATCH ] tcm_qla2xxx Add SCSI command jammer/discard capabilty to the tcm_qla2xxx module Quinn Tran
  1 sibling, 1 reply; 9+ messages in thread
From: Bart Van Assche @ 2015-03-08  8:10 UTC (permalink / raw)
  To: Laurence Oberman, Andy Grover, linux-scsi, nab; +Cc: Laurence Oberman


On 03/08/2015 04:26 AM, Laurence Oberman wrote:
> Hello
>
> I use target LIO for all my storage array test targets and customer problem resolution here at Red Hat.
> This patch resulted from a requirement to mimic behaviour of an expensive hardware jammer for a customer.
> I have used this for some time with good success to simulate and reproduce latency and slow drain fabric issues and
> for testing and validating error handling behaviour in the Emulex, Qlogic and other F/C drivers.
>
> Works by checking jammer_flag==1 and host # and discards SCSI command, controlled using echo to sys parameter.
>
> I decided to share the patch, in the hope it may be useful for others but I do understand this is a special use case.
> If this is useful and Nab wants to include it I will create a proper documentation patch as well.
>
> filename:       /lib/modules/3.17.7-200.jammer.fc20.x86_64/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko
> license:        GPL
> description:    TCM QLA2XXX series NPIV enabled fabric driver
> depends:        target_core_mod,qla2xxx,scsi_transport_fc
> intree:         Y
> vermagic:       3.17.7-200.jammer.fc20.x86_64 SMP mod_unload
> parm:           jammer_flag:Set to 1: Enable jammer (int)
> parm:           host_flag:host number to match on (int)
>
>
> Enable host 6 to be jammed
> echo 6 > /sys/module/tcm_qla2xxx/parameters/host_flag
>
> Usage example script:
>
> #!/bin/bash
> host=`cat /sys/module/tcm_qla2xxx/parameters/host_flag`
> sleep_time=120  ### Time to jam for
> echo "We start to discard commands on SCSI host $host"
> logger "Jammer started"
> echo 1 >  /sys/module/tcm_qla2xxx/parameters/jammer_flag
> sleep $sleep_time
> echo 0 >  /sys/module/tcm_qla2xxx/parameters/jammer_flag
> echo "We stopped the jammer"
> logger "Jammer stopped"
>
> This Patch diff against 3.19.1
>
> Tested by: Laurence Oberman <loberman@redhat.com>
> Signed-off-by: Laurence Oberman <loberman@redhat.com>
>
> diff -Nurp a/drivers/scsi/qla2xxx/tcm_qla2xxx.c b/drivers/scsi/qla2xxx/tcm_qla2xxx.c
> --- a/drivers/scsi/qla2xxx/tcm_qla2xxx.c	2015-03-07 18:35:15.246737589 -0500
> +++ b/drivers/scsi/qla2xxx/tcm_qla2xxx.c	2015-03-07 18:35:40.168599630 -0500
> @@ -50,6 +50,14 @@
>   #include "qla_target.h"
>   #include "tcm_qla2xxx.h"
>
> +int message_flag=0;
> +int jammer_flag = 0;
> +module_param(jammer_flag, int,0644);
> +MODULE_PARM_DESC(jammer_flag, "If set to 1: Enable jammer");
> +int host_flag=0;
> +module_param(host_flag, int,0644);
> +MODULE_PARM_DESC(host_flag, "host number to match on");
> +
>   static struct workqueue_struct *tcm_qla2xxx_free_wq;
>   static struct workqueue_struct *tcm_qla2xxx_cmd_wq;
>
> @@ -570,6 +578,22 @@ static int tcm_qla2xxx_handle_cmd(scsi_q
>   		pr_err("Unable to locate active struct se_session\n");
>   		return -EINVAL;
>   	}
> +	
> + 	// Control messaging here
> +	message_flag += jammer_flag;
> +	if(message_flag == 1)
> +		printk("tcm_qla2xx:SCSI Jammer enabled on host %d\n",host_flag);
> +	if((jammer_flag == 0) && (message_flag >=0)) {
> +		printk("tcm_qla2xx:SCSI Jammer stopped, %d SCSI commands discarded for host %d\n",message_flag,host_flag);
> +		message_flag=-1;
> +	}
> +		
> +	if ((vha->host_no == host_flag) && (jammer_flag == 1))
> +	{
> +		// return, and don't run target_submit_cmd, effectively discarding command
> +		return 0;
> +	}
> +
>
>   	return target_submit_cmd(se_cmd, se_sess, cdb, &cmd->sense_buffer[0],
>   				cmd->unpacked_lun, data_length, fcp_task_attr,
> @@ -2165,6 +2189,7 @@ static void tcm_qla2xxx_deregister_confi
>   static int __init tcm_qla2xxx_init(void)
>   {
>   	int ret;
> +	jammer_flag = 0;
>
>   	ret = tcm_qla2xxx_register_configfs();
>   	if (ret < 0)

The above code is added in the hot path so it's worth to optimize that 
code. Had you already considered to combine the jammer_flag and 
host_flag kernel module parameters into a single parameter, e.g. 
"jam_host" ? The value "-1" could be used to represent "jamming 
disabled" and values >= 0 to represent a host number.

Additionally, please add unlikely() around the test that verifies 
whether jamming is enabled. Please also verify your patches with 
checkpatch before submission. The above patch does not follow the Linux 
kernel coding style completely.

Bart.

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH ] tcm_qla2xxx  Add SCSI command jammer/discard capabilty to the tcm_qla2xxx module - revision2
  2015-03-08  8:10   ` Bart Van Assche
@ 2015-03-08 15:38     ` Laurence Oberman
  2015-03-12 13:13       ` Bart Van Assche
  0 siblings, 1 reply; 9+ messages in thread
From: Laurence Oberman @ 2015-03-08 15:38 UTC (permalink / raw)
  To: Bart Van Assche; +Cc: Andy Grover, linux-scsi, nab, Laurence Oberman

Hello Bart,
Thanks

Here is revision2

I added unlikely and removed messaging control as it not necessary and adds overhead.

I use target LIO for all my storage array test targets and customer problem resolution here at Red Hat.
This patch resulted from a requirement to mimic behaviour of an expensive hardware jammer for a customer.
I have used this for some time with good success to simulate and reproduce latency and slow drain fabric issues and
for testing and validating error handling behaviour in the Emulex, Qlogic and other F/C drivers.

Works by checking new parameter jam_host if its >= 0 and matches vha->host_no , jamming is enabled when jam_host >=0
If parameter set to -1 (default) no jamming is enabled. 
I decided to share the patch, in the hope it may be useful for others but I do understand this is a special use case.


This Patch diff against 3.19.1

$ linux-3.19.1/scripts/checkpatch.pl latest-upstream-jammer-path 
total: 0 errors, 0 warnings, 60 lines checked

latest-upstream-jammer-path has no obvious style problems and is ready for submission.

Tested by: Laurence Oberman <loberman@redhat.com>
Signed-off-by: Laurence Oberman <loberman@redhat.com>

diff -Nurp a/Documentation/scsi/tcm_qla2xxx.txt b/Documentation/scsi/tcm_qla2xxx.txt
--- a/Documentation/scsi/tcm_qla2xxx.txt	1969-12-31 19:00:00.000000000 -0500
+++ b/Documentation/scsi/tcm_qla2xxx.txt	2015-03-08 11:32:42.262181821 -0400
@@ -0,0 +1,30 @@
+tcm_qla2xxx jammer parameter usage
+----------------------------------
+There is now a new module parameter added to the tcm_qla2xx module
+parm:           jam_host:Host to jam >=0 Enable jammer (int)
+
+Use this parameter to control the discarding of SCSI commands to a selected host.
+This may be useful for testing error handling and simulating slow drain and other
+fabric issues.
+
+Any value >=0 that matches a fc_host # will discard the commands for that host.
+Reset back to -1 to stop the jamming.
+
+Enable host 6 to be jammed
+echo 6 > /sys/module/tcm_qla2xxx/parameters/jam_host
+
+Disable jamming on host 6
+echo -1 > /sys/module/tcm_qla2xxx/parameters/jam_host
+
+Usage example script:
+
+#!/bin/bash
+sleep_time=120  ### Time to jam for
+echo 6 >  /sys/module/tcm_qla2xxx/parameters/jam_host
+host=`cat /sys/module/tcm_qla2xxx/parameters/jam_host`
+echo "We start to discard commands on SCSI host $host"
+logger "Jammer started"
+sleep $sleep_time
+echo -1 >  /sys/module/tcm_qla2xxx/parameters/jam_host
+echo "We stopped the jammer"
+logger "Jammer stopped"
diff -Nurp a/drivers/scsi/qla2xxx/tcm_qla2xxx.c b/drivers/scsi/qla2xxx/tcm_qla2xxx.c
--- a/drivers/scsi/qla2xxx/tcm_qla2xxx.c	2015-03-08 10:13:31.798400426 -0400
+++ b/drivers/scsi/qla2xxx/tcm_qla2xxx.c	2015-03-08 11:00:53.002419568 -0400
@@ -50,6 +50,10 @@
 #include "qla_target.h"
 #include "tcm_qla2xxx.h"
 
+int jam_host = -1;
+module_param(jam_host, int, 0644);
+MODULE_PARM_DESC(jam_host, "Host to jam >=0 Enable jammer");
+
 static struct workqueue_struct *tcm_qla2xxx_free_wq;
 static struct workqueue_struct *tcm_qla2xxx_cmd_wq;
 
@@ -571,6 +575,13 @@ static int tcm_qla2xxx_handle_cmd(scsi_q
 		return -EINVAL;
 	}
 
+	if (unlikely(vha->host_no == jam_host)) {
+		/*
+		return, and dont run target_submit_cmd, discarding command
+		*/
+		return 0;
+	}
+
 	return target_submit_cmd(se_cmd, se_sess, cdb, &cmd->sense_buffer[0],
 				cmd->unpacked_lun, data_length, fcp_task_attr,
 				data_dir, flags);
@@ -2165,6 +2176,7 @@ static void tcm_qla2xxx_deregister_confi
 static int __init tcm_qla2xxx_init(void)
 {
 	int ret;
+	jam_host = -1;
 
 	ret = tcm_qla2xxx_register_configfs();
 	if (ret < 0)





Thanks you for the consideration

Laurence Oberman
Red Hat Global Support Service
SEG Team

----- Original Message -----
From: "Bart Van Assche" <bart.vanassche@sandisk.com>
To: "Laurence Oberman" <loberman@redhat.com>, "Andy Grover" <agrover@redhat.com>, linux-scsi@vger.kernel.org, nab@daterainc.com
Cc: "Laurence Oberman" <oberman.l@gmail.com>
Sent: Sunday, March 8, 2015 4:10:34 AM
Subject: Re: [PATCH ] tcm_qla2xxx  Add SCSI command jammer/discard capabilty to the tcm_qla2xxx module


On 03/08/2015 04:26 AM, Laurence Oberman wrote:
> Hello
>
> I use target LIO for all my storage array test targets and customer problem resolution here at Red Hat.
> This patch resulted from a requirement to mimic behaviour of an expensive hardware jammer for a customer.
> I have used this for some time with good success to simulate and reproduce latency and slow drain fabric issues and
> for testing and validating error handling behaviour in the Emulex, Qlogic and other F/C drivers.
>
> Works by checking jammer_flag==1 and host # and discards SCSI command, controlled using echo to sys parameter.
>
> I decided to share the patch, in the hope it may be useful for others but I do understand this is a special use case.
> If this is useful and Nab wants to include it I will create a proper documentation patch as well.
>
> filename:       /lib/modules/3.17.7-200.jammer.fc20.x86_64/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko
> license:        GPL
> description:    TCM QLA2XXX series NPIV enabled fabric driver
> depends:        target_core_mod,qla2xxx,scsi_transport_fc
> intree:         Y
> vermagic:       3.17.7-200.jammer.fc20.x86_64 SMP mod_unload
> parm:           jammer_flag:Set to 1: Enable jammer (int)
> parm:           host_flag:host number to match on (int)
>
>
> Enable host 6 to be jammed
> echo 6 > /sys/module/tcm_qla2xxx/parameters/host_flag
>
> Usage example script:
>
> #!/bin/bash
> host=`cat /sys/module/tcm_qla2xxx/parameters/host_flag`
> sleep_time=120  ### Time to jam for
> echo "We start to discard commands on SCSI host $host"
> logger "Jammer started"
> echo 1 >  /sys/module/tcm_qla2xxx/parameters/jammer_flag
> sleep $sleep_time
> echo 0 >  /sys/module/tcm_qla2xxx/parameters/jammer_flag
> echo "We stopped the jammer"
> logger "Jammer stopped"
>
> This Patch diff against 3.19.1
>
> Tested by: Laurence Oberman <loberman@redhat.com>
> Signed-off-by: Laurence Oberman <loberman@redhat.com>
>
> diff -Nurp a/drivers/scsi/qla2xxx/tcm_qla2xxx.c b/drivers/scsi/qla2xxx/tcm_qla2xxx.c
> --- a/drivers/scsi/qla2xxx/tcm_qla2xxx.c	2015-03-07 18:35:15.246737589 -0500
> +++ b/drivers/scsi/qla2xxx/tcm_qla2xxx.c	2015-03-07 18:35:40.168599630 -0500
> @@ -50,6 +50,14 @@
>   #include "qla_target.h"
>   #include "tcm_qla2xxx.h"
>
> +int message_flag=0;
> +int jammer_flag = 0;
> +module_param(jammer_flag, int,0644);
> +MODULE_PARM_DESC(jammer_flag, "If set to 1: Enable jammer");
> +int host_flag=0;
> +module_param(host_flag, int,0644);
> +MODULE_PARM_DESC(host_flag, "host number to match on");
> +
>   static struct workqueue_struct *tcm_qla2xxx_free_wq;
>   static struct workqueue_struct *tcm_qla2xxx_cmd_wq;
>
> @@ -570,6 +578,22 @@ static int tcm_qla2xxx_handle_cmd(scsi_q
>   		pr_err("Unable to locate active struct se_session\n");
>   		return -EINVAL;
>   	}
> +	
> + 	// Control messaging here
> +	message_flag += jammer_flag;
> +	if(message_flag == 1)
> +		printk("tcm_qla2xx:SCSI Jammer enabled on host %d\n",host_flag);
> +	if((jammer_flag == 0) && (message_flag >=0)) {
> +		printk("tcm_qla2xx:SCSI Jammer stopped, %d SCSI commands discarded for host %d\n",message_flag,host_flag);
> +		message_flag=-1;
> +	}
> +		
> +	if ((vha->host_no == host_flag) && (jammer_flag == 1))
> +	{
> +		// return, and don't run target_submit_cmd, effectively discarding command
> +		return 0;
> +	}
> +
>
>   	return target_submit_cmd(se_cmd, se_sess, cdb, &cmd->sense_buffer[0],
>   				cmd->unpacked_lun, data_length, fcp_task_attr,
> @@ -2165,6 +2189,7 @@ static void tcm_qla2xxx_deregister_confi
>   static int __init tcm_qla2xxx_init(void)
>   {
>   	int ret;
> +	jammer_flag = 0;
>
>   	ret = tcm_qla2xxx_register_configfs();
>   	if (ret < 0)

The above code is added in the hot path so it's worth to optimize that 
code. Had you already considered to combine the jammer_flag and 
host_flag kernel module parameters into a single parameter, e.g. 
"jam_host" ? The value "-1" could be used to represent "jamming 
disabled" and values >= 0 to represent a host number.

Additionally, please add unlikely() around the test that verifies 
whether jamming is enabled. Please also verify your patches with 
checkpatch before submission. The above patch does not follow the Linux 
kernel coding style completely.

Bart.

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH ] tcm_qla2xxx  Add SCSI command jammer/discard capabilty to the tcm_qla2xxx module - revision2
  2015-03-08 15:38     ` [PATCH ] tcm_qla2xxx Add SCSI command jammer/discard capabilty to the tcm_qla2xxx module - revision2 Laurence Oberman
@ 2015-03-12 13:13       ` Bart Van Assche
  2015-03-13  2:13         ` [PATCH ] qla2xxx Add SCSI command jammer/discard capabilty to the qla2xxx target module - revision3 Laurence Oberman
  0 siblings, 1 reply; 9+ messages in thread
From: Bart Van Assche @ 2015-03-12 13:13 UTC (permalink / raw)
  To: Laurence Oberman; +Cc: Andy Grover, linux-scsi, nab, Laurence Oberman

On 03/08/2015 11:38 AM, Laurence Oberman wrote:
> Here is revision2
>
> I added unlikely and removed messaging control as it not necessary and adds overhead.
>
> I use target LIO for all my storage array test targets and customer problem resolution here at Red Hat.
> This patch resulted from a requirement to mimic behaviour of an expensive hardware jammer for a customer.
> I have used this for some time with good success to simulate and reproduce latency and slow drain fabric issues and
> for testing and validating error handling behaviour in the Emulex, Qlogic and other F/C drivers.
>
> Works by checking new parameter jam_host if its >= 0 and matches vha->host_no , jamming is enabled when jam_host >=0
> If parameter set to -1 (default) no jamming is enabled.
> I decided to share the patch, in the hope it may be useful for others but I do understand this is a special use case.

Hello Laurence,

Thanks for reworking this patch quickly. This patch looks fine to me. 
The only remaining concern I have is that I'm wondering what the best 
place would be to add this functionality - the qla2xxx driver or the LIO 
core ?

Bart.

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH ] tcm_qla2xxx  Add SCSI command jammer/discard capabilty to the tcm_qla2xxx module
  2015-03-08  3:26 ` [PATCH ] tcm_qla2xxx Add SCSI command jammer/discard capabilty to the tcm_qla2xxx module Laurence Oberman
  2015-03-08  8:10   ` Bart Van Assche
@ 2015-03-12 22:07   ` Quinn Tran
  2015-03-12 23:55     ` Laurence Oberman
  1 sibling, 1 reply; 9+ messages in thread
From: Quinn Tran @ 2015-03-12 22:07 UTC (permalink / raw)
  To: Laurence Oberman, Andy Grover, linux-scsi, nab; +Cc: Laurence Oberman

[-- Attachment #1: Type: text/plain, Size: 4690 bytes --]

This idea definitely help flush out additional interaction issues between
fabric drivers and TCM.

However, the current spot where the error injection is placed will cause
memory leak.  The error injection tries to drop the command before
submission to TCM.  TCM & QLA drivers will loose track of this command.
The test will be short live if enough memory have been leaked.  May be the
command should be drop before mem allocation.

In addition, it would nice if the other spots can be included such as:
queue_status(), queue_data_in, aborted_task(), queue_tm_rsp() &
target_submit_tmr().

If the intend is to test all adapters, then the error injection need to be
move higher up into TCM driver.


Regards,
Quinn Tran




On 3/7/15, 8:26 PM, "Laurence Oberman" <loberman@redhat.com> wrote:

>Hello
>
>I use target LIO for all my storage array test targets and customer
>problem resolution here at Red Hat.
>This patch resulted from a requirement to mimic behaviour of an expensive
>hardware jammer for a customer.
>I have used this for some time with good success to simulate and
>reproduce latency and slow drain fabric issues and
>for testing and validating error handling behaviour in the Emulex, Qlogic
>and other F/C drivers.
>
>Works by checking jammer_flag==1 and host # and discards SCSI command,
>controlled using echo to sys parameter.
>
>I decided to share the patch, in the hope it may be useful for others but
>I do understand this is a special use case.
>If this is useful and Nab wants to include it I will create a proper
>documentation patch as well.
>
>filename:       
>/lib/modules/3.17.7-200.jammer.fc20.x86_64/kernel/drivers/scsi/qla2xxx/tcm
>_qla2xxx.ko
>license:        GPL
>description:    TCM QLA2XXX series NPIV enabled fabric driver
>depends:        target_core_mod,qla2xxx,scsi_transport_fc
>intree:         Y
>vermagic:       3.17.7-200.jammer.fc20.x86_64 SMP mod_unload
>parm:           jammer_flag:Set to 1: Enable jammer (int)
>parm:           host_flag:host number to match on (int)
>
>
>Enable host 6 to be jammed
>echo 6 > /sys/module/tcm_qla2xxx/parameters/host_flag
>
>Usage example script:
>
>#!/bin/bash
>host=`cat /sys/module/tcm_qla2xxx/parameters/host_flag`
>sleep_time=120  ### Time to jam for
>echo "We start to discard commands on SCSI host $host"
>logger "Jammer started"
>echo 1 >  /sys/module/tcm_qla2xxx/parameters/jammer_flag
>sleep $sleep_time
>echo 0 >  /sys/module/tcm_qla2xxx/parameters/jammer_flag
>echo "We stopped the jammer"
>logger "Jammer stopped"
>
>This Patch diff against 3.19.1
>
>Tested by: Laurence Oberman <loberman@redhat.com>
>Signed-off-by: Laurence Oberman <loberman@redhat.com>
>
>diff -Nurp a/drivers/scsi/qla2xxx/tcm_qla2xxx.c
>b/drivers/scsi/qla2xxx/tcm_qla2xxx.c
>--- a/drivers/scsi/qla2xxx/tcm_qla2xxx.c	2015-03-07 18:35:15.246737589
>-0500
>+++ b/drivers/scsi/qla2xxx/tcm_qla2xxx.c	2015-03-07 18:35:40.168599630
>-0500
>@@ -50,6 +50,14 @@
> #include "qla_target.h"
> #include "tcm_qla2xxx.h"
> 
>+int message_flag=0;
>+int jammer_flag = 0;
>+module_param(jammer_flag, int,0644);
>+MODULE_PARM_DESC(jammer_flag, "If set to 1: Enable jammer");
>+int host_flag=0;
>+module_param(host_flag, int,0644);
>+MODULE_PARM_DESC(host_flag, "host number to match on");
>+
> static struct workqueue_struct *tcm_qla2xxx_free_wq;
> static struct workqueue_struct *tcm_qla2xxx_cmd_wq;
> 
>@@ -570,6 +578,22 @@ static int tcm_qla2xxx_handle_cmd(scsi_q
> 		pr_err("Unable to locate active struct se_session\n");
> 		return -EINVAL;
> 	}
>+	
>+ 	// Control messaging here
>+	message_flag += jammer_flag;
>+	if(message_flag == 1)
>+		printk("tcm_qla2xx:SCSI Jammer enabled on host %d\n",host_flag);
>+	if((jammer_flag == 0) && (message_flag >=0)) {
>+		printk("tcm_qla2xx:SCSI Jammer stopped, %d SCSI commands discarded for
>host %d\n",message_flag,host_flag);
>+		message_flag=-1;
>+	}
>+		
>+	if ((vha->host_no == host_flag) && (jammer_flag == 1))
>+	{
>+		// return, and don't run target_submit_cmd, effectively discarding
>command
>+		return 0;
>+	}
>+
> 
> 	return target_submit_cmd(se_cmd, se_sess, cdb, &cmd->sense_buffer[0],
> 				cmd->unpacked_lun, data_length, fcp_task_attr,
>@@ -2165,6 +2189,7 @@ static void tcm_qla2xxx_deregister_confi
> static int __init tcm_qla2xxx_init(void)
> {
> 	int ret;
>+	jammer_flag = 0;
> 
> 	ret = tcm_qla2xxx_register_configfs();
> 	if (ret < 0)
>
>
>--
>To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html


[-- Attachment #2: winmail.dat --]
[-- Type: application/ms-tnef, Size: 6622 bytes --]

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH ] tcm_qla2xxx  Add SCSI command jammer/discard capabilty to the tcm_qla2xxx module
  2015-03-12 22:07   ` [PATCH ] tcm_qla2xxx Add SCSI command jammer/discard capabilty to the tcm_qla2xxx module Quinn Tran
@ 2015-03-12 23:55     ` Laurence Oberman
  0 siblings, 0 replies; 9+ messages in thread
From: Laurence Oberman @ 2015-03-12 23:55 UTC (permalink / raw)
  To: Quinn Tran
  Cc: Andy Grover, linux-scsi, nab, Laurence Oberman, Bart Van Assche

Hello Quinn Tran

Thank you for the feedback. There is a revision2 of this patch I sent as a follow on to Bart that is much cleaner but its still exposed to the memory leaks.
The newer version has a single jam_host parameter as suggested by Bart and the messaging removed. Have a look for it.
Bart also suggested moving the discard to a higher layer in his most recent response to allow other transports to benefit as well.

I have used this a lot but and its been extremely useful, but never used it for extended periods and specifically to test servers connected via F/C to to the LIO host.
I was concerned that we had a dangling allocation after discard but never saw the leak show up significantly in my testing. 
Mostly because my test servers are in error recovery and waiting on timeouts.
Where I placed the discard seemed to be the safest pace for my particular use case. 
I did use other options like zeroing the cdb and passing the command on to avoid the dangling allocation, to force lots of underruns on the host during testing.

Let me revisit my most recent version and take care of the memory leak exposure and look into your other suggestions.
I will reply in that latest thread with a new version.

Many Thanks for the consideration

Laurence

Laurence Oberman
Red Hat Global Support Service
SEG Team

----- Original Message -----
From: "Quinn Tran" <quinn.tran@qlogic.com>
To: "Laurence Oberman" <loberman@redhat.com>, "Andy Grover" <agrover@redhat.com>, "linux-scsi" <linux-scsi@vger.kernel.org>, nab@daterainc.com
Cc: "Laurence Oberman" <oberman.l@gmail.com>
Sent: Thursday, March 12, 2015 6:07:08 PM
Subject: Re: [PATCH ] tcm_qla2xxx  Add SCSI command jammer/discard capabilty to the tcm_qla2xxx module

This idea definitely help flush out additional interaction issues between
fabric drivers and TCM.

However, the current spot where the error injection is placed will cause
memory leak.  The error injection tries to drop the command before
submission to TCM.  TCM & QLA drivers will loose track of this command.
The test will be short live if enough memory have been leaked.  May be the
command should be drop before mem allocation.

In addition, it would nice if the other spots can be included such as:
queue_status(), queue_data_in, aborted_task(), queue_tm_rsp() &
target_submit_tmr().

If the intend is to test all adapters, then the error injection need to be
move higher up into TCM driver.


Regards,
Quinn Tran




On 3/7/15, 8:26 PM, "Laurence Oberman" <loberman@redhat.com> wrote:

>Hello
>
>I use target LIO for all my storage array test targets and customer
>problem resolution here at Red Hat.
>This patch resulted from a requirement to mimic behaviour of an expensive
>hardware jammer for a customer.
>I have used this for some time with good success to simulate and
>reproduce latency and slow drain fabric issues and
>for testing and validating error handling behaviour in the Emulex, Qlogic
>and other F/C drivers.
>
>Works by checking jammer_flag==1 and host # and discards SCSI command,
>controlled using echo to sys parameter.
>
>I decided to share the patch, in the hope it may be useful for others but
>I do understand this is a special use case.
>If this is useful and Nab wants to include it I will create a proper
>documentation patch as well.
>
>filename:       
>/lib/modules/3.17.7-200.jammer.fc20.x86_64/kernel/drivers/scsi/qla2xxx/tcm
>_qla2xxx.ko
>license:        GPL
>description:    TCM QLA2XXX series NPIV enabled fabric driver
>depends:        target_core_mod,qla2xxx,scsi_transport_fc
>intree:         Y
>vermagic:       3.17.7-200.jammer.fc20.x86_64 SMP mod_unload
>parm:           jammer_flag:Set to 1: Enable jammer (int)
>parm:           host_flag:host number to match on (int)
>
>
>Enable host 6 to be jammed
>echo 6 > /sys/module/tcm_qla2xxx/parameters/host_flag
>
>Usage example script:
>
>#!/bin/bash
>host=`cat /sys/module/tcm_qla2xxx/parameters/host_flag`
>sleep_time=120  ### Time to jam for
>echo "We start to discard commands on SCSI host $host"
>logger "Jammer started"
>echo 1 >  /sys/module/tcm_qla2xxx/parameters/jammer_flag
>sleep $sleep_time
>echo 0 >  /sys/module/tcm_qla2xxx/parameters/jammer_flag
>echo "We stopped the jammer"
>logger "Jammer stopped"
>
>This Patch diff against 3.19.1
>
>Tested by: Laurence Oberman <loberman@redhat.com>
>Signed-off-by: Laurence Oberman <loberman@redhat.com>
>
>diff -Nurp a/drivers/scsi/qla2xxx/tcm_qla2xxx.c
>b/drivers/scsi/qla2xxx/tcm_qla2xxx.c
>--- a/drivers/scsi/qla2xxx/tcm_qla2xxx.c	2015-03-07 18:35:15.246737589
>-0500
>+++ b/drivers/scsi/qla2xxx/tcm_qla2xxx.c	2015-03-07 18:35:40.168599630
>-0500
>@@ -50,6 +50,14 @@
> #include "qla_target.h"
> #include "tcm_qla2xxx.h"
> 
>+int message_flag=0;
>+int jammer_flag = 0;
>+module_param(jammer_flag, int,0644);
>+MODULE_PARM_DESC(jammer_flag, "If set to 1: Enable jammer");
>+int host_flag=0;
>+module_param(host_flag, int,0644);
>+MODULE_PARM_DESC(host_flag, "host number to match on");
>+
> static struct workqueue_struct *tcm_qla2xxx_free_wq;
> static struct workqueue_struct *tcm_qla2xxx_cmd_wq;
> 
>@@ -570,6 +578,22 @@ static int tcm_qla2xxx_handle_cmd(scsi_q
> 		pr_err("Unable to locate active struct se_session\n");
> 		return -EINVAL;
> 	}
>+	
>+ 	// Control messaging here
>+	message_flag += jammer_flag;
>+	if(message_flag == 1)
>+		printk("tcm_qla2xx:SCSI Jammer enabled on host %d\n",host_flag);
>+	if((jammer_flag == 0) && (message_flag >=0)) {
>+		printk("tcm_qla2xx:SCSI Jammer stopped, %d SCSI commands discarded for
>host %d\n",message_flag,host_flag);
>+		message_flag=-1;
>+	}
>+		
>+	if ((vha->host_no == host_flag) && (jammer_flag == 1))
>+	{
>+		// return, and don't run target_submit_cmd, effectively discarding
>command
>+		return 0;
>+	}
>+
> 
> 	return target_submit_cmd(se_cmd, se_sess, cdb, &cmd->sense_buffer[0],
> 				cmd->unpacked_lun, data_length, fcp_task_attr,
>@@ -2165,6 +2189,7 @@ static void tcm_qla2xxx_deregister_confi
> static int __init tcm_qla2xxx_init(void)
> {
> 	int ret;
>+	jammer_flag = 0;
> 
> 	ret = tcm_qla2xxx_register_configfs();
> 	if (ret < 0)
>
>
>--
>To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html


^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH ] qla2xxx  Add SCSI command jammer/discard capabilty to the qla2xxx target module - revision3
  2015-03-12 13:13       ` Bart Van Assche
@ 2015-03-13  2:13         ` Laurence Oberman
  2015-03-30 14:36           ` Resend: " Laurence Oberman
  0 siblings, 1 reply; 9+ messages in thread
From: Laurence Oberman @ 2015-03-13  2:13 UTC (permalink / raw)
  To: Bart Van Assche, Quinn Tran
  Cc: Andy Grover, linux-scsi, nab, Laurence Oberman

Hello Bart, Quinn Tran

Thanks for the feedback.

Revision3
Moved the discard to the __qlt_do_work code to prevent the memory leak, this cleans up the allocations.
I will look at seeing how best this can be done for the other transports, or in the core but for me the most useful case has been F/C.
I wanted to get feedback so far, and suggest that we should start with this as the initial jamming patch as its the least risky change for now.
I did test this and ran the same set of tests I normally use this error injection for and it looks good.


Patch notes
-----------
I use target LIO for all my storage array test targets and customer problem resolution here at Red Hat.
This patch resulted from a requirement to mimic behaviour of an expensive hardware jammer for a customer.
I have used this for some time with good success to simulate and reproduce latency and slow drain fabric issues and
for testing and validating error handling behaviour in the Emulex, Qlogic and other F/C drivers.
While the jammer is enabled SCSI commands are discarded for the selected host and this allows all the multipath error recovery and other
LLD driver error recovery and timeout code to be debugged and tested.

Works by checking new parameter jam_host if its >= 0 and matches vha->host_no , jamming is enabled when jam_host >=0
If parameter set to -1 (default) no jamming is enabled.
I decided to share the patch, in the hope it may be useful for others but I do understand this is a special use case.

Tested by: Laurence Oberman <loberman@redhat.com>
Signed-off-by: Laurence Oberman <loberman@redhat.com>

diff -Nurp a/Documentation/scsi/qla2xxx.txt b/Documentation/scsi/qla2xxx.txt
--- a/Documentation/scsi/qla2xxx.txt	1969-12-31 19:00:00.000000000 -0500
+++ b/Documentation/scsi/qla2xxx.txt	2015-03-12 21:42:49.828788582 -0400
@@ -0,0 +1,34 @@
+qla2xxx target mode parameters
+------------------------------
+parm:           qlini_mode:Determines when initiator mode will be enabled. Possible values: "exclusive" - initiator mode will be enabled on load, disabled on enabling target mode and then on disabling target mode enabled back; "disabled" - initiator mode will never be enabled; "enabled" (default) - initiator mode will always stay enabled. (charp)
+
+Enables qla2xxx target mode by setting to disabled on module load
+
+There is now a new module parameter added to the qla2xxx module
+parm:           jam_host:Host to jam >=0 Enable jammer (int)
+
+Use this parameter to control the discarding of SCSI commands to a selected host.
+This may be useful for testing error handling and simulating slow drain and other
+fabric issues.
+
+Any value >=0 that matches a fc_host # will discard the commands for that host.
+Reset back to -1 to stop the jamming.
+
+Enable host 6 to be jammed
+echo 6 > /sys/module/qla2xxx/parameters/jam_host
+
+Disable jamming on host 6
+echo -1 > /sys/module/qla2xxx/parameters/jam_host
+
+Usage example script:
+
+#!/bin/bash
+sleep_time=120  ### Time to jam for
+echo 6 >  /sys/module/qla2xxx/parameters/jam_host
+host=`cat /sys/module/qla2xxx/parameters/jam_host`
+echo "We start to discard commands on SCSI host $host"
+logger "Jammer started"
+sleep $sleep_time
+echo -1 >  /sys/module/qla2xxx/parameters/jam_host
+echo "We stopped the jammer"
+logger "Jammer stopped"

diff -Nurp a/drivers/scsi/qla2xxx/qla_target.c b/drivers/scsi/qla2xxx/qla_target.c
--- a/drivers/scsi/qla2xxx/qla_target.c	2015-03-12 21:44:04.691314527 -0400
+++ b/drivers/scsi/qla2xxx/qla_target.c	2015-03-12 21:52:27.551557133 -0400
@@ -59,6 +59,11 @@ MODULE_PARM_DESC(qlini_mode,
 
 int ql2x_ini_mode = QLA2XXX_INI_MODE_EXCLUSIVE;
 
+int jam_host = -1;
+module_param(jam_host, int, 0644);
+MODULE_PARM_DESC(jam_host, "Host to jam >=0 Enable jammer");
+
+
 static int temp_sam_status = SAM_STAT_BUSY;
 
 /*
@@ -3264,6 +3269,11 @@ static void __qlt_do_work(struct qla_tgt
 	cmd->cmd_flags |= BIT_1;
 	if (tgt->tgt_stop)
 		goto out_term;
+	/*
+	* If jam_host >=0, goto out_term discarding command for matching host
+	*/
+	if (unlikely(vha->host_no == jam_host))
+		goto out_term;
 
 	cdb = &atio->u.isp24.fcp_cmnd.cdb[0];
 	cmd->tag = atio->u.isp24.exchange_addr;


Laurence Oberman
Red Hat Global Support Service
SEG Team

----- Original Message -----
From: "Bart Van Assche" <Bart.VanAssche@sandisk.com>
To: "Laurence Oberman" <loberman@redhat.com>
Cc: "Andy Grover" <agrover@redhat.com>, linux-scsi@vger.kernel.org, nab@daterainc.com, "Laurence Oberman" <oberman.l@gmail.com>
Sent: Thursday, March 12, 2015 9:13:28 AM
Subject: Re: [PATCH ] tcm_qla2xxx  Add SCSI command jammer/discard capabilty to the tcm_qla2xxx module - revision2

On 03/08/2015 11:38 AM, Laurence Oberman wrote:
> Here is revision2
>
> I added unlikely and removed messaging control as it not necessary and adds overhead.
>
> I use target LIO for all my storage array test targets and customer problem resolution here at Red Hat.
> This patch resulted from a requirement to mimic behaviour of an expensive hardware jammer for a customer.
> I have used this for some time with good success to simulate and reproduce latency and slow drain fabric issues and
> for testing and validating error handling behaviour in the Emulex, Qlogic and other F/C drivers.
>
> Works by checking new parameter jam_host if its >= 0 and matches vha->host_no , jamming is enabled when jam_host >=0
> If parameter set to -1 (default) no jamming is enabled.
> I decided to share the patch, in the hope it may be useful for others but I do understand this is a special use case.

Hello Laurence,

Thanks for reworking this patch quickly. This patch looks fine to me. 
The only remaining concern I have is that I'm wondering what the best 
place would be to add this functionality - the qla2xxx driver or the LIO 
core ?

Bart.

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Resend: [PATCH ] qla2xxx  Add SCSI command jammer/discard capabilty to the qla2xxx target module - revision3
  2015-03-13  2:13         ` [PATCH ] qla2xxx Add SCSI command jammer/discard capabilty to the qla2xxx target module - revision3 Laurence Oberman
@ 2015-03-30 14:36           ` Laurence Oberman
  2015-05-18 12:50             ` Laurence Oberman
  0 siblings, 1 reply; 9+ messages in thread
From: Laurence Oberman @ 2015-03-30 14:36 UTC (permalink / raw)
  To: Bart Van Assche, Quinn Tran
  Cc: Andy Grover, linux-scsi, nab, Laurence Oberman

Hello Bart, Quinn Tran,

I have been using this jammer facility since I posted the below updated patch with no memory leaks and no issues.
Is there any interest to take this patch in, its certainly been critical for me in some of the error recovery testing I have been doing.

Thanks

Laurence Oberman
Red Hat Global Support Service
SEG Team

----- Original Message -----
From: "Laurence Oberman" <loberman@redhat.com>
To: "Bart Van Assche" <Bart.VanAssche@sandisk.com>, "Quinn Tran" <quinn.tran@qlogic.com>
Cc: "Andy Grover" <agrover@redhat.com>, linux-scsi@vger.kernel.org, nab@daterainc.com, "Laurence Oberman" <oberman.l@gmail.com>
Sent: Thursday, March 12, 2015 10:13:57 PM
Subject: Re: [PATCH ] qla2xxx  Add SCSI command jammer/discard capabilty to the qla2xxx target module - revision3

Hello Bart, Quinn Tran

Thanks for the feedback.

Revision3
Moved the discard to the __qlt_do_work code to prevent the memory leak, this cleans up the allocations.
I will look at seeing how best this can be done for the other transports, or in the core but for me the most useful case has been F/C.
I wanted to get feedback so far, and suggest that we should start with this as the initial jamming patch as its the least risky change for now.
I did test this and ran the same set of tests I normally use this error injection for and it looks good.


Patch notes
-----------
I use target LIO for all my storage array test targets and customer problem resolution here at Red Hat.
This patch resulted from a requirement to mimic behaviour of an expensive hardware jammer for a customer.
I have used this for some time with good success to simulate and reproduce latency and slow drain fabric issues and
for testing and validating error handling behaviour in the Emulex, Qlogic and other F/C drivers.
While the jammer is enabled SCSI commands are discarded for the selected host and this allows all the multipath error recovery and other
LLD driver error recovery and timeout code to be debugged and tested.

Works by checking new parameter jam_host if its >= 0 and matches vha->host_no , jamming is enabled when jam_host >=0
If parameter set to -1 (default) no jamming is enabled.
I decided to share the patch, in the hope it may be useful for others but I do understand this is a special use case.

Tested by: Laurence Oberman <loberman@redhat.com>
Signed-off-by: Laurence Oberman <loberman@redhat.com>

diff -Nurp a/Documentation/scsi/qla2xxx.txt b/Documentation/scsi/qla2xxx.txt
--- a/Documentation/scsi/qla2xxx.txt	1969-12-31 19:00:00.000000000 -0500
+++ b/Documentation/scsi/qla2xxx.txt	2015-03-12 21:42:49.828788582 -0400
@@ -0,0 +1,34 @@
+qla2xxx target mode parameters
+------------------------------
+parm:           qlini_mode:Determines when initiator mode will be enabled. Possible values: "exclusive" - initiator mode will be enabled on load, disabled on enabling target mode and then on disabling target mode enabled back; "disabled" - initiator mode will never be enabled; "enabled" (default) - initiator mode will always stay enabled. (charp)
+
+Enables qla2xxx target mode by setting to disabled on module load
+
+There is now a new module parameter added to the qla2xxx module
+parm:           jam_host:Host to jam >=0 Enable jammer (int)
+
+Use this parameter to control the discarding of SCSI commands to a selected host.
+This may be useful for testing error handling and simulating slow drain and other
+fabric issues.
+
+Any value >=0 that matches a fc_host # will discard the commands for that host.
+Reset back to -1 to stop the jamming.
+
+Enable host 6 to be jammed
+echo 6 > /sys/module/qla2xxx/parameters/jam_host
+
+Disable jamming on host 6
+echo -1 > /sys/module/qla2xxx/parameters/jam_host
+
+Usage example script:
+
+#!/bin/bash
+sleep_time=120  ### Time to jam for
+echo 6 >  /sys/module/qla2xxx/parameters/jam_host
+host=`cat /sys/module/qla2xxx/parameters/jam_host`
+echo "We start to discard commands on SCSI host $host"
+logger "Jammer started"
+sleep $sleep_time
+echo -1 >  /sys/module/qla2xxx/parameters/jam_host
+echo "We stopped the jammer"
+logger "Jammer stopped"

diff -Nurp a/drivers/scsi/qla2xxx/qla_target.c b/drivers/scsi/qla2xxx/qla_target.c
--- a/drivers/scsi/qla2xxx/qla_target.c	2015-03-12 21:44:04.691314527 -0400
+++ b/drivers/scsi/qla2xxx/qla_target.c	2015-03-12 21:52:27.551557133 -0400
@@ -59,6 +59,11 @@ MODULE_PARM_DESC(qlini_mode,
 
 int ql2x_ini_mode = QLA2XXX_INI_MODE_EXCLUSIVE;
 
+int jam_host = -1;
+module_param(jam_host, int, 0644);
+MODULE_PARM_DESC(jam_host, "Host to jam >=0 Enable jammer");
+
+
 static int temp_sam_status = SAM_STAT_BUSY;
 
 /*
@@ -3264,6 +3269,11 @@ static void __qlt_do_work(struct qla_tgt
 	cmd->cmd_flags |= BIT_1;
 	if (tgt->tgt_stop)
 		goto out_term;
+	/*
+	* If jam_host >=0, goto out_term discarding command for matching host
+	*/
+	if (unlikely(vha->host_no == jam_host))
+		goto out_term;
 
 	cdb = &atio->u.isp24.fcp_cmnd.cdb[0];
 	cmd->tag = atio->u.isp24.exchange_addr;


Laurence Oberman
Red Hat Global Support Service
SEG Team

----- Original Message -----
From: "Bart Van Assche" <Bart.VanAssche@sandisk.com>
To: "Laurence Oberman" <loberman@redhat.com>
Cc: "Andy Grover" <agrover@redhat.com>, linux-scsi@vger.kernel.org, nab@daterainc.com, "Laurence Oberman" <oberman.l@gmail.com>
Sent: Thursday, March 12, 2015 9:13:28 AM
Subject: Re: [PATCH ] tcm_qla2xxx  Add SCSI command jammer/discard capabilty to the tcm_qla2xxx module - revision2

On 03/08/2015 11:38 AM, Laurence Oberman wrote:
> Here is revision2
>
> I added unlikely and removed messaging control as it not necessary and adds overhead.
>
> I use target LIO for all my storage array test targets and customer problem resolution here at Red Hat.
> This patch resulted from a requirement to mimic behaviour of an expensive hardware jammer for a customer.
> I have used this for some time with good success to simulate and reproduce latency and slow drain fabric issues and
> for testing and validating error handling behaviour in the Emulex, Qlogic and other F/C drivers.
>
> Works by checking new parameter jam_host if its >= 0 and matches vha->host_no , jamming is enabled when jam_host >=0
> If parameter set to -1 (default) no jamming is enabled.
> I decided to share the patch, in the hope it may be useful for others but I do understand this is a special use case.

Hello Laurence,

Thanks for reworking this patch quickly. This patch looks fine to me. 
The only remaining concern I have is that I'm wondering what the best 
place would be to add this functionality - the qla2xxx driver or the LIO 
core ?

Bart.

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: Resend: [PATCH ] qla2xxx  Add SCSI command jammer/discard capabilty to the qla2xxx target module - revision3
  2015-03-30 14:36           ` Resend: " Laurence Oberman
@ 2015-05-18 12:50             ` Laurence Oberman
  0 siblings, 0 replies; 9+ messages in thread
From: Laurence Oberman @ 2015-05-18 12:50 UTC (permalink / raw)
  To: Bart Van Assche, Quinn Tran; +Cc: linux-scsi, Laurence Oberman

Quinn Tran

Any interest in getting this pulled in.
Revision3 has been in use at customers and in our lab for some time now with no issues.

Revision3
Moved the discard to the __qlt_do_work code to prevent the memory leak, this cleans up the allocations.
I will look at seeing how best this can be done for the other transports, or in the core but for me the most useful case has been F/C.
I wanted to get feedback so far, and suggest that we should start with this as the initial jamming patch as its the least risky change for now.
I did test this and ran the same set of tests I normally use this error injection for and it looks good.


Patch notes
-----------
I use target LIO for all my storage array test targets and customer problem resolution here at Red Hat.
This patch resulted from a requirement to mimic behaviour of an expensive hardware jammer for a customer.
I have used this for some time with good success to simulate and reproduce latency and slow drain fabric issues and
for testing and validating error handling behaviour in the Emulex, Qlogic and other F/C drivers.
While the jammer is enabled SCSI commands are discarded for the selected host and this allows all the multipath error recovery and other
LLD driver error recovery and timeout code to be debugged and tested.

Works by checking new parameter jam_host if its >= 0 and matches vha->host_no , jamming is enabled when jam_host >=0
If parameter set to -1 (default) no jamming is enabled.
I decided to share the patch, in the hope it may be useful for others but I do understand this is a special use case.

Tested by: Laurence Oberman <loberman@redhat.com>
Signed-off-by: Laurence Oberman <loberman@redhat.com>

diff -Nurp a/Documentation/scsi/qla2xxx.txt b/Documentation/scsi/qla2xxx.txt
--- a/Documentation/scsi/qla2xxx.txt	1969-12-31 19:00:00.000000000 -0500
+++ b/Documentation/scsi/qla2xxx.txt	2015-03-12 21:42:49.828788582 -0400
@@ -0,0 +1,34 @@
+qla2xxx target mode parameters
+------------------------------
+parm:           qlini_mode:Determines when initiator mode will be enabled. Possible values: "exclusive" - initiator mode will be enabled on load, disabled on enabling target mode and then on disabling target mode enabled back; "disabled" - initiator mode will never be enabled; "enabled" (default) - initiator mode will always stay enabled. (charp)
+
+Enables qla2xxx target mode by setting to disabled on module load
+
+There is now a new module parameter added to the qla2xxx module
+parm:           jam_host:Host to jam >=0 Enable jammer (int)
+
+Use this parameter to control the discarding of SCSI commands to a selected host.
+This may be useful for testing error handling and simulating slow drain and other
+fabric issues.
+
+Any value >=0 that matches a fc_host # will discard the commands for that host.
+Reset back to -1 to stop the jamming.
+
+Enable host 6 to be jammed
+echo 6 > /sys/module/qla2xxx/parameters/jam_host
+
+Disable jamming on host 6
+echo -1 > /sys/module/qla2xxx/parameters/jam_host
+
+Usage example script:
+
+#!/bin/bash
+sleep_time=120  ### Time to jam for
+echo 6 >  /sys/module/qla2xxx/parameters/jam_host
+host=`cat /sys/module/qla2xxx/parameters/jam_host`
+echo "We start to discard commands on SCSI host $host"
+logger "Jammer started"
+sleep $sleep_time
+echo -1 >  /sys/module/qla2xxx/parameters/jam_host
+echo "We stopped the jammer"
+logger "Jammer stopped"

diff -Nurp a/drivers/scsi/qla2xxx/qla_target.c b/drivers/scsi/qla2xxx/qla_target.c
--- a/drivers/scsi/qla2xxx/qla_target.c	2015-03-12 21:44:04.691314527 -0400
+++ b/drivers/scsi/qla2xxx/qla_target.c	2015-03-12 21:52:27.551557133 -0400
@@ -59,6 +59,11 @@ MODULE_PARM_DESC(qlini_mode,
 
 int ql2x_ini_mode = QLA2XXX_INI_MODE_EXCLUSIVE;
 
+int jam_host = -1;
+module_param(jam_host, int, 0644);
+MODULE_PARM_DESC(jam_host, "Host to jam >=0 Enable jammer");
+
+
 static int temp_sam_status = SAM_STAT_BUSY;
 
 /*
@@ -3264,6 +3269,11 @@ static void __qlt_do_work(struct qla_tgt
 	cmd->cmd_flags |= BIT_1;
 	if (tgt->tgt_stop)
 		goto out_term;
+	/*
+	* If jam_host >=0, goto out_term discarding command for matching host
+	*/
+	if (unlikely(vha->host_no == jam_host))
+		goto out_term;
 
 	cdb = &atio->u.isp24.fcp_cmnd.cdb[0];
 	cmd->tag = atio->u.isp24.exchange_addr;


Laurence Oberman
Red Hat Global Support Service
SEG Team


^ permalink raw reply	[flat|nested] 9+ messages in thread

end of thread, other threads:[~2015-05-18 12:50 UTC | newest]

Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
     [not found] <588344732.13485469.1425769450275.JavaMail.zimbra@redhat.com>
2015-03-08  3:26 ` [PATCH ] tcm_qla2xxx Add SCSI command jammer/discard capabilty to the tcm_qla2xxx module Laurence Oberman
2015-03-08  8:10   ` Bart Van Assche
2015-03-08 15:38     ` [PATCH ] tcm_qla2xxx Add SCSI command jammer/discard capabilty to the tcm_qla2xxx module - revision2 Laurence Oberman
2015-03-12 13:13       ` Bart Van Assche
2015-03-13  2:13         ` [PATCH ] qla2xxx Add SCSI command jammer/discard capabilty to the qla2xxx target module - revision3 Laurence Oberman
2015-03-30 14:36           ` Resend: " Laurence Oberman
2015-05-18 12:50             ` Laurence Oberman
2015-03-12 22:07   ` [PATCH ] tcm_qla2xxx Add SCSI command jammer/discard capabilty to the tcm_qla2xxx module Quinn Tran
2015-03-12 23:55     ` Laurence Oberman

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.