* Cant write to max_sectors_kb on 4.5.0 SRP target
[not found] ` <480311118.27942868.1460063633409.JavaMail.zimbra-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
@ 2016-04-07 21:15 ` Laurence Oberman
[not found] ` <302427900.27942894.1460063713447.JavaMail.zimbra-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
2016-04-08 5:30 ` Nicholas A. Bellinger
0 siblings, 2 replies; 11+ messages in thread
From: Laurence Oberman @ 2016-04-07 21:15 UTC (permalink / raw)
To: linux-scsi, linux-rdma-u79uwXL29TY76Z2rM5mHXA
Hello
I have been testing the SRP initiator code to an LIO array here and part of the testing requires me to set the max_sectors_kb size to get 4k I/O's.
This has been due to me having to debug various sg_map issues.
Linux srptest 4.5.0 #2 SMP Thu Apr 7 16:14:38 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux
This kernel has the scan patch from Hannes, as well as the "[PATCH] IB/mlx5: Expose correct max_sge_rd limit" patch.
However, I also tested with vanilla 4.5.0 as well and its the same issue.
For some reason I cannot change the max_sectors_kb size on 4.5.0 here.
I chatted with Ewan about it as well and he reminded me about Martins changes so wondering if that's playing into this.
Take /dev/sdb as an example
[root@srptest queue]# sg_inq --p 0xb0 /dev/sdb
VPD INQUIRY: Block limits page (SBC)
Maximum compare and write length: 1 blocks
Optimal transfer length granularity: 256 blocks
Maximum transfer length: 256 blocks
Optimal transfer length: 768 blocks
Maximum prefetch, xdread, xdwrite transfer length: 0 blocks
[root@srptest queue]# sg_inq --p 0x83 /dev/sdb
VPD INQUIRY: Device Identification page
Designation descriptor number 1, descriptor length: 20
designator_type: NAA, code_set: Binary
associated with the addressed logical unit
NAA 6, IEEE Company_id: 0x1405
Vendor Specific Identifier: 0x3ec95b43d
Vendor Specific Identifier Extension: 0xaf74871a5f1a0117
[0x60014053ec95b43daf74871a5f1a0117]
Designation descriptor number 2, descriptor length: 57
designator_type: T10 vendor identification, code_set: ASCII
associated with the addressed logical unit
vendor id: LIO-ORG
vendor specific: block-1:3ec95b43-daf7-4871-a5f1-a0117ecc9c87
Designation descriptor number 3, descriptor length: 8
transport: SCSI RDMA Protocol (SRP)
designator_type: Relative target port, code_set: Binary
associated with the target port
Relative target port: 0x2
Designation descriptor number 4, descriptor length: 8
transport: SCSI RDMA Protocol (SRP)
designator_type: Target port group, code_set: Binary
associated with the target port
Target port group: 0x0
Designation descriptor number 5, descriptor length: 8
designator_type: Logical unit group, code_set: Binary
associated with the addressed logical unit
Logical unit group: 0x0
Designation descriptor number 6, descriptor length: 48
transport: SCSI RDMA Protocol (SRP)
designator_type: SCSI name string, code_set: UTF-8
associated with the target port
SCSI name string:
0xfe800000000000007cfe900300726e4f,t,0x0001
Designation descriptor number 7, descriptor length: 40
transport: SCSI RDMA Protocol (SRP)
designator_type: SCSI name string, code_set: UTF-8
associated with the target device that contains addressed lu
SCSI name string:
0xfe800000000000007cfe900300726e4f
[root@srptest queue]# cat max_hw_sectors_kb max_sectors_kb
4096
1280
[root@srptest queue]# echo 4096 > max_sectors_kb
-bash: echo: write error: Invalid argument
The exact same targets served by the same array work fine when I test with the RHEL 7.2 kernel
Linux srptest 3.10.0-327.10.1.el7.bz1313814.x86_64 #1 SMP Fri Mar 11 14:10:52 EST 2016 x86_64 x86_64 x86_64 GNU/Linux
[root@srptest ~]# cd /sys/block/sdb/queue
cat max_hw_sectors_kb max_sectors_kb
4096
512
echo 4096 > max_sectors_kb
cat max_hw_sectors_kb max_sectors_kb
4096
4096
[root@srptest ~]# sg_inq --p 0xb0 /dev/sdb
VPD INQUIRY: Block limits page (SBC)
Maximum compare and write length: 1 blocks
Optimal transfer length granularity: 256 blocks
Maximum transfer length: 256 blocks
Optimal transfer length: 768 blocks
Maximum prefetch, xdread, xdwrite transfer length: 0 blocks
This is an SRP device served by LVM via target LIO on my SRp target server
[root@srptest ~]# sg_inq --p 0x83 /dev/sdb
VPD INQUIRY: Device Identification page
Designation descriptor number 1, descriptor length: 20
designator_type: NAA, code_set: Binary
associated with the addressed logical unit
NAA 6, IEEE Company_id: 0x1405
Vendor Specific Identifier: 0x3ec95b43d
Vendor Specific Identifier Extension: 0xaf74871a5f1a0117
[0x60014053ec95b43daf74871a5f1a0117]
Designation descriptor number 2, descriptor length: 57
designator_type: T10 vendor identification, code_set: ASCII
associated with the addressed logical unit
vendor id: LIO-ORG
vendor specific: block-1:3ec95b43-daf7-4871-a5f1-a0117ecc9c87
Designation descriptor number 3, descriptor length: 8
transport: SCSI RDMA Protocol (SRP)
designator_type: Relative target port, code_set: Binary
associated with the target port
Relative target port: 0x1
Designation descriptor number 4, descriptor length: 8
transport: SCSI RDMA Protocol (SRP)
designator_type: Target port group, code_set: Binary
associated with the target port
Target port group: 0x0
Designation descriptor number 5, descriptor length: 8
designator_type: Logical unit group, code_set: Binary
associated with the addressed logical unit
Logical unit group: 0x0
Designation descriptor number 6, descriptor length: 48
transport: SCSI RDMA Protocol (SRP)
designator_type: SCSI name string, code_set: UTF-8
associated with the target port
SCSI name string:
0xfe800000000000007cfe900300726e4e,t,0x0001
Designation descriptor number 7, descriptor length: 40
transport: SCSI RDMA Protocol (SRP)
designator_type: SCSI name string, code_set: UTF-8
associated with the target device that contains addressed lu
SCSI name string:
0xfe800000000000007cfe900300726e4e
Laurence Oberman
Principal Software Maintenance Engineer
Red Hat Global Support Services
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Cant write to max_sectors_kb on 4.5.0 SRP target
[not found] ` <302427900.27942894.1460063713447.JavaMail.zimbra-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
@ 2016-04-08 2:49 ` Bart Van Assche
2016-04-08 7:58 ` Laurence Oberman
2016-04-08 3:00 ` Martin K. Petersen
1 sibling, 1 reply; 11+ messages in thread
From: Bart Van Assche @ 2016-04-08 2:49 UTC (permalink / raw)
To: Laurence Oberman, linux-scsi, linux-rdma-u79uwXL29TY76Z2rM5mHXA
On 04/07/16 14:16, Laurence Oberman wrote:
> I have been testing the SRP initiator code to an LIO array here and
> part of the testing requires me to set the max_sectors_kb size to
> get 4k I/O's.
.
Hello Laurence,
Have you already tried to set the max_sect parameter in
/etc/srp_daemon.conf (assuming you are using srptools >= 1.0.3 for SRP
login) ? Additionally, writing something like "options ib_srp
cmd_sg_entries=255" into /etc/modprobe.d/ib_srp.conf will increase the
maximum SRP transfer size.
Bart.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Cant write to max_sectors_kb on 4.5.0 SRP target
[not found] ` <302427900.27942894.1460063713447.JavaMail.zimbra-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
2016-04-08 2:49 ` Bart Van Assche
@ 2016-04-08 3:00 ` Martin K. Petersen
2016-04-08 8:31 ` Laurence Oberman
1 sibling, 1 reply; 11+ messages in thread
From: Martin K. Petersen @ 2016-04-08 3:00 UTC (permalink / raw)
To: Laurence Oberman; +Cc: linux-scsi, linux-rdma-u79uwXL29TY76Z2rM5mHXA
>>>>> "Laurence" == Laurence Oberman <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> writes:
Laurence,
The target is reporting inconsistent values here:
> [root@srptest queue]# sg_inq --p 0xb0 /dev/sdb
> VPD INQUIRY: Block limits page (SBC)
> Maximum compare and write length: 1 blocks
> Optimal transfer length granularity: 256 blocks
> Maximum transfer length: 256 blocks
> Optimal transfer length: 768 blocks
OPTIMAL TRANSFER LENGTH GRANULARITY roughly translates to physical block
size or RAID chunk size. It's the smallest I/O unit that does not
require read-modify-write. It would typically be either 1 or 8 blocks
for a drive and maybe 64, 128 or 256 for a RAID5 array. io_min in
queue_limits.
OPTIMAL TRANSFER LENGTH indicates the stripe width and is a multiple of
OPTIMAL TRANSFER LENGTH GRANULARITY. io_opt in queue_limits.
MAXIMUM TRANSFER LENGTH indicates the biggest READ/WRITE command the
device can handle in a single command. In this case 256 blocks so that's
128K. max_dev_sectors in queue_limits.
>From SBC:
"A MAXIMUM TRANSFER LENGTH field set to a non-zero value indicates the
maximum transfer length in logical blocks that the device server accepts
for a single command shown in table 250. If a device server receives one
of these commands with a transfer size greater than this value, then the
device server shall terminate the command with CHECK CONDITION status
[...]"
So those reported values are off.
logical block size <= physical block size <= OTLG <= OTL <= MTL
Or in terms of queue_limits:
lbs <= pbs <= io_min <= io_opt <=
min_not_zero(max_dev_sectors, max_hw_sectors, max_sectors)
--
Martin K. Petersen Oracle Linux Engineering
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Cant write to max_sectors_kb on 4.5.0 SRP target
2016-04-07 21:15 ` Cant write to max_sectors_kb on 4.5.0 SRP target Laurence Oberman
[not found] ` <302427900.27942894.1460063713447.JavaMail.zimbra-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
@ 2016-04-08 5:30 ` Nicholas A. Bellinger
2016-04-08 8:15 ` Laurence Oberman
1 sibling, 1 reply; 11+ messages in thread
From: Nicholas A. Bellinger @ 2016-04-08 5:30 UTC (permalink / raw)
To: Laurence Oberman; +Cc: linux-scsi, linux-rdma, target-devel
Hi Laurence,
On Thu, 2016-04-07 at 17:15 -0400, Laurence Oberman wrote:
> Hello
>
> I have been testing the SRP initiator code to an LIO array here and
> part of the testing requires me to set the max_sectors_kb size to get
> 4k I/O's.
> This has been due to me having to debug various sg_map issues.
>
> Linux srptest 4.5.0 #2 SMP Thu Apr 7 16:14:38 EDT 2016 x86_64 x86_64
> x86_64 GNU/Linux
> This kernel has the scan patch from Hannes, as well as the "[PATCH]
> IB/mlx5: Expose correct max_sge_rd limit" patch.
> However, I also tested with vanilla 4.5.0 as well and its the same
> issue.
>
> For some reason I cannot change the max_sectors_kb size on 4.5.0 here.
>
> I chatted with Ewan about it as well and he reminded me about Martins
> changes so wondering if that's playing into this.
>
> Take /dev/sdb as an example
>
> [root@srptest queue]# sg_inq --p 0xb0 /dev/sdb
> VPD INQUIRY: Block limits page (SBC)
> Maximum compare and write length: 1 blocks
> Optimal transfer length granularity: 256 blocks
> Maximum transfer length: 256 blocks
> Optimal transfer length: 768 blocks
> Maximum prefetch, xdread, xdwrite transfer length: 0 blocks
>
Just curious what target backend this is with..?
Specifically the optimal transfer length granularity and optimal
transfer length may be reported by underlying backend device (eg:
IBLOCK) in spc_emulate_evpd_b0().
What does 'head /sys/kernel/config/target/core/$HBA/$DEV/attrib/*'
of the backend device in question look like..?
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Cant write to max_sectors_kb on 4.5.0 SRP target
2016-04-08 2:49 ` Bart Van Assche
@ 2016-04-08 7:58 ` Laurence Oberman
0 siblings, 0 replies; 11+ messages in thread
From: Laurence Oberman @ 2016-04-08 7:58 UTC (permalink / raw)
To: Bart Van Assche; +Cc: linux-scsi, linux-rdma
Hi Bart
"Have you already tried to set the max_sect parameter in
/etc/srp_daemon.conf (assuming you are using srptools >= 1.0.3 for SRP
login) ? Additionally, writing something like "options ib_srp
cmd_sg_entries=255" into /etc/modprobe.d/ib_srp.conf will increase the
maximum SRP transfer size.
"
Bart, I am using the exact same configuration for the SRP initiator for both kernels.
Simply rebooting into upstream for baseline. All that is changing is the kernel.
The same parameters are applied to the initiator modules in both kernels.
When booting into the RHEL kernel the same applies and I am not changing the target LIO server.
(For Nicholas will follow up with the LIO target details in another email)
Here are my parameters and startup
srptools-1.0.2-1.el7.x86_64
[root@srptest modprobe.d]# cat ib_srp.conf
options ib_srp cmd_sg_entries=128 indirect_sg_entries=1024
[root@srptest modprobe.d]# cat ib_srpt.conf
options ib_srpt srp_max_req_size=8296
Setting up the LUNS I am using
[root@srptest ~]# run_srp_daemon -f /etc/ddn/srp_daemon.conf -R 30 -T 10 -t 7000 -ance -i mlx5_0 -p 1
id_ext=7cfe900300726e4e,ioc_guid=7cfe900300726e4e,dgid=fe800000000000007cfe900300726e4e,pkey=ffff,service_id=7cfe900300726e4e,initiator_ext=4e6e72000390fe7c,queue_size=128,max_cmd_per_lun=128,max_sect=8192
id_ext=7cfe900300726ed2,ioc_guid=7cfe900300726ed2,dgid=fe800000000000007cfe900300726ed2,pkey=ffff,service_id=7cfe900300726ed2,initiator_ext=d26e72000390fe7c,queue_size=128,max_cmd_per_lun=128,max_sect=8192
[root@srptest ~]# cat /etc/ddn/srp_daemon.conf
a queue_size=128,max_cmd_per_lun=128 max_sect=8192
These allow me to set and use the 4MB I/O size on the RHEL kernel.
Example on the RHEL kernel
Linux srptest 3.10.0-327.10.1.el7.bz1313814.x86_64
mpathhh (3600140538661058fbcd4dcca8222f5d5) dm-26 LIO-ORG ,block-2
size=9.0G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=50 status=active
`- 4:0:0:1 sdc 8:32 active ready running
/sys/block/sdc/queue
[root@srptest queue]# echo 4096 > max_sectors_kb
[root@srptest queue]# cat max_sectors_kb
4096
/sys/block/dm-26/queue
[root@srptest queue]# echo 4096 > max_sectors_kb
[root@srptest queue]# cat max_sectors_kb
4096
dd if=/dev/zero of=/dev/mapper/mpathhh bs=4096k oflag=direct count=1000
# DISK STATISTICS (/sec)
# <---------reads---------><---------writes---------><--------averages--------> Pct
#Time Name KBytes Merged IOs Size KBytes Merged IOs Size RWSize QLen Wait SvcTim Util
03:42:45 sdc 0 0 0 0 405504 0 99 4096 4096 1 3 3 39
03:42:45 dm-26 0 0 0 0 405504 300 99 4096 4096 1 4 4 42
03:42:46 sdc 0 0 0 0 815104 0 199 4096 4096 1 4 4 80
03:42:46 dm-26 0 0 0 0 815104 597 199 4096 4096 1 4 4 84
03:42:47 sdc 0 0 0 0 839680 0 205 4096 4096 1 3 3 79
So no issues here, I am able to change the max_sectors_kb to 4096 and then issue 4MB writes to the target from the initiator view.
Now rebooting into the upstream 4.5.0
Linux srptest 4.5.0 #2 SMP Thu Apr 7 16:14:38 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux
[root@srptest ~]# run_srp_daemon -f /etc/ddn/srp_daemon.conf -R 30 -T 10 -t 7000 -ance -i mlx5_0 -p 1
id_ext=7cfe900300726e4e,ioc_guid=7cfe900300726e4e,dgid=fe800000000000007cfe900300726e4e,pkey=ffff,service_id=7cfe900300726e4e,initiator_ext=4e6e72000390fe7c,queue_size=128,max_cmd_per_lun=128,max_sect=8192
id_ext=7cfe900300726ed2,ioc_guid=7cfe900300726ed2,dgid=fe800000000000007cfe900300726ed2,pkey=ffff,service_id=7cfe900300726ed2,initiator_ext=d26e72000390fe7c,queue_size=128,max_cmd_per_lun=128,max_sect=8192
mpathhh (3600140538661058fbcd4dcca8222f5d5) dm-4 LIO-ORG ,block-2
size=9.0G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=50 status=active
`- 6:0:0:1 sdc 8:32 active ready running
[root@srptest queue]# pwd
/sys/block/sdc/queue
[root@srptest queue]# echo 4096 > max_sectors_kb
-bash: echo: write error: Invalid argument
[root@srptest queue]# cat max_sectors_kb
1280
[root@srptest queue]# pwd
/sys/block/dm-4/queue
[root@srptest queue]# echo 4096 > max_sectors_kb
-bash: echo: write error: Invalid argument
[root@srptest queue]# cat max_sectors_kb
1280
So trying 4MB on the initiator I get it pinned to 1MB and cannot write 4MB
[root@srptest ~]# dd if=/dev/zero of=/dev/mapper/mpathhh bs=4096k oflag=direct count=1000
1000+0 records in
1000+0 records out
4194304000 bytes (4.2 GB) copied, 3.72675 s, 1.1 GB/s
# DISK STATISTICS (/sec)
# <---------reads---------><---------writes---------><--------averages--------> Pct
#Time Name KBytes Merged IOs Size KBytes Merged IOs Size RWSize QLen Wait SvcTim Util
03:56:57 sdc 0 0 0 0 1092608 0 1067 1024 1024 3 2 0 74
03:56:57 dm-4 0 0 0 0 1092608 0 1067 1024 1024 3 2 0 79
03:56:58 sdc 0 0 0 0 1070080 0 1045 1024 1024 3 2 0 73
03:56:58 dm-4 0 0 0 0 1070080 0 1045 1024 1024 3 2 0 78
03:56:59 sdc 0 0 0 0 1101824 0 1076 1024 1024 3 2 0 72
03:56:59 dm-4 0 0 0 0 1101824 0 1076 1024 1024 3 2 0 77
Laurence Oberman
Principal Software Maintenance Engineer
Red Hat Global Support Services
Laurence Oberman
Principal Software Maintenance Engineer
Red Hat Global Support Services
----- Original Message -----
From: "Bart Van Assche" <bart.vanassche@sandisk.com>
To: "Laurence Oberman" <loberman@redhat.com>, "linux-scsi" <linux-scsi@vger.kernel.org>, linux-rdma@vger.kernel.org
Sent: Thursday, April 7, 2016 10:49:58 PM
Subject: Re: Cant write to max_sectors_kb on 4.5.0 SRP target
On 04/07/16 14:16, Laurence Oberman wrote:
> I have been testing the SRP initiator code to an LIO array here and
> part of the testing requires me to set the max_sectors_kb size to
> get 4k I/O's.
.
Hello Laurence,
Have you already tried to set the max_sect parameter in
/etc/srp_daemon.conf (assuming you are using srptools >= 1.0.3 for SRP
login) ? Additionally, writing something like "options ib_srp
cmd_sg_entries=255" into /etc/modprobe.d/ib_srp.conf will increase the
maximum SRP transfer size.
Bart.
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Cant write to max_sectors_kb on 4.5.0 SRP target
2016-04-08 5:30 ` Nicholas A. Bellinger
@ 2016-04-08 8:15 ` Laurence Oberman
0 siblings, 0 replies; 11+ messages in thread
From: Laurence Oberman @ 2016-04-08 8:15 UTC (permalink / raw)
To: Nicholas A. Bellinger; +Cc: linux-scsi, linux-rdma, target-devel
Hello Nicholas
Using fedora here and LIO/
The same array is used in testing the RHEL and upstream kernels.
Linux fedstorage 4.5.0-rc7+ #1 SMP Sun Mar 13 16:30:39 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux
I have 3 NVME cards striped in an LVM configuration and I create the block devices using LVM and then provision using targetcli.
I had not tuned anything on the LIO target because I was able to change the max_sectors_kb on the RHEL kernel.
Looking at the I/O size I was issuing on the initiator I confirmed the 4MB I/O was working on the RHEL kernel.
I realized of course it was possible the I/O could be broken down on the target by the time it handled the I/O but I was not focusing on that.
When I started testing upstream because I bumped into the various sg_map issues this is when I found the different behavior on the LUNS on the initiator.
Let me know if you think this is an array side tuning issue.
optimal_sectors = 256
hw_max_sectors = 256
/sys/kernel/config/target/core/iblock_0/block-1
[root@fedstorage iblock_0]# cat hba_info
HBA Index: 1 plugin: iblock version: v5.0
[root@fedstorage iblock_0]# cat hba_mode
0
[root@fedstorage iblock_0]# cd block-1/
/sys/kernel/config/target/core/iblock_0/block-1/attrib
[root@fedstorage attrib]# for i in *
> do
> echo $i
> cat $i
> echo
> done
block_size
512
emulate_3pc
1
emulate_caw
1
emulate_dpo
1
emulate_fua_read
1
emulate_fua_write
1
emulate_model_alias
1
emulate_rest_reord
0
emulate_tas
1
emulate_tpu
0
emulate_tpws
0
emulate_ua_intlck_ctrl
0
emulate_write_cache
0
enforce_pr_isids
1
force_pr_aptpl
0
hw_block_size
512
hw_max_sectors *****
256
hw_pi_prot_type
0
hw_queue_depth
128
is_nonrot
1
max_unmap_block_desc_count
1
max_unmap_lba_count
8388607
max_write_same_len
65535
optimal_sectors
256 *****
pi_prot_format
0
pi_prot_type
0
queue_depth
128
unmap_granularity
1
unmap_granularity_alignment
0
unmap_zeroes_data
0
[root@fedstorage ~]# targetcli ls
o- / ........................................................................................................... [...]
o- backstores ................................................................................................ [...]
| o- block ................................................................................... [Storage Objects: 30]
| | o- block-1 ................................................... [/dev/data/block-1 (9.0GiB) write-thru activated]
| | o- block-2 ................................................... [/dev/data/block-2 (9.0GiB) write-thru activated]
| | o- block-3 ................................................... [/dev/data/block-3 (9.0GiB) write-thru activated]
..
..
| | o- block-28 ................................................. [/dev/data/block-28 (9.0GiB) write-thru activated]
| | o- block-29 ................................................. [/dev/data/block-29 (9.0GiB) write-thru activated]
| | o- block-30 ................................................. [/dev/data/block-30 (9.0GiB) write-thru activated]
| o- fileio ................................................................................... [Storage Objects: 0]
| o- pscsi .................................................................................... [Storage Objects: 0]
| o- ramdisk .................................................................................. [Storage Objects: 0]
o- iscsi .............................................................................................. [Targets: 0]
o- loopback ........................................................................................... [Targets: 0]
o- qla2xxx ............................................................................................ [Targets: 0]
o- srpt ............................................................................................... [Targets: 2]
| o- ib.fe800000000000007cfe900300726e4e ............................................................. [no-gen-acls]
| | o- acls .............................................................................................. [ACLs: 2]
| | | o- ib.4e6e72000390fe7c7cfe900300726ed2 ..................................................... [Mapped LUNs: 30]
| | | | o- mapped_lun0 ................................................................... [lun0 block/block-1 (rw)]
| | | | o- mapped_lun1 ................................................................... [lun1 block/block-2 (rw)]
| | | | o- mapped_lun2 ................................................................... [lun2 block/block-3 (rw)]
..
..
| | | | o- mapped_lun26 ................................................................ [lun26 block/block-27 (rw)]
| | | | o- mapped_lun27 ................................................................ [lun27 block/block-28 (rw)]
| | | | o- mapped_lun28 ................................................................ [lun28 block/block-29 (rw)]
| | | | o- mapped_lun29 ................................................................ [lun29 block/block-30 (rw)]
| | | o- ib.4f6e72000390fe7c7cfe900300726ed3 ..................................................... [Mapped LUNs: 30]
| | | o- mapped_lun0 ................................................................... [lun0 block/block-1 (rw)]
| | | o- mapped_lun1 ................................................................... [lun1 block/block-2 (rw)]
| | | o- mapped_lun2 ................................................................... [lun2 block/block-3 (rw)]
| | | o- mapped_lun3 ................................................................... [lun3 block/block-4 (rw)]
| | | o- mapped_lun4 ................................................................... [lun4 block/block-5 (rw)]
..
,,
Laurence Oberman
Principal Software Maintenance Engineer
Red Hat Global Support Services
----- Original Message -----
From: "Nicholas A. Bellinger" <nab@linux-iscsi.org>
To: "Laurence Oberman" <loberman@redhat.com>
Cc: "linux-scsi" <linux-scsi@vger.kernel.org>, linux-rdma@vger.kernel.org, "target-devel" <target-devel@vger.kernel.org>
Sent: Friday, April 8, 2016 1:30:28 AM
Subject: Re: Cant write to max_sectors_kb on 4.5.0 SRP target
Hi Laurence,
On Thu, 2016-04-07 at 17:15 -0400, Laurence Oberman wrote:
> Hello
>
> I have been testing the SRP initiator code to an LIO array here and
> part of the testing requires me to set the max_sectors_kb size to get
> 4k I/O's.
> This has been due to me having to debug various sg_map issues.
>
> Linux srptest 4.5.0 #2 SMP Thu Apr 7 16:14:38 EDT 2016 x86_64 x86_64
> x86_64 GNU/Linux
> This kernel has the scan patch from Hannes, as well as the "[PATCH]
> IB/mlx5: Expose correct max_sge_rd limit" patch.
> However, I also tested with vanilla 4.5.0 as well and its the same
> issue.
>
> For some reason I cannot change the max_sectors_kb size on 4.5.0 here.
>
> I chatted with Ewan about it as well and he reminded me about Martins
> changes so wondering if that's playing into this.
>
> Take /dev/sdb as an example
>
> [root@srptest queue]# sg_inq --p 0xb0 /dev/sdb
> VPD INQUIRY: Block limits page (SBC)
> Maximum compare and write length: 1 blocks
> Optimal transfer length granularity: 256 blocks
> Maximum transfer length: 256 blocks
> Optimal transfer length: 768 blocks
> Maximum prefetch, xdread, xdwrite transfer length: 0 blocks
>
Just curious what target backend this is with..?
Specifically the optimal transfer length granularity and optimal
transfer length may be reported by underlying backend device (eg:
IBLOCK) in spc_emulate_evpd_b0().
What does 'head /sys/kernel/config/target/core/$HBA/$DEV/attrib/*'
of the backend device in question look like..?
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Cant write to max_sectors_kb on 4.5.0 SRP target
2016-04-08 3:00 ` Martin K. Petersen
@ 2016-04-08 8:31 ` Laurence Oberman
2016-04-08 12:39 ` Ewan D. Milne
0 siblings, 1 reply; 11+ messages in thread
From: Laurence Oberman @ 2016-04-08 8:31 UTC (permalink / raw)
To: Martin K. Petersen; +Cc: linux-scsi, linux-rdma
Hello Martin
Yes, Ewan also noticed that.
This started out as me testing the SRP stack on RHEL 7.2 and baselining against upstream.
We have a customer that requires 4MB I/O.
I bumped into a number of SRP issues including sg_map failures so started reviewing upstream changes to the SRP code and patches.
The RHEL kernel is ignoring this so perhaps we have an issue on our side (RHEL kernel) and upstream is behaving as it should.
What is intersting is that I cannot change the max_sectors_kb at all on the upstream for the SRP LUNS.
Here is an HP SmartArray LUN
[root@srptest ~]# sg_inq --p 0xb0 /dev/sda
VPD INQUIRY: page=0xb0
inquiry: field in cdb illegal (page not supported) **** Known that its not supported
However
/sys/block/sda/queue
[root@srptest queue]# cat max_hw_sectors_kb max_sectors_kb
4096
1280
[root@srptest queue]# echo 4096 > max_sectors_kb
[root@srptest queue]# cat max_hw_sectors_kb max_sectors_kb
4096
4096
On the SRP LUNS I am unable to change to a lower value than max_sectors_kb unless I change it to 128
So perhaps the size on the array is the issue here as Nicholas said and the RHEL kernel has a bug and ignores it.
/sys/block/sdc/queue
[root@srptest queue]# cat max_hw_sectors_kb max_sectors_kb
4096
1280
[root@srptest queue]# echo 512 > max_sectors_kb
-bash: echo: write error: Invalid argument
[root@srptest queue]# echo 256 > max_sectors_kb
-bash: echo: write error: Invalid argument
128 works
[root@srptest queue]# echo 128 > max_sectors_kb
Laurence Oberman
Principal Software Maintenance Engineer
Red Hat Global Support Services
----- Original Message -----
From: "Martin K. Petersen" <martin.petersen@oracle.com>
To: "Laurence Oberman" <loberman@redhat.com>
Cc: "linux-scsi" <linux-scsi@vger.kernel.org>, linux-rdma@vger.kernel.org
Sent: Thursday, April 7, 2016 11:00:16 PM
Subject: Re: Cant write to max_sectors_kb on 4.5.0 SRP target
>>>>> "Laurence" == Laurence Oberman <loberman@redhat.com> writes:
Laurence,
The target is reporting inconsistent values here:
> [root@srptest queue]# sg_inq --p 0xb0 /dev/sdb
> VPD INQUIRY: Block limits page (SBC)
> Maximum compare and write length: 1 blocks
> Optimal transfer length granularity: 256 blocks
> Maximum transfer length: 256 blocks
> Optimal transfer length: 768 blocks
OPTIMAL TRANSFER LENGTH GRANULARITY roughly translates to physical block
size or RAID chunk size. It's the smallest I/O unit that does not
require read-modify-write. It would typically be either 1 or 8 blocks
for a drive and maybe 64, 128 or 256 for a RAID5 array. io_min in
queue_limits.
OPTIMAL TRANSFER LENGTH indicates the stripe width and is a multiple of
OPTIMAL TRANSFER LENGTH GRANULARITY. io_opt in queue_limits.
MAXIMUM TRANSFER LENGTH indicates the biggest READ/WRITE command the
device can handle in a single command. In this case 256 blocks so that's
128K. max_dev_sectors in queue_limits.
>From SBC:
"A MAXIMUM TRANSFER LENGTH field set to a non-zero value indicates the
maximum transfer length in logical blocks that the device server accepts
for a single command shown in table 250. If a device server receives one
of these commands with a transfer size greater than this value, then the
device server shall terminate the command with CHECK CONDITION status
[...]"
So those reported values are off.
logical block size <= physical block size <= OTLG <= OTL <= MTL
Or in terms of queue_limits:
lbs <= pbs <= io_min <= io_opt <=
min_not_zero(max_dev_sectors, max_hw_sectors, max_sectors)
--
Martin K. Petersen Oracle Linux Engineering
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Cant write to max_sectors_kb on 4.5.0 SRP target
2016-04-08 8:31 ` Laurence Oberman
@ 2016-04-08 12:39 ` Ewan D. Milne
[not found] ` <1460119192.25335.40.camel-bi+AKbBUZKY6gyzm1THtWbp2dZbC/Bob@public.gmane.org>
0 siblings, 1 reply; 11+ messages in thread
From: Ewan D. Milne @ 2016-04-08 12:39 UTC (permalink / raw)
To: Laurence Oberman; +Cc: Martin K. Petersen, linux-scsi, linux-rdma
The version of RHEL you are using does not have:
commit ca369d51b3e1649be4a72addd6d6a168cfb3f537
Author: Martin K. Petersen <martin.petersen@oracle.com>
Date: Fri Nov 13 16:46:48 2015 -0500
block/sd: Fix device-imposed transfer length limits
(which will be added during the next update).
In the upstream kernel queue_max_sectors_store() does not
permit you to set a value larger than the device-imposed
limit. This value, stored in q->limits.max_dev_sectors,
is not visible via the block queue sysfs interface.
The code that sets q->limits.max_sectors and q->limits.io_opt
in sd.c does not take the device limit into account, but
the sysfs code to change max_sectors ("max_sectors_kb") does.
So there are a couple of problems here, one is that RHEL
is not clamping to the device limit, and the other one is
that neither RHEL nor upstream kernels take the device limit
into account when setting q->limits.io_opt. This only seems
to be a problem for you because your target is reporting
an optimal I/O size in VPD page B0 that is *smaller* than
the reported maximum I/O size.
The target is clearly reporting inconsistent data, the
question is whether we should change the code to clamp the
optimal I/O size, or whether we should assume the value
the target is reporting is wrong.
So the question is: does the target actually process
requests that are larger than the VPD page B0 reported
maximum size? If so, maybe we should just issue a warning
message rather than reducing the optimal I/O size.
-Ewan
On Fri, 2016-04-08 at 04:31 -0400, Laurence Oberman wrote:
> Hello Martin
>
> Yes, Ewan also noticed that.
>
> This started out as me testing the SRP stack on RHEL 7.2 and baselining against upstream.
> We have a customer that requires 4MB I/O.
> I bumped into a number of SRP issues including sg_map failures so started reviewing upstream changes to the SRP code and patches.
>
> The RHEL kernel is ignoring this so perhaps we have an issue on our side (RHEL kernel) and upstream is behaving as it should.
>
> What is intersting is that I cannot change the max_sectors_kb at all on the upstream for the SRP LUNS.
>
> Here is an HP SmartArray LUN
>
> [root@srptest ~]# sg_inq --p 0xb0 /dev/sda
> VPD INQUIRY: page=0xb0
> inquiry: field in cdb illegal (page not supported) **** Known that its not supported
>
> However
>
> /sys/block/sda/queue
>
> [root@srptest queue]# cat max_hw_sectors_kb max_sectors_kb
> 4096
> 1280
> [root@srptest queue]# echo 4096 > max_sectors_kb
> [root@srptest queue]# cat max_hw_sectors_kb max_sectors_kb
> 4096
> 4096
>
> On the SRP LUNS I am unable to change to a lower value than max_sectors_kb unless I change it to 128
> So perhaps the size on the array is the issue here as Nicholas said and the RHEL kernel has a bug and ignores it.
>
> /sys/block/sdc/queue
>
> [root@srptest queue]# cat max_hw_sectors_kb max_sectors_kb
> 4096
> 1280
>
> [root@srptest queue]# echo 512 > max_sectors_kb
> -bash: echo: write error: Invalid argument
>
> [root@srptest queue]# echo 256 > max_sectors_kb
> -bash: echo: write error: Invalid argument
>
> 128 works
> [root@srptest queue]# echo 128 > max_sectors_kb
>
>
>
>
> Laurence Oberman
> Principal Software Maintenance Engineer
> Red Hat Global Support Services
>
> ----- Original Message -----
> From: "Martin K. Petersen" <martin.petersen@oracle.com>
> To: "Laurence Oberman" <loberman@redhat.com>
> Cc: "linux-scsi" <linux-scsi@vger.kernel.org>, linux-rdma@vger.kernel.org
> Sent: Thursday, April 7, 2016 11:00:16 PM
> Subject: Re: Cant write to max_sectors_kb on 4.5.0 SRP target
>
> >>>>> "Laurence" == Laurence Oberman <loberman@redhat.com> writes:
>
> Laurence,
>
> The target is reporting inconsistent values here:
>
> > [root@srptest queue]# sg_inq --p 0xb0 /dev/sdb
> > VPD INQUIRY: Block limits page (SBC)
> > Maximum compare and write length: 1 blocks
> > Optimal transfer length granularity: 256 blocks
> > Maximum transfer length: 256 blocks
> > Optimal transfer length: 768 blocks
>
> OPTIMAL TRANSFER LENGTH GRANULARITY roughly translates to physical block
> size or RAID chunk size. It's the smallest I/O unit that does not
> require read-modify-write. It would typically be either 1 or 8 blocks
> for a drive and maybe 64, 128 or 256 for a RAID5 array. io_min in
> queue_limits.
>
> OPTIMAL TRANSFER LENGTH indicates the stripe width and is a multiple of
> OPTIMAL TRANSFER LENGTH GRANULARITY. io_opt in queue_limits.
>
> MAXIMUM TRANSFER LENGTH indicates the biggest READ/WRITE command the
> device can handle in a single command. In this case 256 blocks so that's
> 128K. max_dev_sectors in queue_limits.
>
> From SBC:
>
> "A MAXIMUM TRANSFER LENGTH field set to a non-zero value indicates the
> maximum transfer length in logical blocks that the device server accepts
> for a single command shown in table 250. If a device server receives one
> of these commands with a transfer size greater than this value, then the
> device server shall terminate the command with CHECK CONDITION status
> [...]"
>
> So those reported values are off.
>
> logical block size <= physical block size <= OTLG <= OTL <= MTL
>
> Or in terms of queue_limits:
>
> lbs <= pbs <= io_min <= io_opt <=
> min_not_zero(max_dev_sectors, max_hw_sectors, max_sectors)
>
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Cant write to max_sectors_kb on 4.5.0 SRP target
[not found] ` <1460119192.25335.40.camel-bi+AKbBUZKY6gyzm1THtWbp2dZbC/Bob@public.gmane.org>
@ 2016-04-08 13:11 ` Laurence Oberman
[not found] ` <1975890115.28041373.1460121079252.JavaMail.zimbra-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
2016-04-11 21:29 ` Martin K. Petersen
1 sibling, 1 reply; 11+ messages in thread
From: Laurence Oberman @ 2016-04-08 13:11 UTC (permalink / raw)
To: emilne-H+wXaHxf7aLQT0dZR+AlfA
Cc: Martin K. Petersen, linux-scsi, linux-rdma-u79uwXL29TY76Z2rM5mHXA
Hi Ewan,
OK, that makes sense.
I suspected after everybody's responses that RHEL was somehow ignoring the array imposed limit here.
I actually got lucky because I needed to be able to issue 4MB IO'S to reproduce the failures seen
at the customer on the initiator side.
Looking at the target-LIO array now its clamped to 1MB I/O sizes which makes sense.
I really was not focusing on the array at the time expecting it may chop the I/O up as many do.
Knowing what's up now I can continue to test and figure out what patches I need to pull in to SRP on RHEL to make progress.
Thank you to all that responded.
Laurence Oberman
Principal Software Maintenance Engineer
Red Hat Global Support Services
----- Original Message -----
From: "Ewan D. Milne" <emilne-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
To: "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
Cc: "Martin K. Petersen" <martin.petersen-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>, "linux-scsi" <linux-scsi-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
Sent: Friday, April 8, 2016 8:39:52 AM
Subject: Re: Cant write to max_sectors_kb on 4.5.0 SRP target
The version of RHEL you are using does not have:
commit ca369d51b3e1649be4a72addd6d6a168cfb3f537
Author: Martin K. Petersen <martin.petersen-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>
Date: Fri Nov 13 16:46:48 2015 -0500
block/sd: Fix device-imposed transfer length limits
(which will be added during the next update).
In the upstream kernel queue_max_sectors_store() does not
permit you to set a value larger than the device-imposed
limit. This value, stored in q->limits.max_dev_sectors,
is not visible via the block queue sysfs interface.
The code that sets q->limits.max_sectors and q->limits.io_opt
in sd.c does not take the device limit into account, but
the sysfs code to change max_sectors ("max_sectors_kb") does.
So there are a couple of problems here, one is that RHEL
is not clamping to the device limit, and the other one is
that neither RHEL nor upstream kernels take the device limit
into account when setting q->limits.io_opt. This only seems
to be a problem for you because your target is reporting
an optimal I/O size in VPD page B0 that is *smaller* than
the reported maximum I/O size.
The target is clearly reporting inconsistent data, the
question is whether we should change the code to clamp the
optimal I/O size, or whether we should assume the value
the target is reporting is wrong.
So the question is: does the target actually process
requests that are larger than the VPD page B0 reported
maximum size? If so, maybe we should just issue a warning
message rather than reducing the optimal I/O size.
-Ewan
On Fri, 2016-04-08 at 04:31 -0400, Laurence Oberman wrote:
> Hello Martin
>
> Yes, Ewan also noticed that.
>
> This started out as me testing the SRP stack on RHEL 7.2 and baselining against upstream.
> We have a customer that requires 4MB I/O.
> I bumped into a number of SRP issues including sg_map failures so started reviewing upstream changes to the SRP code and patches.
>
> The RHEL kernel is ignoring this so perhaps we have an issue on our side (RHEL kernel) and upstream is behaving as it should.
>
> What is intersting is that I cannot change the max_sectors_kb at all on the upstream for the SRP LUNS.
>
> Here is an HP SmartArray LUN
>
> [root@srptest ~]# sg_inq --p 0xb0 /dev/sda
> VPD INQUIRY: page=0xb0
> inquiry: field in cdb illegal (page not supported) **** Known that its not supported
>
> However
>
> /sys/block/sda/queue
>
> [root@srptest queue]# cat max_hw_sectors_kb max_sectors_kb
> 4096
> 1280
> [root@srptest queue]# echo 4096 > max_sectors_kb
> [root@srptest queue]# cat max_hw_sectors_kb max_sectors_kb
> 4096
> 4096
>
> On the SRP LUNS I am unable to change to a lower value than max_sectors_kb unless I change it to 128
> So perhaps the size on the array is the issue here as Nicholas said and the RHEL kernel has a bug and ignores it.
>
> /sys/block/sdc/queue
>
> [root@srptest queue]# cat max_hw_sectors_kb max_sectors_kb
> 4096
> 1280
>
> [root@srptest queue]# echo 512 > max_sectors_kb
> -bash: echo: write error: Invalid argument
>
> [root@srptest queue]# echo 256 > max_sectors_kb
> -bash: echo: write error: Invalid argument
>
> 128 works
> [root@srptest queue]# echo 128 > max_sectors_kb
>
>
>
>
> Laurence Oberman
> Principal Software Maintenance Engineer
> Red Hat Global Support Services
>
> ----- Original Message -----
> From: "Martin K. Petersen" <martin.petersen-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>
> To: "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> Cc: "linux-scsi" <linux-scsi-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> Sent: Thursday, April 7, 2016 11:00:16 PM
> Subject: Re: Cant write to max_sectors_kb on 4.5.0 SRP target
>
> >>>>> "Laurence" == Laurence Oberman <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> writes:
>
> Laurence,
>
> The target is reporting inconsistent values here:
>
> > [root@srptest queue]# sg_inq --p 0xb0 /dev/sdb
> > VPD INQUIRY: Block limits page (SBC)
> > Maximum compare and write length: 1 blocks
> > Optimal transfer length granularity: 256 blocks
> > Maximum transfer length: 256 blocks
> > Optimal transfer length: 768 blocks
>
> OPTIMAL TRANSFER LENGTH GRANULARITY roughly translates to physical block
> size or RAID chunk size. It's the smallest I/O unit that does not
> require read-modify-write. It would typically be either 1 or 8 blocks
> for a drive and maybe 64, 128 or 256 for a RAID5 array. io_min in
> queue_limits.
>
> OPTIMAL TRANSFER LENGTH indicates the stripe width and is a multiple of
> OPTIMAL TRANSFER LENGTH GRANULARITY. io_opt in queue_limits.
>
> MAXIMUM TRANSFER LENGTH indicates the biggest READ/WRITE command the
> device can handle in a single command. In this case 256 blocks so that's
> 128K. max_dev_sectors in queue_limits.
>
> From SBC:
>
> "A MAXIMUM TRANSFER LENGTH field set to a non-zero value indicates the
> maximum transfer length in logical blocks that the device server accepts
> for a single command shown in table 250. If a device server receives one
> of these commands with a transfer size greater than this value, then the
> device server shall terminate the command with CHECK CONDITION status
> [...]"
>
> So those reported values are off.
>
> logical block size <= physical block size <= OTLG <= OTL <= MTL
>
> Or in terms of queue_limits:
>
> lbs <= pbs <= io_min <= io_opt <=
> min_not_zero(max_dev_sectors, max_hw_sectors, max_sectors)
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Cant write to max_sectors_kb on 4.5.0 SRP target
[not found] ` <1975890115.28041373.1460121079252.JavaMail.zimbra-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
@ 2016-04-11 14:57 ` Laurence Oberman
0 siblings, 0 replies; 11+ messages in thread
From: Laurence Oberman @ 2016-04-11 14:57 UTC (permalink / raw)
To: emilne-H+wXaHxf7aLQT0dZR+AlfA
Cc: Martin K. Petersen, linux-scsi, linux-rdma-u79uwXL29TY76Z2rM5mHXA
As a follow up to this issue.
I looked at modifying the LIO target code to allow a larger max_sectors_kb exported to the initiator for the nvme devices but had some issues.
In the end I created 15 fileio devices using 200GB of ramdisk and exported those so I could test 4MB I/O from the initiator.
These allow the 4MB setting on the upstream kernel.
[root@srptest ~]# sg_inq -p 0xb0 /dev/sdk
VPD INQUIRY: Block limits page (SBC)
Maximum compare and write length: 1 blocks
Optimal transfer length granularity: 1 blocks
Maximum transfer length: 16384 blocks
Optimal transfer length: 16384 blocks
Maximum prefetch, xdread, xdwrite transfer length: 0 blocks
The sg_map issues I am having on the RHEL kernel are likely due to the "proper" max sector size being ignored.
I am testing latest upstream now 4.5.0 with all the sg related patches to see if that's stable.
Thanks
Laurence Oberman
Principal Software Maintenance Engineer
Red Hat Global Support Services
----- Original Message -----
From: "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
To: emilne-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org
Cc: "Martin K. Petersen" <martin.petersen-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>, "linux-scsi" <linux-scsi-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
Sent: Friday, April 8, 2016 9:11:19 AM
Subject: Re: Cant write to max_sectors_kb on 4.5.0 SRP target
Hi Ewan,
OK, that makes sense.
I suspected after everybody's responses that RHEL was somehow ignoring the array imposed limit here.
I actually got lucky because I needed to be able to issue 4MB IO'S to reproduce the failures seen
at the customer on the initiator side.
Looking at the target-LIO array now its clamped to 1MB I/O sizes which makes sense.
I really was not focusing on the array at the time expecting it may chop the I/O up as many do.
Knowing what's up now I can continue to test and figure out what patches I need to pull in to SRP on RHEL to make progress.
Thank you to all that responded.
Laurence Oberman
Principal Software Maintenance Engineer
Red Hat Global Support Services
----- Original Message -----
From: "Ewan D. Milne" <emilne-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
To: "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
Cc: "Martin K. Petersen" <martin.petersen-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>, "linux-scsi" <linux-scsi-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
Sent: Friday, April 8, 2016 8:39:52 AM
Subject: Re: Cant write to max_sectors_kb on 4.5.0 SRP target
The version of RHEL you are using does not have:
commit ca369d51b3e1649be4a72addd6d6a168cfb3f537
Author: Martin K. Petersen <martin.petersen-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>
Date: Fri Nov 13 16:46:48 2015 -0500
block/sd: Fix device-imposed transfer length limits
(which will be added during the next update).
In the upstream kernel queue_max_sectors_store() does not
permit you to set a value larger than the device-imposed
limit. This value, stored in q->limits.max_dev_sectors,
is not visible via the block queue sysfs interface.
The code that sets q->limits.max_sectors and q->limits.io_opt
in sd.c does not take the device limit into account, but
the sysfs code to change max_sectors ("max_sectors_kb") does.
So there are a couple of problems here, one is that RHEL
is not clamping to the device limit, and the other one is
that neither RHEL nor upstream kernels take the device limit
into account when setting q->limits.io_opt. This only seems
to be a problem for you because your target is reporting
an optimal I/O size in VPD page B0 that is *smaller* than
the reported maximum I/O size.
The target is clearly reporting inconsistent data, the
question is whether we should change the code to clamp the
optimal I/O size, or whether we should assume the value
the target is reporting is wrong.
So the question is: does the target actually process
requests that are larger than the VPD page B0 reported
maximum size? If so, maybe we should just issue a warning
message rather than reducing the optimal I/O size.
-Ewan
On Fri, 2016-04-08 at 04:31 -0400, Laurence Oberman wrote:
> Hello Martin
>
> Yes, Ewan also noticed that.
>
> This started out as me testing the SRP stack on RHEL 7.2 and baselining against upstream.
> We have a customer that requires 4MB I/O.
> I bumped into a number of SRP issues including sg_map failures so started reviewing upstream changes to the SRP code and patches.
>
> The RHEL kernel is ignoring this so perhaps we have an issue on our side (RHEL kernel) and upstream is behaving as it should.
>
> What is intersting is that I cannot change the max_sectors_kb at all on the upstream for the SRP LUNS.
>
> Here is an HP SmartArray LUN
>
> [root@srptest ~]# sg_inq --p 0xb0 /dev/sda
> VPD INQUIRY: page=0xb0
> inquiry: field in cdb illegal (page not supported) **** Known that its not supported
>
> However
>
> /sys/block/sda/queue
>
> [root@srptest queue]# cat max_hw_sectors_kb max_sectors_kb
> 4096
> 1280
> [root@srptest queue]# echo 4096 > max_sectors_kb
> [root@srptest queue]# cat max_hw_sectors_kb max_sectors_kb
> 4096
> 4096
>
> On the SRP LUNS I am unable to change to a lower value than max_sectors_kb unless I change it to 128
> So perhaps the size on the array is the issue here as Nicholas said and the RHEL kernel has a bug and ignores it.
>
> /sys/block/sdc/queue
>
> [root@srptest queue]# cat max_hw_sectors_kb max_sectors_kb
> 4096
> 1280
>
> [root@srptest queue]# echo 512 > max_sectors_kb
> -bash: echo: write error: Invalid argument
>
> [root@srptest queue]# echo 256 > max_sectors_kb
> -bash: echo: write error: Invalid argument
>
> 128 works
> [root@srptest queue]# echo 128 > max_sectors_kb
>
>
>
>
> Laurence Oberman
> Principal Software Maintenance Engineer
> Red Hat Global Support Services
>
> ----- Original Message -----
> From: "Martin K. Petersen" <martin.petersen-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>
> To: "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> Cc: "linux-scsi" <linux-scsi-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> Sent: Thursday, April 7, 2016 11:00:16 PM
> Subject: Re: Cant write to max_sectors_kb on 4.5.0 SRP target
>
> >>>>> "Laurence" == Laurence Oberman <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> writes:
>
> Laurence,
>
> The target is reporting inconsistent values here:
>
> > [root@srptest queue]# sg_inq --p 0xb0 /dev/sdb
> > VPD INQUIRY: Block limits page (SBC)
> > Maximum compare and write length: 1 blocks
> > Optimal transfer length granularity: 256 blocks
> > Maximum transfer length: 256 blocks
> > Optimal transfer length: 768 blocks
>
> OPTIMAL TRANSFER LENGTH GRANULARITY roughly translates to physical block
> size or RAID chunk size. It's the smallest I/O unit that does not
> require read-modify-write. It would typically be either 1 or 8 blocks
> for a drive and maybe 64, 128 or 256 for a RAID5 array. io_min in
> queue_limits.
>
> OPTIMAL TRANSFER LENGTH indicates the stripe width and is a multiple of
> OPTIMAL TRANSFER LENGTH GRANULARITY. io_opt in queue_limits.
>
> MAXIMUM TRANSFER LENGTH indicates the biggest READ/WRITE command the
> device can handle in a single command. In this case 256 blocks so that's
> 128K. max_dev_sectors in queue_limits.
>
> From SBC:
>
> "A MAXIMUM TRANSFER LENGTH field set to a non-zero value indicates the
> maximum transfer length in logical blocks that the device server accepts
> for a single command shown in table 250. If a device server receives one
> of these commands with a transfer size greater than this value, then the
> device server shall terminate the command with CHECK CONDITION status
> [...]"
>
> So those reported values are off.
>
> logical block size <= physical block size <= OTLG <= OTL <= MTL
>
> Or in terms of queue_limits:
>
> lbs <= pbs <= io_min <= io_opt <=
> min_not_zero(max_dev_sectors, max_hw_sectors, max_sectors)
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Cant write to max_sectors_kb on 4.5.0 SRP target
[not found] ` <1460119192.25335.40.camel-bi+AKbBUZKY6gyzm1THtWbp2dZbC/Bob@public.gmane.org>
2016-04-08 13:11 ` Laurence Oberman
@ 2016-04-11 21:29 ` Martin K. Petersen
1 sibling, 0 replies; 11+ messages in thread
From: Martin K. Petersen @ 2016-04-11 21:29 UTC (permalink / raw)
To: Ewan D. Milne
Cc: Laurence Oberman, Martin K. Petersen, linux-scsi,
linux-rdma-u79uwXL29TY76Z2rM5mHXA
>>>>> "Ewan" == Ewan D Milne <emilne-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> writes:
Ewan> In the upstream kernel queue_max_sectors_store() does not permit
Ewan> you to set a value larger than the device-imposed limit. This
Ewan> value, stored in q->limits.max_dev_sectors, is not visible via the
Ewan> block queue sysfs interface.
I should refresh the patch that exposes that. There were a few comments
and I never got around to rolling those in.
--
Martin K. Petersen Oracle Linux Engineering
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2016-04-11 21:29 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <480311118.27942868.1460063633409.JavaMail.zimbra@redhat.com>
[not found] ` <480311118.27942868.1460063633409.JavaMail.zimbra-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
2016-04-07 21:15 ` Cant write to max_sectors_kb on 4.5.0 SRP target Laurence Oberman
[not found] ` <302427900.27942894.1460063713447.JavaMail.zimbra-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
2016-04-08 2:49 ` Bart Van Assche
2016-04-08 7:58 ` Laurence Oberman
2016-04-08 3:00 ` Martin K. Petersen
2016-04-08 8:31 ` Laurence Oberman
2016-04-08 12:39 ` Ewan D. Milne
[not found] ` <1460119192.25335.40.camel-bi+AKbBUZKY6gyzm1THtWbp2dZbC/Bob@public.gmane.org>
2016-04-08 13:11 ` Laurence Oberman
[not found] ` <1975890115.28041373.1460121079252.JavaMail.zimbra-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
2016-04-11 14:57 ` Laurence Oberman
2016-04-11 21:29 ` Martin K. Petersen
2016-04-08 5:30 ` Nicholas A. Bellinger
2016-04-08 8:15 ` Laurence Oberman
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.