Linux-NVME Archive on lore.kernel.org
 help / color / Atom feed
* [PATCH 0/2] allow for busy poll improvements
@ 2019-10-30 22:27 Wunderlich, Mark
  2019-11-06 16:44 ` Sagi Grimberg
  0 siblings, 1 reply; 3+ messages in thread
From: Wunderlich, Mark @ 2019-10-30 22:27 UTC (permalink / raw)
  To: linux-nvme

Proposing a small series of two patches that provide for improved packet processing for a fabric network interface that is operating with poll mode enabled

Patch 1:  Modifies the do/while terminate condition in nvmet_tcp_io_work() to be time based  vs. the existing code that is operations count based, which will match more to how it is performed for the host version of io_work().   If poll mode is not active then original function behavior is preserved.

Being time based allows setting the loop period to a value indicated by the network socket if CONFIG_NET_RX_BUSY_POLL is enabled [poll_mode].  Operating in poll_mode provides for increased opportunity to reap send or recv completions without premature exit when a single iteration of loop is idle (pending being false).

After exiting the do/while loop, the work item will be re-queued if there was previous 'pending' activity.  In the case of poll mode, this is determined by any accumulated ops completions over the complete do/while period, vs if the last iteration through the loop showed successful activity.

Patch 2:  This patch builds upon previous kernel network patches listed below that enabled enhanced symmetric queuing, which is leveraged while in poll_mode:

- a4fd1f4 Documentation: Add explanation for XPS using Rx-queue(s) map
- 8af2c06 net-sysfs: Add interface for Rx queue(s) map per Tx queue
- fc9bab2 net: Enable Tx queue selection based on Rx queues
- c6345ce net: Record receive queue number for a connection

Setting the socket priority to a non-zero value, via the proposed module parameter, will trigger indication to the network NIC that optimized network processing and queue selection can or should be considered.  The default value for priority remains zero to support default NIC behavior related to priority.

When applied, and running with an optimized busy polling NIC, there is a measurable improvement in I/O performance and reduction in latency.  The sample data below shows FIO results For 4K random read operations to a single remote nvme device.  The queue depth Is 32 and the batch size is 8.  One set of data represents a baseline for the standard Linux kernel running on the host and target (5.2.1 stable).  In the second, the two patches were applied on the target but the NIC poll mode was not enabled.  This to show default target performance is not impacted by adding the changes.  And finally, the two Proposed patches were applied to the target, and poll mode was enabled.

For comparison, The number of threads for the fio job are scaled from 1 to 8 until we reach nvme device I/O saturation.  The data shows no performance impact by adding the patches and run in default non-polled mode.  While in polled mode device saturation is reached faster with lower average and 99.99 latency.

Sample FIO invocation line:
fio --filename=/dev/nvme0n1 --time_based --thread --runtime=60 --ramp_time=10 --rw=randrw --rwmixread=100 --refill_buffers --direct=1 --ioengine=libaio --bs=4k --iodepth=32 --iodepth_batch_complete_min=1 --iodepth_batch_complete_max=32 --iodepth_batch=8 --numjobs=1 --group_reporting --gtod_reduce=0 --disable_lat=0 --name=cpu3 --cpus_allowed=3

Baseline 5.2.1 stable kernel:
Threads  |  IOPS (K)  |  Avg Lat (usec)  | 99.99 (usec)
*****************************
1  |  195  |  157.69  |  553
2  |  215  |  284.31  |  758
3  |  404  |  229.04  |  742
4  |  515  |  239.98  |  742
5  |  549  |  282.22  |  750
6  |  581  |  321.64  |  750
7  |  587  |  376.68  |  750
8  |  587  |  432.58  |  971

With patches applied on Target kernel,
But poll mode off, so_priority set to 0.
******************************
1  |  197  |  156.40  |  545
2  |  286  |  215.14  |  758
3  |  422  |  218.75  |  734
4  |  491  |  251.34  |  742
5  |  504  |  306.68  |  750
6  |  583  |  319.91  |  766
7  |  587  |  378.32  |  742
8  |  587  |  434.57  |  660

With patches applied on Target kernel,
Poll mode enabled:
******************************
1  |  227  |  129.43  |  537
2  |  404  |  144.33  |  537
3  |  523  |  170.25  |  529
4  |  563  |  215.94  |  537
5  |  587  |  263.78  |  510
6  |  587  |  321.41  |  506
7  |  587  |  378.67  |  502
8  |  587  |  435.27  |  502

Note:  Data was gathered using kernel 5.2.1 stable.  The patches posted to this mailing list were merged to the infradead tree pull 10/30/19.

Mark Wunderlich (2):
- nvmet-tcp: enable polling option in io_work
- nvmet-tcp: set SO_PRIORITY for accepted sockets


_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: [PATCH 0/2] allow for busy poll improvements
  2019-10-30 22:27 [PATCH 0/2] allow for busy poll improvements Wunderlich, Mark
@ 2019-11-06 16:44 ` Sagi Grimberg
  2019-11-07 16:09   ` Wunderlich, Mark
  0 siblings, 1 reply; 3+ messages in thread
From: Sagi Grimberg @ 2019-11-06 16:44 UTC (permalink / raw)
  To: Wunderlich, Mark, linux-nvme


> Proposing a small series of two patches that provide for improved packet processing for a fabric network interface that is operating with poll mode enabled

Mark,

Can we start with a series for the SO_PRIORITY for the host and the
target first? IIRC you mentioned that enabling this alone will
produce an improvement and the change is completely harmless.

Then we can add the polling change in an incremental series on that.

_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

^ permalink raw reply	[flat|nested] 3+ messages in thread

* RE: [PATCH 0/2] allow for busy poll improvements
  2019-11-06 16:44 ` Sagi Grimberg
@ 2019-11-07 16:09   ` Wunderlich, Mark
  0 siblings, 0 replies; 3+ messages in thread
From: Wunderlich, Mark @ 2019-11-07 16:09 UTC (permalink / raw)
  To: Sagi Grimberg, linux-nvme


>> Proposing a small series of two patches that provide for improved 
>> packet processing for a fabric network interface that is operating 
>> with poll mode enabled

>Mark,

>Can we start with a series for the SO_PRIORITY for the host and the target first? IIRC you mentioned that enabling this alone will produce an improvement >and the change is completely harmless.

>Then we can add the polling change in an incremental series on that.

OK, will re-submit new series for host and target so_priority change.
Cheers
_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

^ permalink raw reply	[flat|nested] 3+ messages in thread

end of thread, back to index

Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-10-30 22:27 [PATCH 0/2] allow for busy poll improvements Wunderlich, Mark
2019-11-06 16:44 ` Sagi Grimberg
2019-11-07 16:09   ` Wunderlich, Mark

Linux-NVME Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-nvme/0 linux-nvme/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-nvme linux-nvme/ https://lore.kernel.org/linux-nvme \
		linux-nvme@lists.infradead.org
	public-inbox-index linux-nvme

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.infradead.lists.linux-nvme


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git