Linux-NVME Archive on
 help / color / Atom feed
From: "Wunderlich, Mark" <>
To: "" <>
Cc: Sagi Grimberg <>
Subject: [PATCH 0/2] nvme-tcp: Set SO_PRIORITY for all sockets
Date: Thu, 16 Jan 2020 00:46:05 +0000
Message-ID: <> (raw)

Host and target patch series to allow setting socket priority.
Enables ability to associate all sockets related to NVMf TCP
traffic to a priority group that will perform optimized
network processing for this traffic class. Maintain initial
default priority of zero, only setting socket priority when
module option set to non-zero value, when optimized
network processing conditions exist.

Basic performance measurements for io_uring, pvsync2, and libaio
show benefit of improved lower latency performance when enhanced
processing support is triggered via socket priority.

The data gathered using infradead branch nvme-5.5-rc.  Data shows
Existing OS baseline performance without patch applied to both host
and target,  performance when patch applied and priority value set but
no support for enhanced traffic processing exists,  and finally when
patch applied and some form of advanced traffic processing exists
such as symmetric queueing.

The patches were defined to allow for any priority value to be used
to support future flexibility to specify any unique value for optional
advanced network processing.  If having this flexibility is reviewed as
not beneficial to the community at this time, then the desired option
would be to set the default module priority to non-zero, to always
allow for advanced network traffic processing if possible.  But data
does show a slight performance dip when the priority is set when
there is no advanced network support available.

All data gathered using 48 I/O queues and 48 poll queues established
Between host and target.  Host busy_poll value set at 5000.  Target
busy_read value set at 60000.  Single Optane device used.

Baseline:  No patches applied
QueueDepth/Batch    IOPS(K)    Avg. Latency(usec)    99.99(usec)
1/1    24.2    40.90    70.14
32/8  194    156.88   441

Patch added, priority=1, But enhanced processing not available:
1/1    22     43.16    77.31
32/8  222     137.89    277

Patch added, priority=1, enhanced NIC processing added
1/1    30.9    32.01    59.64
32/8   259     119.26    174

Baseline:  No patches applied
1/1    24.3    40.85    80.38
32/8   24.3    40.84   73.21

Patch added, priority=1:
1/1    23.1    42.87    79.36
32/8   23.1    43     79.36

Patch added, priority=1, enhanced NIC processing added
1/1    31.3     31.69    62.20
32/8   31.3    31.63   62.20

Baseline:  No patches applied
1/1    26.2    37.67    77.31
32/8   139    220.10   807

Patch added, priority=1:
1/1    24.6    40    78.33
32/8   138    220.91   791

Patch added, priority=1, enhanced NIC processing added
1/1    28     34.03    69.12
32/8   278    90.77    139

linux-nvme mailing list

             reply index

Thread overview: 2+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-01-16  0:46 Wunderlich, Mark [this message]
2020-01-17  1:13 ` Sagi Grimberg

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \ \ \ \ \

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Linux-NVME Archive on

Archives are clonable:
	git clone --mirror linux-nvme/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-nvme linux-nvme/ \
	public-inbox-index linux-nvme

Example config snippet for mirrors

Newsgroup available over NNTP:

AGPL code for this site: git clone