Re: [PATCH 0/2] nvme-tcp: Set SO_PRIORITY for all sockets

[Sagi Grimberg <sagi@grimberg.me>]

Mark,

> Host and target patch series to allow setting socket priority.
> Enables ability to associate all sockets related to NVMf TCP
> traffic to a priority group that will perform optimized
> network processing for this traffic class. Maintain initial
> default priority of zero, only setting socket priority when
> module option set to non-zero value, when optimized
> network processing conditions exist.
> 
> Basic performance measurements for io_uring, pvsync2, and libaio
> show benefit of improved lower latency performance when enhanced
> processing support is triggered via socket priority.
> 
> The data gathered using infradead branch nvme-5.5-rc.  Data shows
> Existing OS baseline performance without patch applied to both host
> and target,  performance when patch applied and priority value set but
> no support for enhanced traffic processing exists,  and finally when
> patch applied and some form of advanced traffic processing exists
> such as symmetric queueing.
> 
> The patches were defined to allow for any priority value to be used
> to support future flexibility to specify any unique value for optional
> advanced network processing.  If having this flexibility is reviewed as
> not beneficial to the community at this time, then the desired option
> would be to set the default module priority to non-zero, to always
> allow for advanced network traffic processing if possible.  But data
> does show a slight performance dip when the priority is set when
> there is no advanced network support available.

I think we can keep the special mode in a modparam for the time being,
especially given that other configuration is needed as well. The fact
that there is a small delta if adq is not enabled but so_priority is
set is acceptable, and if this option is used we can assume the user
relies on adq anyhow.

> All data gathered using 48 I/O queues and 48 poll queues established
> Between host and target.

I'm assuming that overall IOPs/throughput on a multithreaded workload
also improved (when not capped by the single Optane device).

>  Host busy_poll value set at 5000.  Target
> busy_read value set at 60000.

This shouldn't make a difference for nvme-tcp which overrides busy
polling for its own blk_poll hook.

>  Single Optane device used.
> 
> IO_URING:
> Baseline:  No patches applied
> QueueDepth/Batch    IOPS(K)    Avg. Latency(usec)    99.99(usec)
> 1/1    24.2    40.90    70.14
> 32/8  194    156.88   441
> 
> Patch added, priority=1, But enhanced processing not available:
> 1/1    22     43.16    77.31
> 32/8  222     137.89    277
> 
> Patch added, priority=1, enhanced NIC processing added
> 1/1    30.9    32.01    59.64
> 32/8   259     119.26    174
> 
> PVSYNC2:
> Baseline:  No patches applied
> 1/1    24.3    40.85    80.38
> 32/8   24.3    40.84   73.21
> 
> Patch added, priority=1:
> 1/1    23.1    42.87    79.36
> 32/8   23.1    43     79.36
> 
> Patch added, priority=1, enhanced NIC processing added
> 1/1    31.3     31.69    62.20
> 32/8   31.3    31.63   62.20

No need to report 32/8 for pvsync2

> 
> LIBAIO:
> Baseline:  No patches applied
> 1/1    26.2    37.67    77.31
> 32/8   139    220.10   807
> 
> Patch added, priority=1:
> 1/1    24.6    40    78.33
> 32/8   138    220.91   791
> 
> Patch added, priority=1, enhanced NIC processing added
> 1/1    28     34.03    69.12
> 32/8   278    90.77    139

Given how trivial the patch set is with the obvious improvement,
I say we should take it as is.

For the series:
Reviewed-by: Sagi Grimberg <sagi@grimberg.me>

_______________________________________________
linux-nvme mailing list
linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme