archive mirror
 help / color / mirror / Atom feed
* Possible to configure nvme irq cpu affinity?
@ 2019-10-31 22:20 Jeffrey Baker
  0 siblings, 0 replies; only message in thread
From: Jeffrey Baker @ 2019-10-31 22:20 UTC (permalink / raw)
  To: linux-nvme

I'm running kernel 4.15, the vendor kernel from Ubuntu 16.04. On a
system I have with four samsung nvme devices and 20 CPU cores (40 CPU
threads). I have a weird mapping of nvme irqs to cpus.  Each device
has 32 queues, each queue maps to 1 or 2 CPUs:

# grep -H . /sys/block/nvme0n1/mq/*/cpu_list | sort -t/ -k6n
/sys/block/nvme0n1/mq/0/cpu_list:0, 20
/sys/block/nvme0n1/mq/1/cpu_list:1, 21
/sys/block/nvme0n1/mq/2/cpu_list:2, 22
/sys/block/nvme0n1/mq/3/cpu_list:3, 23
/sys/block/nvme0n1/mq/16/cpu_list:10, 30
/sys/block/nvme0n1/mq/17/cpu_list:11, 31
/sys/block/nvme0n1/mq/18/cpu_list:12, 32
/sys/block/nvme0n1/mq/19/cpu_list:13, 33

There's 33 interrupts for these 32 queues, nvme0q0-q32. I don't
understand that but it's not the main problem. My problem is I can't
change them. I get EIO.

# cat /proc/irq/43/affinity_hint
# cat /proc/irq/43/smp_affinity
# echo 1 > /proc/irq/43/smp_affinity
-su: echo: write error: Input/output error

The reason I want to change this is the I/O rate on my application is
pretty modest but constant. On account of the way these IRQs are
spread around, such that there are 1-8 of them on every core, none of
my cores ever go to sleep (C6 state) and consequently none of the
active cores are ever able to clock up to higher P-states ("turbo").
In short, all 40 CPUs are stuck running at 2500MHz all the time, even
though under this workload I should be seeing 3100MHz
opportunistically. This measurably harms my service latency, not to
mention the power consumption.

If I could concentrate these queues on fewer CPUs, I would.  Is there
a newer kernel where this can be configured?


Linux-nvme mailing list

^ permalink raw reply	[flat|nested] only message in thread

only message in thread, other threads:[~2019-10-31 22:21 UTC | newest]

Thread overview: (only message) (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-10-31 22:20 Possible to configure nvme irq cpu affinity? Jeffrey Baker

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).