* [PATCH 0/4][RFC v2] x86/irq: Spread vectors on different CPUs
@ 2017-09-01 5:03 Chen Yu
2017-09-01 5:03 ` [PATCH 1/4][RFC v2] x86/apic: Extend the defination for vector_irq Chen Yu
` (3 more replies)
0 siblings, 4 replies; 21+ messages in thread
From: Chen Yu @ 2017-09-01 5:03 UTC (permalink / raw)
To: x86
Cc: Thomas Gleixner, Ingo Molnar, H. Peter Anvin, Rui Zhang,
linux-kernel, Chen Yu
This patch set tries to spread the vectors assigned on different
CPUs as much as possible. The requirement to do this came from
a hibernation issue encountered on a vector-heavily-used system
which has many multi queue devices on that, the CPUs failed to be
brought offline due to insufficient free vector slots on the
remaining CPUs. In theory this problem should be fixed in lastest
kernel, because the managed interrupt mechanism for multi queue
devices, which no longer migrates these interrupts. It simply shuts
them down and restarts them when the CPU comes back online. However
there is still one corner case that the solution mentioned above
does not cover, so this patch set tries to address this issue
by spreading the vectors for that case too.
The previous discussion is here:
https://patchwork.kernel.org/patch/9725227/
Thomas has given many useful suggestion on this, and this patch set
tries to address them one-by-one, please refer to the commit
description in '[PATCH 4/4] Spread the vectors by choosing the idlest
CPU' for detail.
Chen Yu (4):
x86/apic: Extend the defination for vector_irq
x86/apic: Record the number of vectors assigned on a CPU
x86/apic: Introduce the per vector cpumask array
x86/apic: Spread the vectors by choosing the idlest CPU
arch/x86/include/asm/hw_irq.h | 16 +++++-
arch/x86/kernel/apic/vector.c | 129 +++++++++++++++++++++++++++++++++++++-----
arch/x86/kernel/irq.c | 21 ++++---
arch/x86/kernel/irqinit.c | 10 ++--
arch/x86/lguest/boot.c | 3 +-
5 files changed, 148 insertions(+), 31 deletions(-)
--
2.7.4
^ permalink raw reply [flat|nested] 21+ messages in thread
* [PATCH 1/4][RFC v2] x86/apic: Extend the defination for vector_irq
2017-09-01 5:03 [PATCH 0/4][RFC v2] x86/irq: Spread vectors on different CPUs Chen Yu
@ 2017-09-01 5:03 ` Chen Yu
2017-09-01 5:04 ` [PATCH 2/4][RFC v2] x86/apic: Record the number of vectors assigned on a CPU Chen Yu
` (2 subsequent siblings)
3 siblings, 0 replies; 21+ messages in thread
From: Chen Yu @ 2017-09-01 5:03 UTC (permalink / raw)
To: x86
Cc: Thomas Gleixner, Ingo Molnar, H. Peter Anvin, Rui Zhang,
linux-kernel, Chen Yu, Rafael J. Wysocki, Len Brown,
Dan Williams
Introduce a variable inside the vector_irq, to record the number
of vectors assigned per CPU. This is to prepare for the vector
spreading work.
No functional change.
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: "Rafael J. Wysocki" <rjw@rjwysocki.net>
Cc: Len Brown <lenb@kernel.org>
Cc: Dan Williams <dan.j.williams@intel.com>
Signed-off-by: Chen Yu <yu.c.chen@intel.com>
---
arch/x86/include/asm/hw_irq.h | 8 ++++++--
arch/x86/kernel/apic/vector.c | 25 +++++++++++++------------
arch/x86/kernel/irq.c | 18 +++++++++---------
arch/x86/kernel/irqinit.c | 9 +++++----
arch/x86/lguest/boot.c | 2 +-
5 files changed, 34 insertions(+), 28 deletions(-)
diff --git a/arch/x86/include/asm/hw_irq.h b/arch/x86/include/asm/hw_irq.h
index d6dbafb..b2243fe 100644
--- a/arch/x86/include/asm/hw_irq.h
+++ b/arch/x86/include/asm/hw_irq.h
@@ -179,8 +179,12 @@ extern char irq_entries_start[];
#define VECTOR_UNUSED NULL
#define VECTOR_RETRIGGERED ((void *)~0UL)
-typedef struct irq_desc* vector_irq_t[NR_VECTORS];
-DECLARE_PER_CPU(vector_irq_t, vector_irq);
+struct vector_irq_info {
+ struct irq_desc *desc[NR_VECTORS];
+ int alloc;
+};
+
+DECLARE_PER_CPU(struct vector_irq_info, vector_irq);
#endif /* !ASSEMBLY_ */
diff --git a/arch/x86/kernel/apic/vector.c b/arch/x86/kernel/apic/vector.c
index b3af457..2ce1021 100644
--- a/arch/x86/kernel/apic/vector.c
+++ b/arch/x86/kernel/apic/vector.c
@@ -179,7 +179,7 @@ static int __assign_irq_vector(int irq, struct apic_chip_data *d,
goto next;
for_each_cpu(new_cpu, vector_searchmask) {
- if (!IS_ERR_OR_NULL(per_cpu(vector_irq, new_cpu)[vector]))
+ if (!IS_ERR_OR_NULL(per_cpu(vector_irq, new_cpu).desc[vector]))
goto next;
}
/* Found one! */
@@ -189,7 +189,8 @@ static int __assign_irq_vector(int irq, struct apic_chip_data *d,
if (d->cfg.vector)
cpumask_copy(d->old_domain, d->domain);
for_each_cpu(new_cpu, vector_searchmask)
- per_cpu(vector_irq, new_cpu)[vector] = irq_to_desc(irq);
+ per_cpu(vector_irq, new_cpu).desc[vector] = irq_to_desc(irq);
+
goto update;
next_cpu:
@@ -268,7 +269,7 @@ static void clear_irq_vector(int irq, struct apic_chip_data *data)
vector = data->cfg.vector;
for_each_cpu_and(cpu, data->domain, cpu_online_mask)
- per_cpu(vector_irq, cpu)[vector] = VECTOR_UNUSED;
+ per_cpu(vector_irq, cpu).desc[vector] = VECTOR_UNUSED;
data->cfg.vector = 0;
cpumask_clear(data->domain);
@@ -285,9 +286,9 @@ static void clear_irq_vector(int irq, struct apic_chip_data *data)
for_each_cpu_and(cpu, data->old_domain, cpu_online_mask) {
for (vector = FIRST_EXTERNAL_VECTOR; vector < NR_VECTORS;
vector++) {
- if (per_cpu(vector_irq, cpu)[vector] != desc)
+ if (per_cpu(vector_irq, cpu).desc[vector] != desc)
continue;
- per_cpu(vector_irq, cpu)[vector] = VECTOR_UNUSED;
+ per_cpu(vector_irq, cpu).desc[vector] = VECTOR_UNUSED;
break;
}
}
@@ -481,17 +482,17 @@ static void __setup_vector_irq(int cpu)
if (!data || !cpumask_test_cpu(cpu, data->domain))
continue;
vector = data->cfg.vector;
- per_cpu(vector_irq, cpu)[vector] = desc;
+ per_cpu(vector_irq, cpu).desc[vector] = desc;
}
/* Mark the free vectors */
for (vector = 0; vector < NR_VECTORS; ++vector) {
- desc = per_cpu(vector_irq, cpu)[vector];
+ desc = per_cpu(vector_irq, cpu).desc[vector];
if (IS_ERR_OR_NULL(desc))
continue;
data = apic_chip_data(irq_desc_get_irq_data(desc));
if (!cpumask_test_cpu(cpu, data->domain))
- per_cpu(vector_irq, cpu)[vector] = VECTOR_UNUSED;
+ per_cpu(vector_irq, cpu).desc[vector] = VECTOR_UNUSED;
}
}
@@ -511,7 +512,7 @@ void setup_vector_irq(int cpu)
* legacy vector to irq mapping:
*/
for (irq = 0; irq < nr_legacy_irqs(); irq++)
- per_cpu(vector_irq, cpu)[ISA_IRQ_VECTOR(irq)] = irq_to_desc(irq);
+ per_cpu(vector_irq, cpu).desc[ISA_IRQ_VECTOR(irq)] = irq_to_desc(irq);
__setup_vector_irq(cpu);
}
@@ -596,7 +597,7 @@ asmlinkage __visible void __irq_entry smp_irq_move_cleanup_interrupt(void)
unsigned int irr;
retry:
- desc = __this_cpu_read(vector_irq[vector]);
+ desc = __this_cpu_read(vector_irq.desc[vector]);
if (IS_ERR_OR_NULL(desc))
continue;
@@ -647,7 +648,7 @@ asmlinkage __visible void __irq_entry smp_irq_move_cleanup_interrupt(void)
apic->send_IPI_self(IRQ_MOVE_CLEANUP_VECTOR);
goto unlock;
}
- __this_cpu_write(vector_irq[vector], VECTOR_UNUSED);
+ __this_cpu_write(vector_irq.desc[vector], VECTOR_UNUSED);
cpumask_clear_cpu(me, data->old_domain);
unlock:
raw_spin_unlock(&desc->lock);
@@ -781,7 +782,7 @@ void irq_force_complete_move(struct irq_desc *desc)
* descriptor set in their vector array. Clean it up.
*/
for_each_cpu(cpu, data->old_domain)
- per_cpu(vector_irq, cpu)[cfg->old_vector] = VECTOR_UNUSED;
+ per_cpu(vector_irq, cpu).desc[cfg->old_vector] = VECTOR_UNUSED;
/* Cleanup the left overs of the (half finished) move */
cpumask_clear(data->old_domain);
diff --git a/arch/x86/kernel/irq.c b/arch/x86/kernel/irq.c
index 4ed0aba..ae11e86 100644
--- a/arch/x86/kernel/irq.c
+++ b/arch/x86/kernel/irq.c
@@ -239,7 +239,7 @@ __visible unsigned int __irq_entry do_IRQ(struct pt_regs *regs)
/* entering_irq() tells RCU that we're not quiescent. Check it. */
RCU_LOCKDEP_WARN(!rcu_is_watching(), "IRQ failed to wake up RCU");
- desc = __this_cpu_read(vector_irq[vector]);
+ desc = __this_cpu_read(vector_irq.desc[vector]);
if (!handle_irq(desc, regs)) {
ack_APIC_irq();
@@ -249,7 +249,7 @@ __visible unsigned int __irq_entry do_IRQ(struct pt_regs *regs)
__func__, smp_processor_id(),
vector);
} else {
- __this_cpu_write(vector_irq[vector], VECTOR_UNUSED);
+ __this_cpu_write(vector_irq.desc[vector], VECTOR_UNUSED);
}
}
@@ -375,7 +375,7 @@ int check_irq_vectors_for_cpu_disable(void)
this_count = 0;
for (vector = FIRST_EXTERNAL_VECTOR; vector < NR_VECTORS; vector++) {
- desc = __this_cpu_read(vector_irq[vector]);
+ desc = __this_cpu_read(vector_irq.desc[vector]);
if (IS_ERR_OR_NULL(desc))
continue;
/*
@@ -433,7 +433,7 @@ int check_irq_vectors_for_cpu_disable(void)
for (vector = FIRST_EXTERNAL_VECTOR;
vector < first_system_vector; vector++) {
if (!test_bit(vector, used_vectors) &&
- IS_ERR_OR_NULL(per_cpu(vector_irq, cpu)[vector])) {
+ IS_ERR_OR_NULL(per_cpu(vector_irq, cpu).desc[vector])) {
if (++count == this_count)
return 0;
}
@@ -475,24 +475,24 @@ void fixup_irqs(void)
* nothing else will touch it.
*/
for (vector = FIRST_EXTERNAL_VECTOR; vector < NR_VECTORS; vector++) {
- if (IS_ERR_OR_NULL(__this_cpu_read(vector_irq[vector])))
+ if (IS_ERR_OR_NULL(__this_cpu_read(vector_irq.desc[vector])))
continue;
irr = apic_read(APIC_IRR + (vector / 32 * 0x10));
if (irr & (1 << (vector % 32))) {
- desc = __this_cpu_read(vector_irq[vector]);
+ desc = __this_cpu_read(vector_irq.desc[vector]);
raw_spin_lock(&desc->lock);
data = irq_desc_get_irq_data(desc);
chip = irq_data_get_irq_chip(data);
if (chip->irq_retrigger) {
chip->irq_retrigger(data);
- __this_cpu_write(vector_irq[vector], VECTOR_RETRIGGERED);
+ __this_cpu_write(vector_irq.desc[vector], VECTOR_RETRIGGERED);
}
raw_spin_unlock(&desc->lock);
}
- if (__this_cpu_read(vector_irq[vector]) != VECTOR_RETRIGGERED)
- __this_cpu_write(vector_irq[vector], VECTOR_UNUSED);
+ if (__this_cpu_read(vector_irq.desc[vector]) != VECTOR_RETRIGGERED)
+ __this_cpu_write(vector_irq.desc[vector], VECTOR_UNUSED);
}
}
#endif
diff --git a/arch/x86/kernel/irqinit.c b/arch/x86/kernel/irqinit.c
index c7fd185..734b54f 100644
--- a/arch/x86/kernel/irqinit.c
+++ b/arch/x86/kernel/irqinit.c
@@ -51,8 +51,9 @@ static struct irqaction irq2 = {
.flags = IRQF_NO_THREAD,
};
-DEFINE_PER_CPU(vector_irq_t, vector_irq) = {
- [0 ... NR_VECTORS - 1] = VECTOR_UNUSED,
+DEFINE_PER_CPU(struct vector_irq_info, vector_irq) = {
+ .desc[0 ... NR_VECTORS - 1] = VECTOR_UNUSED,
+ .alloc = 0,
};
int vector_used_by_percpu_irq(unsigned int vector)
@@ -60,7 +61,7 @@ int vector_used_by_percpu_irq(unsigned int vector)
int cpu;
for_each_online_cpu(cpu) {
- if (!IS_ERR_OR_NULL(per_cpu(vector_irq, cpu)[vector]))
+ if (!IS_ERR_OR_NULL(per_cpu(vector_irq, cpu).desc[vector]))
return 1;
}
@@ -94,7 +95,7 @@ void __init init_IRQ(void)
* irq's migrate etc.
*/
for (i = 0; i < nr_legacy_irqs(); i++)
- per_cpu(vector_irq, 0)[ISA_IRQ_VECTOR(i)] = irq_to_desc(i);
+ per_cpu(vector_irq, 0).desc[ISA_IRQ_VECTOR(i)] = irq_to_desc(i);
x86_init.irqs.intr_init();
}
diff --git a/arch/x86/lguest/boot.c b/arch/x86/lguest/boot.c
index 9947269..e80758a 100644
--- a/arch/x86/lguest/boot.c
+++ b/arch/x86/lguest/boot.c
@@ -846,7 +846,7 @@ static int lguest_setup_irq(unsigned int irq)
/* Some systems map "vectors" to interrupts weirdly. Not us! */
desc = irq_to_desc(irq);
- __this_cpu_write(vector_irq[FIRST_EXTERNAL_VECTOR + irq], desc);
+ __this_cpu_write(vector_irq.desc[FIRST_EXTERNAL_VECTOR + irq], desc);
return 0;
}
--
2.7.4
^ permalink raw reply related [flat|nested] 21+ messages in thread
* [PATCH 2/4][RFC v2] x86/apic: Record the number of vectors assigned on a CPU
2017-09-01 5:03 [PATCH 0/4][RFC v2] x86/irq: Spread vectors on different CPUs Chen Yu
2017-09-01 5:03 ` [PATCH 1/4][RFC v2] x86/apic: Extend the defination for vector_irq Chen Yu
@ 2017-09-01 5:04 ` Chen Yu
2017-09-01 5:04 ` [PATCH 3/4] x86/apic: Introduce the per vector cpumask array Chen Yu
2017-09-01 5:04 ` [PATCH 4/4][RFC v2] x86/apic: Spread the vectors by choosing the idlest CPU Chen Yu
3 siblings, 0 replies; 21+ messages in thread
From: Chen Yu @ 2017-09-01 5:04 UTC (permalink / raw)
To: x86
Cc: Thomas Gleixner, Ingo Molnar, H. Peter Anvin, Rui Zhang,
linux-kernel, Chen Yu, Rafael J. Wysocki, Len Brown,
Dan Williams
Update the number of vectors assigned on each CPU during
vector allocation/free process. This is to prepare for
the vector spreading work that, we can find out the CPU
with least vectors assigned.
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: "Rafael J. Wysocki" <rjw@rjwysocki.net>
Cc: Len Brown <lenb@kernel.org>
Cc: Dan Williams <dan.j.williams@intel.com>
Signed-off-by: Chen Yu <yu.c.chen@intel.com>
---
arch/x86/include/asm/hw_irq.h | 8 ++++++++
arch/x86/kernel/apic/vector.c | 45 ++++++++++++++++++++++++++++++++++++++++++-
arch/x86/kernel/irq.c | 5 ++++-
arch/x86/kernel/irqinit.c | 1 +
arch/x86/lguest/boot.c | 1 +
5 files changed, 58 insertions(+), 2 deletions(-)
diff --git a/arch/x86/include/asm/hw_irq.h b/arch/x86/include/asm/hw_irq.h
index b2243fe..d1b3c61 100644
--- a/arch/x86/include/asm/hw_irq.h
+++ b/arch/x86/include/asm/hw_irq.h
@@ -151,6 +151,10 @@ extern struct irq_cfg *irqd_cfg(struct irq_data *irq_data);
extern void lock_vector_lock(void);
extern void unlock_vector_lock(void);
extern void setup_vector_irq(int cpu);
+extern void inc_vector_alloc(const struct cpumask *mask,
+ int count);
+extern void dec_vector_alloc(const struct cpumask *mask,
+ int count);
#ifdef CONFIG_SMP
extern void send_cleanup_vector(struct irq_cfg *);
extern void irq_complete_move(struct irq_cfg *cfg);
@@ -163,6 +167,10 @@ extern void apic_ack_edge(struct irq_data *data);
#else /* CONFIG_X86_LOCAL_APIC */
static inline void lock_vector_lock(void) {}
static inline void unlock_vector_lock(void) {}
+static inline void inc_vector_alloc(const struct cpumask *mask,
+ int count) {}
+static inline void dec_vector_alloc(const struct cpumask *mask,
+ int count) {}
#endif /* CONFIG_X86_LOCAL_APIC */
/* Statistics */
diff --git a/arch/x86/kernel/apic/vector.c b/arch/x86/kernel/apic/vector.c
index 2ce1021..4ff84c0 100644
--- a/arch/x86/kernel/apic/vector.c
+++ b/arch/x86/kernel/apic/vector.c
@@ -50,6 +50,36 @@ void unlock_vector_lock(void)
raw_spin_unlock(&vector_lock);
}
+static void update_vectors_alloc(const struct cpumask *mask,
+ int count, bool add)
+{
+ int cpu;
+
+ for_each_cpu(cpu, mask) {
+ int cur_alloc = per_cpu(vector_irq, cpu).alloc;
+
+ /* Update the number of vectors assigned on this CPU. */
+ if (add && (cur_alloc + count <= NR_VECTORS))
+ per_cpu(vector_irq, cpu).alloc += count;
+ else if (!add && cur_alloc >= count)
+ per_cpu(vector_irq, cpu).alloc -= count;
+ else
+ continue;
+ }
+}
+
+void inc_vector_alloc(const struct cpumask *mask,
+ int count)
+{
+ update_vectors_alloc(mask, count, true);
+}
+
+void dec_vector_alloc(const struct cpumask *mask,
+ int count)
+{
+ update_vectors_alloc(mask, count, false);
+}
+
static struct apic_chip_data *apic_chip_data(struct irq_data *irq_data)
{
if (!irq_data)
@@ -191,6 +221,7 @@ static int __assign_irq_vector(int irq, struct apic_chip_data *d,
for_each_cpu(new_cpu, vector_searchmask)
per_cpu(vector_irq, new_cpu).desc[vector] = irq_to_desc(irq);
+ inc_vector_alloc(vector_searchmask, 1);
goto update;
next_cpu:
@@ -263,6 +294,7 @@ static void clear_irq_vector(int irq, struct apic_chip_data *data)
{
struct irq_desc *desc;
int cpu, vector;
+ struct cpumask mask;
if (!data->cfg.vector)
return;
@@ -271,6 +303,9 @@ static void clear_irq_vector(int irq, struct apic_chip_data *data)
for_each_cpu_and(cpu, data->domain, cpu_online_mask)
per_cpu(vector_irq, cpu).desc[vector] = VECTOR_UNUSED;
+ cpumask_and(&mask, data->domain, cpu_online_mask);
+ dec_vector_alloc(&mask, 1);
+
data->cfg.vector = 0;
cpumask_clear(data->domain);
@@ -289,6 +324,7 @@ static void clear_irq_vector(int irq, struct apic_chip_data *data)
if (per_cpu(vector_irq, cpu).desc[vector] != desc)
continue;
per_cpu(vector_irq, cpu).desc[vector] = VECTOR_UNUSED;
+ dec_vector_alloc(cpumask_of(cpu), 1);
break;
}
}
@@ -483,6 +519,7 @@ static void __setup_vector_irq(int cpu)
continue;
vector = data->cfg.vector;
per_cpu(vector_irq, cpu).desc[vector] = desc;
+ inc_vector_alloc(cpumask_of(cpu), 1);
}
/* Mark the free vectors */
for (vector = 0; vector < NR_VECTORS; ++vector) {
@@ -491,8 +528,10 @@ static void __setup_vector_irq(int cpu)
continue;
data = apic_chip_data(irq_desc_get_irq_data(desc));
- if (!cpumask_test_cpu(cpu, data->domain))
+ if (!cpumask_test_cpu(cpu, data->domain)) {
per_cpu(vector_irq, cpu).desc[vector] = VECTOR_UNUSED;
+ dec_vector_alloc(cpumask_of(cpu), 1);
+ }
}
}
@@ -514,6 +553,7 @@ void setup_vector_irq(int cpu)
for (irq = 0; irq < nr_legacy_irqs(); irq++)
per_cpu(vector_irq, cpu).desc[ISA_IRQ_VECTOR(irq)] = irq_to_desc(irq);
+ inc_vector_alloc(cpumask_of(cpu), irq);
__setup_vector_irq(cpu);
}
@@ -649,6 +689,7 @@ asmlinkage __visible void __irq_entry smp_irq_move_cleanup_interrupt(void)
goto unlock;
}
__this_cpu_write(vector_irq.desc[vector], VECTOR_UNUSED);
+ dec_vector_alloc(cpumask_of(me), 1);
cpumask_clear_cpu(me, data->old_domain);
unlock:
raw_spin_unlock(&desc->lock);
@@ -784,6 +825,8 @@ void irq_force_complete_move(struct irq_desc *desc)
for_each_cpu(cpu, data->old_domain)
per_cpu(vector_irq, cpu).desc[cfg->old_vector] = VECTOR_UNUSED;
+ dec_vector_alloc(data->old_domain, 1);
+
/* Cleanup the left overs of the (half finished) move */
cpumask_clear(data->old_domain);
data->move_in_progress = 0;
diff --git a/arch/x86/kernel/irq.c b/arch/x86/kernel/irq.c
index ae11e86..67c01b8 100644
--- a/arch/x86/kernel/irq.c
+++ b/arch/x86/kernel/irq.c
@@ -250,6 +250,7 @@ __visible unsigned int __irq_entry do_IRQ(struct pt_regs *regs)
vector);
} else {
__this_cpu_write(vector_irq.desc[vector], VECTOR_UNUSED);
+ inc_vector_alloc(cpumask_of(smp_processor_id()), 1);
}
}
@@ -491,8 +492,10 @@ void fixup_irqs(void)
}
raw_spin_unlock(&desc->lock);
}
- if (__this_cpu_read(vector_irq.desc[vector]) != VECTOR_RETRIGGERED)
+ if (__this_cpu_read(vector_irq.desc[vector]) != VECTOR_RETRIGGERED) {
__this_cpu_write(vector_irq.desc[vector], VECTOR_UNUSED);
+ dec_vector_alloc(cpumask_of(smp_processor_id()), 1);
+ }
}
}
#endif
diff --git a/arch/x86/kernel/irqinit.c b/arch/x86/kernel/irqinit.c
index 734b54f..dd618c1 100644
--- a/arch/x86/kernel/irqinit.c
+++ b/arch/x86/kernel/irqinit.c
@@ -97,6 +97,7 @@ void __init init_IRQ(void)
for (i = 0; i < nr_legacy_irqs(); i++)
per_cpu(vector_irq, 0).desc[ISA_IRQ_VECTOR(i)] = irq_to_desc(i);
+ inc_vector_alloc(cpumask_of(1), i);
x86_init.irqs.intr_init();
}
diff --git a/arch/x86/lguest/boot.c b/arch/x86/lguest/boot.c
index e80758a..0696354 100644
--- a/arch/x86/lguest/boot.c
+++ b/arch/x86/lguest/boot.c
@@ -847,6 +847,7 @@ static int lguest_setup_irq(unsigned int irq)
/* Some systems map "vectors" to interrupts weirdly. Not us! */
desc = irq_to_desc(irq);
__this_cpu_write(vector_irq.desc[FIRST_EXTERNAL_VECTOR + irq], desc);
+ inc_vector_alloc(cpumask_of(smp_processor_id()), 1);
return 0;
}
--
2.7.4
^ permalink raw reply related [flat|nested] 21+ messages in thread
* [PATCH 3/4] x86/apic: Introduce the per vector cpumask array
2017-09-01 5:03 [PATCH 0/4][RFC v2] x86/irq: Spread vectors on different CPUs Chen Yu
2017-09-01 5:03 ` [PATCH 1/4][RFC v2] x86/apic: Extend the defination for vector_irq Chen Yu
2017-09-01 5:04 ` [PATCH 2/4][RFC v2] x86/apic: Record the number of vectors assigned on a CPU Chen Yu
@ 2017-09-01 5:04 ` Chen Yu
2017-09-01 5:04 ` [PATCH 4/4][RFC v2] x86/apic: Spread the vectors by choosing the idlest CPU Chen Yu
3 siblings, 0 replies; 21+ messages in thread
From: Chen Yu @ 2017-09-01 5:04 UTC (permalink / raw)
To: x86
Cc: Thomas Gleixner, Ingo Molnar, H. Peter Anvin, Rui Zhang,
linux-kernel, Chen Yu, Rafael J. Wysocki, Len Brown,
Dan Williams
This array is a bitmap for sorting the number of vectors
assigned on each CPU, thus to quickly retrieve the CPU
which have assigned the least amount of vectors, then
choose that CPU as a hint to spread the vectors among
multiple CPUs.
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: "Rafael J. Wysocki" <rjw@rjwysocki.net>
Cc: Len Brown <lenb@kernel.org>
Cc: Dan Williams <dan.j.williams@intel.com>
Signed-off-by: Chen Yu <yu.c.chen@intel.com>
---
arch/x86/kernel/apic/vector.c | 33 +++++++++++++++++++++++++++++++++
1 file changed, 33 insertions(+)
diff --git a/arch/x86/kernel/apic/vector.c b/arch/x86/kernel/apic/vector.c
index 4ff84c0..b60cc66 100644
--- a/arch/x86/kernel/apic/vector.c
+++ b/arch/x86/kernel/apic/vector.c
@@ -32,6 +32,16 @@ struct irq_domain *x86_vector_domain;
EXPORT_SYMBOL_GPL(x86_vector_domain);
static DEFINE_RAW_SPINLOCK(vector_lock);
static cpumask_var_t vector_cpumask, vector_searchmask, searched_cpumask;
+/*
+ * vectors_alloc_mask[i] records the CPUs which have assigned i vectors each.
+ * For example, let vectors_alloc_mask[3] = cpumask_or(cpumask_of(8), cpumask_of(9))
+ * which means that:
+ * On CPU8 and CPU9, the number of vectors been assigned is 3.
+ * This bitmap can be used to quickly find out the CPUs who has allocated the least
+ * number of vectors, by checking from vectors_alloc_mask[0] to vectors_alloc_mask[255]
+ * until a non-zero value is found. These CPUs are chosen for vector spreading.
+ */
+static cpumask_var_t vectors_alloc_mask[NR_VECTORS];
static struct irq_chip lapic_controller;
#ifdef CONFIG_X86_IO_APIC
static struct apic_chip_data *legacy_irq_data[NR_IRQS_LEGACY];
@@ -65,6 +75,9 @@ static void update_vectors_alloc(const struct cpumask *mask,
per_cpu(vector_irq, cpu).alloc -= count;
else
continue;
+ /* Update the position for this CPU. */
+ cpumask_clear_cpu(cpu, vectors_alloc_mask[cur_alloc]);
+ cpumask_set_cpu(cpu, vectors_alloc_mask[per_cpu(vector_irq, cpu).alloc]);
}
}
@@ -479,6 +492,25 @@ static void __init init_legacy_irqs(void)
static inline void init_legacy_irqs(void) { }
#endif
+static int __init alloc_assigned_vectors_mask(void)
+{
+ int i, ret;
+
+ for (i = 0; i < NR_VECTORS; i++) {
+ ret = alloc_cpumask_var(&vectors_alloc_mask[i], GFP_KERNEL);
+ if (!ret)
+ goto free_mask;
+ }
+ /* Initially all the CPUs have 0 vector assigned. */
+ cpumask_setall(vectors_alloc_mask[0]);
+ return 0;
+
+ free_mask:
+ while (i--)
+ free_cpumask_var(vectors_alloc_mask[i]);
+ return -ENOMEM;
+}
+
int __init arch_early_irq_init(void)
{
struct fwnode_handle *fn;
@@ -499,6 +531,7 @@ int __init arch_early_irq_init(void)
BUG_ON(!alloc_cpumask_var(&vector_cpumask, GFP_KERNEL));
BUG_ON(!alloc_cpumask_var(&vector_searchmask, GFP_KERNEL));
BUG_ON(!alloc_cpumask_var(&searched_cpumask, GFP_KERNEL));
+ BUG_ON(alloc_assigned_vectors_mask());
return arch_early_ioapic_init();
}
--
2.7.4
^ permalink raw reply related [flat|nested] 21+ messages in thread
* [PATCH 4/4][RFC v2] x86/apic: Spread the vectors by choosing the idlest CPU
2017-09-01 5:03 [PATCH 0/4][RFC v2] x86/irq: Spread vectors on different CPUs Chen Yu
` (2 preceding siblings ...)
2017-09-01 5:04 ` [PATCH 3/4] x86/apic: Introduce the per vector cpumask array Chen Yu
@ 2017-09-01 5:04 ` Chen Yu
2017-09-03 18:17 ` Thomas Gleixner
3 siblings, 1 reply; 21+ messages in thread
From: Chen Yu @ 2017-09-01 5:04 UTC (permalink / raw)
To: x86
Cc: Thomas Gleixner, Ingo Molnar, H. Peter Anvin, Rui Zhang,
linux-kernel, Chen Yu, Rafael J. Wysocki, Len Brown,
Dan Williams
This is the major logic to spread the vectors on different CPUs.
The main idea is to choose the 'idlest' CPU which has assigned
the least number of vectors as the candidate/hint for the vector
allocation domain, in the hope that the vector allocation domain
could leverage this hint to generate corresponding cpumask.
One of the requirements to do this vector spreading work comes from the
following hibernation problem found on a 16 cores server:
CPU 31 disable failed: CPU has 62 vectors assigned and there
are only 0 available.
2 issues were found after investigation:
1. The network driver has declared many vector resources via
pci_enable_msix_range(), say, this driver might likely want
to reserve 6 per logical CPU, then there would be 192 of them.
2. Besides, most of the vectors reserved by this driver are assigned
on CPU0 due to the current code strategy, so there would be
insufficient slots left on CPU0 to receive any migrated IRQs
during CPU offine.
In theory after the commit c5cb83bb337c ("genirq/cpuhotplug: Handle
managed IRQs on CPU hotplug") the issue 1 has been solved with the
managed interrupt mechanism for multi queue devices, which no longer
migrate these interrupts. It simply shuts them down and restarts
them when the CPU comes back online.
However, according to the implementation of this network driver,
the commit mentioned above does not fully fix the issue 1.
Here's the framework of the network driver:
step 1. Reserved enough irq vectors and corresponding IRQs.
step 2. If the network is activated, invoke request_irq() to
register the handler.
step 3. Invoke set_affinity() to spread the IRQs onto different
CPUs, thus to spread the vectors too.
The problem is, if the network cable is not connected, step 2
and 3 will not get invoked, thus the IRQ vectors will not spread
on different CPUs and will still be allocated on CPU0. As a result
the CPUs will still get the offline failure.
Previously there were some discussion in the thread [1] about the
vector spread, and here are the suggestion from Thomas:
Q1:
Rewrite that allocation code from scratch, use per cpu bitmaps,
so we can do fast search over masks and keep track of
the vectors which are used/free per cpu.
A1:
per cpu bitmap was onced considered but this might not be
as fast as the solution proposed here. That is, if we introduce the
per cpu bitmap, we need to count the number of vectors assigned on
each CPUs by cpumask_weight() and sort them in order to get the
CPUs who have the least number of vectors assigned. By contrast,
if we introduce the bitmaps for each vector number, we can get the
'idle' CPU at the cost of nearly O(1).
One scenario of this patch can be illustrated below:
a) After initialization, vector_mask[0] is set with {CPU0...CPU31},
other vector_mask[i] remain zero, which means that, CPU0...CPU31
have zero vectors assigned.
b) During driver probe, CPU0 is chosen to be assigned with one vector.
Thus vector_mask[0] becomes {CPU1...CPU31}, and vector_mask[1] becomes
{CPU0}.
c) The next time the system tries to assign another vector, it searches from
vector[0], because it is non-empty, CPU1 is chosen to place
the vector. Thus vector_mask[0] becomes {CPU2...CPU31}, and vector_mask[1]
becomes {CPU0, CPU1}.
d) and so on, the vectors assignment will spread on different CPUs.
Q2:
"respect the allocation domains"
A2:
This solution provides a CPU as the param for the vector allocation domains,
in the hope that the domains can leverage this hint to generate the cpumask
with less vectors assigned. But yes, the CPU we select upfront gets fed into
apic->vector_allocation_domain() might just say no because the CPU
does not belong to it, but anyway this should somehow spread the vectors,
although I can not find out a graceful solution to this.
BTW, the vector allocation domain is default_vector_allocation_domain()
in our case, thus it works.
Q3:
"set a preference for the node on which the interrupt was allocated"
A3:
This can be achieved by further optimization in the following way:
a) Also record the number of vectors used per node.
b) Each time invoking __assign_irq_vector, adjust the input mask
by cpumask_and(mask, mask, node_mask_with_least_vectors())
[1] https://marc.info/?l=linux-kernel&m=149867663002202
or
[1] https://patchwork.kernel.org/patch/9725227/
Thanks for your time.
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: "Rafael J. Wysocki" <rjw@rjwysocki.net>
Cc: Len Brown <lenb@kernel.org>
Cc: Dan Williams <dan.j.williams@intel.com>
Signed-off-by: Chen Yu <yu.c.chen@intel.com>
---
arch/x86/kernel/apic/vector.c | 26 ++++++++++++++++++++++++--
1 file changed, 24 insertions(+), 2 deletions(-)
diff --git a/arch/x86/kernel/apic/vector.c b/arch/x86/kernel/apic/vector.c
index b60cc66..c7f0e8b 100644
--- a/arch/x86/kernel/apic/vector.c
+++ b/arch/x86/kernel/apic/vector.c
@@ -145,6 +145,28 @@ static void free_apic_chip_data(struct apic_chip_data *data)
}
}
+static int pick_next_cpu_hint(const struct cpumask *mask)
+{
+ int i;
+ struct cpumask search, tmp;
+
+ cpumask_and(&search, mask, cpu_online_mask);
+ /*
+ * Search in the order from vectors_alloc_mask[0] to
+ * vectors_alloc_mask[255], try to find an intersection
+ * between the mask and vectors_alloc_mask[],
+ * return the first CPU in this intersection, thus
+ * this CPU has the least number of vectors allocated.
+ */
+ for (i = 0; i < NR_VECTORS; i++) {
+ cpumask_and(&tmp, &search, vectors_alloc_mask[i]);
+ if (!cpumask_empty(&tmp))
+ return cpumask_first(&tmp);
+ }
+ /* This should not happened...*/
+ return cpumask_first_and(mask, cpu_online_mask);
+}
+
static int __assign_irq_vector(int irq, struct apic_chip_data *d,
const struct cpumask *mask,
struct irq_data *irqdata)
@@ -175,7 +197,7 @@ static int __assign_irq_vector(int irq, struct apic_chip_data *d,
/* Only try and allocate irqs on cpus that are present */
cpumask_clear(d->old_domain);
cpumask_clear(searched_cpumask);
- cpu = cpumask_first_and(mask, cpu_online_mask);
+ cpu = pick_next_cpu_hint(mask);
while (cpu < nr_cpu_ids) {
int new_cpu, offset;
@@ -247,7 +269,7 @@ static int __assign_irq_vector(int irq, struct apic_chip_data *d,
*/
cpumask_or(searched_cpumask, searched_cpumask, vector_cpumask);
cpumask_andnot(vector_cpumask, mask, searched_cpumask);
- cpu = cpumask_first_and(vector_cpumask, cpu_online_mask);
+ cpu = pick_next_cpu_hint(vector_cpumask);
continue;
}
return -ENOSPC;
--
2.7.4
^ permalink raw reply related [flat|nested] 21+ messages in thread
* Re: [PATCH 4/4][RFC v2] x86/apic: Spread the vectors by choosing the idlest CPU
2017-09-01 5:04 ` [PATCH 4/4][RFC v2] x86/apic: Spread the vectors by choosing the idlest CPU Chen Yu
@ 2017-09-03 18:17 ` Thomas Gleixner
2017-09-03 19:18 ` RFD: x86: Sanitize the vector allocator Thomas Gleixner
` (2 more replies)
0 siblings, 3 replies; 21+ messages in thread
From: Thomas Gleixner @ 2017-09-03 18:17 UTC (permalink / raw)
To: Chen Yu
Cc: x86, Ingo Molnar, H. Peter Anvin, Rui Zhang, LKML,
Rafael J. Wysocki, Len Brown, Dan Williams, Christoph Hellwig,
Peter Zijlstra
On Fri, 1 Sep 2017, Chen Yu wrote:
> This is the major logic to spread the vectors on different CPUs.
> The main idea is to choose the 'idlest' CPU which has assigned
> the least number of vectors as the candidate/hint for the vector
> allocation domain, in the hope that the vector allocation domain
> could leverage this hint to generate corresponding cpumask.
>
> One of the requirements to do this vector spreading work comes from the
> following hibernation problem found on a 16 cores server:
>
> CPU 31 disable failed: CPU has 62 vectors assigned and there
> are only 0 available.
>
> 2 issues were found after investigation:
>
> 1. The network driver has declared many vector resources via
> pci_enable_msix_range(), say, this driver might likely want
> to reserve 6 per logical CPU, then there would be 192 of them.
> 2. Besides, most of the vectors reserved by this driver are assigned
> on CPU0 due to the current code strategy, so there would be
> insufficient slots left on CPU0 to receive any migrated IRQs
> during CPU offine.
>
> In theory after the commit c5cb83bb337c ("genirq/cpuhotplug: Handle
> managed IRQs on CPU hotplug") the issue 1 has been solved with the
> managed interrupt mechanism for multi queue devices, which no longer
> migrate these interrupts. It simply shuts them down and restarts
> them when the CPU comes back online.
>
> However, according to the implementation of this network driver,
> the commit mentioned above does not fully fix the issue 1.
> Here's the framework of the network driver:
>
> step 1. Reserved enough irq vectors and corresponding IRQs.
> step 2. If the network is activated, invoke request_irq() to
> register the handler.
> step 3. Invoke set_affinity() to spread the IRQs onto different
> CPUs, thus to spread the vectors too.
>
> The problem is, if the network cable is not connected, step 2
> and 3 will not get invoked, thus the IRQ vectors will not spread
> on different CPUs and will still be allocated on CPU0. As a result
> the CPUs will still get the offline failure.
So the proper solution for this network driver is to switch to managed
interrupts instead of trying to work around it in some other place. It's
using the wrong mechanism - end of story.
Why are you insisting on implementing a band aid for this particular driver
instead of fixing the underlying problem of that driver which requires to
have 32 queues and interrupts open even if only a single CPU is online?
> Previously there were some discussion in the thread [1] about the
> vector spread, and here are the suggestion from Thomas:
>
> Q1:
> Rewrite that allocation code from scratch, use per cpu bitmaps,
> so we can do fast search over masks and keep track of
> the vectors which are used/free per cpu.
> A1:
> per cpu bitmap was onced considered but this might not be
> as fast as the solution proposed here. That is, if we introduce the
> per cpu bitmap, we need to count the number of vectors assigned on
> each CPUs by cpumask_weight() and sort them in order to get the
> CPUs who have the least number of vectors assigned. By contrast,
> if we introduce the bitmaps for each vector number, we can get the
> 'idle' CPU at the cost of nearly O(1).
The weight accounting with the cpumasks is an orthogonal issue to the per
cpu vector bitmaps.
Once you selected a CPU the current code still loops in circles instead of
just searching a bitmap.
And if the required affinity mask needs more than one CPU then this is
still not giving you the right answer. You assign perhaps a vector to the
least busy CPU, but the other CPUs which happen to be in the mask are going
to be randomly selected.
I'm not against making the vector allocation better, but certainly not by
adding yet more duct tape to something which is well known as one of the
dumbest search algorithms on the planet with a worst case of
nvectors * nr_online_cpus * nr_online_cpus_in_affinity_mask
and your mechanism nests another loop of potentially NR_VECTORS into
that. Which is pointless as the actual assignable vector space is
smaller. While x86 has 256 vectors the assignable vector space is way
smaller and non continous:
Vectors 0- 31 are reserved for CPU traps and exceptions
Vectors 239-255 are reserved by the kernel for IPIs, local timer etc.
Vector 32 can be reserved by the kernel for the reboot interrupt
Vector 96 can be reserved by the kernel for INT80
So the actually usable array size is between 205 and 207.
Aside of that the code is incorrect vs. the percpu accounting. Remove the
limit in the update function and see what happens. While the limit is
necessary in general, it should at least warn and yell whenever the
accounting goes out of bounds. And if that happens, then the whole thing is
completely useless simply because the numbers are just wrong.
> One scenario of this patch can be illustrated below:
>
> a) After initialization, vector_mask[0] is set with {CPU0...CPU31},
> other vector_mask[i] remain zero, which means that, CPU0...CPU31
> have zero vectors assigned.
> b) During driver probe, CPU0 is chosen to be assigned with one vector.
> Thus vector_mask[0] becomes {CPU1...CPU31}, and vector_mask[1] becomes
> {CPU0}.
> c) The next time the system tries to assign another vector, it searches from
> vector[0], because it is non-empty, CPU1 is chosen to place
> the vector. Thus vector_mask[0] becomes {CPU2...CPU31}, and vector_mask[1]
> becomes {CPU0, CPU1}.
> d) and so on, the vectors assignment will spread on different CPUs.
So this works for your particular network driver scenario, which is the
wrong example as this driver just needs to be reworked to not expose that
issue.
The reality of affinity requirements is not that simple. It's very much
device dependent and making it mandatory can have negative side effects.
> Q2:
> "respect the allocation domains"
> A2:
> This solution provides a CPU as the param for the vector allocation domains,
Solution? This is a crude hack which "solves" one particular problem in the
wrong way.
> in the hope that the domains can leverage this hint to generate the cpumask
Hope is never a good enginering principle.
> with less vectors assigned. But yes, the CPU we select upfront gets fed into
> apic->vector_allocation_domain() might just say no because the CPU
> does not belong to it, but anyway this should somehow spread the vectors,
> although I can not find out a graceful solution to this.
> BTW, the vector allocation domain is default_vector_allocation_domain()
> in our case, thus it works.
Now try that with the logical delivery modes which allow multi CPU
assignements and the cluster mode thereof.
> Q3:
> "set a preference for the node on which the interrupt was allocated"
> A3:
> This can be achieved by further optimization in the following way:
> a) Also record the number of vectors used per node.
> b) Each time invoking __assign_irq_vector, adjust the input mask
> by cpumask_and(mask, mask, node_mask_with_least_vectors())
I'm not looking forward to that heuristics and the next loop inside the
loops of loops.
So no, this is not going to happen. The current implementation sucks and
needs a rewrite.
I already started to look into that, but started at the low level end of
it. See the IDT cleanup series in tip:x86/apic. There is much more vooddo
in the vector management code to clean up before we can actually talk about
making the assignment algorithm smarter.
I'll send out a separate mail raising a few points to discuss before we are
actually starting to look into magic spreading functionality.
Thanks,
tglx
^ permalink raw reply [flat|nested] 21+ messages in thread
* RFD: x86: Sanitize the vector allocator
2017-09-03 18:17 ` Thomas Gleixner
@ 2017-09-03 19:18 ` Thomas Gleixner
2017-09-05 22:57 ` [PATCH 4/4][RFC v2] x86/apic: Spread the vectors by choosing the idlest CPU Thomas Gleixner
2017-09-06 4:13 ` Yu Chen
2 siblings, 0 replies; 21+ messages in thread
From: Thomas Gleixner @ 2017-09-03 19:18 UTC (permalink / raw)
To: Chen Yu
Cc: x86, Ingo Molnar, H. Peter Anvin, Rui Zhang, LKML,
Rafael J. Wysocki, Len Brown, Dan Williams, Christoph Hellwig,
Peter Zijlstra
[-- Attachment #1: Type: text/plain, Size: 6609 bytes --]
The vector allocator of x86 is a pretty stupid linear search algorithm with
a worst case of
nr_vectors * nr_online_cpus * nr_cpus_in_affinity mask
It has some other magic properties and really wants to be replaced by
something smarter.
That needs quite some cleanup of the vector management code outside of the
allocator, which I started to work on with the cleanup of the IDT
management which is headed for 4.14. I have some other things in the
pipeline which eliminate quite some duct tape in that area, but I ran into
a couple of interesting things:
1) Multi CPU affinities
This is only vailable when the APIC is using logical destination
mode. With physical destination mode there is already a restriction to a
single CPU target.
The multi CPU affinity is biased towards the CPU with the lowest APIC ID
in the destination bitfield. Only if that APIC is busy (ISR not empty)
then the next APIC gets it.
A full kernel build on a SKL 4 CPU desktop machine with affinity set to
CPU0-3 shows that more than 90 percent of the AHCI interrupts end up on
CPU0.
Aside of that the same cold build (right after boot) is about 2% faster
when the AHCI interrupt is only affine to CPU0.
I did some experiments on all my machines which have logical destination
mode with various workloads and the results are similiar. The
distribution of interrupts on the CPUs varies with the workloads, but
the vast majority always ends up on CPU0
I've not found a case where the multi CPU affinity is superiour. I might
have the wrong workloads and the wrong machines, but it would be
extremly helpful just to get rid of this and use single CPU affinities
only. That'd simplify the allocator along with the various APIC
implementations.
2) The 'priority level' spreading magic
The comment in __asign_irq_vector says:
* NOTE! The local APIC isn't very good at handling
* multiple interrupts at the same interrupt level.
* As the interrupt level is determined by taking the
* vector number and shifting that right by 4, we
* want to spread these out a bit so that they don't
* all fall in the same interrupt level.
After doing some palaeontological research I found the following in the
PPro Developer Manual Volume 3:
"7.4.2. Valid Interrupts
The local and I/O APICs support 240 distinct vectors in the range of 16
to 255. Interrupt priority is implied by its vector, according to the
following relationship: priority = vector / 16
One is the lowest priority and 15 is the highest. Vectors 16 through
31 are reserved for exclusive use by the processor. The remaining
vectors are for general use. The processor’s local APIC includes an
in-service entry and a holding entry for each priority level. To avoid
losing inter- rupts, software should allocate no more than 2 interrupt
vectors per priority."
The current SDM tells nothing about that, instead it states:
"If more than one interrupt is generated with the same vector number,
the local APIC can set the bit for the vector both in the IRR and the
ISR. This means that for the Pentium 4 and Intel Xeon processors, the
IRR and ISR can queue two interrupts for each interrupt vector: one
in the IRR and one in the ISR. Any additional interrupts issued for
the same interrupt vector are collapsed into the single bit in the
IRR.
For the P6 family and Pentium processors, the IRR and ISR registers
can queue no more than two interrupts per interrupt vector and will
reject other interrupts that are received within the same vector."
Which means, that on P6/Pentium the APIC will reject a new message and
tell the sender to retry, which increases the load on the APIC bus and
nothing more.
There is no affirmative answer from Intel on that, but I think it's sane
to remove that:
1) I've looked through a bunch of other operating systems and none of
them bothers to implement this or mentiones this at all.
2) The current allocator has no enforcement for this and especially the
legacy interrupts, which are the main source of interrupts on these
P6 and older systmes, are allocated linearly in the same priority
level and just work.
3) The current machines have no problem with that at all as I verified
with some experiments.
4) AMD at least confirmed that such an issue is unknown.
5) P6 and older are dinosaurs almost 20 years EOL, so we really should
not worry about that anymore.
So this can be eliminated, which makes the allocation mechanism way
simpler.
Some other issues which are not in the way of cleanups and replacements,
but need to be looked at as well:
1) Automated affinity assignment
This only helps when the underlying device requests it and has the
matching queues per CPU. That's what the managed interrupt affinity
mechanism was made for.
In other cases the automated assignment can have really bad effects.
On the same SKL as above I made the AHCI interrupt affine to CPU3 only
which makes the kernel build slower by whopping 10% than having it
affine on CPU0. Interestingly enough irqbalanced end up with the wrong
decision as well.
So we need to be very careful about that. It depends on the device and
the driver how good 'random' placement works.
That means we need hinting from the drivers about their preferred
allocation scheme. If we don't have that then we should for now default
to the current scheme which puts the interrupt on the node on which the
device is.
2) Vector waste
All 16 legacy interrupt vectors are populated at boot and stay there
forever whether they are used or not. On most modern machines that's 10+
vectors wasted for nothing. If the APIC uses logical destination mode
that means these vectors are per default allocated on up to 8 CPUs or in
the case of clustered X2APIC on all CPUs in a cluster.
It'd be worthwhile to allocate these legacy vectors dynamically when
they are actually used. That might fail, but that's the same on devices
which use MSI etc. For legacy systems this is a non issue as there are
plenty of vectors available. On modern machines the 4-5 really used
legacy vectors are requested early during the boot process and should
not end up in a fully exhausted vector space.
Nothing urgent, but worthwhile to fix I think.
Thoughts?
Thanks,
tglx
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH 4/4][RFC v2] x86/apic: Spread the vectors by choosing the idlest CPU
2017-09-03 18:17 ` Thomas Gleixner
2017-09-03 19:18 ` RFD: x86: Sanitize the vector allocator Thomas Gleixner
@ 2017-09-05 22:57 ` Thomas Gleixner
2017-09-06 4:34 ` Yu Chen
2017-09-06 4:13 ` Yu Chen
2 siblings, 1 reply; 21+ messages in thread
From: Thomas Gleixner @ 2017-09-05 22:57 UTC (permalink / raw)
To: Chen Yu
Cc: x86, Ingo Molnar, H. Peter Anvin, Rui Zhang, LKML,
Rafael J. Wysocki, Len Brown, Dan Williams, Christoph Hellwig,
Peter Zijlstra
On Sun, 3 Sep 2017, Thomas Gleixner wrote:
> On Fri, 1 Sep 2017, Chen Yu wrote:
>
> > This is the major logic to spread the vectors on different CPUs.
> > The main idea is to choose the 'idlest' CPU which has assigned
> > the least number of vectors as the candidate/hint for the vector
> > allocation domain, in the hope that the vector allocation domain
> > could leverage this hint to generate corresponding cpumask.
> >
> > One of the requirements to do this vector spreading work comes from the
> > following hibernation problem found on a 16 cores server:
> >
> > CPU 31 disable failed: CPU has 62 vectors assigned and there
> > are only 0 available.
Thinking more about this, this makes no sense whatsoever.
The total number of interrupts on a system is the same whether they are
all on CPU 0 or evenly spread over all CPUs.
As this machine is using physcial destination mode, the number of vectors
used is the same as the number of interrupts, except for the case where a
move of an interrupt is in progress and the interrupt which cleans up the
old vector has not yet arrived. Lets ignore that for now.
The available vector space is 204 per CPU on such a system.
256 - SYSTEM[0-31, 32, 128, 239-255] - LEGACY[50] = 204
> > CPU 31 disable failed: CPU has 62 vectors assigned and there
> > are only 0 available.
CPU31 is the last AP going offline (CPU0 is still online).
It wants to move 62 vectors to CPU0, but it can't because CPU0 has 0
available vectors. That means CPU0 has 204 vectors used. I doubt that, but
what I doubt even more is that this interrupt spreading helps in any way.
Assumed that we have a total of 204 + 62 = 266 device interrupt vectors in
use and they are evenly spread over 32 CPUs, so each CPU has either 8 or
nine vectors. Fine.
Now if you unplug all CPUs except CPU0 starting from CPU1 up to CPU31 then
at the point where CPU31 is about to be unplugged, CPU0 holds 133 vectors
and CPU31 holds 133 vectors as well - assumed that the spread is exactly
even.
I have a hard time to figure out how the 133 vectors on CPU31 are now
magically fitting in the empty space on CPU0, which is 204 - 133 = 71. In
my limited understanding of math 133 is greater than 71, but your patch
might make that magically be wrong.
Can you please provide detailed information about how many device
interrupts are actually in use/allocated on that system?
Please enable CONFIG_GENERIC_IRQ_DEBUGFS and provide the output of
# cat /sys/kernel/debug/irq/domains/*
and
# ls /sys/kernel/debug/irq/irqs
Thanks,
tglx
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH 4/4][RFC v2] x86/apic: Spread the vectors by choosing the idlest CPU
2017-09-03 18:17 ` Thomas Gleixner
2017-09-03 19:18 ` RFD: x86: Sanitize the vector allocator Thomas Gleixner
2017-09-05 22:57 ` [PATCH 4/4][RFC v2] x86/apic: Spread the vectors by choosing the idlest CPU Thomas Gleixner
@ 2017-09-06 4:13 ` Yu Chen
2017-09-06 6:15 ` Christoph Hellwig
2 siblings, 1 reply; 21+ messages in thread
From: Yu Chen @ 2017-09-06 4:13 UTC (permalink / raw)
To: Thomas Gleixner
Cc: x86, Ingo Molnar, H. Peter Anvin, Rui Zhang, LKML,
Rafael J. Wysocki, Len Brown, Dan Williams, Christoph Hellwig,
Peter Zijlstra
Thanks for looking at this,
(please bear my slow response as I need to check related code
before replying.)
On Sun, Sep 03, 2017 at 08:17:04PM +0200, Thomas Gleixner wrote:
> On Fri, 1 Sep 2017, Chen Yu wrote:
>
> > This is the major logic to spread the vectors on different CPUs.
> > The main idea is to choose the 'idlest' CPU which has assigned
> > the least number of vectors as the candidate/hint for the vector
> > allocation domain, in the hope that the vector allocation domain
> > could leverage this hint to generate corresponding cpumask.
> >
> > One of the requirements to do this vector spreading work comes from the
> > following hibernation problem found on a 16 cores server:
> >
> > CPU 31 disable failed: CPU has 62 vectors assigned and there
> > are only 0 available.
> >
> > 2 issues were found after investigation:
> >
> > 1. The network driver has declared many vector resources via
> > pci_enable_msix_range(), say, this driver might likely want
> > to reserve 6 per logical CPU, then there would be 192 of them.
> > 2. Besides, most of the vectors reserved by this driver are assigned
> > on CPU0 due to the current code strategy, so there would be
> > insufficient slots left on CPU0 to receive any migrated IRQs
> > during CPU offine.
> >
> > In theory after the commit c5cb83bb337c ("genirq/cpuhotplug: Handle
> > managed IRQs on CPU hotplug") the issue 1 has been solved with the
> > managed interrupt mechanism for multi queue devices, which no longer
> > migrate these interrupts. It simply shuts them down and restarts
> > them when the CPU comes back online.
> >
> > However, according to the implementation of this network driver,
> > the commit mentioned above does not fully fix the issue 1.
> > Here's the framework of the network driver:
> >
> > step 1. Reserved enough irq vectors and corresponding IRQs.
> > step 2. If the network is activated, invoke request_irq() to
> > register the handler.
> > step 3. Invoke set_affinity() to spread the IRQs onto different
> > CPUs, thus to spread the vectors too.
> >
> > The problem is, if the network cable is not connected, step 2
> > and 3 will not get invoked, thus the IRQ vectors will not spread
> > on different CPUs and will still be allocated on CPU0. As a result
> > the CPUs will still get the offline failure.
>
> So the proper solution for this network driver is to switch to managed
> interrupts instead of trying to work around it in some other place. It's
> using the wrong mechanism - end of story.
>
> Why are you insisting on implementing a band aid for this particular driver
> instead of fixing the underlying problem of that driver which requires to
> have 32 queues and interrupts open even if only a single CPU is online?
>
I agree, the driver could be rewritten, but it might take some time, so
meanwhile I'm looking at also other possible optimization.
> > Previously there were some discussion in the thread [1] about the
> > vector spread, and here are the suggestion from Thomas:
> >
> > Q1:
> > Rewrite that allocation code from scratch, use per cpu bitmaps,
> > so we can do fast search over masks and keep track of
> > the vectors which are used/free per cpu.
> > A1:
> > per cpu bitmap was onced considered but this might not be
> > as fast as the solution proposed here. That is, if we introduce the
> > per cpu bitmap, we need to count the number of vectors assigned on
> > each CPUs by cpumask_weight() and sort them in order to get the
> > CPUs who have the least number of vectors assigned. By contrast,
> > if we introduce the bitmaps for each vector number, we can get the
> > 'idle' CPU at the cost of nearly O(1).
>
> The weight accounting with the cpumasks is an orthogonal issue to the per
> cpu vector bitmaps.
>
> Once you selected a CPU the current code still loops in circles instead of
> just searching a bitmap.
Yes, I agree.
>
> And if the required affinity mask needs more than one CPU then this is
> still not giving you the right answer. You assign perhaps a vector to the
> least busy CPU, but the other CPUs which happen to be in the mask are going
> to be randomly selected.
Yes, for multi cpumask, it might depends on how vector domain behaves.
>
> I'm not against making the vector allocation better, but certainly not by
> adding yet more duct tape to something which is well known as one of the
> dumbest search algorithms on the planet with a worst case of
>
> nvectors * nr_online_cpus * nr_online_cpus_in_affinity_mask
>
> and your mechanism nests another loop of potentially NR_VECTORS into
> that. Which is pointless as the actual assignable vector space is
> smaller. While x86 has 256 vectors the assignable vector space is way
> smaller and non continous:
>
> Vectors 0- 31 are reserved for CPU traps and exceptions
> Vectors 239-255 are reserved by the kernel for IPIs, local timer etc.
> Vector 32 can be reserved by the kernel for the reboot interrupt
> Vector 96 can be reserved by the kernel for INT80
>
> So the actually usable array size is between 205 and 207.
>
> Aside of that the code is incorrect vs. the percpu accounting. Remove the
> limit in the update function and see what happens. While the limit is
> necessary in general, it should at least warn and yell whenever the
> accounting goes out of bounds. And if that happens, then the whole thing is
> completely useless simply because the numbers are just wrong.
>
Ok.
> > One scenario of this patch can be illustrated below:
> >
> > a) After initialization, vector_mask[0] is set with {CPU0...CPU31},
> > other vector_mask[i] remain zero, which means that, CPU0...CPU31
> > have zero vectors assigned.
> > b) During driver probe, CPU0 is chosen to be assigned with one vector.
> > Thus vector_mask[0] becomes {CPU1...CPU31}, and vector_mask[1] becomes
> > {CPU0}.
> > c) The next time the system tries to assign another vector, it searches from
> > vector[0], because it is non-empty, CPU1 is chosen to place
> > the vector. Thus vector_mask[0] becomes {CPU2...CPU31}, and vector_mask[1]
> > becomes {CPU0, CPU1}.
> > d) and so on, the vectors assignment will spread on different CPUs.
>
> So this works for your particular network driver scenario, which is the
> wrong example as this driver just needs to be reworked to not expose that
> issue.
>
> The reality of affinity requirements is not that simple. It's very much
> device dependent and making it mandatory can have negative side effects.
>
Yes, this patch just 'cure' a special case.
> > Q2:
> > "respect the allocation domains"
> > A2:
> > This solution provides a CPU as the param for the vector allocation domains,
>
> Solution? This is a crude hack which "solves" one particular problem in the
> wrong way.
>
> > in the hope that the domains can leverage this hint to generate the cpumask
>
> Hope is never a good enginering principle.
>
> > with less vectors assigned. But yes, the CPU we select upfront gets fed into
> > apic->vector_allocation_domain() might just say no because the CPU
> > does not belong to it, but anyway this should somehow spread the vectors,
> > although I can not find out a graceful solution to this.
> > BTW, the vector allocation domain is default_vector_allocation_domain()
> > in our case, thus it works.
>
> Now try that with the logical delivery modes which allow multi CPU
> assignements and the cluster mode thereof.
Yes, I feel a little hard to consider the vector domain as it looks
indepent of the current vector allocation process.
>
> > Q3:
> > "set a preference for the node on which the interrupt was allocated"
> > A3:
> > This can be achieved by further optimization in the following way:
> > a) Also record the number of vectors used per node.
> > b) Each time invoking __assign_irq_vector, adjust the input mask
> > by cpumask_and(mask, mask, node_mask_with_least_vectors())
>
> I'm not looking forward to that heuristics and the next loop inside the
> loops of loops.
>
> So no, this is not going to happen. The current implementation sucks and
> needs a rewrite.
>
> I already started to look into that, but started at the low level end of
> it. See the IDT cleanup series in tip:x86/apic. There is much more vooddo
> in the vector management code to clean up before we can actually talk about
> making the assignment algorithm smarter.
Ok, I'm pulling from tip:x86/apic.
>
> I'll send out a separate mail raising a few points to discuss before we are
> actually starting to look into magic spreading functionality.
>
For the vector calculation question you asked I'll reply to that thread.
Thanks,
Yu
> Thanks,
>
> tglx
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH 4/4][RFC v2] x86/apic: Spread the vectors by choosing the idlest CPU
2017-09-05 22:57 ` [PATCH 4/4][RFC v2] x86/apic: Spread the vectors by choosing the idlest CPU Thomas Gleixner
@ 2017-09-06 4:34 ` Yu Chen
2017-09-06 8:03 ` Thomas Gleixner
0 siblings, 1 reply; 21+ messages in thread
From: Yu Chen @ 2017-09-06 4:34 UTC (permalink / raw)
To: Thomas Gleixner
Cc: x86, Ingo Molnar, H. Peter Anvin, Rui Zhang, LKML,
Rafael J. Wysocki, Len Brown, Dan Williams, Christoph Hellwig,
Peter Zijlstra
On Wed, Sep 06, 2017 at 12:57:41AM +0200, Thomas Gleixner wrote:
> On Sun, 3 Sep 2017, Thomas Gleixner wrote:
>
> > On Fri, 1 Sep 2017, Chen Yu wrote:
> >
> > > This is the major logic to spread the vectors on different CPUs.
> > > The main idea is to choose the 'idlest' CPU which has assigned
> > > the least number of vectors as the candidate/hint for the vector
> > > allocation domain, in the hope that the vector allocation domain
> > > could leverage this hint to generate corresponding cpumask.
> > >
> > > One of the requirements to do this vector spreading work comes from the
> > > following hibernation problem found on a 16 cores server:
> > >
> > > CPU 31 disable failed: CPU has 62 vectors assigned and there
> > > are only 0 available.
>
> Thinking more about this, this makes no sense whatsoever.
>
> The total number of interrupts on a system is the same whether they are
> all on CPU 0 or evenly spread over all CPUs.
>
> As this machine is using physcial destination mode, the number of vectors
> used is the same as the number of interrupts, except for the case where a
> move of an interrupt is in progress and the interrupt which cleans up the
> old vector has not yet arrived. Lets ignore that for now.
>
> The available vector space is 204 per CPU on such a system.
>
> 256 - SYSTEM[0-31, 32, 128, 239-255] - LEGACY[50] = 204
>
> > > CPU 31 disable failed: CPU has 62 vectors assigned and there
> > > are only 0 available.
>
> CPU31 is the last AP going offline (CPU0 is still online).
>
> It wants to move 62 vectors to CPU0, but it can't because CPU0 has 0
> available vectors. That means CPU0 has 204 vectors used. I doubt that, but
> what I doubt even more is that this interrupt spreading helps in any way.
>
> Assumed that we have a total of 204 + 62 = 266 device interrupt vectors in
> use and they are evenly spread over 32 CPUs, so each CPU has either 8 or
> nine vectors. Fine.
>
> Now if you unplug all CPUs except CPU0 starting from CPU1 up to CPU31 then
> at the point where CPU31 is about to be unplugged, CPU0 holds 133 vectors
> and CPU31 holds 133 vectors as well - assumed that the spread is exactly
> even.
>
> I have a hard time to figure out how the 133 vectors on CPU31 are now
> magically fitting in the empty space on CPU0, which is 204 - 133 = 71. In
> my limited understanding of math 133 is greater than 71, but your patch
> might make that magically be wrong.
>
The problem is reproduced when the network cable is not plugged in,
because this driver looks like this:
step 1. Reserved enough irq vectors and corresponding IRQs.
step 2. If the network is activated, invoke request_irq() to
register the handler.
step 3. Invoke set_affinity() to spread the IRQs onto different
CPUs, thus to spread the vectors too.
Here's my understanding for why spreading vectors might help for this
special case:
As step 2 will not get invoked, the IRQs of this driver
has not been enabled, thus in migrate_one_irq() this IRQ
will not be considered because there is a check of
irqd_is_started(d), thus there should only be 8 vectors
allocated by this driver on CPU0, and 8 vectors left on
CPU31, and the 8 vectors on CPU31 will not be migrated
to CPU0 neither, so there is room for other 'valid' vectors
to be migrated to CPU0.
> Can you please provide detailed information about how many device
> interrupts are actually in use/allocated on that system?
>
> Please enable CONFIG_GENERIC_IRQ_DEBUGFS and provide the output of
>
> # cat /sys/kernel/debug/irq/domains/*
>
> and
>
> # ls /sys/kernel/debug/irq/irqs
>
Ok, here's the information after system bootup on top
of 4.13:
# cat /sys/kernel/debug/irq/domains/*
name: VECTOR
size: 0
mapped: 388
flags: 0x00000041
name: IO-APIC-0
size: 24
mapped: 16
flags: 0x00000041
parent: VECTOR
name: VECTOR
size: 0
mapped: 388
flags: 0x00000041
name: IO-APIC-1
size: 8
mapped: 2
flags: 0x00000041
parent: VECTOR
name: VECTOR
size: 0
mapped: 388
flags: 0x00000041
name: IO-APIC-2
size: 8
mapped: 0
flags: 0x00000041
parent: VECTOR
name: VECTOR
size: 0
mapped: 388
flags: 0x00000041
name: IO-APIC-3
size: 8
mapped: 0
flags: 0x00000041
parent: VECTOR
name: VECTOR
size: 0
mapped: 388
flags: 0x00000041
name: IO-APIC-4
size: 8
mapped: 5
flags: 0x00000041
parent: VECTOR
name: VECTOR
size: 0
mapped: 388
flags: 0x00000041
name: PCI-HT
size: 0
mapped: 0
flags: 0x00000041
parent: VECTOR
name: VECTOR
size: 0
mapped: 388
flags: 0x00000041
name: PCI-MSI-2
size: 0
mapped: 365
flags: 0x00000051
parent: VECTOR
name: VECTOR
size: 0
mapped: 388
flags: 0x00000041
name: VECTOR
size: 0
mapped: 388
flags: 0x00000041
# ls /sys/kernel/debug/irq/irqs
ls /sys/kernel/debug/irq/irqs
0 10 11 13 142 184 217 259 292 31 33 337 339
340 342 344 346 348 350 352 354 356 358 360 362
364 366 368 370 372 374 376 378 380 382 384 386
388 390 392 394 4 6 7 9 1 109 12 14 15 2
24 26 3 32 335 338 34 341 343 345 347 349
351 353 355 357 359 361 363 365 367 369 371 373
375 377 379 381 383 385 387 389 391 393 395 5
67 8
BTW, do we have sysfs to display how much vectors used on each CPUs?
Thanks,
Yu
> Thanks,
>
> tglx
>
>
>
>
>
>
>
>
>
>
>
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH 4/4][RFC v2] x86/apic: Spread the vectors by choosing the idlest CPU
2017-09-06 4:13 ` Yu Chen
@ 2017-09-06 6:15 ` Christoph Hellwig
2017-09-06 17:46 ` Dan Williams
0 siblings, 1 reply; 21+ messages in thread
From: Christoph Hellwig @ 2017-09-06 6:15 UTC (permalink / raw)
To: Yu Chen
Cc: Thomas Gleixner, x86, Ingo Molnar, H. Peter Anvin, Rui Zhang,
LKML, Rafael J. Wysocki, Len Brown, Dan Williams,
Christoph Hellwig, Peter Zijlstra
On Wed, Sep 06, 2017 at 12:13:38PM +0800, Yu Chen wrote:
> I agree, the driver could be rewritten, but it might take some time, so
> meanwhile I'm looking at also other possible optimization.
Which driver are we talking about anyway? Let's start looking at it
and fix the issue there.
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH 4/4][RFC v2] x86/apic: Spread the vectors by choosing the idlest CPU
2017-09-06 4:34 ` Yu Chen
@ 2017-09-06 8:03 ` Thomas Gleixner
2017-09-07 2:52 ` Yu Chen
0 siblings, 1 reply; 21+ messages in thread
From: Thomas Gleixner @ 2017-09-06 8:03 UTC (permalink / raw)
To: Yu Chen
Cc: x86, Ingo Molnar, H. Peter Anvin, Rui Zhang, LKML,
Rafael J. Wysocki, Len Brown, Dan Williams, Christoph Hellwig,
Peter Zijlstra
On Wed, 6 Sep 2017, Yu Chen wrote:
> On Wed, Sep 06, 2017 at 12:57:41AM +0200, Thomas Gleixner wrote:
> > I have a hard time to figure out how the 133 vectors on CPU31 are now
> > magically fitting in the empty space on CPU0, which is 204 - 133 = 71. In
> > my limited understanding of math 133 is greater than 71, but your patch
> > might make that magically be wrong.
> >
> The problem is reproduced when the network cable is not plugged in,
> because this driver looks like this:
>
> step 1. Reserved enough irq vectors and corresponding IRQs.
> step 2. If the network is activated, invoke request_irq() to
> register the handler.
> step 3. Invoke set_affinity() to spread the IRQs onto different
> CPUs, thus to spread the vectors too.
>
> Here's my understanding for why spreading vectors might help for this
> special case:
> As step 2 will not get invoked, the IRQs of this driver
> has not been enabled, thus in migrate_one_irq() this IRQ
> will not be considered because there is a check of
> irqd_is_started(d), thus there should only be 8 vectors
> allocated by this driver on CPU0, and 8 vectors left on
> CPU31, and the 8 vectors on CPU31 will not be migrated
> to CPU0 neither, so there is room for other 'valid' vectors
> to be migrated to CPU0.
Can you please spare me repeating your theories, as long as you don't have
hard facts to back them up? The network cable is changing the symptoms,
but the underlying root cause is definitely something different.
> # cat /sys/kernel/debug/irq/domains/*
> name: VECTOR
> size: 0
> mapped: 388
> flags: 0x00000041
So we have 388 vectors mapped in total. And those are just device vectors
because system vectors are not accounted there.
> name: IO-APIC-0
> size: 24
> mapped: 16
That's the legacy space
> name: IO-APIC-1
> size: 8
> mapped: 2
> name: IO-APIC-2
> size: 8
> mapped: 0
> name: IO-APIC-3
> size: 8
> mapped: 0
> name: IO-APIC-4
> size: 8
> mapped: 5
And a few GSIs: Total GSIs = 16 + 2 + 5 = 23
> name: PCI-MSI-2
> size: 0
> mapped: 365
Plus 365 PCI-MSI vectors allocated.
> flags: 0x00000051
> parent: VECTOR
> name: VECTOR
> size: 0
> mapped: 388
Which nicely sums up to 388
> # ls /sys/kernel/debug/irq/irqs
> ls /sys/kernel/debug/irq/irqs
> 0 10 11 13 142 184 217 259 292 31 33 337 339
> 340 342 344 346 348 350 352 354 356 358 360 362
> 364 366 368 370 372 374 376 378 380 382 384 386
> 388 390 392 394 4 6 7 9 1 109 12 14 15 2
> 24 26 3 32 335 338 34 341 343 345 347 349
> 351 353 355 357 359 361 363 365 367 369 371 373
> 375 377 379 381 383 385 387 389 391 393 395 5
> 67 8
That are all interrupts which are active. That's a total of 89. Can you
explain where the delta of 299 vectors comes from?
299 allocated, vector mapped, but unused interrupts?
That's where your problem is, not in the vector spreading. You have a
massive leak.
> BTW, do we have sysfs to display how much vectors used on each CPUs?
Not yet.
Can you please apply the debug patch below, boot the machine and right
after login provide the output of
# cat /sys/kernel/debug/tracing/trace
Thanks,
tglx
8<-------------------
--- a/kernel/irq/msi.c
+++ b/kernel/irq/msi.c
@@ -372,6 +372,9 @@ int msi_domain_alloc_irqs(struct irq_dom
return ret;
}
+ trace_printk("dev: %s nvec %d virq %d\n",
+ dev_name(dev), desc->nvec_used, virq);
+
for (i = 0; i < desc->nvec_used; i++)
irq_set_msi_desc_off(virq, i, desc);
}
@@ -419,6 +422,8 @@ void msi_domain_free_irqs(struct irq_dom
* entry. If that's the case, don't do anything.
*/
if (desc->irq) {
+ trace_printk("dev: %s nvec %d virq %d\n",
+ dev_name(dev), desc->nvec_used, desc->irq);
irq_domain_free_irqs(desc->irq, desc->nvec_used);
desc->irq = 0;
}
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH 4/4][RFC v2] x86/apic: Spread the vectors by choosing the idlest CPU
2017-09-06 6:15 ` Christoph Hellwig
@ 2017-09-06 17:46 ` Dan Williams
2017-09-07 2:57 ` Yu Chen
2017-09-07 5:59 ` Thomas Gleixner
0 siblings, 2 replies; 21+ messages in thread
From: Dan Williams @ 2017-09-06 17:46 UTC (permalink / raw)
To: Christoph Hellwig
Cc: Yu Chen, Thomas Gleixner, X86 ML, Ingo Molnar, H. Peter Anvin,
Rui Zhang, LKML, Rafael J. Wysocki, Len Brown, Peter Zijlstra
On Tue, Sep 5, 2017 at 11:15 PM, Christoph Hellwig <hch@lst.de> wrote:
> On Wed, Sep 06, 2017 at 12:13:38PM +0800, Yu Chen wrote:
>> I agree, the driver could be rewritten, but it might take some time, so
>> meanwhile I'm looking at also other possible optimization.
>
> Which driver are we talking about anyway? Let's start looking at it
> and fix the issue there.
As far as I understand, it's already fixed there:
commit 7c9ae7f053e9e896c24fd23595ba369a5fe322e1
Author: Carolyn Wyborny <carolyn.wyborny@intel.com>
Date: Tue Jun 20 15:16:53 2017 -0700
i40e: Fix for trace found with S4 state
This patch fixes a problem found in systems when entering
S4 state. This patch fixes the problem by ensuring that
the misc vector's IRQ is disabled as well. Without this
patch a stack trace can be seen upon entering S4 state.
However this seems like something that should be handled generically
in the irq-core especially since commit c5cb83bb337c
"genirq/cpuhotplug: Handle managed IRQs on CPU hotplug" was headed in
that direction. It's otherwise non-obvious when a driver needs to
release and re-acquire interrupts or be reworked to use managed
interrupts.
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH 4/4][RFC v2] x86/apic: Spread the vectors by choosing the idlest CPU
2017-09-06 8:03 ` Thomas Gleixner
@ 2017-09-07 2:52 ` Yu Chen
2017-09-07 5:54 ` Thomas Gleixner
0 siblings, 1 reply; 21+ messages in thread
From: Yu Chen @ 2017-09-07 2:52 UTC (permalink / raw)
To: Thomas Gleixner
Cc: x86, Ingo Molnar, H. Peter Anvin, Rui Zhang, LKML,
Rafael J. Wysocki, Len Brown, Dan Williams, Christoph Hellwig,
Peter Zijlstra
On Wed, Sep 06, 2017 at 10:03:58AM +0200, Thomas Gleixner wrote:
> On Wed, 6 Sep 2017, Yu Chen wrote:
> > On Wed, Sep 06, 2017 at 12:57:41AM +0200, Thomas Gleixner wrote:
> > > I have a hard time to figure out how the 133 vectors on CPU31 are now
> > > magically fitting in the empty space on CPU0, which is 204 - 133 = 71. In
> > > my limited understanding of math 133 is greater than 71, but your patch
> > > might make that magically be wrong.
> > >
> > The problem is reproduced when the network cable is not plugged in,
> > because this driver looks like this:
> >
> > step 1. Reserved enough irq vectors and corresponding IRQs.
> > step 2. If the network is activated, invoke request_irq() to
> > register the handler.
> > step 3. Invoke set_affinity() to spread the IRQs onto different
> > CPUs, thus to spread the vectors too.
> >
> > Here's my understanding for why spreading vectors might help for this
> > special case:
> > As step 2 will not get invoked, the IRQs of this driver
> > has not been enabled, thus in migrate_one_irq() this IRQ
> > will not be considered because there is a check of
> > irqd_is_started(d), thus there should only be 8 vectors
> > allocated by this driver on CPU0, and 8 vectors left on
> > CPU31, and the 8 vectors on CPU31 will not be migrated
> > to CPU0 neither, so there is room for other 'valid' vectors
> > to be migrated to CPU0.
>
> Can you please spare me repeating your theories, as long as you don't have
> hard facts to back them up? The network cable is changing the symptoms,
> but the underlying root cause is definitely something different.
>
> > # cat /sys/kernel/debug/irq/domains/*
> > name: VECTOR
> > size: 0
> > mapped: 388
> > flags: 0x00000041
>
> So we have 388 vectors mapped in total. And those are just device vectors
> because system vectors are not accounted there.
>
> > name: IO-APIC-0
> > size: 24
> > mapped: 16
>
> That's the legacy space
>
> > name: IO-APIC-1
> > size: 8
> > mapped: 2
>
> > name: IO-APIC-2
> > size: 8
> > mapped: 0
>
> > name: IO-APIC-3
> > size: 8
> > mapped: 0
>
> > name: IO-APIC-4
> > size: 8
> > mapped: 5
>
> And a few GSIs: Total GSIs = 16 + 2 + 5 = 23
>
> > name: PCI-MSI-2
> > size: 0
> > mapped: 365
>
> Plus 365 PCI-MSI vectors allocated.
>
> > flags: 0x00000051
> > parent: VECTOR
> > name: VECTOR
> > size: 0
> > mapped: 388
>
> Which nicely sums up to 388
>
> > # ls /sys/kernel/debug/irq/irqs
> > ls /sys/kernel/debug/irq/irqs
> > 0 10 11 13 142 184 217 259 292 31 33 337 339
> > 340 342 344 346 348 350 352 354 356 358 360 362
> > 364 366 368 370 372 374 376 378 380 382 384 386
> > 388 390 392 394 4 6 7 9 1 109 12 14 15 2
> > 24 26 3 32 335 338 34 341 343 345 347 349
> > 351 353 355 357 359 361 363 365 367 369 371 373
> > 375 377 379 381 383 385 387 389 391 393 395 5
> > 67 8
>
> That are all interrupts which are active. That's a total of 89. Can you
> explain where the delta of 299 vectors comes from?
>
> 299 allocated, vector mapped, but unused interrupts?
>
> That's where your problem is, not in the vector spreading. You have a
> massive leak.
>
> > BTW, do we have sysfs to display how much vectors used on each CPUs?
>
> Not yet.
>
> Can you please apply the debug patch below, boot the machine and right
> after login provide the output of
>
> # cat /sys/kernel/debug/tracing/trace
>
Ok, I've tested on top of 4.13, here're the results:
# cat /sys/kernel/debug/tracing/trace
# tracer: nop
#
# _-----=> irqs-off
# / _----=> need-resched
# | / _---=> hardirq/softirq
# || / _--=> preempt-depth
# ||| / delay
# TASK-PID CPU# |||| TIMESTAMP FUNCTION
# | | | |||| | |
kworker/0:0-3 [000] .... 7.373461: msi_domain_alloc_irqs: dev: 0000:00:1c.0 nvec 1 virq 24
kworker/0:0-3 [000] .... 7.373666: msi_domain_alloc_irqs: dev: 0000:b2:00.0 nvec 1 virq 26
kworker/0:0-3 [000] .... 7.578980: msi_domain_alloc_irqs: dev: 0000:00:11.5 nvec 1 virq 31
kworker/0:0-3 [000] .... 7.676938: msi_domain_alloc_irqs: dev: 0000:00:17.0 nvec 1 virq 32
kworker/0:0-3 [000] .... 8.017664: msi_domain_alloc_irqs: dev: 0000:00:14.0 nvec 1 virq 33
kworker/0:2-303 [000] .... 9.135467: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 34
kworker/0:2-303 [000] .... 9.135476: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 35
kworker/0:2-303 [000] .... 9.135484: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 36
kworker/0:2-303 [000] .... 9.135492: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 37
kworker/0:2-303 [000] .... 9.135499: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 38
kworker/0:2-303 [000] .... 9.135508: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 39
kworker/0:2-303 [000] .... 9.135516: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 40
kworker/0:2-303 [000] .... 9.135526: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 41
kworker/0:2-303 [000] .... 9.135534: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 42
kworker/0:2-303 [000] .... 9.135542: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 43
kworker/0:2-303 [000] .... 9.135550: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 44
kworker/0:2-303 [000] .... 9.135557: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 45
kworker/0:2-303 [000] .... 9.135564: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 46
kworker/0:2-303 [000] .... 9.135572: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 47
kworker/0:2-303 [000] .... 9.135579: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 48
kworker/0:2-303 [000] .... 9.135587: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 49
kworker/0:2-303 [000] .... 9.135594: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 50
kworker/0:2-303 [000] .... 9.135601: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 51
kworker/0:2-303 [000] .... 9.135608: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 52
kworker/0:2-303 [000] .... 9.135616: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 53
kworker/0:2-303 [000] .... 9.135623: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 54
kworker/0:2-303 [000] .... 9.135631: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 55
kworker/0:2-303 [000] .... 9.135638: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 56
kworker/0:2-303 [000] .... 9.135645: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 57
kworker/0:2-303 [000] .... 9.135653: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 58
kworker/0:2-303 [000] .... 9.135660: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 59
kworker/0:2-303 [000] .... 9.135668: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 60
kworker/0:2-303 [000] .... 9.135676: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 61
kworker/0:2-303 [000] .... 9.135683: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 62
kworker/0:2-303 [000] .... 9.135691: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 63
kworker/0:2-303 [000] .... 9.135699: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 64
kworker/0:2-303 [000] .... 9.135707: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 65
kworker/0:2-303 [000] .... 9.135714: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 66
kworker/0:2-303 [000] .... 9.135721: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 67
kworker/0:2-303 [000] .... 9.135729: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 68
kworker/0:2-303 [000] .... 9.135736: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 69
kworker/0:2-303 [000] .... 9.135744: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 70
kworker/0:2-303 [000] .... 9.135751: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 71
kworker/0:2-303 [000] .... 9.135758: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 72
kworker/0:2-303 [000] .... 9.135766: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 73
kworker/0:2-303 [000] .... 9.135773: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 74
kworker/0:2-303 [000] .... 9.135780: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 75
kworker/0:2-303 [000] .... 9.135788: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 76
kworker/0:2-303 [000] .... 9.135795: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 77
kworker/0:2-303 [000] .... 9.135803: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 78
kworker/0:2-303 [000] .... 9.135810: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 79
kworker/0:2-303 [000] .... 9.135818: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 80
kworker/0:2-303 [000] .... 9.135826: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 81
kworker/0:2-303 [000] .... 9.135833: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 82
kworker/0:2-303 [000] .... 9.135841: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 83
kworker/0:2-303 [000] .... 9.135848: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 84
kworker/0:2-303 [000] .... 9.135856: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 85
kworker/0:2-303 [000] .... 9.135863: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 86
kworker/0:2-303 [000] .... 9.135871: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 87
kworker/0:2-303 [000] .... 9.135878: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 88
kworker/0:2-303 [000] .... 9.135885: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 89
kworker/0:2-303 [000] .... 9.135893: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 90
kworker/0:2-303 [000] .... 9.135900: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 91
kworker/0:2-303 [000] .... 9.135908: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 92
kworker/0:2-303 [000] .... 9.135915: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 93
kworker/0:2-303 [000] .... 9.135923: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 94
kworker/0:2-303 [000] .... 9.135930: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 95
kworker/0:2-303 [000] .... 9.135937: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 96
kworker/0:2-303 [000] .... 9.135944: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 97
kworker/0:2-303 [000] .... 9.135951: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 98
kworker/0:2-303 [000] .... 9.135958: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 99
kworker/0:2-303 [000] .... 9.135964: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 100
kworker/0:2-303 [000] .... 9.135971: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 101
kworker/0:2-303 [000] .... 9.135978: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 102
kworker/0:2-303 [000] .... 9.135985: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 103
kworker/0:2-303 [000] .... 9.135991: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 104
kworker/0:2-303 [000] .... 9.135998: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 105
kworker/0:2-303 [000] .... 9.136005: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 106
kworker/0:2-303 [000] .... 9.136012: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 107
kworker/0:2-303 [000] .... 9.136019: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 108
kworker/0:2-303 [000] .... 9.599173: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 109
kworker/0:2-303 [000] .... 9.599181: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 110
kworker/0:2-303 [000] .... 9.599189: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 111
kworker/0:2-303 [000] .... 9.599196: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 112
kworker/0:2-303 [000] .... 9.599203: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 113
kworker/0:2-303 [000] .... 9.599210: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 114
kworker/0:2-303 [000] .... 9.599217: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 115
kworker/0:2-303 [000] .... 9.599224: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 116
kworker/0:2-303 [000] .... 9.599231: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 117
kworker/0:2-303 [000] .... 9.599239: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 118
kworker/0:2-303 [000] .... 9.599246: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 119
kworker/0:2-303 [000] .... 9.599253: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 120
kworker/0:2-303 [000] .... 9.599259: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 121
kworker/0:2-303 [000] .... 9.599266: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 122
kworker/0:2-303 [000] .... 9.599273: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 123
kworker/0:2-303 [000] .... 9.599279: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 124
kworker/0:2-303 [000] .... 9.599286: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 125
kworker/0:2-303 [000] .... 9.599293: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 126
kworker/0:2-303 [000] .... 9.599300: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 127
kworker/0:2-303 [000] .... 9.599307: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 128
kworker/0:2-303 [000] .... 9.599314: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 129
kworker/0:2-303 [000] .... 9.599321: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 130
kworker/0:2-303 [000] .... 9.599327: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 131
kworker/0:2-303 [000] .... 9.599334: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 132
kworker/0:2-303 [000] .... 9.599341: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 133
kworker/0:2-303 [000] .... 9.599349: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 134
kworker/0:2-303 [000] .... 9.599356: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 135
kworker/0:2-303 [000] .... 9.599363: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 136
kworker/0:2-303 [000] .... 9.599369: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 137
kworker/0:2-303 [000] .... 9.599376: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 138
kworker/0:2-303 [000] .... 9.599382: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 139
kworker/0:2-303 [000] .... 9.599389: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 140
kworker/0:2-303 [000] .... 9.599395: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 141
kworker/0:2-303 [000] .... 9.599402: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 142
kworker/0:2-303 [000] .... 9.599409: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 143
kworker/0:2-303 [000] .... 9.599416: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 144
kworker/0:2-303 [000] .... 9.599422: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 145
kworker/0:2-303 [000] .... 9.599428: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 146
kworker/0:2-303 [000] .... 9.599437: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 147
kworker/0:2-303 [000] .... 9.599444: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 148
kworker/0:2-303 [000] .... 9.599450: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 149
kworker/0:2-303 [000] .... 9.599456: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 150
kworker/0:2-303 [000] .... 9.599463: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 151
kworker/0:2-303 [000] .... 9.599470: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 152
kworker/0:2-303 [000] .... 9.599476: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 153
kworker/0:2-303 [000] .... 9.599482: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 154
kworker/0:2-303 [000] .... 9.599488: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 155
kworker/0:2-303 [000] .... 9.599494: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 156
kworker/0:2-303 [000] .... 9.599501: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 157
kworker/0:2-303 [000] .... 9.599507: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 158
kworker/0:2-303 [000] .... 9.599513: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 159
kworker/0:2-303 [000] .... 9.599519: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 160
kworker/0:2-303 [000] .... 9.599527: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 161
kworker/0:2-303 [000] .... 9.599534: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 162
kworker/0:2-303 [000] .... 9.599541: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 163
kworker/0:2-303 [000] .... 9.599547: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 164
kworker/0:2-303 [000] .... 9.599553: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 165
kworker/0:2-303 [000] .... 9.599560: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 166
kworker/0:2-303 [000] .... 9.599567: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 167
kworker/0:2-303 [000] .... 9.599573: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 168
kworker/0:2-303 [000] .... 9.599579: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 169
kworker/0:2-303 [000] .... 9.599585: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 170
kworker/0:2-303 [000] .... 9.599591: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 171
kworker/0:2-303 [000] .... 9.599597: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 172
kworker/0:2-303 [000] .... 9.599604: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 173
kworker/0:2-303 [000] .... 9.599610: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 174
kworker/0:2-303 [000] .... 9.599616: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 175
kworker/0:2-303 [000] .... 9.599622: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 176
kworker/0:2-303 [000] .... 9.599628: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 177
kworker/0:2-303 [000] .... 9.599634: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 178
kworker/0:2-303 [000] .... 9.599640: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 179
kworker/0:2-303 [000] .... 9.599646: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 180
kworker/0:2-303 [000] .... 9.599653: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 181
kworker/0:2-303 [000] .... 9.599660: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 182
kworker/0:2-303 [000] .... 9.599666: msi_domain_alloc_irqs: dev: 0000:bb:00.1 nvec 1 virq 183
kworker/0:2-303 [000] .... 9.699902: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 184
kworker/0:2-303 [000] .... 9.699910: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 185
kworker/0:2-303 [000] .... 9.699917: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 186
kworker/0:2-303 [000] .... 9.699924: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 187
kworker/0:2-303 [000] .... 9.699931: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 188
kworker/0:2-303 [000] .... 9.699939: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 189
kworker/0:2-303 [000] .... 9.699946: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 190
kworker/0:2-303 [000] .... 9.699953: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 191
kworker/0:2-303 [000] .... 9.699960: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 192
kworker/0:2-303 [000] .... 9.699967: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 193
kworker/0:2-303 [000] .... 9.699975: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 194
kworker/0:2-303 [000] .... 9.699983: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 195
kworker/0:2-303 [000] .... 9.699989: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 196
kworker/0:2-303 [000] .... 9.699996: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 197
kworker/0:2-303 [000] .... 9.700003: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 198
kworker/0:2-303 [000] .... 9.700012: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 199
kworker/0:2-303 [000] .... 9.700019: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 200
kworker/0:2-303 [000] .... 9.700026: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 201
kworker/0:2-303 [000] .... 9.700032: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 202
kworker/0:2-303 [000] .... 9.700039: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 203
kworker/0:2-303 [000] .... 9.700047: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 204
kworker/0:2-303 [000] .... 9.700053: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 205
kworker/0:2-303 [000] .... 9.700060: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 206
kworker/0:2-303 [000] .... 9.700067: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 207
kworker/0:2-303 [000] .... 9.700074: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 208
kworker/0:2-303 [000] .... 9.700081: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 209
kworker/0:2-303 [000] .... 9.700088: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 210
kworker/0:2-303 [000] .... 9.700095: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 211
kworker/0:2-303 [000] .... 9.700103: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 212
kworker/0:2-303 [000] .... 9.700112: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 213
kworker/0:2-303 [000] .... 9.700121: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 214
kworker/0:2-303 [000] .... 9.700129: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 215
kworker/0:2-303 [000] .... 9.700137: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 216
kworker/0:2-303 [000] .... 9.700145: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 217
kworker/0:2-303 [000] .... 9.700154: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 218
kworker/0:2-303 [000] .... 9.700163: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 219
kworker/0:2-303 [000] .... 9.700172: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 220
kworker/0:2-303 [000] .... 9.700180: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 221
kworker/0:2-303 [000] .... 9.700189: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 222
kworker/0:2-303 [000] .... 9.700200: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 223
kworker/0:2-303 [000] .... 9.700209: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 224
kworker/0:2-303 [000] .... 9.700217: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 225
kworker/0:2-303 [000] .... 9.700225: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 226
kworker/0:2-303 [000] .... 9.700235: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 227
kworker/0:2-303 [000] .... 9.700244: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 228
kworker/0:2-303 [000] .... 9.700253: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 229
kworker/0:2-303 [000] .... 9.700261: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 230
kworker/0:2-303 [000] .... 9.700269: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 231
kworker/0:2-303 [000] .... 9.700278: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 232
kworker/0:2-303 [000] .... 9.700287: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 233
kworker/0:2-303 [000] .... 9.700295: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 234
kworker/0:2-303 [000] .... 9.700304: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 235
kworker/0:2-303 [000] .... 9.700313: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 236
kworker/0:2-303 [000] .... 9.700321: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 237
kworker/0:2-303 [000] .... 9.700330: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 238
kworker/0:2-303 [000] .... 9.700339: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 239
kworker/0:2-303 [000] .... 9.700347: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 240
kworker/0:2-303 [000] .... 9.700355: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 241
kworker/0:2-303 [000] .... 9.700363: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 242
kworker/0:2-303 [000] .... 9.700372: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 243
kworker/0:2-303 [000] .... 9.700380: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 244
kworker/0:2-303 [000] .... 9.700390: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 245
kworker/0:2-303 [000] .... 9.700398: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 246
kworker/0:2-303 [000] .... 9.700406: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 247
kworker/0:2-303 [000] .... 9.700415: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 248
kworker/0:2-303 [000] .... 9.700424: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 249
kworker/0:2-303 [000] .... 9.700432: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 250
kworker/0:2-303 [000] .... 9.700440: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 251
kworker/0:2-303 [000] .... 9.700449: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 252
kworker/0:2-303 [000] .... 9.700458: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 253
kworker/0:2-303 [000] .... 9.700466: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 254
kworker/0:2-303 [000] .... 9.700474: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 255
kworker/0:2-303 [000] .... 9.700483: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 256
kworker/0:2-303 [000] .... 9.700492: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 257
kworker/0:2-303 [000] .... 9.700500: msi_domain_alloc_irqs: dev: 0000:bb:00.2 nvec 1 virq 258
kworker/0:2-303 [000] .... 9.761556: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 259
kworker/0:2-303 [000] .... 9.761570: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 260
kworker/0:2-303 [000] .... 9.761583: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 261
kworker/0:2-303 [000] .... 9.761595: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 262
kworker/0:2-303 [000] .... 9.761608: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 263
kworker/0:2-303 [000] .... 9.761619: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 264
kworker/0:2-303 [000] .... 9.761629: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 265
kworker/0:2-303 [000] .... 9.761639: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 266
kworker/0:2-303 [000] .... 9.761649: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 267
kworker/0:2-303 [000] .... 9.761658: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 268
kworker/0:2-303 [000] .... 9.761669: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 269
kworker/0:2-303 [000] .... 9.761678: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 270
kworker/0:2-303 [000] .... 9.761688: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 271
kworker/0:2-303 [000] .... 9.761697: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 272
kworker/0:2-303 [000] .... 9.761707: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 273
kworker/0:2-303 [000] .... 9.761717: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 274
kworker/0:2-303 [000] .... 9.761726: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 275
kworker/0:2-303 [000] .... 9.761736: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 276
kworker/0:2-303 [000] .... 9.761745: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 277
kworker/0:2-303 [000] .... 9.761756: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 278
kworker/0:2-303 [000] .... 9.761765: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 279
kworker/0:2-303 [000] .... 9.761775: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 280
kworker/0:2-303 [000] .... 9.761784: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 281
kworker/0:2-303 [000] .... 9.761793: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 282
kworker/0:2-303 [000] .... 9.761803: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 283
kworker/0:2-303 [000] .... 9.761812: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 284
kworker/0:2-303 [000] .... 9.761822: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 285
kworker/0:2-303 [000] .... 9.761831: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 286
kworker/0:2-303 [000] .... 9.761840: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 287
kworker/0:2-303 [000] .... 9.761854: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 288
kworker/0:2-303 [000] .... 9.761863: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 289
kworker/0:2-303 [000] .... 9.761872: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 290
kworker/0:2-303 [000] .... 9.761881: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 291
kworker/0:2-303 [000] .... 9.761892: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 292
kworker/0:2-303 [000] .... 9.761903: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 293
kworker/0:2-303 [000] .... 9.761913: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 294
kworker/0:2-303 [000] .... 9.761923: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 295
kworker/0:2-303 [000] .... 9.761932: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 296
kworker/0:2-303 [000] .... 9.761941: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 297
kworker/0:2-303 [000] .... 9.761951: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 298
kworker/0:2-303 [000] .... 9.761961: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 299
kworker/0:2-303 [000] .... 9.761970: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 300
kworker/0:2-303 [000] .... 9.761979: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 301
kworker/0:2-303 [000] .... 9.761988: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 302
kworker/0:2-303 [000] .... 9.761998: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 303
kworker/0:2-303 [000] .... 9.762008: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 304
kworker/0:2-303 [000] .... 9.762017: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 305
kworker/0:2-303 [000] .... 9.762027: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 306
kworker/0:2-303 [000] .... 9.762037: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 307
kworker/0:2-303 [000] .... 9.762046: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 308
kworker/0:2-303 [000] .... 9.762055: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 309
kworker/0:2-303 [000] .... 9.762065: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 310
kworker/0:2-303 [000] .... 9.762074: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 311
kworker/0:2-303 [000] .... 9.762085: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 312
kworker/0:2-303 [000] .... 9.762094: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 313
kworker/0:2-303 [000] .... 9.762104: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 314
kworker/0:2-303 [000] .... 9.762113: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 315
kworker/0:2-303 [000] .... 9.762122: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 316
kworker/0:2-303 [000] .... 9.762133: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 317
kworker/0:2-303 [000] .... 9.762142: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 318
kworker/0:2-303 [000] .... 9.762151: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 319
kworker/0:2-303 [000] .... 9.762162: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 320
kworker/0:2-303 [000] .... 9.762171: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 321
kworker/0:2-303 [000] .... 9.762182: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 322
kworker/0:2-303 [000] .... 9.762191: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 323
kworker/0:2-303 [000] .... 9.762200: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 324
kworker/0:2-303 [000] .... 9.762209: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 325
kworker/0:2-303 [000] .... 9.762220: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 326
kworker/0:2-303 [000] .... 9.762231: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 327
kworker/0:2-303 [000] .... 9.762240: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 328
kworker/0:2-303 [000] .... 9.762249: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 329
kworker/0:2-303 [000] .... 9.762258: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 330
kworker/0:2-303 [000] .... 9.762268: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 331
kworker/0:2-303 [000] .... 9.762278: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 332
kworker/0:2-303 [000] .... 9.762288: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 333
kworker/0:0-3 [000] .... 12.026100: msi_domain_alloc_irqs: dev: 0000:00:04.0 nvec 1 virq 335
kworker/0:0-3 [000] .... 12.143303: msi_domain_alloc_irqs: dev: 0000:00:04.1 nvec 1 virq 337
kworker/0:0-3 [000] .... 12.158686: msi_domain_alloc_irqs: dev: 0000:00:04.2 nvec 1 virq 338
kworker/0:0-3 [000] .... 12.168628: msi_domain_alloc_irqs: dev: 0000:00:04.3 nvec 1 virq 339
kworker/0:0-3 [000] .... 12.176980: msi_domain_alloc_irqs: dev: 0000:00:04.4 nvec 1 virq 340
kworker/0:0-3 [000] .... 12.185280: msi_domain_alloc_irqs: dev: 0000:00:04.5 nvec 1 virq 341
kworker/0:0-3 [000] .... 12.204748: msi_domain_alloc_irqs: dev: 0000:00:04.6 nvec 1 virq 342
kworker/0:0-3 [000] .... 12.214549: msi_domain_alloc_irqs: dev: 0000:00:04.7 nvec 1 virq 343
kworker/0:1-169 [000] .... 12.440642: msi_domain_alloc_irqs: dev: 0000:b5:00.0 nvec 1 virq 344
kworker/0:1-169 [000] .... 12.440664: msi_domain_alloc_irqs: dev: 0000:b5:00.0 nvec 1 virq 345
kworker/0:1-169 [000] .... 12.440685: msi_domain_alloc_irqs: dev: 0000:b5:00.0 nvec 1 virq 346
kworker/0:1-169 [000] .... 12.440704: msi_domain_alloc_irqs: dev: 0000:b5:00.0 nvec 1 virq 347
kworker/0:1-169 [000] .... 12.440724: msi_domain_alloc_irqs: dev: 0000:b5:00.0 nvec 1 virq 348
kworker/0:1-169 [000] .... 12.440742: msi_domain_alloc_irqs: dev: 0000:b5:00.0 nvec 1 virq 349
kworker/0:1-169 [000] .... 12.440761: msi_domain_alloc_irqs: dev: 0000:b5:00.0 nvec 1 virq 350
kworker/0:1-169 [000] .... 12.440780: msi_domain_alloc_irqs: dev: 0000:b5:00.0 nvec 1 virq 351
kworker/0:1-169 [000] .... 12.440802: msi_domain_alloc_irqs: dev: 0000:b5:00.0 nvec 1 virq 352
kworker/0:1-169 [000] .... 12.440822: msi_domain_alloc_irqs: dev: 0000:b5:00.0 nvec 1 virq 353
kworker/0:1-169 [000] .... 12.440841: msi_domain_alloc_irqs: dev: 0000:b5:00.0 nvec 1 virq 354
kworker/0:1-169 [000] .... 12.440860: msi_domain_alloc_irqs: dev: 0000:b5:00.0 nvec 1 virq 355
kworker/0:1-169 [000] .... 12.440878: msi_domain_alloc_irqs: dev: 0000:b5:00.0 nvec 1 virq 356
kworker/0:1-169 [000] .... 12.440897: msi_domain_alloc_irqs: dev: 0000:b5:00.0 nvec 1 virq 357
kworker/0:1-169 [000] .... 12.440917: msi_domain_alloc_irqs: dev: 0000:b5:00.0 nvec 1 virq 358
kworker/0:1-169 [000] .... 12.440936: msi_domain_alloc_irqs: dev: 0000:b5:00.0 nvec 1 virq 359
kworker/0:1-169 [000] .... 12.440954: msi_domain_alloc_irqs: dev: 0000:b5:00.0 nvec 1 virq 360
kworker/0:1-169 [000] .... 13.251009: msi_domain_alloc_irqs: dev: 0000:b7:00.0 nvec 1 virq 361
kworker/0:1-169 [000] .... 13.251033: msi_domain_alloc_irqs: dev: 0000:b7:00.0 nvec 1 virq 362
kworker/0:1-169 [000] .... 13.251055: msi_domain_alloc_irqs: dev: 0000:b7:00.0 nvec 1 virq 363
kworker/0:1-169 [000] .... 13.251076: msi_domain_alloc_irqs: dev: 0000:b7:00.0 nvec 1 virq 364
kworker/0:1-169 [000] .... 13.251100: msi_domain_alloc_irqs: dev: 0000:b7:00.0 nvec 1 virq 365
kworker/0:1-169 [000] .... 13.251120: msi_domain_alloc_irqs: dev: 0000:b7:00.0 nvec 1 virq 366
kworker/0:1-169 [000] .... 13.251142: msi_domain_alloc_irqs: dev: 0000:b7:00.0 nvec 1 virq 367
kworker/0:1-169 [000] .... 13.251161: msi_domain_alloc_irqs: dev: 0000:b7:00.0 nvec 1 virq 368
kworker/0:1-169 [000] .... 13.251182: msi_domain_alloc_irqs: dev: 0000:b7:00.0 nvec 1 virq 369
kworker/0:1-169 [000] .... 13.251204: msi_domain_alloc_irqs: dev: 0000:b7:00.0 nvec 1 virq 370
kworker/0:1-169 [000] .... 13.251224: msi_domain_alloc_irqs: dev: 0000:b7:00.0 nvec 1 virq 371
kworker/0:1-169 [000] .... 13.251263: msi_domain_alloc_irqs: dev: 0000:b7:00.0 nvec 1 virq 372
kworker/0:1-169 [000] .... 13.251288: msi_domain_alloc_irqs: dev: 0000:b7:00.0 nvec 1 virq 373
kworker/0:1-169 [000] .... 13.251308: msi_domain_alloc_irqs: dev: 0000:b7:00.0 nvec 1 virq 374
kworker/0:1-169 [000] .... 13.251330: msi_domain_alloc_irqs: dev: 0000:b7:00.0 nvec 1 virq 375
kworker/0:1-169 [000] .... 13.251349: msi_domain_alloc_irqs: dev: 0000:b7:00.0 nvec 1 virq 376
kworker/0:1-169 [000] .... 13.251374: msi_domain_alloc_irqs: dev: 0000:b7:00.0 nvec 1 virq 377
kworker/0:1-169 [000] .... 14.043690: msi_domain_alloc_irqs: dev: 0000:b9:00.0 nvec 1 virq 378
kworker/0:1-169 [000] .... 14.043721: msi_domain_alloc_irqs: dev: 0000:b9:00.0 nvec 1 virq 379
kworker/0:1-169 [000] .... 14.043745: msi_domain_alloc_irqs: dev: 0000:b9:00.0 nvec 1 virq 380
kworker/0:1-169 [000] .... 14.043767: msi_domain_alloc_irqs: dev: 0000:b9:00.0 nvec 1 virq 381
kworker/0:1-169 [000] .... 14.043793: msi_domain_alloc_irqs: dev: 0000:b9:00.0 nvec 1 virq 382
kworker/0:1-169 [000] .... 14.043817: msi_domain_alloc_irqs: dev: 0000:b9:00.0 nvec 1 virq 383
kworker/0:1-169 [000] .... 14.043840: msi_domain_alloc_irqs: dev: 0000:b9:00.0 nvec 1 virq 384
kworker/0:1-169 [000] .... 14.043862: msi_domain_alloc_irqs: dev: 0000:b9:00.0 nvec 1 virq 385
kworker/0:1-169 [000] .... 14.043884: msi_domain_alloc_irqs: dev: 0000:b9:00.0 nvec 1 virq 386
kworker/0:1-169 [000] .... 14.043909: msi_domain_alloc_irqs: dev: 0000:b9:00.0 nvec 1 virq 387
kworker/0:1-169 [000] .... 14.043932: msi_domain_alloc_irqs: dev: 0000:b9:00.0 nvec 1 virq 388
kworker/0:1-169 [000] .... 14.043953: msi_domain_alloc_irqs: dev: 0000:b9:00.0 nvec 1 virq 389
kworker/0:1-169 [000] .... 14.043975: msi_domain_alloc_irqs: dev: 0000:b9:00.0 nvec 1 virq 390
kworker/0:1-169 [000] .... 14.043997: msi_domain_alloc_irqs: dev: 0000:b9:00.0 nvec 1 virq 391
kworker/0:1-169 [000] .... 14.044021: msi_domain_alloc_irqs: dev: 0000:b9:00.0 nvec 1 virq 392
kworker/0:1-169 [000] .... 14.044043: msi_domain_alloc_irqs: dev: 0000:b9:00.0 nvec 1 virq 393
kworker/0:1-169 [000] .... 14.044065: msi_domain_alloc_irqs: dev: 0000:b9:00.0 nvec 1 virq 394
kworker/0:1-169 [000] .... 5181.743530: msi_domain_alloc_irqs: dev: 0000:00:1f.6 nvec 1 virq 395
ip-2037 [029] .... 5182.263982: msi_domain_alloc_irqs: dev: 0000:00:1f.6 nvec 1 virq 395
ip-2037 [029] .... 5182.367473: msi_domain_alloc_irqs: dev: 0000:00:1f.6 nvec 1 virq 395
bb:00.[0-3] Ethernet controller: Intel Corporation Device 37d0 (rev 03)
-+-[0000:b2]-+-00.0-[b3-bc]----00.0-[b4-bc]--+-00.0-[b5-b6]----00.0
| | +-01.0-[b7-b8]----00.0
| | +-02.0-[b9-ba]----00.0
| | \-03.0-[bb-bc]--+-00.0
| | +-00.1
| | +-00.2
| | \-00.3
and they are using i40e driver, the vectors should be reserved by:
i40e_probe() ->
i40e_init_interrupt_scheme() ->
i40e_init_msix() ->
i40e_reserve_msix_vectors() ->
pci_enable_msix_range()
# ls /sys/kernel/debug/irq/irqs
0 10 11 13 142 184 217 259 292 31 33
337 339 340 342 344 346 348 350 352 354 356
358 360 362 364 366 368 370 372 374 376 378
380 382 384 386 388 390 392 394 4 6 7 9
1 109 12 14 15 2 24 26 3 32 335
338 34 341 343 345 347 349 351 353 355 357
359 361 363 365 367 369 371 373 375 377 379
381 383 385 387 389 391 393 395 5 67 8
# cat /sys/kernel/debug/irq/domains/*
name: VECTOR
size: 0
mapped: 388
flags: 0x00000041
name: IO-APIC-0
size: 24
mapped: 16
flags: 0x00000041
parent: VECTOR
name: VECTOR
size: 0
mapped: 388
flags: 0x00000041
name: IO-APIC-1
size: 8
mapped: 2
flags: 0x00000041
parent: VECTOR
name: VECTOR
size: 0
mapped: 388
flags: 0x00000041
name: IO-APIC-2
size: 8
mapped: 0
flags: 0x00000041
parent: VECTOR
name: VECTOR
size: 0
mapped: 388
flags: 0x00000041
name: IO-APIC-3
size: 8
mapped: 0
flags: 0x00000041
parent: VECTOR
name: VECTOR
size: 0
mapped: 388
flags: 0x00000041
name: IO-APIC-4
size: 8
mapped: 5
flags: 0x00000041
parent: VECTOR
name: VECTOR
size: 0
mapped: 388
flags: 0x00000041
name: PCI-HT
size: 0
mapped: 0
flags: 0x00000041
parent: VECTOR
name: VECTOR
size: 0
mapped: 388
flags: 0x00000041
name: PCI-MSI-2
size: 0
mapped: 365
flags: 0x00000051
parent: VECTOR
name: VECTOR
size: 0
mapped: 388
flags: 0x00000041
name: VECTOR
size: 0
mapped: 388
flags: 0x00000041
Thanks,
Yu
> Thanks,
>
> tglx
>
> 8<-------------------
> --- a/kernel/irq/msi.c
> +++ b/kernel/irq/msi.c
> @@ -372,6 +372,9 @@ int msi_domain_alloc_irqs(struct irq_dom
> return ret;
> }
>
> + trace_printk("dev: %s nvec %d virq %d\n",
> + dev_name(dev), desc->nvec_used, virq);
> +
> for (i = 0; i < desc->nvec_used; i++)
> irq_set_msi_desc_off(virq, i, desc);
> }
> @@ -419,6 +422,8 @@ void msi_domain_free_irqs(struct irq_dom
> * entry. If that's the case, don't do anything.
> */
> if (desc->irq) {
> + trace_printk("dev: %s nvec %d virq %d\n",
> + dev_name(dev), desc->nvec_used, desc->irq);
> irq_domain_free_irqs(desc->irq, desc->nvec_used);
> desc->irq = 0;
> }
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH 4/4][RFC v2] x86/apic: Spread the vectors by choosing the idlest CPU
2017-09-06 17:46 ` Dan Williams
@ 2017-09-07 2:57 ` Yu Chen
2017-09-07 5:59 ` Thomas Gleixner
1 sibling, 0 replies; 21+ messages in thread
From: Yu Chen @ 2017-09-07 2:57 UTC (permalink / raw)
To: Dan Williams
Cc: Christoph Hellwig, Thomas Gleixner, X86 ML, Ingo Molnar,
H. Peter Anvin, Rui Zhang, LKML, Rafael J. Wysocki, Len Brown,
Peter Zijlstra
On Wed, Sep 06, 2017 at 10:46:17AM -0700, Dan Williams wrote:
> On Tue, Sep 5, 2017 at 11:15 PM, Christoph Hellwig <hch@lst.de> wrote:
> > On Wed, Sep 06, 2017 at 12:13:38PM +0800, Yu Chen wrote:
> >> I agree, the driver could be rewritten, but it might take some time, so
> >> meanwhile I'm looking at also other possible optimization.
> >
> > Which driver are we talking about anyway? Let's start looking at it
> > and fix the issue there.
>
> As far as I understand, it's already fixed there:
>
> commit 7c9ae7f053e9e896c24fd23595ba369a5fe322e1
> Author: Carolyn Wyborny <carolyn.wyborny@intel.com>
> Date: Tue Jun 20 15:16:53 2017 -0700
>
> i40e: Fix for trace found with S4 state
>
> This patch fixes a problem found in systems when entering
> S4 state. This patch fixes the problem by ensuring that
> the misc vector's IRQ is disabled as well. Without this
> patch a stack trace can be seen upon entering S4 state.
>
> However this seems like something that should be handled generically
> in the irq-core especially since commit c5cb83bb337c
> "genirq/cpuhotplug: Handle managed IRQs on CPU hotplug" was headed in
> that direction. It's otherwise non-obvious when a driver needs to
> release and re-acquire interrupts or be reworked to use managed
> interrupts.
Yes, thanks for the explaination! I did not notice this patch has
been merged already.
I'm using the normal CPU hotplug to reproduce the issue:
#!/bin/bash
n=1
while [ $n -le 31 ]
do
echo 0 > /sys/devices/system/cpu/cpu${n}/online
n=$(( n+1 ))
done
Thanks,
Yu
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH 4/4][RFC v2] x86/apic: Spread the vectors by choosing the idlest CPU
2017-09-07 2:52 ` Yu Chen
@ 2017-09-07 5:54 ` Thomas Gleixner
2017-09-07 8:34 ` Yu Chen
0 siblings, 1 reply; 21+ messages in thread
From: Thomas Gleixner @ 2017-09-07 5:54 UTC (permalink / raw)
To: Yu Chen
Cc: x86, Ingo Molnar, H. Peter Anvin, Rui Zhang, LKML,
Rafael J. Wysocki, Len Brown, Dan Williams, Christoph Hellwig,
Peter Zijlstra, Jeff Kirsher
On Thu, 7 Sep 2017, Yu Chen wrote:
> On Wed, Sep 06, 2017 at 10:03:58AM +0200, Thomas Gleixner wrote:
> > Can you please apply the debug patch below, boot the machine and right
> > after login provide the output of
> >
> > # cat /sys/kernel/debug/tracing/trace
> >
> kworker/0:2-303 [000] .... 9.135467: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 34
> kworker/0:2-303 [000] .... 9.135476: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 35
> kworker/0:2-303 [000] .... 9.135484: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 36
<SNIP>
> kworker/0:2-303 [000] .... 9.762268: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 331
> kworker/0:2-303 [000] .... 9.762278: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 332
> kworker/0:2-303 [000] .... 9.762288: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 333
That's 300 vectors.
> bb:00.[0-3] Ethernet controller: Intel Corporation Device 37d0 (rev 03)
>
> -+-[0000:b2]-+-00.0-[b3-bc]----00.0-[b4-bc]--+-00.0-[b5-b6]----00.0
> | | +-01.0-[b7-b8]----00.0
> | | +-02.0-[b9-ba]----00.0
> | | \-03.0-[bb-bc]--+-00.0
> | | +-00.1
> | | +-00.2
> | | \-00.3
>
> and they are using i40e driver, the vectors should be reserved by:
> i40e_probe() ->
> i40e_init_interrupt_scheme() ->
> i40e_init_msix() ->
> i40e_reserve_msix_vectors() ->
> pci_enable_msix_range()
>
> # ls /sys/kernel/debug/irq/irqs
> 0 10 11 13 142 184 217 259 292 31 33
> 337 339 340 342 344 346 348 350 352 354 356
> 358 360 362 364 366 368 370 372 374 376 378
> 380 382 384 386 388 390 392 394 4 6 7 9
> 1 109 12 14 15 2 24 26 3 32 335
> 338 34 341 343 345 347 349 351 353 355 357
> 359 361 363 365 367 369 371 373 375 377 379
> 381 383 385 387 389 391 393 395 5 67 8
Out of these 300 interrupts exactly 8 randomly selected ones are actively
used. And the other 292 interrupts are just there because it might need
them in the future when the 32 CPU machine gets magically upgraded to 4096
cores at runtime?
Can the i40e people @intel please fix this waste of resources and sanitize
their interrupt allocation scheme?
Please switch it over to managed interrupts so the affinity spreading
happens in a sane way and the interrupts are properly managed on CPU
hotplug.
Thanks,
tglx
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH 4/4][RFC v2] x86/apic: Spread the vectors by choosing the idlest CPU
2017-09-06 17:46 ` Dan Williams
2017-09-07 2:57 ` Yu Chen
@ 2017-09-07 5:59 ` Thomas Gleixner
2017-09-07 6:23 ` Dan Williams
1 sibling, 1 reply; 21+ messages in thread
From: Thomas Gleixner @ 2017-09-07 5:59 UTC (permalink / raw)
To: Dan Williams
Cc: Christoph Hellwig, Yu Chen, X86 ML, Ingo Molnar, H. Peter Anvin,
Rui Zhang, LKML, Rafael J. Wysocki, Len Brown, Peter Zijlstra,
Jeff Kirsher
On Wed, 6 Sep 2017, Dan Williams wrote:
> On Tue, Sep 5, 2017 at 11:15 PM, Christoph Hellwig <hch@lst.de> wrote:
> > On Wed, Sep 06, 2017 at 12:13:38PM +0800, Yu Chen wrote:
> >> I agree, the driver could be rewritten, but it might take some time, so
> >> meanwhile I'm looking at also other possible optimization.
> >
> > Which driver are we talking about anyway? Let's start looking at it
> > and fix the issue there.
>
> As far as I understand, it's already fixed there:
>
> commit 7c9ae7f053e9e896c24fd23595ba369a5fe322e1
-ENOSUCHCOMMIT
> Author: Carolyn Wyborny <carolyn.wyborny@intel.com>
> Date: Tue Jun 20 15:16:53 2017 -0700
>
> i40e: Fix for trace found with S4 state
>
> This patch fixes a problem found in systems when entering
> S4 state. This patch fixes the problem by ensuring that
> the misc vector's IRQ is disabled as well. Without this
> patch a stack trace can be seen upon entering S4 state.
>
> However this seems like something that should be handled generically
> in the irq-core especially since commit c5cb83bb337c
> "genirq/cpuhotplug: Handle managed IRQs on CPU hotplug" was headed in
> that direction. It's otherwise non-obvious when a driver needs to
> release and re-acquire interrupts or be reworked to use managed
> interrupts.
There are two problems here:
1) The driver allocates 300 interrupts and uses exactly 8 randomly chosen
ones.
2) It's not using the managed affinity mechanics, so the interrupts cannot
be sanely handled by the kernel, neither affinity wise nor at hotplug
time.
Thanks,
tglx
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH 4/4][RFC v2] x86/apic: Spread the vectors by choosing the idlest CPU
2017-09-07 5:59 ` Thomas Gleixner
@ 2017-09-07 6:23 ` Dan Williams
2017-09-07 6:59 ` Thomas Gleixner
0 siblings, 1 reply; 21+ messages in thread
From: Dan Williams @ 2017-09-07 6:23 UTC (permalink / raw)
To: Thomas Gleixner
Cc: Christoph Hellwig, Yu Chen, X86 ML, Ingo Molnar, H. Peter Anvin,
Rui Zhang, LKML, Rafael J. Wysocki, Len Brown, Peter Zijlstra,
Jeff Kirsher
On Wed, Sep 6, 2017 at 10:59 PM, Thomas Gleixner <tglx@linutronix.de> wrote:
> On Wed, 6 Sep 2017, Dan Williams wrote:
>
>> On Tue, Sep 5, 2017 at 11:15 PM, Christoph Hellwig <hch@lst.de> wrote:
>> > On Wed, Sep 06, 2017 at 12:13:38PM +0800, Yu Chen wrote:
>> >> I agree, the driver could be rewritten, but it might take some time, so
>> >> meanwhile I'm looking at also other possible optimization.
>> >
>> > Which driver are we talking about anyway? Let's start looking at it
>> > and fix the issue there.
>>
>> As far as I understand, it's already fixed there:
>>
>> commit 7c9ae7f053e9e896c24fd23595ba369a5fe322e1
>
> -ENOSUCHCOMMIT
Sorry, that's still pending in -next.
>> Author: Carolyn Wyborny <carolyn.wyborny@intel.com>
>> Date: Tue Jun 20 15:16:53 2017 -0700
>>
>> i40e: Fix for trace found with S4 state
>>
>> This patch fixes a problem found in systems when entering
>> S4 state. This patch fixes the problem by ensuring that
>> the misc vector's IRQ is disabled as well. Without this
>> patch a stack trace can be seen upon entering S4 state.
>>
>> However this seems like something that should be handled generically
>> in the irq-core especially since commit c5cb83bb337c
>> "genirq/cpuhotplug: Handle managed IRQs on CPU hotplug" was headed in
>> that direction. It's otherwise non-obvious when a driver needs to
>> release and re-acquire interrupts or be reworked to use managed
>> interrupts.
>
> There are two problems here:
>
> 1) The driver allocates 300 interrupts and uses exactly 8 randomly chosen
> ones.
>
> 2) It's not using the managed affinity mechanics, so the interrupts cannot
> be sanely handled by the kernel, neither affinity wise nor at hotplug
> time.
Ok, this driver is an obvious candidate, but is there a general
guideline of when a driver must use affinity management? Should we be
emitting a message when a driver exceeds a certain threshold of
unmanaged interrupts to flag this in the future?
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH 4/4][RFC v2] x86/apic: Spread the vectors by choosing the idlest CPU
2017-09-07 6:23 ` Dan Williams
@ 2017-09-07 6:59 ` Thomas Gleixner
0 siblings, 0 replies; 21+ messages in thread
From: Thomas Gleixner @ 2017-09-07 6:59 UTC (permalink / raw)
To: Dan Williams
Cc: Christoph Hellwig, Yu Chen, X86 ML, Ingo Molnar, H. Peter Anvin,
Rui Zhang, LKML, Rafael J. Wysocki, Len Brown, Peter Zijlstra,
Jeff Kirsher
On Wed, 6 Sep 2017, Dan Williams wrote:
> On Wed, Sep 6, 2017 at 10:59 PM, Thomas Gleixner <tglx@linutronix.de> wrote:
> >> commit 7c9ae7f053e9e896c24fd23595ba369a5fe322e1
> >
> > -ENOSUCHCOMMIT
>
> Sorry, that's still pending in -next.
Ok.
> >> Author: Carolyn Wyborny <carolyn.wyborny@intel.com>
> >> Date: Tue Jun 20 15:16:53 2017 -0700
> >>
> >> i40e: Fix for trace found with S4 state
> >>
> >> This patch fixes a problem found in systems when entering
> >> S4 state. This patch fixes the problem by ensuring that
> >> the misc vector's IRQ is disabled as well. Without this
> >> patch a stack trace can be seen upon entering S4 state.
Btw. This changelog is pretty useless.....
> >> However this seems like something that should be handled generically
> >> in the irq-core especially since commit c5cb83bb337c
> >> "genirq/cpuhotplug: Handle managed IRQs on CPU hotplug" was headed in
> >> that direction. It's otherwise non-obvious when a driver needs to
> >> release and re-acquire interrupts or be reworked to use managed
> >> interrupts.
> >
> > There are two problems here:
> >
> > 1) The driver allocates 300 interrupts and uses exactly 8 randomly chosen
> > ones.
> >
> > 2) It's not using the managed affinity mechanics, so the interrupts cannot
> > be sanely handled by the kernel, neither affinity wise nor at hotplug
> > time.
>
> Ok, this driver is an obvious candidate, but is there a general
> guideline of when a driver must use affinity management? Should we be
> emitting a message when a driver exceeds a certain threshold of
> unmanaged interrupts to flag this in the future?
I guess everything which uses multi queues and therefor allocates 8+
vectors is something which falls into that category.
Aside of that, drivers should be sane in terms of allocations. Allocating
metric tons of interrupts for nothing is not really a sign of a proper
thought out resource management.
Thanks,
tglx
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH 4/4][RFC v2] x86/apic: Spread the vectors by choosing the idlest CPU
2017-09-07 5:54 ` Thomas Gleixner
@ 2017-09-07 8:34 ` Yu Chen
2017-09-07 9:45 ` Thomas Gleixner
0 siblings, 1 reply; 21+ messages in thread
From: Yu Chen @ 2017-09-07 8:34 UTC (permalink / raw)
To: Thomas Gleixner
Cc: x86, Ingo Molnar, H. Peter Anvin, Rui Zhang, LKML,
Rafael J. Wysocki, Len Brown, Dan Williams, Christoph Hellwig,
Peter Zijlstra, Jeff Kirsher
On Thu, Sep 07, 2017 at 07:54:09AM +0200, Thomas Gleixner wrote:
> On Thu, 7 Sep 2017, Yu Chen wrote:
> > On Wed, Sep 06, 2017 at 10:03:58AM +0200, Thomas Gleixner wrote:
> > > Can you please apply the debug patch below, boot the machine and right
> > > after login provide the output of
> > >
> > > # cat /sys/kernel/debug/tracing/trace
> > >
> > kworker/0:2-303 [000] .... 9.135467: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 34
> > kworker/0:2-303 [000] .... 9.135476: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 35
> > kworker/0:2-303 [000] .... 9.135484: msi_domain_alloc_irqs: dev: 0000:bb:00.0 nvec 1 virq 36
>
> <SNIP>
>
> > kworker/0:2-303 [000] .... 9.762268: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 331
> > kworker/0:2-303 [000] .... 9.762278: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 332
> > kworker/0:2-303 [000] .... 9.762288: msi_domain_alloc_irqs: dev: 0000:bb:00.3 nvec 1 virq 333
>
> That's 300 vectors.
>
> > bb:00.[0-3] Ethernet controller: Intel Corporation Device 37d0 (rev 03)
> >
> > -+-[0000:b2]-+-00.0-[b3-bc]----00.0-[b4-bc]--+-00.0-[b5-b6]----00.0
> > | | +-01.0-[b7-b8]----00.0
> > | | +-02.0-[b9-ba]----00.0
> > | | \-03.0-[bb-bc]--+-00.0
> > | | +-00.1
> > | | +-00.2
> > | | \-00.3
> >
> > and they are using i40e driver, the vectors should be reserved by:
> > i40e_probe() ->
> > i40e_init_interrupt_scheme() ->
> > i40e_init_msix() ->
> > i40e_reserve_msix_vectors() ->
> > pci_enable_msix_range()
> >
> > # ls /sys/kernel/debug/irq/irqs
> > 0 10 11 13 142 184 217 259 292 31 33
> > 337 339 340 342 344 346 348 350 352 354 356
> > 358 360 362 364 366 368 370 372 374 376 378
> > 380 382 384 386 388 390 392 394 4 6 7 9
> > 1 109 12 14 15 2 24 26 3 32 335
> > 338 34 341 343 345 347 349 351 353 355 357
> > 359 361 363 365 367 369 371 373 375 377 379
> > 381 383 385 387 389 391 393 395 5 67 8
>
> Out of these 300 interrupts exactly 8 randomly selected ones are actively
> used. And the other 292 interrupts are just there because it might need
> them in the future when the 32 CPU machine gets magically upgraded to 4096
> cores at runtime?
>
Humm, the 292 vectors remain disabled due to the network devices have
not been enabled(say,ifconfig up does not get invoked), so request_irq()
does not get invoked for these vectors? I have an impression that once
I've borrowed some fiber cables to connect the platform, the active IRQ
from i40e raised a lot, although I don't have these expensive cables
now...
> Can the i40e people @intel please fix this waste of resources and sanitize
> their interrupt allocation scheme?
>
> Please switch it over to managed interrupts so the affinity spreading
> happens in a sane way and the interrupts are properly managed on CPU
> hotplug.
Ok, I think currently in i40e driver the reservation of vectors
leverages pci_enable_msix_range() and did not provide the affinity
hit to low level IRQ system thus the managed interrupts is not enabled
there(although later in i40e driver we use irq_set_affinity_hint() to
spread the IRQs)
Thanks,
Yu
>
> Thanks,
>
> tglx
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH 4/4][RFC v2] x86/apic: Spread the vectors by choosing the idlest CPU
2017-09-07 8:34 ` Yu Chen
@ 2017-09-07 9:45 ` Thomas Gleixner
0 siblings, 0 replies; 21+ messages in thread
From: Thomas Gleixner @ 2017-09-07 9:45 UTC (permalink / raw)
To: Yu Chen
Cc: x86, Ingo Molnar, H. Peter Anvin, Rui Zhang, LKML,
Rafael J. Wysocki, Len Brown, Dan Williams, Christoph Hellwig,
Peter Zijlstra, Jeff Kirsher
On Thu, 7 Sep 2017, Yu Chen wrote:
> On Thu, Sep 07, 2017 at 07:54:09AM +0200, Thomas Gleixner wrote:
> > Please switch it over to managed interrupts so the affinity spreading
> > happens in a sane way and the interrupts are properly managed on CPU
> > hotplug.
> Ok, I think currently in i40e driver the reservation of vectors
> leverages pci_enable_msix_range() and did not provide the affinity
> hit to low level IRQ system thus the managed interrupts is not enabled
> there(although later in i40e driver we use irq_set_affinity_hint() to
> spread the IRQs)
The affinity hint has nothing to do with that. It's a hint which tells user
space irqbalanced what the desired placement of the interrupt should
be. That was never used for spreading the affinity automatically in the
kernel and will never be used to do so. It was a design failure from the
very beginning and should be eliminated ASAP.
The general problem here is the way how the whole MSI(X) machinery works in
the kernel.
pci_enable_msix()
allocate_interrupts()
allocate_irqdescs()
allocate_resources()
allocate_DMAR_entries()
allocate_vectors()
initialize_MSI_entries()
The reason for this is historical. Drivers expect, that request_irq()
works, when they allocated the required resources upfront.
Of course this could be changed, but there are issues with that:
1) The driver must ensure that it does not enable any of the internal
interrupt delivery mechanisms in the device before request_irq() has
succeeded.
That needs auditing drivers all over the place or we just ignore that
and leave everyone puzzled why things suddenly stop to work.
2) Reservation accounting
When no vectors are allocated, we still need to make reservations so
we can tell a driver that the vector space is exhausted when it
invokes pci_enable_msix(). But how do we size the reservation space?
Based on nr_possible_cpus(), nr_online_cpus() or some other
heuristics?
Sure, we can just ignore that and resort to overcommitment and fail
request_irq() when resources are not available, which brings us back
to #1
But resorting to overcommitment does not make the cpu hotplug problem
magically go away. If queues and interrupts are used, then the non managed
variants are going to break affinities and move stuff to the still online
CPUs, which is going to fail.
Managed irqs just work because the driver stops the queue and the interrupt
(which can even stay requested) is shut down and 'kept' on the outgoing
CPU. If the CPU comes back then the vector is reestablished and the
interrupt started up on the fly. Stuff just works.....
Thanks,
tglx
^ permalink raw reply [flat|nested] 21+ messages in thread
end of thread, other threads:[~2017-09-07 9:45 UTC | newest]
Thread overview: 21+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-09-01 5:03 [PATCH 0/4][RFC v2] x86/irq: Spread vectors on different CPUs Chen Yu
2017-09-01 5:03 ` [PATCH 1/4][RFC v2] x86/apic: Extend the defination for vector_irq Chen Yu
2017-09-01 5:04 ` [PATCH 2/4][RFC v2] x86/apic: Record the number of vectors assigned on a CPU Chen Yu
2017-09-01 5:04 ` [PATCH 3/4] x86/apic: Introduce the per vector cpumask array Chen Yu
2017-09-01 5:04 ` [PATCH 4/4][RFC v2] x86/apic: Spread the vectors by choosing the idlest CPU Chen Yu
2017-09-03 18:17 ` Thomas Gleixner
2017-09-03 19:18 ` RFD: x86: Sanitize the vector allocator Thomas Gleixner
2017-09-05 22:57 ` [PATCH 4/4][RFC v2] x86/apic: Spread the vectors by choosing the idlest CPU Thomas Gleixner
2017-09-06 4:34 ` Yu Chen
2017-09-06 8:03 ` Thomas Gleixner
2017-09-07 2:52 ` Yu Chen
2017-09-07 5:54 ` Thomas Gleixner
2017-09-07 8:34 ` Yu Chen
2017-09-07 9:45 ` Thomas Gleixner
2017-09-06 4:13 ` Yu Chen
2017-09-06 6:15 ` Christoph Hellwig
2017-09-06 17:46 ` Dan Williams
2017-09-07 2:57 ` Yu Chen
2017-09-07 5:59 ` Thomas Gleixner
2017-09-07 6:23 ` Dan Williams
2017-09-07 6:59 ` Thomas Gleixner
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.