From: "Roger Pau Monné" <roger.pau@citrix.com>
To: Jan Beulich <JBeulich@suse.com>
Cc: xen-devel <xen-devel@lists.xenproject.org>,
Wei Liu <wei.liu2@citrix.com>,
Andrew Cooper <andrew.cooper3@citrix.com>
Subject: Re: [Xen-devel] [PATCH v2 06/12] x86/IRQ: consolidate use of ->arch.cpu_mask
Date: Mon, 13 May 2019 13:32:23 +0200 [thread overview]
Message-ID: <20190513113223.pyvrp6nrhbczgbjj@Air-de-Roger> (raw)
Message-ID: <20190513113223.cLwo5tnx326-9Z5v1G3sCyyPnmGCcWBhSu8KwiFnaGo@z> (raw)
In-Reply-To: <5CD2D545020000780022CD3D@prv1-mh.provo.novell.com>
On Wed, May 08, 2019 at 07:10:29AM -0600, Jan Beulich wrote:
> Mixed meaning was implied so far by different pieces of code -
> disagreement was in particular about whether to expect offline CPUs'
> bits to possibly be set. Switch to a mostly consistent meaning
> (exception being high priority interrupts, which would perhaps better
> be switched to the same model as well in due course). Use the field to
> record the vector allocation mask, i.e. potentially including bits of
> offline (parked) CPUs. This implies that before passing the mask to
> certain functions (most notably cpu_mask_to_apicid()) it needs to be
> further reduced to the online subset.
>
> The exception of high priority interrupts is also why for the moment
> _bind_irq_vector() is left as is, despite looking wrong: It's used
> exclusively for IRQ0, which isn't supposed to move off CPU0 at any time.
>
> The prior lack of restricting to online CPUs in set_desc_affinity()
> before calling cpu_mask_to_apicid() in particular allowed (in x2APIC
> clustered mode) offlined CPUs to end up enabled in an IRQ's destination
> field. (I wonder whether vector_allocation_cpumask_flat() shouldn't
> follow a similar model, using cpu_present_map in favor of
> cpu_online_map.)
>
> For IO-APIC code it was definitely wrong to potentially store, as a
> fallback, TARGET_CPUS (i.e. all online ones) into the field, as that
> would have caused problems when determining on which CPUs to release
> vectors when they've gone out of use. Disable interrupts instead when
> no valid target CPU can be established (which code elsewhere should
> guarantee to never happen), and log a message in such an unlikely event.
>
> Signed-off-by: Jan Beulich <jbeulich@suse.com>
Thanks.
Reviewed-by: Roger Pau Monné <roger.pau@citrix.com>
Some comments below.
> ---
> v2: New.
>
> --- a/xen/arch/x86/io_apic.c
> +++ b/xen/arch/x86/io_apic.c
> @@ -680,7 +680,7 @@ void /*__init*/ setup_ioapic_dest(void)
> continue;
> irq = pin_2_irq(irq_entry, ioapic, pin);
> desc = irq_to_desc(irq);
> - BUG_ON(cpumask_empty(desc->arch.cpu_mask));
> + BUG_ON(!cpumask_intersects(desc->arch.cpu_mask, &cpu_online_map));
I wonder if maybe you could instead do:
if ( cpumask_intersects(desc->arch.cpu_mask, &cpu_online_map) )
set_ioapic_affinity_irq(desc, desc->arch.cpu_mask);
else
ASSERT_UNREACHABLE();
I guess if the IRQ is in use by Xen itself the failure ought to be
fatal.
> set_ioapic_affinity_irq(desc, desc->arch.cpu_mask);
> }
>
> @@ -2197,7 +2197,6 @@ int io_apic_set_pci_routing (int ioapic,
> {
> struct irq_desc *desc = irq_to_desc(irq);
> struct IO_APIC_route_entry entry;
> - cpumask_t mask;
> unsigned long flags;
> int vector;
>
> @@ -2232,11 +2231,17 @@ int io_apic_set_pci_routing (int ioapic,
> return vector;
> entry.vector = vector;
>
> - cpumask_copy(&mask, TARGET_CPUS);
> - /* Don't chance ending up with an empty mask. */
> - if (cpumask_intersects(&mask, desc->arch.cpu_mask))
> - cpumask_and(&mask, &mask, desc->arch.cpu_mask);
> - SET_DEST(entry, logical, cpu_mask_to_apicid(&mask));
> + if (cpumask_intersects(desc->arch.cpu_mask, TARGET_CPUS)) {
> + cpumask_t *mask = this_cpu(scratch_cpumask);
> +
> + cpumask_and(mask, desc->arch.cpu_mask, TARGET_CPUS);
> + SET_DEST(entry, logical, cpu_mask_to_apicid(mask));
> + } else {
> + printk(XENLOG_ERR "IRQ%d: no target CPU (%*pb vs %*pb)\n",
> + irq, nr_cpu_ids, cpumask_bits(desc->arch.cpu_mask),
> + nr_cpu_ids, cpumask_bits(TARGET_CPUS));
> + desc->status |= IRQ_DISABLED;
> + }
Hm, part of this file doesn't seem to use Xen coding style, but the
chunk you add below does use it. And there are functions (like
mask_and_ack_level_ioapic_irq that seem to use a mix of coding
styles).
I'm not sure what's the policy here, should new chunks follow Xen's
coding style?
>
> apic_printk(APIC_DEBUG, KERN_DEBUG "IOAPIC[%d]: Set PCI routing entry "
> "(%d-%d -> %#x -> IRQ %d Mode:%i Active:%i)\n", ioapic,
> @@ -2422,7 +2427,21 @@ int ioapic_guest_write(unsigned long phy
> /* Set the vector field to the real vector! */
> rte.vector = desc->arch.vector;
>
> - SET_DEST(rte, logical, cpu_mask_to_apicid(desc->arch.cpu_mask));
> + if ( cpumask_intersects(desc->arch.cpu_mask, TARGET_CPUS) )
> + {
> + cpumask_t *mask = this_cpu(scratch_cpumask);
> +
> + cpumask_and(mask, desc->arch.cpu_mask, TARGET_CPUS);
> + SET_DEST(rte, logical, cpu_mask_to_apicid(mask));
> + }
> + else
> + {
> + gprintk(XENLOG_ERR, "IRQ%d: no target CPU (%*pb vs %*pb)\n",
> + irq, nr_cpu_ids, cpumask_bits(desc->arch.cpu_mask),
> + nr_cpu_ids, cpumask_bits(TARGET_CPUS));
> + desc->status |= IRQ_DISABLED;
> + rte.mask = 1;
> + }
>
> __ioapic_write_entry(apic, pin, 0, rte);
>
> --- a/xen/arch/x86/irq.c
> +++ b/xen/arch/x86/irq.c
> @@ -471,11 +471,13 @@ static int __assign_irq_vector(
> */
> static int current_vector = FIRST_DYNAMIC_VECTOR, current_offset = 0;
> int cpu, err, old_vector;
> - cpumask_t tmp_mask;
> vmask_t *irq_used_vectors = NULL;
>
> old_vector = irq_to_vector(irq);
> - if (old_vector > 0) {
> + if ( old_vector > 0 )
Another candidate to switch to valid_irq_vector or at least make an
explicit comparison with IRQ_VECTOR_UNASSIGNED.
Seeing your reply to my comment in that direction on a previous patch
this can be done as a follow up.
Roger.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
next prev parent reply other threads:[~2019-05-13 11:32 UTC|newest]
Thread overview: 196+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-04-29 11:16 [PATCH 0/9] x86: IRQ management adjustments Jan Beulich
2019-04-29 11:16 ` [Xen-devel] " Jan Beulich
2019-04-29 11:22 ` [PATCH RFC 1/9] x86/IRQ: deal with move-in-progress state in fixup_irqs() Jan Beulich
2019-04-29 11:22 ` [Xen-devel] " Jan Beulich
2019-04-29 12:55 ` Jan Beulich
2019-04-29 12:55 ` [Xen-devel] " Jan Beulich
2019-04-29 13:08 ` Jan Beulich
2019-04-29 13:08 ` [Xen-devel] " Jan Beulich
2019-04-29 11:23 ` [PATCH 2/9] x86/IRQ: deal with move cleanup count " Jan Beulich
2019-04-29 11:23 ` [Xen-devel] " Jan Beulich
2019-05-03 15:21 ` Roger Pau Monné
2019-05-03 15:21 ` [Xen-devel] " Roger Pau Monné
2019-05-06 7:44 ` Jan Beulich
2019-05-06 7:44 ` [Xen-devel] " Jan Beulich
2019-05-07 7:28 ` Jan Beulich
2019-05-07 7:28 ` [Xen-devel] " Jan Beulich
2019-05-07 8:12 ` Roger Pau Monné
2019-05-07 8:12 ` [Xen-devel] " Roger Pau Monné
2019-05-07 9:28 ` Jan Beulich
2019-05-07 9:28 ` [Xen-devel] " Jan Beulich
2019-04-29 11:23 ` [PATCH 3/9] x86/IRQ: improve dump_irqs() Jan Beulich
2019-04-29 11:23 ` [Xen-devel] " Jan Beulich
2019-05-03 15:43 ` Roger Pau Monné
2019-05-03 15:43 ` [Xen-devel] " Roger Pau Monné
2019-05-06 8:06 ` Jan Beulich
2019-05-06 8:06 ` [Xen-devel] " Jan Beulich
2019-04-29 11:24 ` [PATCH 4/9] x86/IRQ: desc->affinity should strictly represent the requested value Jan Beulich
2019-04-29 11:24 ` [Xen-devel] " Jan Beulich
2019-05-03 16:21 ` Roger Pau Monné
2019-05-03 16:21 ` [Xen-devel] " Roger Pau Monné
2019-05-06 8:14 ` Jan Beulich
2019-05-06 8:14 ` [Xen-devel] " Jan Beulich
2019-04-29 11:25 ` [PATCH 5/9] x86/IRQ: fix locking around vector management Jan Beulich
2019-04-29 11:25 ` [Xen-devel] " Jan Beulich
2019-05-06 11:48 ` Roger Pau Monné
2019-05-06 11:48 ` [Xen-devel] " Roger Pau Monné
2019-05-06 13:06 ` Jan Beulich
2019-05-06 13:06 ` [Xen-devel] " Jan Beulich
2019-04-29 11:25 ` [PATCH 6/9] x86/IRQ: reduce unused space in struct arch_irq_desc Jan Beulich
2019-04-29 11:25 ` [Xen-devel] " Jan Beulich
2019-04-29 11:46 ` Andrew Cooper
2019-04-29 11:46 ` [Xen-devel] " Andrew Cooper
2019-04-29 11:26 ` [PATCH 7/9] x86/IRQ: drop redundant cpumask_empty() from move_masked_irq() Jan Beulich
2019-04-29 11:26 ` [Xen-devel] " Jan Beulich
2019-05-06 13:39 ` Roger Pau Monné
2019-05-06 13:39 ` [Xen-devel] " Roger Pau Monné
2019-04-29 11:26 ` [PATCH 8/9] x86/IRQ: make fixup_irqs() skip unconnected internally used interrupts Jan Beulich
2019-04-29 11:26 ` [Xen-devel] " Jan Beulich
2019-05-06 13:52 ` Roger Pau Monné
2019-05-06 13:52 ` [Xen-devel] " Roger Pau Monné
2019-05-06 14:25 ` Jan Beulich
2019-05-06 14:25 ` [Xen-devel] " Jan Beulich
2019-05-06 14:37 ` Roger Pau Monné
2019-05-06 14:37 ` [Xen-devel] " Roger Pau Monné
2019-04-29 11:27 ` [PATCH 9/9] x86/IO-APIC: drop an unused variable from setup_IO_APIC_irqs() Jan Beulich
2019-04-29 11:27 ` [Xen-devel] " Jan Beulich
2019-04-29 11:40 ` Andrew Cooper
2019-04-29 11:40 ` [Xen-devel] " Andrew Cooper
2019-04-29 15:40 ` [PATCH v1b 1/9] x86/IRQ: deal with move-in-progress state in fixup_irqs() Jan Beulich
2019-04-29 15:40 ` [Xen-devel] " Jan Beulich
2019-05-03 9:19 ` Roger Pau Monné
2019-05-03 9:19 ` [Xen-devel] " Roger Pau Monné
2019-05-03 14:10 ` Jan Beulich
2019-05-03 14:10 ` [Xen-devel] " Jan Beulich
2019-05-06 7:15 ` Jan Beulich
2019-05-06 7:15 ` [Xen-devel] " Jan Beulich
2019-05-06 14:28 ` Roger Pau Monné
2019-05-06 14:28 ` [Xen-devel] " Roger Pau Monné
2019-05-06 15:00 ` Jan Beulich
2019-05-06 15:00 ` [Xen-devel] " Jan Beulich
2019-05-08 12:59 ` [PATCH v2 00/12] x86: IRQ management adjustments Jan Beulich
2019-05-08 12:59 ` [Xen-devel] " Jan Beulich
2019-05-08 13:03 ` [PATCH v2 01/12] x86/IRQ: deal with move-in-progress state in fixup_irqs() Jan Beulich
2019-05-08 13:03 ` [Xen-devel] " Jan Beulich
2019-05-13 9:04 ` Roger Pau Monné
2019-05-13 9:04 ` [Xen-devel] " Roger Pau Monné
2019-05-13 9:09 ` Jan Beulich
2019-05-13 9:09 ` [Xen-devel] " Jan Beulich
2019-05-08 13:03 ` [PATCH v2 02/12] x86/IRQ: deal with move cleanup count " Jan Beulich
2019-05-08 13:03 ` [Xen-devel] " Jan Beulich
2019-05-08 13:07 ` [PATCH v2 03/12] x86/IRQ: avoid UB (or worse) in trace_irq_mask() Jan Beulich
2019-05-08 13:07 ` [Xen-devel] " Jan Beulich
2019-05-13 9:08 ` Roger Pau Monné
2019-05-13 9:08 ` [Xen-devel] " Roger Pau Monné
2019-05-13 10:42 ` George Dunlap
2019-05-13 10:42 ` [Xen-devel] " George Dunlap
2019-05-13 12:05 ` Jan Beulich
2019-05-13 12:05 ` [Xen-devel] " Jan Beulich
2019-05-08 13:08 ` [PATCH v2 04/12] x86/IRQ: improve dump_irqs() Jan Beulich
2019-05-08 13:08 ` [Xen-devel] " Jan Beulich
2019-05-08 13:09 ` [PATCH v2 05/12] x86/IRQ: desc->affinity should strictly represent the requested value Jan Beulich
2019-05-08 13:09 ` [Xen-devel] " Jan Beulich
2019-05-08 13:10 ` [PATCH v2 06/12] x86/IRQ: consolidate use of ->arch.cpu_mask Jan Beulich
2019-05-08 13:10 ` [Xen-devel] " Jan Beulich
2019-05-13 11:32 ` Roger Pau Monné [this message]
2019-05-13 11:32 ` Roger Pau Monné
2019-05-13 15:21 ` Jan Beulich
2019-05-13 15:21 ` [Xen-devel] " Jan Beulich
2019-05-08 13:10 ` [PATCH v2 07/12] x86/IRQ: fix locking around vector management Jan Beulich
2019-05-08 13:10 ` [Xen-devel] " Jan Beulich
2019-05-08 13:16 ` Jan Beulich
2019-05-08 13:16 ` [Xen-devel] " Jan Beulich
2019-05-11 0:11 ` Tian, Kevin
2019-05-11 0:11 ` [Xen-devel] " Tian, Kevin
2019-05-13 13:48 ` Roger Pau Monné
2019-05-13 13:48 ` [Xen-devel] " Roger Pau Monné
2019-05-13 14:19 ` Jan Beulich
2019-05-13 14:19 ` [Xen-devel] " Jan Beulich
2019-05-13 14:45 ` Roger Pau Monné
2019-05-13 14:45 ` [Xen-devel] " Roger Pau Monné
2019-05-13 15:05 ` Jan Beulich
2019-05-13 15:05 ` [Xen-devel] " Jan Beulich
2019-05-08 13:11 ` [PATCH v2 08/12] x86/IRQs: correct/tighten vector check in _clear_irq_vector() Jan Beulich
2019-05-08 13:11 ` [Xen-devel] " Jan Beulich
2019-05-13 14:01 ` Roger Pau Monné
2019-05-13 14:01 ` [Xen-devel] " Roger Pau Monné
2019-05-08 13:12 ` [PATCH v2 09/12] x86/IRQ: make fixup_irqs() skip unconnected internally used interrupts Jan Beulich
2019-05-08 13:12 ` [Xen-devel] " Jan Beulich
2019-05-08 13:13 ` [PATCH v2 10/12] x86/IRQ: reduce unused space in struct arch_irq_desc Jan Beulich
2019-05-08 13:13 ` [Xen-devel] " Jan Beulich
2019-05-08 13:13 ` [PATCH v2 11/12] x86/IRQ: drop redundant cpumask_empty() from move_masked_irq() Jan Beulich
2019-05-08 13:13 ` [Xen-devel] " Jan Beulich
2019-05-08 13:14 ` [PATCH v2 12/12] x86/IRQ: simplify and rename pirq_acktype() Jan Beulich
2019-05-08 13:14 ` [Xen-devel] " Jan Beulich
2019-05-13 14:14 ` Roger Pau Monné
2019-05-13 14:14 ` [Xen-devel] " Roger Pau Monné
2019-05-17 10:39 ` [PATCH v3 00/15] x86: IRQ management adjustments Jan Beulich
2019-05-17 10:39 ` [Xen-devel] " Jan Beulich
2019-05-17 10:44 ` [PATCH v3 01/15] x86/IRQ: deal with move-in-progress state in fixup_irqs() Jan Beulich
2019-05-17 10:44 ` [Xen-devel] " Jan Beulich
2019-07-03 15:39 ` Andrew Cooper
2019-07-04 9:32 ` Jan Beulich
2019-05-17 10:45 ` [PATCH v3 02/15] x86/IRQ: deal with move cleanup count " Jan Beulich
2019-05-17 10:45 ` [Xen-devel] " Jan Beulich
2019-07-03 16:32 ` Andrew Cooper
2019-05-17 10:46 ` [PATCH v3 03/15] x86/IRQ: improve dump_irqs() Jan Beulich
2019-05-17 10:46 ` [Xen-devel] " Jan Beulich
2019-07-03 16:39 ` Andrew Cooper
2019-05-17 10:46 ` [PATCH v3 04/15] x86/IRQ: desc->affinity should strictly represent the requested value Jan Beulich
2019-05-17 10:46 ` [Xen-devel] " Jan Beulich
2019-07-03 17:58 ` Andrew Cooper
2019-07-04 9:37 ` Jan Beulich
2019-05-17 10:47 ` [PATCH v3 05/15] x86/IRQ: consolidate use of ->arch.cpu_mask Jan Beulich
2019-05-17 10:47 ` [Xen-devel] " Jan Beulich
2019-07-03 18:07 ` Andrew Cooper
2019-05-17 10:47 ` [PATCH v3 06/15] x86/IRQ: fix locking around vector management Jan Beulich
2019-05-17 10:47 ` [Xen-devel] " Jan Beulich
2019-07-03 18:23 ` Andrew Cooper
2019-07-04 9:54 ` Jan Beulich
2019-05-17 10:48 ` [PATCH v3 07/15] x86/IRQ: target online CPUs when binding guest IRQ Jan Beulich
2019-05-17 10:48 ` [Xen-devel] " Jan Beulich
2019-05-20 11:40 ` Roger Pau Monné
2019-05-20 11:40 ` [Xen-devel] " Roger Pau Monné
2019-05-20 15:17 ` Jan Beulich
2019-05-20 15:17 ` [Xen-devel] " Jan Beulich
2019-05-22 9:41 ` Roger Pau Monné
2019-05-22 9:41 ` [Xen-devel] " Roger Pau Monné
2019-07-03 18:30 ` Andrew Cooper
2019-05-17 10:49 ` [PATCH v3 08/15] x86/IRQs: correct/tighten vector check in _clear_irq_vector() Jan Beulich
2019-05-17 10:49 ` [Xen-devel] " Jan Beulich
2019-07-03 18:31 ` Andrew Cooper
2019-05-17 10:49 ` [PATCH v3 09/15] x86/IRQ: make fixup_irqs() skip unconnected internally used interrupts Jan Beulich
2019-05-17 10:49 ` [Xen-devel] " Jan Beulich
2019-07-03 18:36 ` Andrew Cooper
2019-05-17 10:50 ` [PATCH v3 10/15] x86/IRQ: drop redundant cpumask_empty() from move_masked_irq() Jan Beulich
2019-05-17 10:50 ` [Xen-devel] " Jan Beulich
2019-07-03 18:38 ` Andrew Cooper
2019-05-17 10:51 ` [PATCH v3 11/15] x86/IRQ: simplify and rename pirq_acktype() Jan Beulich
2019-05-17 10:51 ` [Xen-devel] " Jan Beulich
2019-07-03 18:39 ` Andrew Cooper
2019-05-17 10:51 ` [PATCH v3 12/15] x86/IRQ: add explicit tracing-enabled check to trace_irq_mask() Jan Beulich
2019-05-17 10:51 ` [Xen-devel] " Jan Beulich
2019-05-20 11:46 ` Roger Pau Monné
2019-05-20 11:46 ` [Xen-devel] " Roger Pau Monné
2019-07-03 18:41 ` Andrew Cooper
2019-07-04 10:01 ` Jan Beulich
2019-05-17 10:52 ` [PATCH v3 13/15] x86/IRQ: tighten vector checks Jan Beulich
2019-05-17 10:52 ` [Xen-devel] " Jan Beulich
2019-05-20 14:04 ` Roger Pau Monné
2019-05-20 14:04 ` [Xen-devel] " Roger Pau Monné
2019-05-20 15:26 ` Jan Beulich
2019-05-20 15:26 ` [Xen-devel] " Jan Beulich
2019-05-22 16:42 ` Roger Pau Monné
2019-05-22 16:42 ` [Xen-devel] " Roger Pau Monné
2019-05-23 8:36 ` Jan Beulich
2019-05-23 8:36 ` [Xen-devel] " Jan Beulich
2019-07-03 18:42 ` Andrew Cooper
2019-05-17 10:52 ` [PATCH v3 14/15] x86/IRQ: eliminate some on-stack cpumask_t instances Jan Beulich
2019-05-17 10:52 ` [Xen-devel] " Jan Beulich
2019-05-20 14:22 ` Roger Pau Monné
2019-05-20 14:22 ` [Xen-devel] " Roger Pau Monné
2019-07-03 18:44 ` Andrew Cooper
2019-07-04 10:04 ` Jan Beulich
2019-05-17 10:53 ` [PATCH v3 15/15] x86/IRQ: move {,_}clear_irq_vector() Jan Beulich
2019-05-17 10:53 ` [Xen-devel] " Jan Beulich
2019-07-03 18:45 ` [Xen-devel] [PATCH v3 15/15] x86/IRQ: move {, _}clear_irq_vector() Andrew Cooper
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20190513113223.pyvrp6nrhbczgbjj@Air-de-Roger \
--to=roger.pau@citrix.com \
--cc=JBeulich@suse.com \
--cc=andrew.cooper3@citrix.com \
--cc=wei.liu2@citrix.com \
--cc=xen-devel@lists.xenproject.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).