From: "Jan Beulich" <JBeulich@suse.com> To: Roger Pau Monne <roger.pau@citrix.com> Cc: Andrew Cooper <andrew.cooper3@citrix.com>, Wei Liu <wei.liu2@citrix.com>, xen-devel <xen-devel@lists.xenproject.org> Subject: Re: [PATCH 5/9] x86/IRQ: fix locking around vector management Date: Mon, 06 May 2019 07:06:10 -0600 [thread overview] Message-ID: <5CD03142020000780022C2F6@prv1-mh.provo.novell.com> (raw) In-Reply-To: <20190506114809.56e2sl4z6w2z4na2@Air-de-Roger> >>> On 06.05.19 at 13:48, <roger.pau@citrix.com> wrote: > On Mon, Apr 29, 2019 at 05:25:03AM -0600, Jan Beulich wrote: >> All of __{assign,bind,clear}_irq_vector() manipulate struct irq_desc >> fields, and hence ought to be called with the descriptor lock held in >> addition to vector_lock. This is currently the case for only >> set_desc_affinity() and destroy_irq(), which also clarifies what the > > AFAICT set_desc_affinity is called from set_ioapic_affinity_irq which in > turn is called from setup_ioapic_dest without holding the desc lock. > Is this fine because that's only used a boot time? No, this isn't fine, and it's also not only called at boot time. I simply didn't spot this case of function re-use - I had come to the conclusion that all calls to set_desc_affinity() would come through the .set_affinity hook pointers (or happen sufficiently early). VT-d's adjust_irq_affinity() has a similar issue. At boot time alone would be insufficient anyway. Not taking locks can only be safe prior to bringing up APs; any later skipping of locking would at least require additional justification. >> nesting behavior between the locks has to be. Reflect the new >> expectation by having these functions all take a descriptor as >> parameter instead of an interrupt number. >> >> Drop one of the two leading underscores from all three functions at >> the same time. >> >> There's one case left where descriptors get manipulated with just >> vector_lock held: setup_vector_irq() assumes its caller to acquire >> vector_lock, and hence can't itself acquire the descriptor locks (wrong >> lock order). I don't currently see how to address this. > > Can you take the desc lock and vector lock for each irq in the second > loop of setup_vector_irq and remove the vector locking from the caller? > > That might be inefficient, but it's just done for CPU initialization. > > AFAICT the first loop of setup_vector_irq doesn't require any locking > since it's per-cpu initialization. It's not so much the first lock afaict. It's the combined action of calling this function and setting the online bit which needs the lock held around it. I.e. the function setting bits in various descriptors' CPU masks (and the tracking of the vector -> IRQ relationships) has to be atomic (to the outside world) with the setting of the CPU's bit in cpu_online_map. >> Signed-off-by: Jan Beulich <jbeulich@suse.com> > > Change looks good to me: > > Reviewed-by: Roger Pau Monné <roger.pau@citrix.com> Thanks, but I'll not add this for now, given the further locking to be added as per above. Jan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel
WARNING: multiple messages have this Message-ID (diff)
From: "Jan Beulich" <JBeulich@suse.com> To: "Roger Pau Monne" <roger.pau@citrix.com> Cc: Andrew Cooper <andrew.cooper3@citrix.com>, Wei Liu <wei.liu2@citrix.com>, xen-devel <xen-devel@lists.xenproject.org> Subject: Re: [Xen-devel] [PATCH 5/9] x86/IRQ: fix locking around vector management Date: Mon, 06 May 2019 07:06:10 -0600 [thread overview] Message-ID: <5CD03142020000780022C2F6@prv1-mh.provo.novell.com> (raw) Message-ID: <20190506130610.O_Tj6aulowDofNDrg9ki-pOuotNUJZuwi4YacoBtOfk@z> (raw) In-Reply-To: <20190506114809.56e2sl4z6w2z4na2@Air-de-Roger> >>> On 06.05.19 at 13:48, <roger.pau@citrix.com> wrote: > On Mon, Apr 29, 2019 at 05:25:03AM -0600, Jan Beulich wrote: >> All of __{assign,bind,clear}_irq_vector() manipulate struct irq_desc >> fields, and hence ought to be called with the descriptor lock held in >> addition to vector_lock. This is currently the case for only >> set_desc_affinity() and destroy_irq(), which also clarifies what the > > AFAICT set_desc_affinity is called from set_ioapic_affinity_irq which in > turn is called from setup_ioapic_dest without holding the desc lock. > Is this fine because that's only used a boot time? No, this isn't fine, and it's also not only called at boot time. I simply didn't spot this case of function re-use - I had come to the conclusion that all calls to set_desc_affinity() would come through the .set_affinity hook pointers (or happen sufficiently early). VT-d's adjust_irq_affinity() has a similar issue. At boot time alone would be insufficient anyway. Not taking locks can only be safe prior to bringing up APs; any later skipping of locking would at least require additional justification. >> nesting behavior between the locks has to be. Reflect the new >> expectation by having these functions all take a descriptor as >> parameter instead of an interrupt number. >> >> Drop one of the two leading underscores from all three functions at >> the same time. >> >> There's one case left where descriptors get manipulated with just >> vector_lock held: setup_vector_irq() assumes its caller to acquire >> vector_lock, and hence can't itself acquire the descriptor locks (wrong >> lock order). I don't currently see how to address this. > > Can you take the desc lock and vector lock for each irq in the second > loop of setup_vector_irq and remove the vector locking from the caller? > > That might be inefficient, but it's just done for CPU initialization. > > AFAICT the first loop of setup_vector_irq doesn't require any locking > since it's per-cpu initialization. It's not so much the first lock afaict. It's the combined action of calling this function and setting the online bit which needs the lock held around it. I.e. the function setting bits in various descriptors' CPU masks (and the tracking of the vector -> IRQ relationships) has to be atomic (to the outside world) with the setting of the CPU's bit in cpu_online_map. >> Signed-off-by: Jan Beulich <jbeulich@suse.com> > > Change looks good to me: > > Reviewed-by: Roger Pau Monné <roger.pau@citrix.com> Thanks, but I'll not add this for now, given the further locking to be added as per above. Jan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel
next prev parent reply other threads:[~2019-05-06 13:06 UTC|newest] Thread overview: 196+ messages / expand[flat|nested] mbox.gz Atom feed top 2019-04-29 11:16 [PATCH 0/9] x86: IRQ management adjustments Jan Beulich 2019-04-29 11:16 ` [Xen-devel] " Jan Beulich 2019-04-29 11:22 ` [PATCH RFC 1/9] x86/IRQ: deal with move-in-progress state in fixup_irqs() Jan Beulich 2019-04-29 11:22 ` [Xen-devel] " Jan Beulich 2019-04-29 12:55 ` Jan Beulich 2019-04-29 12:55 ` [Xen-devel] " Jan Beulich 2019-04-29 13:08 ` Jan Beulich 2019-04-29 13:08 ` [Xen-devel] " Jan Beulich 2019-04-29 11:23 ` [PATCH 2/9] x86/IRQ: deal with move cleanup count " Jan Beulich 2019-04-29 11:23 ` [Xen-devel] " Jan Beulich 2019-05-03 15:21 ` Roger Pau Monné 2019-05-03 15:21 ` [Xen-devel] " Roger Pau Monné 2019-05-06 7:44 ` Jan Beulich 2019-05-06 7:44 ` [Xen-devel] " Jan Beulich 2019-05-07 7:28 ` Jan Beulich 2019-05-07 7:28 ` [Xen-devel] " Jan Beulich 2019-05-07 8:12 ` Roger Pau Monné 2019-05-07 8:12 ` [Xen-devel] " Roger Pau Monné 2019-05-07 9:28 ` Jan Beulich 2019-05-07 9:28 ` [Xen-devel] " Jan Beulich 2019-04-29 11:23 ` [PATCH 3/9] x86/IRQ: improve dump_irqs() Jan Beulich 2019-04-29 11:23 ` [Xen-devel] " Jan Beulich 2019-05-03 15:43 ` Roger Pau Monné 2019-05-03 15:43 ` [Xen-devel] " Roger Pau Monné 2019-05-06 8:06 ` Jan Beulich 2019-05-06 8:06 ` [Xen-devel] " Jan Beulich 2019-04-29 11:24 ` [PATCH 4/9] x86/IRQ: desc->affinity should strictly represent the requested value Jan Beulich 2019-04-29 11:24 ` [Xen-devel] " Jan Beulich 2019-05-03 16:21 ` Roger Pau Monné 2019-05-03 16:21 ` [Xen-devel] " Roger Pau Monné 2019-05-06 8:14 ` Jan Beulich 2019-05-06 8:14 ` [Xen-devel] " Jan Beulich 2019-04-29 11:25 ` [PATCH 5/9] x86/IRQ: fix locking around vector management Jan Beulich 2019-04-29 11:25 ` [Xen-devel] " Jan Beulich 2019-05-06 11:48 ` Roger Pau Monné 2019-05-06 11:48 ` [Xen-devel] " Roger Pau Monné 2019-05-06 13:06 ` Jan Beulich [this message] 2019-05-06 13:06 ` Jan Beulich 2019-04-29 11:25 ` [PATCH 6/9] x86/IRQ: reduce unused space in struct arch_irq_desc Jan Beulich 2019-04-29 11:25 ` [Xen-devel] " Jan Beulich 2019-04-29 11:46 ` Andrew Cooper 2019-04-29 11:46 ` [Xen-devel] " Andrew Cooper 2019-04-29 11:26 ` [PATCH 7/9] x86/IRQ: drop redundant cpumask_empty() from move_masked_irq() Jan Beulich 2019-04-29 11:26 ` [Xen-devel] " Jan Beulich 2019-05-06 13:39 ` Roger Pau Monné 2019-05-06 13:39 ` [Xen-devel] " Roger Pau Monné 2019-04-29 11:26 ` [PATCH 8/9] x86/IRQ: make fixup_irqs() skip unconnected internally used interrupts Jan Beulich 2019-04-29 11:26 ` [Xen-devel] " Jan Beulich 2019-05-06 13:52 ` Roger Pau Monné 2019-05-06 13:52 ` [Xen-devel] " Roger Pau Monné 2019-05-06 14:25 ` Jan Beulich 2019-05-06 14:25 ` [Xen-devel] " Jan Beulich 2019-05-06 14:37 ` Roger Pau Monné 2019-05-06 14:37 ` [Xen-devel] " Roger Pau Monné 2019-04-29 11:27 ` [PATCH 9/9] x86/IO-APIC: drop an unused variable from setup_IO_APIC_irqs() Jan Beulich 2019-04-29 11:27 ` [Xen-devel] " Jan Beulich 2019-04-29 11:40 ` Andrew Cooper 2019-04-29 11:40 ` [Xen-devel] " Andrew Cooper 2019-04-29 15:40 ` [PATCH v1b 1/9] x86/IRQ: deal with move-in-progress state in fixup_irqs() Jan Beulich 2019-04-29 15:40 ` [Xen-devel] " Jan Beulich 2019-05-03 9:19 ` Roger Pau Monné 2019-05-03 9:19 ` [Xen-devel] " Roger Pau Monné 2019-05-03 14:10 ` Jan Beulich 2019-05-03 14:10 ` [Xen-devel] " Jan Beulich 2019-05-06 7:15 ` Jan Beulich 2019-05-06 7:15 ` [Xen-devel] " Jan Beulich 2019-05-06 14:28 ` Roger Pau Monné 2019-05-06 14:28 ` [Xen-devel] " Roger Pau Monné 2019-05-06 15:00 ` Jan Beulich 2019-05-06 15:00 ` [Xen-devel] " Jan Beulich 2019-05-08 12:59 ` [PATCH v2 00/12] x86: IRQ management adjustments Jan Beulich 2019-05-08 12:59 ` [Xen-devel] " Jan Beulich 2019-05-08 13:03 ` [PATCH v2 01/12] x86/IRQ: deal with move-in-progress state in fixup_irqs() Jan Beulich 2019-05-08 13:03 ` [Xen-devel] " Jan Beulich 2019-05-13 9:04 ` Roger Pau Monné 2019-05-13 9:04 ` [Xen-devel] " Roger Pau Monné 2019-05-13 9:09 ` Jan Beulich 2019-05-13 9:09 ` [Xen-devel] " Jan Beulich 2019-05-08 13:03 ` [PATCH v2 02/12] x86/IRQ: deal with move cleanup count " Jan Beulich 2019-05-08 13:03 ` [Xen-devel] " Jan Beulich 2019-05-08 13:07 ` [PATCH v2 03/12] x86/IRQ: avoid UB (or worse) in trace_irq_mask() Jan Beulich 2019-05-08 13:07 ` [Xen-devel] " Jan Beulich 2019-05-13 9:08 ` Roger Pau Monné 2019-05-13 9:08 ` [Xen-devel] " Roger Pau Monné 2019-05-13 10:42 ` George Dunlap 2019-05-13 10:42 ` [Xen-devel] " George Dunlap 2019-05-13 12:05 ` Jan Beulich 2019-05-13 12:05 ` [Xen-devel] " Jan Beulich 2019-05-08 13:08 ` [PATCH v2 04/12] x86/IRQ: improve dump_irqs() Jan Beulich 2019-05-08 13:08 ` [Xen-devel] " Jan Beulich 2019-05-08 13:09 ` [PATCH v2 05/12] x86/IRQ: desc->affinity should strictly represent the requested value Jan Beulich 2019-05-08 13:09 ` [Xen-devel] " Jan Beulich 2019-05-08 13:10 ` [PATCH v2 06/12] x86/IRQ: consolidate use of ->arch.cpu_mask Jan Beulich 2019-05-08 13:10 ` [Xen-devel] " Jan Beulich 2019-05-13 11:32 ` Roger Pau Monné 2019-05-13 11:32 ` [Xen-devel] " Roger Pau Monné 2019-05-13 15:21 ` Jan Beulich 2019-05-13 15:21 ` [Xen-devel] " Jan Beulich 2019-05-08 13:10 ` [PATCH v2 07/12] x86/IRQ: fix locking around vector management Jan Beulich 2019-05-08 13:10 ` [Xen-devel] " Jan Beulich 2019-05-08 13:16 ` Jan Beulich 2019-05-08 13:16 ` [Xen-devel] " Jan Beulich 2019-05-11 0:11 ` Tian, Kevin 2019-05-11 0:11 ` [Xen-devel] " Tian, Kevin 2019-05-13 13:48 ` Roger Pau Monné 2019-05-13 13:48 ` [Xen-devel] " Roger Pau Monné 2019-05-13 14:19 ` Jan Beulich 2019-05-13 14:19 ` [Xen-devel] " Jan Beulich 2019-05-13 14:45 ` Roger Pau Monné 2019-05-13 14:45 ` [Xen-devel] " Roger Pau Monné 2019-05-13 15:05 ` Jan Beulich 2019-05-13 15:05 ` [Xen-devel] " Jan Beulich 2019-05-08 13:11 ` [PATCH v2 08/12] x86/IRQs: correct/tighten vector check in _clear_irq_vector() Jan Beulich 2019-05-08 13:11 ` [Xen-devel] " Jan Beulich 2019-05-13 14:01 ` Roger Pau Monné 2019-05-13 14:01 ` [Xen-devel] " Roger Pau Monné 2019-05-08 13:12 ` [PATCH v2 09/12] x86/IRQ: make fixup_irqs() skip unconnected internally used interrupts Jan Beulich 2019-05-08 13:12 ` [Xen-devel] " Jan Beulich 2019-05-08 13:13 ` [PATCH v2 10/12] x86/IRQ: reduce unused space in struct arch_irq_desc Jan Beulich 2019-05-08 13:13 ` [Xen-devel] " Jan Beulich 2019-05-08 13:13 ` [PATCH v2 11/12] x86/IRQ: drop redundant cpumask_empty() from move_masked_irq() Jan Beulich 2019-05-08 13:13 ` [Xen-devel] " Jan Beulich 2019-05-08 13:14 ` [PATCH v2 12/12] x86/IRQ: simplify and rename pirq_acktype() Jan Beulich 2019-05-08 13:14 ` [Xen-devel] " Jan Beulich 2019-05-13 14:14 ` Roger Pau Monné 2019-05-13 14:14 ` [Xen-devel] " Roger Pau Monné 2019-05-17 10:39 ` [PATCH v3 00/15] x86: IRQ management adjustments Jan Beulich 2019-05-17 10:39 ` [Xen-devel] " Jan Beulich 2019-05-17 10:44 ` [PATCH v3 01/15] x86/IRQ: deal with move-in-progress state in fixup_irqs() Jan Beulich 2019-05-17 10:44 ` [Xen-devel] " Jan Beulich 2019-07-03 15:39 ` Andrew Cooper 2019-07-04 9:32 ` Jan Beulich 2019-05-17 10:45 ` [PATCH v3 02/15] x86/IRQ: deal with move cleanup count " Jan Beulich 2019-05-17 10:45 ` [Xen-devel] " Jan Beulich 2019-07-03 16:32 ` Andrew Cooper 2019-05-17 10:46 ` [PATCH v3 03/15] x86/IRQ: improve dump_irqs() Jan Beulich 2019-05-17 10:46 ` [Xen-devel] " Jan Beulich 2019-07-03 16:39 ` Andrew Cooper 2019-05-17 10:46 ` [PATCH v3 04/15] x86/IRQ: desc->affinity should strictly represent the requested value Jan Beulich 2019-05-17 10:46 ` [Xen-devel] " Jan Beulich 2019-07-03 17:58 ` Andrew Cooper 2019-07-04 9:37 ` Jan Beulich 2019-05-17 10:47 ` [PATCH v3 05/15] x86/IRQ: consolidate use of ->arch.cpu_mask Jan Beulich 2019-05-17 10:47 ` [Xen-devel] " Jan Beulich 2019-07-03 18:07 ` Andrew Cooper 2019-05-17 10:47 ` [PATCH v3 06/15] x86/IRQ: fix locking around vector management Jan Beulich 2019-05-17 10:47 ` [Xen-devel] " Jan Beulich 2019-07-03 18:23 ` Andrew Cooper 2019-07-04 9:54 ` Jan Beulich 2019-05-17 10:48 ` [PATCH v3 07/15] x86/IRQ: target online CPUs when binding guest IRQ Jan Beulich 2019-05-17 10:48 ` [Xen-devel] " Jan Beulich 2019-05-20 11:40 ` Roger Pau Monné 2019-05-20 11:40 ` [Xen-devel] " Roger Pau Monné 2019-05-20 15:17 ` Jan Beulich 2019-05-20 15:17 ` [Xen-devel] " Jan Beulich 2019-05-22 9:41 ` Roger Pau Monné 2019-05-22 9:41 ` [Xen-devel] " Roger Pau Monné 2019-07-03 18:30 ` Andrew Cooper 2019-05-17 10:49 ` [PATCH v3 08/15] x86/IRQs: correct/tighten vector check in _clear_irq_vector() Jan Beulich 2019-05-17 10:49 ` [Xen-devel] " Jan Beulich 2019-07-03 18:31 ` Andrew Cooper 2019-05-17 10:49 ` [PATCH v3 09/15] x86/IRQ: make fixup_irqs() skip unconnected internally used interrupts Jan Beulich 2019-05-17 10:49 ` [Xen-devel] " Jan Beulich 2019-07-03 18:36 ` Andrew Cooper 2019-05-17 10:50 ` [PATCH v3 10/15] x86/IRQ: drop redundant cpumask_empty() from move_masked_irq() Jan Beulich 2019-05-17 10:50 ` [Xen-devel] " Jan Beulich 2019-07-03 18:38 ` Andrew Cooper 2019-05-17 10:51 ` [PATCH v3 11/15] x86/IRQ: simplify and rename pirq_acktype() Jan Beulich 2019-05-17 10:51 ` [Xen-devel] " Jan Beulich 2019-07-03 18:39 ` Andrew Cooper 2019-05-17 10:51 ` [PATCH v3 12/15] x86/IRQ: add explicit tracing-enabled check to trace_irq_mask() Jan Beulich 2019-05-17 10:51 ` [Xen-devel] " Jan Beulich 2019-05-20 11:46 ` Roger Pau Monné 2019-05-20 11:46 ` [Xen-devel] " Roger Pau Monné 2019-07-03 18:41 ` Andrew Cooper 2019-07-04 10:01 ` Jan Beulich 2019-05-17 10:52 ` [PATCH v3 13/15] x86/IRQ: tighten vector checks Jan Beulich 2019-05-17 10:52 ` [Xen-devel] " Jan Beulich 2019-05-20 14:04 ` Roger Pau Monné 2019-05-20 14:04 ` [Xen-devel] " Roger Pau Monné 2019-05-20 15:26 ` Jan Beulich 2019-05-20 15:26 ` [Xen-devel] " Jan Beulich 2019-05-22 16:42 ` Roger Pau Monné 2019-05-22 16:42 ` [Xen-devel] " Roger Pau Monné 2019-05-23 8:36 ` Jan Beulich 2019-05-23 8:36 ` [Xen-devel] " Jan Beulich 2019-07-03 18:42 ` Andrew Cooper 2019-05-17 10:52 ` [PATCH v3 14/15] x86/IRQ: eliminate some on-stack cpumask_t instances Jan Beulich 2019-05-17 10:52 ` [Xen-devel] " Jan Beulich 2019-05-20 14:22 ` Roger Pau Monné 2019-05-20 14:22 ` [Xen-devel] " Roger Pau Monné 2019-07-03 18:44 ` Andrew Cooper 2019-07-04 10:04 ` Jan Beulich 2019-05-17 10:53 ` [PATCH v3 15/15] x86/IRQ: move {,_}clear_irq_vector() Jan Beulich 2019-05-17 10:53 ` [Xen-devel] " Jan Beulich 2019-07-03 18:45 ` [Xen-devel] [PATCH v3 15/15] x86/IRQ: move {, _}clear_irq_vector() Andrew Cooper
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=5CD03142020000780022C2F6@prv1-mh.provo.novell.com \ --to=jbeulich@suse.com \ --cc=andrew.cooper3@citrix.com \ --cc=roger.pau@citrix.com \ --cc=wei.liu2@citrix.com \ --cc=xen-devel@lists.xenproject.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).