* [RFC PATCH] irqchip/gic-v3: wait irq done to set affinity @ 2023-01-06 8:21 ` Yipeng Zou 0 siblings, 0 replies; 8+ messages in thread From: Yipeng Zou @ 2023-01-06 8:21 UTC (permalink / raw) To: tglx, maz, linux-kernel, linux-arm-kernel Cc: hewenliang4, zouyipeng, chris.zjh, liaochang1 Recently we have some problem about gic set affinity in our test. This patch just aim to make some discuss about this problem. For now, the implementation of gic set affinity going to take effects immediately, and without check if any irq are being processed. So, This leads to some problem, think about this scenario: 1. First, we have an irq was generated by an device. 2. In the processing of this irq(after handle event, before clear IRQD_IRQ_INPROGRESS flag), we modify the route and the gic takes effect immediately,at the same time the new one was generated again. 3. The new irq will be processing in other cpu which different form the old one. 4. The new irq going to be discarded because of the flag IRQD_IRQ_INPROGRESS has been set. I notice that if we set IRQF_ONESHOT when register the irq, this problem will gone. But I'm also thinking about change the gic_set_affinity function, to wait current irq done on all cpus before gic_write_irouter. I'm not sure if that's appropriate. Is the best workaround to use IRQF_ONESHOT to prevent reentrancy? Please let me know, if have any other suggestions on this issue. Signed-off-by: Yipeng Zou <zouyipeng@huawei.com> --- drivers/irqchip/irq-gic-v3.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/drivers/irqchip/irq-gic-v3.c b/drivers/irqchip/irq-gic-v3.c index 997104d4338e..e9b9f15f07f8 100644 --- a/drivers/irqchip/irq-gic-v3.c +++ b/drivers/irqchip/irq-gic-v3.c @@ -1348,6 +1348,8 @@ static int gic_set_affinity(struct irq_data *d, const struct cpumask *mask_val, reg = gic_dist_base(d) + offset + (index * 8); val = gic_mpidr_to_affinity(cpu_logical_map(cpu)); + // wait irq done on all cpus + gic_write_irouter(val, reg); /* -- 2.17.1 ^ permalink raw reply related [flat|nested] 8+ messages in thread
* [RFC PATCH] irqchip/gic-v3: wait irq done to set affinity @ 2023-01-06 8:21 ` Yipeng Zou 0 siblings, 0 replies; 8+ messages in thread From: Yipeng Zou @ 2023-01-06 8:21 UTC (permalink / raw) To: tglx, maz, linux-kernel, linux-arm-kernel Cc: hewenliang4, zouyipeng, chris.zjh, liaochang1 Recently we have some problem about gic set affinity in our test. This patch just aim to make some discuss about this problem. For now, the implementation of gic set affinity going to take effects immediately, and without check if any irq are being processed. So, This leads to some problem, think about this scenario: 1. First, we have an irq was generated by an device. 2. In the processing of this irq(after handle event, before clear IRQD_IRQ_INPROGRESS flag), we modify the route and the gic takes effect immediately,at the same time the new one was generated again. 3. The new irq will be processing in other cpu which different form the old one. 4. The new irq going to be discarded because of the flag IRQD_IRQ_INPROGRESS has been set. I notice that if we set IRQF_ONESHOT when register the irq, this problem will gone. But I'm also thinking about change the gic_set_affinity function, to wait current irq done on all cpus before gic_write_irouter. I'm not sure if that's appropriate. Is the best workaround to use IRQF_ONESHOT to prevent reentrancy? Please let me know, if have any other suggestions on this issue. Signed-off-by: Yipeng Zou <zouyipeng@huawei.com> --- drivers/irqchip/irq-gic-v3.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/drivers/irqchip/irq-gic-v3.c b/drivers/irqchip/irq-gic-v3.c index 997104d4338e..e9b9f15f07f8 100644 --- a/drivers/irqchip/irq-gic-v3.c +++ b/drivers/irqchip/irq-gic-v3.c @@ -1348,6 +1348,8 @@ static int gic_set_affinity(struct irq_data *d, const struct cpumask *mask_val, reg = gic_dist_base(d) + offset + (index * 8); val = gic_mpidr_to_affinity(cpu_logical_map(cpu)); + // wait irq done on all cpus + gic_write_irouter(val, reg); /* -- 2.17.1 _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply related [flat|nested] 8+ messages in thread
* Re: [RFC PATCH] irqchip/gic-v3: wait irq done to set affinity 2023-01-06 8:21 ` Yipeng Zou @ 2023-01-06 11:55 ` Marc Zyngier -1 siblings, 0 replies; 8+ messages in thread From: Marc Zyngier @ 2023-01-06 11:55 UTC (permalink / raw) To: Yipeng Zou Cc: tglx, linux-kernel, linux-arm-kernel, hewenliang4, chris.zjh, liaochang1 On Fri, 06 Jan 2023 08:21:36 +0000, Yipeng Zou <zouyipeng@huawei.com> wrote: > > Recently we have some problem about gic set affinity in our test. > > This patch just aim to make some discuss about this problem. > > For now, the implementation of gic set affinity going to take effects > immediately, and without check if any irq are being processed. > > So, This leads to some problem, think about this scenario: > > 1. First, we have an irq was generated by an device. > > 2. In the processing of this irq(after handle event, before clear > IRQD_IRQ_INPROGRESS flag), we modify the route and the gic takes effect > immediately,at the same time the new one was generated again. How is that possible? If it is affected by GICD_IROUTERn (as your patch suggests), then it is a SPI. If it is a SPI, it has an active state. Which means it cannot fire again without a deactivation (EOI if EOImode=0, EOI+DIR if EOImode=1) having taken place. So either something has deactivated the interrupt without masking it beforehand, or the active state is not honoured. Either way, this is wrong. > > 3. The new irq will be processing in other cpu which different form the > old one. > > 4. The new irq going to be discarded because of the flag IRQD_IRQ_INPROGRESS > has been set. > > I notice that if we set IRQF_ONESHOT when register the irq, this problem > will gone. > > But I'm also thinking about change the gic_set_affinity function, to wait > current irq done on all cpus before gic_write_irouter. > I'm not sure if that's appropriate. The base architecture should guarantee that this is not a problem, thanks to the active state. If that was a LPI (which do not have an active state), that'd be a different problem. But this doesn't seem to be the case here. I'm afraid to say that what you describe seem like a bug of some sort, either HW or SW. Thanks, M. -- Without deviation from the norm, progress is not possible. ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [RFC PATCH] irqchip/gic-v3: wait irq done to set affinity @ 2023-01-06 11:55 ` Marc Zyngier 0 siblings, 0 replies; 8+ messages in thread From: Marc Zyngier @ 2023-01-06 11:55 UTC (permalink / raw) To: Yipeng Zou Cc: tglx, linux-kernel, linux-arm-kernel, hewenliang4, chris.zjh, liaochang1 On Fri, 06 Jan 2023 08:21:36 +0000, Yipeng Zou <zouyipeng@huawei.com> wrote: > > Recently we have some problem about gic set affinity in our test. > > This patch just aim to make some discuss about this problem. > > For now, the implementation of gic set affinity going to take effects > immediately, and without check if any irq are being processed. > > So, This leads to some problem, think about this scenario: > > 1. First, we have an irq was generated by an device. > > 2. In the processing of this irq(after handle event, before clear > IRQD_IRQ_INPROGRESS flag), we modify the route and the gic takes effect > immediately,at the same time the new one was generated again. How is that possible? If it is affected by GICD_IROUTERn (as your patch suggests), then it is a SPI. If it is a SPI, it has an active state. Which means it cannot fire again without a deactivation (EOI if EOImode=0, EOI+DIR if EOImode=1) having taken place. So either something has deactivated the interrupt without masking it beforehand, or the active state is not honoured. Either way, this is wrong. > > 3. The new irq will be processing in other cpu which different form the > old one. > > 4. The new irq going to be discarded because of the flag IRQD_IRQ_INPROGRESS > has been set. > > I notice that if we set IRQF_ONESHOT when register the irq, this problem > will gone. > > But I'm also thinking about change the gic_set_affinity function, to wait > current irq done on all cpus before gic_write_irouter. > I'm not sure if that's appropriate. The base architecture should guarantee that this is not a problem, thanks to the active state. If that was a LPI (which do not have an active state), that'd be a different problem. But this doesn't seem to be the case here. I'm afraid to say that what you describe seem like a bug of some sort, either HW or SW. Thanks, M. -- Without deviation from the norm, progress is not possible. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [RFC PATCH] irqchip/gic-v3: wait irq done to set affinity 2023-01-06 11:55 ` Marc Zyngier @ 2023-01-09 12:26 ` Yipeng Zou -1 siblings, 0 replies; 8+ messages in thread From: Yipeng Zou @ 2023-01-09 12:26 UTC (permalink / raw) To: Marc Zyngier Cc: tglx, linux-kernel, linux-arm-kernel, hewenliang4, chris.zjh, liaochang1 在 2023/1/6 19:55, Marc Zyngier 写道: > On Fri, 06 Jan 2023 08:21:36 +0000, > Yipeng Zou <zouyipeng@huawei.com> wrote: >> Recently we have some problem about gic set affinity in our test. >> >> This patch just aim to make some discuss about this problem. >> >> For now, the implementation of gic set affinity going to take effects >> immediately, and without check if any irq are being processed. >> >> So, This leads to some problem, think about this scenario: >> >> 1. First, we have an irq was generated by an device. >> >> 2. In the processing of this irq(after handle event, before clear >> IRQD_IRQ_INPROGRESS flag), we modify the route and the gic takes effect >> immediately,at the same time the new one was generated again. > How is that possible? > > If it is affected by GICD_IROUTERn (as your patch suggests), then it > is a SPI. If it is a SPI, it has an active state. Which means it > cannot fire again without a deactivation (EOI if EOImode=0, EOI+DIR if > EOImode=1) having taken place. > > So either something has deactivated the interrupt without masking it > beforehand, or the active state is not honoured. Either way, this is > wrong. Yes, agree, There is no possible in SPI case. >> 3. The new irq will be processing in other cpu which different form the >> old one. >> >> 4. The new irq going to be discarded because of the flag IRQD_IRQ_INPROGRESS >> has been set. >> >> I notice that if we set IRQF_ONESHOT when register the irq, this problem >> will gone. >> >> But I'm also thinking about change the gic_set_affinity function, to wait >> current irq done on all cpus before gic_write_irouter. >> I'm not sure if that's appropriate. > The base architecture should guarantee that this is not a problem, > thanks to the active state. If that was a LPI (which do not have an > active state), that'd be a different problem. But this doesn't seem to > be the case here. Hi , Thanks for reply very much. I have rechecked our test. Actually, that was a LPI in out test case. It cause the problem since its_send_movi command. I made a mistake when i modified the code. It should be as follow. Sorry for misleading you. diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c index 973ede0197e3..fad08ccb7fd9 100644 --- a/drivers/irqchip/irq-gic-v3-its.c +++ b/drivers/irqchip/irq-gic-v3-its.c @@ -1667,6 +1667,9 @@ static int its_set_affinity(struct irq_data *d, const struct cpumask *mask_val, /* don't set the affinity when the target cpu is same as current one */ if (cpu != prev_cpu) { + + // wait irq done on all cpus + target_col = &its_dev->its->collections[cpu]; its_send_movi(its_dev, target_col, id); its_dev->event_map.col_map[id] = cpu > I'm afraid to say that what you describe seem like a bug of some sort, > either HW or SW. > > Thanks, > > M. -- Regards, Yipeng Zou ^ permalink raw reply related [flat|nested] 8+ messages in thread
* Re: [RFC PATCH] irqchip/gic-v3: wait irq done to set affinity @ 2023-01-09 12:26 ` Yipeng Zou 0 siblings, 0 replies; 8+ messages in thread From: Yipeng Zou @ 2023-01-09 12:26 UTC (permalink / raw) To: Marc Zyngier Cc: tglx, linux-kernel, linux-arm-kernel, hewenliang4, chris.zjh, liaochang1 在 2023/1/6 19:55, Marc Zyngier 写道: > On Fri, 06 Jan 2023 08:21:36 +0000, > Yipeng Zou <zouyipeng@huawei.com> wrote: >> Recently we have some problem about gic set affinity in our test. >> >> This patch just aim to make some discuss about this problem. >> >> For now, the implementation of gic set affinity going to take effects >> immediately, and without check if any irq are being processed. >> >> So, This leads to some problem, think about this scenario: >> >> 1. First, we have an irq was generated by an device. >> >> 2. In the processing of this irq(after handle event, before clear >> IRQD_IRQ_INPROGRESS flag), we modify the route and the gic takes effect >> immediately,at the same time the new one was generated again. > How is that possible? > > If it is affected by GICD_IROUTERn (as your patch suggests), then it > is a SPI. If it is a SPI, it has an active state. Which means it > cannot fire again without a deactivation (EOI if EOImode=0, EOI+DIR if > EOImode=1) having taken place. > > So either something has deactivated the interrupt without masking it > beforehand, or the active state is not honoured. Either way, this is > wrong. Yes, agree, There is no possible in SPI case. >> 3. The new irq will be processing in other cpu which different form the >> old one. >> >> 4. The new irq going to be discarded because of the flag IRQD_IRQ_INPROGRESS >> has been set. >> >> I notice that if we set IRQF_ONESHOT when register the irq, this problem >> will gone. >> >> But I'm also thinking about change the gic_set_affinity function, to wait >> current irq done on all cpus before gic_write_irouter. >> I'm not sure if that's appropriate. > The base architecture should guarantee that this is not a problem, > thanks to the active state. If that was a LPI (which do not have an > active state), that'd be a different problem. But this doesn't seem to > be the case here. Hi , Thanks for reply very much. I have rechecked our test. Actually, that was a LPI in out test case. It cause the problem since its_send_movi command. I made a mistake when i modified the code. It should be as follow. Sorry for misleading you. diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c index 973ede0197e3..fad08ccb7fd9 100644 --- a/drivers/irqchip/irq-gic-v3-its.c +++ b/drivers/irqchip/irq-gic-v3-its.c @@ -1667,6 +1667,9 @@ static int its_set_affinity(struct irq_data *d, const struct cpumask *mask_val, /* don't set the affinity when the target cpu is same as current one */ if (cpu != prev_cpu) { + + // wait irq done on all cpus + target_col = &its_dev->its->collections[cpu]; its_send_movi(its_dev, target_col, id); its_dev->event_map.col_map[id] = cpu > I'm afraid to say that what you describe seem like a bug of some sort, > either HW or SW. > > Thanks, > > M. -- Regards, Yipeng Zou _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply related [flat|nested] 8+ messages in thread
* Re: [RFC PATCH] irqchip/gic-v3: wait irq done to set affinity 2023-01-09 12:26 ` Yipeng Zou @ 2023-01-17 9:38 ` Thomas Gleixner -1 siblings, 0 replies; 8+ messages in thread From: Thomas Gleixner @ 2023-01-17 9:38 UTC (permalink / raw) To: Yipeng Zou, Marc Zyngier Cc: linux-kernel, linux-arm-kernel, hewenliang4, chris.zjh, liaochang1 On Mon, Jan 09 2023 at 20:26, Yipeng Zou wrote: > 在 2023/1/6 19:55, Marc Zyngier 写道: > index 973ede0197e3..fad08ccb7fd9 100644 > --- a/drivers/irqchip/irq-gic-v3-its.c > +++ b/drivers/irqchip/irq-gic-v3-its.c > @@ -1667,6 +1667,9 @@ static int its_set_affinity(struct irq_data *d, > const struct cpumask *mask_val, > > /* don't set the affinity when the target cpu is same as > current one */ > if (cpu != prev_cpu) { > + > + // wait irq done on all cpus > + There is no way to wait here. The caller holds the interrupt descriptor lock. If this is really an issue for LPI, then the only way to deal with that is CONFIG_GENERIC_PENDING_IRQ, which delays the affinity change to interrupt context Why on earth must all the known hardware mistakes be repeated over and over? Thanks, tglx _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [RFC PATCH] irqchip/gic-v3: wait irq done to set affinity @ 2023-01-17 9:38 ` Thomas Gleixner 0 siblings, 0 replies; 8+ messages in thread From: Thomas Gleixner @ 2023-01-17 9:38 UTC (permalink / raw) To: Yipeng Zou, Marc Zyngier Cc: linux-kernel, linux-arm-kernel, hewenliang4, chris.zjh, liaochang1 On Mon, Jan 09 2023 at 20:26, Yipeng Zou wrote: > 在 2023/1/6 19:55, Marc Zyngier 写道: > index 973ede0197e3..fad08ccb7fd9 100644 > --- a/drivers/irqchip/irq-gic-v3-its.c > +++ b/drivers/irqchip/irq-gic-v3-its.c > @@ -1667,6 +1667,9 @@ static int its_set_affinity(struct irq_data *d, > const struct cpumask *mask_val, > > /* don't set the affinity when the target cpu is same as > current one */ > if (cpu != prev_cpu) { > + > + // wait irq done on all cpus > + There is no way to wait here. The caller holds the interrupt descriptor lock. If this is really an issue for LPI, then the only way to deal with that is CONFIG_GENERIC_PENDING_IRQ, which delays the affinity change to interrupt context Why on earth must all the known hardware mistakes be repeated over and over? Thanks, tglx ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2023-01-17 9:41 UTC | newest] Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2023-01-06 8:21 [RFC PATCH] irqchip/gic-v3: wait irq done to set affinity Yipeng Zou 2023-01-06 8:21 ` Yipeng Zou 2023-01-06 11:55 ` Marc Zyngier 2023-01-06 11:55 ` Marc Zyngier 2023-01-09 12:26 ` Yipeng Zou 2023-01-09 12:26 ` Yipeng Zou 2023-01-17 9:38 ` Thomas Gleixner 2023-01-17 9:38 ` Thomas Gleixner
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.