[PATCH] irqchip/gic-v3-its: remove the shareability of ITS

[PATCH] irqchip/gic-v3-its: remove the shareability of ITS

[Harry Song]

I know this is a very wrong patch, but my platform
has an abnormal ITS problem caused by data consistency:
My chip does not support Cache Coherent Interconnect (CCI).
By default, ITS driver is the inner memory attribute.
gits_write_cbaser() is used to write the inner memory
attribute. But hw doesn't return the hardware's non-shareable
property,so I don't think reading GITS_CBASER and GICR_PROPBASER
here will get the real property of the current hardware: inner
or outer shareable is not supported, so I would like to know
whether ITS driver cannot be used on chips without CCI, or
what method can be used to use ITS driver on chips without CCI?

The current patch is designed to make ITS think that the current
chip has no inner or outer memory properties, and then use
its by flushing dcache.

This is the log for bug reports without patches:

[    0.000000] GICv3: CPU0: found redistributor 0 region 0:0x0000000003460000
[    0.000000] ITS [mem 0x03440000-0x0345ffff]
[    0.000000] ITS@0x0000000003440000: allocated 8192 Devices @41850000 (indirect, esz 8, psz 64K, shr 0)
[    0.000000] ITS@0x0000000003440000: allocated 32768 Interrupt Collections @41860000 (flat, esz 2, psz 64K, shr 0)
[    0.000000] GICv3: using LPI property table @0x0000000041870000
[    0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000041880000
[    0.000000] ITS queue timeout (64 1)
[    0.000000] ITS cmd its_build_mapc_cmd failed
[    0.000000] ITS queue timeout (128 1)
[    0.000000] ITS cmd its_build_invall_cmd failed

Signed-off-by: Harry Song <jundongsong1@gmail.com>
---

I am very sorry to bother you. This problem has been bothering me for several weeks.
I am looking forward to your reply.

Thanks,
Harry

 drivers/irqchip/irq-gic-v3-its.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
index 973ede019..780099ce2 100644
--- a/drivers/irqchip/irq-gic-v3-its.c
+++ b/drivers/irqchip/irq-gic-v3-its.c
@@ -2359,6 +2359,11 @@ static int its_setup_baser(struct its_node *its, struct its_baser *baser,
 	its_write_baser(its, baser, val);
 	tmp = baser->val;
 
+	if (tmp & GITS_BASER_SHAREABILITY_MASK)
+		tmp &= ~GITS_BASER_SHAREABILITY_MASK;
+	else
+		gic_flush_dcache_to_poc(base, PAGE_ORDER_TO_SIZE(order));
+
 	if ((val ^ tmp) & GITS_BASER_SHAREABILITY_MASK) {
 		/*
 		 * Shareability didn't stick. Just use
@@ -3096,6 +3101,8 @@ static void its_cpu_init_lpis(void)
 	gicr_write_propbaser(val, rbase + GICR_PROPBASER);
 	tmp = gicr_read_propbaser(rbase + GICR_PROPBASER);
 
+	tmp &= ~GICR_PROPBASER_SHAREABILITY_MASK;
+
 	if ((tmp ^ val) & GICR_PROPBASER_SHAREABILITY_MASK) {
 		if (!(tmp & GICR_PROPBASER_SHAREABILITY_MASK)) {
 			/*
@@ -5095,6 +5102,8 @@ static int __init its_probe_one(struct resource *res,
 	gits_write_cbaser(baser, its->base + GITS_CBASER);
 	tmp = gits_read_cbaser(its->base + GITS_CBASER);
 
+	tmp &= ~GICR_PROPBASER_SHAREABILITY_MASK;
+
 	if ((tmp ^ baser) & GITS_CBASER_SHAREABILITY_MASK) {
 		if (!(tmp & GITS_CBASER_SHAREABILITY_MASK)) {
 			/*
-- 
2.25.1

Re: [PATCH] irqchip/gic-v3-its: remove the shareability of ITS

[Marc Zyngier]

On Wed, 07 Dec 2022 13:52:23 +0000,
Harry Song <jundongsong1@gmail.com> wrote:
> 
> I know this is a very wrong patch, but my platform
> has an abnormal ITS problem caused by data consistency:
> My chip does not support Cache Coherent Interconnect (CCI).

That doesn't mean much. Nothing mandates to have a CCI, and plenty of
systems have other ways to maintain coherency.

> By default, ITS driver is the inner memory attribute.
> gits_write_cbaser() is used to write the inner memory
> attribute. But hw doesn't return the hardware's non-shareable
> property,so I don't think reading GITS_CBASER and GICR_PROPBASER
> here will get the real property of the current hardware: inner
> or outer shareable is not supported, so I would like to know
> whether ITS driver cannot be used on chips without CCI, or
> what method can be used to use ITS driver on chips without CCI?

It's not about CCI or not CCI. It is about which shareability domain
your ITS is in.

And it doesn't only affect the ITS. It also affects the
redistributors, and anything that accesses memory.

> 
> The current patch is designed to make ITS think that the current
> chip has no inner or outer memory properties, and then use
> its by flushing dcache.
> 
> This is the log for bug reports without patches:
> 
> [    0.000000] GICv3: CPU0: found redistributor 0 region 0:0x0000000003460000
> [    0.000000] ITS [mem 0x03440000-0x0345ffff]
> [    0.000000] ITS@0x0000000003440000: allocated 8192 Devices @41850000 (indirect, esz 8, psz 64K, shr 0)
> [    0.000000] ITS@0x0000000003440000: allocated 32768 Interrupt Collections @41860000 (flat, esz 2, psz 64K, shr 0)
> [    0.000000] GICv3: using LPI property table @0x0000000041870000
> [    0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000041880000
> [    0.000000] ITS queue timeout (64 1)
> [    0.000000] ITS cmd its_build_mapc_cmd failed
> [    0.000000] ITS queue timeout (128 1)
> [    0.000000] ITS cmd its_build_invall_cmd failed

Ah, this suspiciously looks like a Rockchip machine...

>
> Signed-off-by: Harry Song <jundongsong1@gmail.com>
> ---
> 
> I am very sorry to bother you. This problem has been bothering me
> for several weeks.  I am looking forward to your reply.

If you have such issue, this needs to be handled as per-platform
quirk. I'm not putting such generic hacks in the driver.

	M.

-- 
Without deviation from the norm, progress is not possible.

Re: [PATCH] irqchip/gic-v3-its: remove the shareability of ITS

[Harry Song]

On Wed, Dec 7, 2022 at 11:19 PM Marc Zyngier <maz@kernel.org> wrote:
>
> On Wed, 07 Dec 2022 13:52:23 +0000,
> Harry Song <jundongsong1@gmail.com> wrote:
> >
> > I know this is a very wrong patch, but my platform
> > has an abnormal ITS problem caused by data consistency:
> > My chip does not support Cache Coherent Interconnect (CCI).
>
> That doesn't mean much. Nothing mandates to have a CCI, and plenty of
> systems have other ways to maintain coherency.
>
> > By default, ITS driver is the inner memory attribute.
> > gits_write_cbaser() is used to write the inner memory
> > attribute. But hw doesn't return the hardware's non-shareable
> > property,so I don't think reading GITS_CBASER and GICR_PROPBASER
> > here will get the real property of the current hardware: inner
> > or outer shareable is not supported, so I would like to know
> > whether ITS driver cannot be used on chips without CCI, or
> > what method can be used to use ITS driver on chips without CCI?
>
> It's not about CCI or not CCI. It is about which shareability domain
> your ITS is in.
>
> And it doesn't only affect the ITS. It also affects the
> redistributors, and anything that accesses memory.
>

Yes, my current chip is Rockchip platform (rk3588), so is it because the chip
is not designed as a proper shared domain for ITS, so the exception of
ITS is caused?

> >
> > The current patch is designed to make ITS think that the current
> > chip has no inner or outer memory properties, and then use
> > its by flushing dcache.
> >
> > This is the log for bug reports without patches:
> >
> > [    0.000000] GICv3: CPU0: found redistributor 0 region 0:0x0000000003460000
> > [    0.000000] ITS [mem 0x03440000-0x0345ffff]
> > [    0.000000] ITS@0x0000000003440000: allocated 8192 Devices @41850000 (indirect, esz 8, psz 64K, shr 0)
> > [    0.000000] ITS@0x0000000003440000: allocated 32768 Interrupt Collections @41860000 (flat, esz 2, psz 64K, shr 0)
> > [    0.000000] GICv3: using LPI property table @0x0000000041870000
> > [    0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000041880000
> > [    0.000000] ITS queue timeout (64 1)
> > [    0.000000] ITS cmd its_build_mapc_cmd failed
> > [    0.000000] ITS queue timeout (128 1)
> > [    0.000000] ITS cmd its_build_invall_cmd failed
>
> Ah, this suspiciously looks like a Rockchip machine...
>
> >
> > Signed-off-by: Harry Song <jundongsong1@gmail.com>
> > ---
> >
> > I am very sorry to bother you. This problem has been bothering me
> > for several weeks.  I am looking forward to your reply.
>
> If you have such issue, this needs to be handled as per-platform
> quirk. I'm not putting such generic hacks in the driver.
>
>         M.
>
> --
> Without deviation from the norm, progress is not possible.

Are there currently other chip platforms that have this problem, and then ITS
is already compatible with the problem? Please tell me how to submit
hacks patch for rk3588 platform?

If the hacks patch cannot be submitted, please tell me whether it
requires the next generation
chip to have any design requirements for ITS?

Design ITS and CPU to be the same inner memory property?


Thank you very much for your answer,
Harry

Re: [PATCH] irqchip/gic-v3-its: remove the shareability of ITS

[Sebastian Reichel]

[-- Attachment #1: Type: text/plain, Size: 3786 bytes --]

Hi,

On Thu, Dec 08, 2022 at 10:28:29AM +0800, Harry Song wrote:
> On Wed, Dec 7, 2022 at 11:19 PM Marc Zyngier <maz@kernel.org> wrote:
> >
> > On Wed, 07 Dec 2022 13:52:23 +0000,
> > Harry Song <jundongsong1@gmail.com> wrote:
> > >
> > > I know this is a very wrong patch, but my platform
> > > has an abnormal ITS problem caused by data consistency:
> > > My chip does not support Cache Coherent Interconnect (CCI).
> >
> > That doesn't mean much. Nothing mandates to have a CCI, and plenty of
> > systems have other ways to maintain coherency.
> >
> > > By default, ITS driver is the inner memory attribute.
> > > gits_write_cbaser() is used to write the inner memory
> > > attribute. But hw doesn't return the hardware's non-shareable
> > > property,so I don't think reading GITS_CBASER and GICR_PROPBASER
> > > here will get the real property of the current hardware: inner
> > > or outer shareable is not supported, so I would like to know
> > > whether ITS driver cannot be used on chips without CCI, or
> > > what method can be used to use ITS driver on chips without CCI?
> >
> > It's not about CCI or not CCI. It is about which shareability domain
> > your ITS is in.
> >
> > And it doesn't only affect the ITS. It also affects the
> > redistributors, and anything that accesses memory.
> >
> 
> Yes, my current chip is Rockchip platform (rk3588), so is it because the chip
> is not designed as a proper shared domain for ITS, so the exception of
> ITS is caused?
> 
> > >
> > > The current patch is designed to make ITS think that the current
> > > chip has no inner or outer memory properties, and then use
> > > its by flushing dcache.
> > >
> > > This is the log for bug reports without patches:
> > >
> > > [    0.000000] GICv3: CPU0: found redistributor 0 region 0:0x0000000003460000
> > > [    0.000000] ITS [mem 0x03440000-0x0345ffff]
> > > [    0.000000] ITS@0x0000000003440000: allocated 8192 Devices @41850000 (indirect, esz 8, psz 64K, shr 0)
> > > [    0.000000] ITS@0x0000000003440000: allocated 32768 Interrupt Collections @41860000 (flat, esz 2, psz 64K, shr 0)
> > > [    0.000000] GICv3: using LPI property table @0x0000000041870000
> > > [    0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000041880000
> > > [    0.000000] ITS queue timeout (64 1)
> > > [    0.000000] ITS cmd its_build_mapc_cmd failed
> > > [    0.000000] ITS queue timeout (128 1)
> > > [    0.000000] ITS cmd its_build_invall_cmd failed
> >
> > Ah, this suspiciously looks like a Rockchip machine...
> >
> > >
> > > Signed-off-by: Harry Song <jundongsong1@gmail.com>
> > > ---
> > >
> > > I am very sorry to bother you. This problem has been bothering me
> > > for several weeks.  I am looking forward to your reply.
> >
> > If you have such issue, this needs to be handled as per-platform
> > quirk. I'm not putting such generic hacks in the driver.
> >
> >         M.
> >
> > --
> > Without deviation from the norm, progress is not possible.
> 
> Are there currently other chip platforms that have this problem, and then ITS
> is already compatible with the problem? Please tell me how to submit
> hacks patch for rk3588 platform?
> 
> If the hacks patch cannot be submitted, please tell me whether it
> requires the next generation
> chip to have any design requirements for ITS?
> 
> Design ITS and CPU to be the same inner memory property?

Previous rockchip generation has the same bug and it got discussed
here:

https://lore.kernel.org/lkml/871rbdt4tu.wl-maz@kernel.org/T/#m5dbc70ff308d81e98dd0d797e23d3fbf9c353245

You can use this DT as base with mainline to avoid ITS:

https://lore.kernel.org/all/20221205172350.75234-1-sebastian.reichel@collabora.com/

-- Sebastian

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH] irqchip/gic-v3-its: remove the shareability of ITS

[Harry Song]

Thank you for your reply. I know these two links.
My email is to ask about the root cause of this bug.

I would like to know whether the driver design of ITS requires that
the CPU and ITS
must be in a shared domain. Such as using CCI in chips;

From marc's reply, I think so. In addition to CCI, other methods can
also be used to
ensure consistency, which is a requirement for IC design.

Thanks,
Harry

On Fri, Dec 9, 2022 at 12:58 AM Sebastian Reichel
<sebastian.reichel@collabora.com> wrote:
>
> Hi,
>
> On Thu, Dec 08, 2022 at 10:28:29AM +0800, Harry Song wrote:
> > On Wed, Dec 7, 2022 at 11:19 PM Marc Zyngier <maz@kernel.org> wrote:
> > >
> > > On Wed, 07 Dec 2022 13:52:23 +0000,
> > > Harry Song <jundongsong1@gmail.com> wrote:
> > > >
> > > > I know this is a very wrong patch, but my platform
> > > > has an abnormal ITS problem caused by data consistency:
> > > > My chip does not support Cache Coherent Interconnect (CCI).
> > >
> > > That doesn't mean much. Nothing mandates to have a CCI, and plenty of
> > > systems have other ways to maintain coherency.
> > >
> > > > By default, ITS driver is the inner memory attribute.
> > > > gits_write_cbaser() is used to write the inner memory
> > > > attribute. But hw doesn't return the hardware's non-shareable
> > > > property,so I don't think reading GITS_CBASER and GICR_PROPBASER
> > > > here will get the real property of the current hardware: inner
> > > > or outer shareable is not supported, so I would like to know
> > > > whether ITS driver cannot be used on chips without CCI, or
> > > > what method can be used to use ITS driver on chips without CCI?
> > >
> > > It's not about CCI or not CCI. It is about which shareability domain
> > > your ITS is in.
> > >
> > > And it doesn't only affect the ITS. It also affects the
> > > redistributors, and anything that accesses memory.
> > >
> >
> > Yes, my current chip is Rockchip platform (rk3588), so is it because the chip
> > is not designed as a proper shared domain for ITS, so the exception of
> > ITS is caused?
> >
> > > >
> > > > The current patch is designed to make ITS think that the current
> > > > chip has no inner or outer memory properties, and then use
> > > > its by flushing dcache.
> > > >
> > > > This is the log for bug reports without patches:
> > > >
> > > > [    0.000000] GICv3: CPU0: found redistributor 0 region 0:0x0000000003460000
> > > > [    0.000000] ITS [mem 0x03440000-0x0345ffff]
> > > > [    0.000000] ITS@0x0000000003440000: allocated 8192 Devices @41850000 (indirect, esz 8, psz 64K, shr 0)
> > > > [    0.000000] ITS@0x0000000003440000: allocated 32768 Interrupt Collections @41860000 (flat, esz 2, psz 64K, shr 0)
> > > > [    0.000000] GICv3: using LPI property table @0x0000000041870000
> > > > [    0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000041880000
> > > > [    0.000000] ITS queue timeout (64 1)
> > > > [    0.000000] ITS cmd its_build_mapc_cmd failed
> > > > [    0.000000] ITS queue timeout (128 1)
> > > > [    0.000000] ITS cmd its_build_invall_cmd failed
> > >
> > > Ah, this suspiciously looks like a Rockchip machine...
> > >
> > > >
> > > > Signed-off-by: Harry Song <jundongsong1@gmail.com>
> > > > ---
> > > >
> > > > I am very sorry to bother you. This problem has been bothering me
> > > > for several weeks.  I am looking forward to your reply.
> > >
> > > If you have such issue, this needs to be handled as per-platform
> > > quirk. I'm not putting such generic hacks in the driver.
> > >
> > >         M.
> > >
> > > --
> > > Without deviation from the norm, progress is not possible.
> >
> > Are there currently other chip platforms that have this problem, and then ITS
> > is already compatible with the problem? Please tell me how to submit
> > hacks patch for rk3588 platform?
> >
> > If the hacks patch cannot be submitted, please tell me whether it
> > requires the next generation
> > chip to have any design requirements for ITS?
> >
> > Design ITS and CPU to be the same inner memory property?
>
> Previous rockchip generation has the same bug and it got discussed
> here:
>
> https://lore.kernel.org/lkml/871rbdt4tu.wl-maz@kernel.org/T/#m5dbc70ff308d81e98dd0d797e23d3fbf9c353245
>
> You can use this DT as base with mainline to avoid ITS:
>
> https://lore.kernel.org/all/20221205172350.75234-1-sebastian.reichel@collabora.com/
>
> -- Sebastian

Re: [PATCH] irqchip/gic-v3-its: remove the shareability of ITS

[Marc Zyngier]

On Thu, 08 Dec 2022 02:28:29 +0000,
Harry Song <jundongsong1@gmail.com> wrote:
> 
> On Wed, Dec 7, 2022 at 11:19 PM Marc Zyngier <maz@kernel.org> wrote:
> >
> > On Wed, 07 Dec 2022 13:52:23 +0000,
> > Harry Song <jundongsong1@gmail.com> wrote:
> > >
> > > I know this is a very wrong patch, but my platform
> > > has an abnormal ITS problem caused by data consistency:
> > > My chip does not support Cache Coherent Interconnect (CCI).
> >
> > That doesn't mean much. Nothing mandates to have a CCI, and plenty of
> > systems have other ways to maintain coherency.
> >
> > > By default, ITS driver is the inner memory attribute.
> > > gits_write_cbaser() is used to write the inner memory
> > > attribute. But hw doesn't return the hardware's non-shareable
> > > property,so I don't think reading GITS_CBASER and GICR_PROPBASER
> > > here will get the real property of the current hardware: inner
> > > or outer shareable is not supported, so I would like to know
> > > whether ITS driver cannot be used on chips without CCI, or
> > > what method can be used to use ITS driver on chips without CCI?
> >
> > It's not about CCI or not CCI. It is about which shareability domain
> > your ITS is in.
> >
> > And it doesn't only affect the ITS. It also affects the
> > redistributors, and anything that accesses memory.
> >
> 
> Yes, my current chip is Rockchip platform (rk3588), so is it because the chip
> is not designed as a proper shared domain for ITS, so the exception of
> ITS is caused?

This Rockchip platforms have two problems:

- The GIC doesn't report whether is is in a shareable domain or not

- The GIC relies on memory allocation being under 4GB

In both cases, this needs to be quirked. Last time the subject came
up, Rockchip didn't want to do this properly, so the ITS is
unsupported on these systems until someone writes a decent patch.

> > > The current patch is designed to make ITS think that the current
> > > chip has no inner or outer memory properties, and then use
> > > its by flushing dcache.
> > >
> > > This is the log for bug reports without patches:
> > >
> > > [    0.000000] GICv3: CPU0: found redistributor 0 region 0:0x0000000003460000
> > > [    0.000000] ITS [mem 0x03440000-0x0345ffff]
> > > [    0.000000] ITS@0x0000000003440000: allocated 8192 Devices @41850000 (indirect, esz 8, psz 64K, shr 0)
> > > [    0.000000] ITS@0x0000000003440000: allocated 32768 Interrupt Collections @41860000 (flat, esz 2, psz 64K, shr 0)
> > > [    0.000000] GICv3: using LPI property table @0x0000000041870000
> > > [    0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000041880000
> > > [    0.000000] ITS queue timeout (64 1)
> > > [    0.000000] ITS cmd its_build_mapc_cmd failed
> > > [    0.000000] ITS queue timeout (128 1)
> > > [    0.000000] ITS cmd its_build_invall_cmd failed
> >
> > Ah, this suspiciously looks like a Rockchip machine...
> >
> > >
> > > Signed-off-by: Harry Song <jundongsong1@gmail.com>
> > > ---
> > >
> > > I am very sorry to bother you. This problem has been bothering me
> > > for several weeks.  I am looking forward to your reply.
> >
> > If you have such issue, this needs to be handled as per-platform
> > quirk. I'm not putting such generic hacks in the driver.
> >
> >         M.
> >
> > --
> > Without deviation from the norm, progress is not possible.
> 
> Are there currently other chip platforms that have this problem, and then ITS
> is already compatible with the problem?

No. Only the RK35xx systems are affected. Other systems with the exact
same GIC600 are working just fine.

> Please tell me how to submit hacks patch for rk3588 platform?  If
> the hacks patch cannot be submitted, please tell me whether it
> requires the next generation chip to have any design requirements
> for ITS?

A patch *can* be submitted. What I want to see is described as part of

https://lore.kernel.org/lkml/878s5i2qyw.wl-maz@kernel.org/

If you write such a patch, I'll gladly merge it.

> Design ITS and CPU to be the same inner memory property?

No, you need to integrate it so that the attribute are
*discoverable*, and all the memory reachable from the GIC.

	M.

-- 
Without deviation from the norm, progress is not possible.

Re: [PATCH] irqchip/gic-v3-its: remove the shareability of ITS

[Marc Zyngier]

On Fri, 09 Dec 2022 03:34:21 +0000,
Harry Song <jundongsong1@gmail.com> wrote:
> 
> Thank you for your reply. I know these two links.
> My email is to ask about the root cause of this bug.
> 
> I would like to know whether the driver design of ITS requires that
> the CPU and ITS must be in a shared domain. Such as using CCI in
> chips;

This problem has nothing to do with CCI or coherency. It has to do
with how the GIC is plugged in the interconnect and what attributes it
advertises.

	M.

-- 
Without deviation from the norm, progress is not possible.

Re: [PATCH] irqchip/gic-v3-its: remove the shareability of ITS

[Marc Zyngier]

On Thu, 08 Dec 2022 16:58:20 +0000,
Sebastian Reichel <sebastian.reichel@collabora.com> wrote:
> 
> [1  <text/plain; us-ascii (quoted-printable)>]
> Hi,
> 
> On Thu, Dec 08, 2022 at 10:28:29AM +0800, Harry Song wrote:
> > On Wed, Dec 7, 2022 at 11:19 PM Marc Zyngier <maz@kernel.org> wrote:
> > >
> > > On Wed, 07 Dec 2022 13:52:23 +0000,
> > > Harry Song <jundongsong1@gmail.com> wrote:
> > > >
> > > > I know this is a very wrong patch, but my platform
> > > > has an abnormal ITS problem caused by data consistency:
> > > > My chip does not support Cache Coherent Interconnect (CCI).
> > >
> > > That doesn't mean much. Nothing mandates to have a CCI, and plenty of
> > > systems have other ways to maintain coherency.
> > >
> > > > By default, ITS driver is the inner memory attribute.
> > > > gits_write_cbaser() is used to write the inner memory
> > > > attribute. But hw doesn't return the hardware's non-shareable
> > > > property,so I don't think reading GITS_CBASER and GICR_PROPBASER
> > > > here will get the real property of the current hardware: inner
> > > > or outer shareable is not supported, so I would like to know
> > > > whether ITS driver cannot be used on chips without CCI, or
> > > > what method can be used to use ITS driver on chips without CCI?
> > >
> > > It's not about CCI or not CCI. It is about which shareability domain
> > > your ITS is in.
> > >
> > > And it doesn't only affect the ITS. It also affects the
> > > redistributors, and anything that accesses memory.
> > >
> > 
> > Yes, my current chip is Rockchip platform (rk3588), so is it because the chip
> > is not designed as a proper shared domain for ITS, so the exception of
> > ITS is caused?
> > 
> > > >
> > > > The current patch is designed to make ITS think that the current
> > > > chip has no inner or outer memory properties, and then use
> > > > its by flushing dcache.
> > > >
> > > > This is the log for bug reports without patches:
> > > >
> > > > [    0.000000] GICv3: CPU0: found redistributor 0 region 0:0x0000000003460000
> > > > [    0.000000] ITS [mem 0x03440000-0x0345ffff]
> > > > [    0.000000] ITS@0x0000000003440000: allocated 8192 Devices @41850000 (indirect, esz 8, psz 64K, shr 0)
> > > > [    0.000000] ITS@0x0000000003440000: allocated 32768 Interrupt Collections @41860000 (flat, esz 2, psz 64K, shr 0)
> > > > [    0.000000] GICv3: using LPI property table @0x0000000041870000
> > > > [    0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000041880000
> > > > [    0.000000] ITS queue timeout (64 1)
> > > > [    0.000000] ITS cmd its_build_mapc_cmd failed
> > > > [    0.000000] ITS queue timeout (128 1)
> > > > [    0.000000] ITS cmd its_build_invall_cmd failed
> > >
> > > Ah, this suspiciously looks like a Rockchip machine...
> > >
> > > >
> > > > Signed-off-by: Harry Song <jundongsong1@gmail.com>
> > > > ---
> > > >
> > > > I am very sorry to bother you. This problem has been bothering me
> > > > for several weeks.  I am looking forward to your reply.
> > >
> > > If you have such issue, this needs to be handled as per-platform
> > > quirk. I'm not putting such generic hacks in the driver.
> > >
> > >         M.
> > >
> > > --
> > > Without deviation from the norm, progress is not possible.
> > 
> > Are there currently other chip platforms that have this problem, and then ITS
> > is already compatible with the problem? Please tell me how to submit
> > hacks patch for rk3588 platform?
> > 
> > If the hacks patch cannot be submitted, please tell me whether it
> > requires the next generation
> > chip to have any design requirements for ITS?
> > 
> > Design ITS and CPU to be the same inner memory property?
> 
> Previous rockchip generation has the same bug and it got discussed
> here:
> 
> https://lore.kernel.org/lkml/871rbdt4tu.wl-maz@kernel.org/T/#m5dbc70ff308d81e98dd0d797e23d3fbf9c353245
> 
> You can use this DT as base with mainline to avoid ITS:
> 
> https://lore.kernel.org/all/20221205172350.75234-1-sebastian.reichel@collabora.com/

Using the MBI region really isn't the same thing. You're limited to a
handful of MSI instead of 64k, and you don't get any device isolation.

Also, your usage of the GIC binding is pretty broken. A single PPI
partition covering all the CPUs, the two PMU having the same PPI and
#interrupt-calls=3, that's all completely wrong.

	M.

-- 
Without deviation from the norm, progress is not possible.

Re: [PATCH] irqchip/gic-v3-its: remove the shareability of ITS

[Harry Song]

Thank you again.

Harry

On Fri, Dec 9, 2022 at 7:13 PM Marc Zyngier <maz@kernel.org> wrote:
>
> On Fri, 09 Dec 2022 03:34:21 +0000,
> Harry Song <jundongsong1@gmail.com> wrote:
> >
> > Thank you for your reply. I know these two links.
> > My email is to ask about the root cause of this bug.
> >
> > I would like to know whether the driver design of ITS requires that
> > the CPU and ITS must be in a shared domain. Such as using CCI in
> > chips;
>
> This problem has nothing to do with CCI or coherency. It has to do
> with how the GIC is plugged in the interconnect and what attributes it
> advertises.
>
>         M.
>

Wow, that's a great link:
https://lore.kernel.org/lkml/878s5i2qyw.wl-maz@kernel.org/

Many thanks for your detailed reply.
I got the answer I wanted to know, That was very helpful.

> --
> Without deviation from the norm, progress is not possible.

Thank you again,
Harry

Re: [PATCH] irqchip/gic-v3-its: remove the shareability of ITS

[Harry Song]

On Fri, Dec 9, 2022 at 9:37 PM Harry Song <jundongsong1@gmail.com> wrote:
>
> Thank you again.
>
> Harry
>
> On Fri, Dec 9, 2022 at 7:13 PM Marc Zyngier <maz@kernel.org> wrote:
> >
> > On Fri, 09 Dec 2022 03:34:21 +0000,
> > Harry Song <jundongsong1@gmail.com> wrote:
> > >
> > > Thank you for your reply. I know these two links.
> > > My email is to ask about the root cause of this bug.
> > >
> > > I would like to know whether the driver design of ITS requires that
> > > the CPU and ITS must be in a shared domain. Such as using CCI in
> > > chips;
> >
> > This problem has nothing to do with CCI or coherency. It has to do
> > with how the GIC is plugged in the interconnect and what attributes it
> > advertises.

This problem has nothing to do with CCI or coherency ??

Now , I have a question about this sentence：
If CCI is not used, how does the hardware realize the interconnection
between GIC-600 and cache?
If CCI is not used, our hardware colleagues said that the internal ITS
of the GIC-600 sends out operations with cache attributes (Inner/Outer
Shareable),
and there is no way to be captured by the cache and directly enter the
DDR. How does arm realize the interconnection between GIC-600 and
cache without CCI?

Waiting for your reply.

Harry Song.

> >
> >         M.
> >
>
> Wow, that's a great link:
> https://lore.kernel.org/lkml/878s5i2qyw.wl-maz@kernel.org/
>
> Many thanks for your detailed reply.
> I got the answer I wanted to know, That was very helpful.
>
> > --
> > Without deviation from the norm, progress is not possible.
>
> Thank you again,
> Harry

Re: [PATCH] irqchip/gic-v3-its: remove the shareability of ITS

[Marc Zyngier]

On Mon, 13 Feb 2023 07:30:08 +0000,
Harry Song <jundongsong1@gmail.com> wrote:
> 
> On Fri, Dec 9, 2022 at 9:37 PM Harry Song <jundongsong1@gmail.com> wrote:
> >
> > Thank you again.
> >
> > Harry
> >
> > On Fri, Dec 9, 2022 at 7:13 PM Marc Zyngier <maz@kernel.org> wrote:
> > >
> > > On Fri, 09 Dec 2022 03:34:21 +0000,
> > > Harry Song <jundongsong1@gmail.com> wrote:
> > > >
> > > > Thank you for your reply. I know these two links.
> > > > My email is to ask about the root cause of this bug.
> > > >
> > > > I would like to know whether the driver design of ITS requires that
> > > > the CPU and ITS must be in a shared domain. Such as using CCI in
> > > > chips;
> > >
> > > This problem has nothing to do with CCI or coherency. It has to do
> > > with how the GIC is plugged in the interconnect and what attributes it
> > > advertises.
> 
> This problem has nothing to do with CCI or coherency ??
>
> Now , I have a question about this sentence：
> If CCI is not used, how does the hardware realize the interconnection
> between GIC-600 and cache?
> If CCI is not used, our hardware colleagues said that the internal ITS
> of the GIC-600 sends out operations with cache attributes (Inner/Outer
> Shareable),
> and there is no way to be captured by the cache and directly enter the
> DDR. How does arm realize the interconnection between GIC-600 and
> cache without CCI?

This is becoming tedious.

Why do you need things to be cacheable/shareable? The HW doesn't need
it, and the SW doesn't need it either.

All that SW needs is to be told *how* the HW behaves, and it relies on
the GIC to tell it by not accepting configurations it cannot support.
That's all. This is all described in the thread I pointed you to last
year.

If your HW is accepting configurations it cannot deal with, then it is
a bug. You can work around it (again see the thread I pointed you to),
or you can continue to moan about it.

But I'm not interested in arguing further about this.

Also, for ARM integration problems, please contact the ARM technical
support. I'm here for Linux, and nothing else.

	M.

-- 
Without deviation from the norm, progress is not possible.