LDREX and STREX in heterogeneous system？

LDREX and STREX in heterogeneous system？

[richard clark]

Hi Catalin and Mark,

Suppose in a heterogeneous system, there're cortex-M7 and cortex-A72
sharing the same bus. Does the below code sequence work as (ldr/str)ex
expected?

r2 point to a uncached shared memory between M7 and A72

M7                                      A72
ldrex r1, [r2]
      ------------------------->  strex r0, r1, [r2]

Richard

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: LDREX and STREX in heterogeneous system？

[Catalin Marinas]

On Wed, Nov 09, 2022 at 04:34:13PM +0800, richard clark wrote:
> Suppose in a heterogeneous system, there're cortex-M7 and cortex-A72
> sharing the same bus. Does the below code sequence work as (ldr/str)ex
> expected?
> 
> r2 point to a uncached shared memory between M7 and A72
> 
> M7                                      A72
> ldrex r1, [r2]
>       ------------------------->  strex r0, r1, [r2]

In general, it won't. The exclusives are supposed to work in the same
inner shareable domain, so it depends on how the SoC has the M7 and A72
wired up. Are they cache coherent with each-other? Is there a global
exclusive monitor? The M7 may also need the MPU regions set up with the
Shareable attribute.

-- 
Catalin

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: LDREX and STREX in heterogeneous system？

[richard clark]

On Wed, Nov 9, 2022 at 5:26 PM Catalin Marinas <catalin.marinas@arm.com> wrote:
>
> On Wed, Nov 09, 2022 at 04:34:13PM +0800, richard clark wrote:
> > Suppose in a heterogeneous system, there're cortex-M7 and cortex-A72
> > sharing the same bus. Does the below code sequence work as (ldr/str)ex
> > expected?
> >
> > r2 point to a uncached shared memory between M7 and A72
> >
> > M7                                      A72
> > ldrex r1, [r2]
> >       ------------------------->  strex r0, r1, [r2]
>
> In general, it won't. The exclusives are supposed to work in the same
> inner shareable domain, so it depends on how the SoC has the M7 and A72
> wired up. Are they cache coherent with each-other? Is there a global
> exclusive monitor? The M7 may also need the MPU regions set up with the
> Shareable attribute.
Thanks Catalin! AFAIK the M7 and A53 are not in the same Inner
shareable domain: if A53 modifies the SRAM with D$ on, the M7 will
still get the stale data from the same SRAM location. Except that
probably there is not a global exclusive monitor there.
>
> --
> Catalin

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: LDREX and STREX in heterogeneous system？

[Catalin Marinas]

On Thu, Nov 10, 2022 at 11:27:23AM +0800, richard clark wrote:
> On Wed, Nov 9, 2022 at 5:26 PM Catalin Marinas <catalin.marinas@arm.com> wrote:
> >
> > On Wed, Nov 09, 2022 at 04:34:13PM +0800, richard clark wrote:
> > > Suppose in a heterogeneous system, there're cortex-M7 and cortex-A72
> > > sharing the same bus. Does the below code sequence work as (ldr/str)ex
> > > expected?
> > >
> > > r2 point to a uncached shared memory between M7 and A72
> > >
> > > M7                                      A72
> > > ldrex r1, [r2]
> > >       ------------------------->  strex r0, r1, [r2]
> >
> > In general, it won't. The exclusives are supposed to work in the same
> > inner shareable domain, so it depends on how the SoC has the M7 and A72
> > wired up. Are they cache coherent with each-other? Is there a global
> > exclusive monitor? The M7 may also need the MPU regions set up with the
> > Shareable attribute.
> 
> Thanks Catalin! AFAIK the M7 and A53 are not in the same Inner
> shareable domain: if A53 modifies the SRAM with D$ on, the M7 will
> still get the stale data from the same SRAM location. Except that
> probably there is not a global exclusive monitor there.

An SoC may allow exclusives on a small range of non-cacheable memory but
it's not something to infer from the CPUID registers. You'd have to ask
the hardware people whether they built a global exclusive monitor and
whether that's shared with the M7.

Otherwise, you might want to look at other algorithms like Lamport's
bakery.

-- 
Catalin

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: LDREX and STREX in heterogeneous system？

[richard clark]

On Sat, Nov 12, 2022 at 6:40 AM Catalin Marinas <catalin.marinas@arm.com> wrote:
>
> On Thu, Nov 10, 2022 at 11:27:23AM +0800, richard clark wrote:
> > On Wed, Nov 9, 2022 at 5:26 PM Catalin Marinas <catalin.marinas@arm.com> wrote:
> > >
> > > On Wed, Nov 09, 2022 at 04:34:13PM +0800, richard clark wrote:
> > > > Suppose in a heterogeneous system, there're cortex-M7 and cortex-A72
> > > > sharing the same bus. Does the below code sequence work as (ldr/str)ex
> > > > expected?
> > > >
> > > > r2 point to a uncached shared memory between M7 and A72
> > > >
> > > > M7                                      A72
> > > > ldrex r1, [r2]
> > > >       ------------------------->  strex r0, r1, [r2]
> > >
> > > In general, it won't. The exclusives are supposed to work in the same
> > > inner shareable domain, so it depends on how the SoC has the M7 and A72
> > > wired up. Are they cache coherent with each-other? Is there a global
> > > exclusive monitor? The M7 may also need the MPU regions set up with the
> > > Shareable attribute.
> >
> > Thanks Catalin! AFAIK the M7 and A53 are not in the same Inner
> > shareable domain: if A53 modifies the SRAM with D$ on, the M7 will
> > still get the stale data from the same SRAM location. Except that
> > probably there is not a global exclusive monitor there.
>
> An SoC may allow exclusives on a small range of non-cacheable memory but
> it's not something to infer from the CPUID registers. You'd have to ask
> the hardware people whether they built a global exclusive monitor and
> whether that's shared with the M7.

The hardware people confirmed there's no global exclusive monitor in
the SoC and the M and A are not in the same inner-shareable-domain, so
order to access the shared resources between M and A in an exclusive
way, the SoC provides another method as so-called 'hw gate'...

>
> Otherwise, you might want to look at other algorithms like Lamport's
> bakery.
>
> --
> Catalin

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: LDREX and STREX in heterogeneous system？

[Catalin Marinas]

On Thu, Nov 17, 2022 at 09:32:50AM +0800, richard clark wrote:
> On Sat, Nov 12, 2022 at 6:40 AM Catalin Marinas <catalin.marinas@arm.com> wrote:
> > On Thu, Nov 10, 2022 at 11:27:23AM +0800, richard clark wrote:
> > > On Wed, Nov 9, 2022 at 5:26 PM Catalin Marinas <catalin.marinas@arm.com> wrote:
> > > > On Wed, Nov 09, 2022 at 04:34:13PM +0800, richard clark wrote:
> > > > > Suppose in a heterogeneous system, there're cortex-M7 and cortex-A72
> > > > > sharing the same bus. Does the below code sequence work as (ldr/str)ex
> > > > > expected?
> > > > >
> > > > > r2 point to a uncached shared memory between M7 and A72
> > > > >
> > > > > M7                                      A72
> > > > > ldrex r1, [r2]
> > > > >       ------------------------->  strex r0, r1, [r2]
> > > >
> > > > In general, it won't. The exclusives are supposed to work in the same
> > > > inner shareable domain, so it depends on how the SoC has the M7 and A72
> > > > wired up. Are they cache coherent with each-other? Is there a global
> > > > exclusive monitor? The M7 may also need the MPU regions set up with the
> > > > Shareable attribute.
> > >
> > > Thanks Catalin! AFAIK the M7 and A53 are not in the same Inner
> > > shareable domain: if A53 modifies the SRAM with D$ on, the M7 will
> > > still get the stale data from the same SRAM location. Except that
> > > probably there is not a global exclusive monitor there.
> >
> > An SoC may allow exclusives on a small range of non-cacheable memory but
> > it's not something to infer from the CPUID registers. You'd have to ask
> > the hardware people whether they built a global exclusive monitor and
> > whether that's shared with the M7.
> 
> The hardware people confirmed there's no global exclusive monitor in
> the SoC and the M and A are not in the same inner-shareable-domain, so
> order to access the shared resources between M and A in an exclusive
> way, the SoC provides another method as so-called 'hw gate'...

You may want to look at hwspinlock and implement a driver that deals
with this 'hw gate'.

-- 
Catalin

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel