EDAC on arm64

EDAC on arm64

[Jon Masters]

Hi Rob,

Have you considered reviving the patch you posted previously for EDAC
support (the atomic_scrub read/write test piece dependency)?

http://lists.infradead.org/pipermail/linux-arm-kernel/2014-April/249039.html

Jon.

EDAC on arm64

[Will Deacon]

On Sat, Feb 28, 2015 at 12:52:03AM +0000, Jon Masters wrote:
> Have you considered reviving the patch you posted previously for EDAC
> support (the atomic_scrub read/write test piece dependency)?
> 
> http://lists.infradead.org/pipermail/linux-arm-kernel/2014-April/249039.html

Well, we'd need a way to handle the non-coherent DMA case and it's really
not clear how to fix that.

Will

EDAC on arm64

[Catalin Marinas]

On Mon, Mar 02, 2015 at 10:59:32AM +0000, Will Deacon wrote:
> On Sat, Feb 28, 2015 at 12:52:03AM +0000, Jon Masters wrote:
> > Have you considered reviving the patch you posted previously for EDAC
> > support (the atomic_scrub read/write test piece dependency)?
> > 
> > http://lists.infradead.org/pipermail/linux-arm-kernel/2014-April/249039.html
> 
> Well, we'd need a way to handle the non-coherent DMA case and it's really
> not clear how to fix that.

I agree, that's where the discussions stopped. Basically the EDAC memory
writing is racy with any non-cacheable memory accesses (by CPU or
device). The only way we could safely use this is only if all the
devices are coherent *and* KVM is disabled. With KVM, guests may access
the memory uncached, so we hit the same problem.

-- 
Catalin

EDAC on arm64

[Rob Herring]

On Mon, Mar 2, 2015 at 8:58 AM, Catalin Marinas <catalin.marinas@arm.com> wrote:
> On Mon, Mar 02, 2015 at 10:59:32AM +0000, Will Deacon wrote:
>> On Sat, Feb 28, 2015 at 12:52:03AM +0000, Jon Masters wrote:
>> > Have you considered reviving the patch you posted previously for EDAC
>> > support (the atomic_scrub read/write test piece dependency)?
>> >
>> > http://lists.infradead.org/pipermail/linux-arm-kernel/2014-April/249039.html
>>
>> Well, we'd need a way to handle the non-coherent DMA case and it's really
>> not clear how to fix that.
>
> I agree, that's where the discussions stopped. Basically the EDAC memory
> writing is racy with any non-cacheable memory accesses (by CPU or
> device). The only way we could safely use this is only if all the
> devices are coherent *and* KVM is disabled. With KVM, guests may access
> the memory uncached, so we hit the same problem.

Scrubbing only prevents repeated error reporting of correctable errors
which only repeat on a cache miss. Perhaps we should just add an empty
version that is a nop. At least then we can enable EDAC and be able to
report errors. Otherwise, they may just be silently ignored (fine for
correctable, but not so good to do with uncorrectable errors).

Anyway, I'll leave it to someone who still has this itch to sort out.
My itch was cured.

Rob

>
> --
> Catalin

EDAC on arm64

[Catalin Marinas]

On Mon, Mar 02, 2015 at 10:34:16AM -0600, Rob Herring wrote:
> On Mon, Mar 2, 2015 at 8:58 AM, Catalin Marinas <catalin.marinas@arm.com> wrote:
> > On Mon, Mar 02, 2015 at 10:59:32AM +0000, Will Deacon wrote:
> >> On Sat, Feb 28, 2015 at 12:52:03AM +0000, Jon Masters wrote:
> >> > Have you considered reviving the patch you posted previously for EDAC
> >> > support (the atomic_scrub read/write test piece dependency)?
> >> >
> >> > http://lists.infradead.org/pipermail/linux-arm-kernel/2014-April/249039.html
> >>
> >> Well, we'd need a way to handle the non-coherent DMA case and it's really
> >> not clear how to fix that.
> >
> > I agree, that's where the discussions stopped. Basically the EDAC memory
> > writing is racy with any non-cacheable memory accesses (by CPU or
> > device). The only way we could safely use this is only if all the
> > devices are coherent *and* KVM is disabled. With KVM, guests may access
> > the memory uncached, so we hit the same problem.
> 
> Scrubbing only prevents repeated error reporting of correctable errors
> which only repeat on a cache miss. Perhaps we should just add an empty
> version that is a nop. At least then we can enable EDAC and be able to
> report errors. Otherwise, they may just be silently ignored (fine for
> correctable, but not so good to do with uncorrectable errors).

This looks fine to me.

> Anyway, I'll leave it to someone who still has this itch to sort out.

Maybe Jon? ;)

-- 
Catalin

EDAC on arm64

[Catalin Marinas]

On Mon, Mar 02, 2015 at 10:34:16AM -0600, Rob Herring wrote:
> On Mon, Mar 2, 2015 at 8:58 AM, Catalin Marinas <catalin.marinas@arm.com> wrote:
> > On Mon, Mar 02, 2015 at 10:59:32AM +0000, Will Deacon wrote:
> >> On Sat, Feb 28, 2015 at 12:52:03AM +0000, Jon Masters wrote:
> >> > Have you considered reviving the patch you posted previously for EDAC
> >> > support (the atomic_scrub read/write test piece dependency)?
> >> >
> >> > http://lists.infradead.org/pipermail/linux-arm-kernel/2014-April/249039.html
> >>
> >> Well, we'd need a way to handle the non-coherent DMA case and it's really
> >> not clear how to fix that.
> >
> > I agree, that's where the discussions stopped. Basically the EDAC memory
> > writing is racy with any non-cacheable memory accesses (by CPU or
> > device). The only way we could safely use this is only if all the
> > devices are coherent *and* KVM is disabled. With KVM, guests may access
> > the memory uncached, so we hit the same problem.
> 
> Scrubbing only prevents repeated error reporting of correctable errors
> which only repeat on a cache miss. Perhaps we should just add an empty
> version that is a nop.

Can the error be cleared by a cache clean&invalidate by VA?

-- 
Catalin

EDAC on arm64

[Rob Herring]

On Mon, Mar 2, 2015 at 4:27 PM, Catalin Marinas <catalin.marinas@arm.com> wrote:
> On Mon, Mar 02, 2015 at 10:34:16AM -0600, Rob Herring wrote:
>> On Mon, Mar 2, 2015 at 8:58 AM, Catalin Marinas <catalin.marinas@arm.com> wrote:
>> > On Mon, Mar 02, 2015 at 10:59:32AM +0000, Will Deacon wrote:
>> >> On Sat, Feb 28, 2015 at 12:52:03AM +0000, Jon Masters wrote:
>> >> > Have you considered reviving the patch you posted previously for EDAC
>> >> > support (the atomic_scrub read/write test piece dependency)?
>> >> >
>> >> > http://lists.infradead.org/pipermail/linux-arm-kernel/2014-April/249039.html
>> >>
>> >> Well, we'd need a way to handle the non-coherent DMA case and it's really
>> >> not clear how to fix that.
>> >
>> > I agree, that's where the discussions stopped. Basically the EDAC memory
>> > writing is racy with any non-cacheable memory accesses (by CPU or
>> > device). The only way we could safely use this is only if all the
>> > devices are coherent *and* KVM is disabled. With KVM, guests may access
>> > the memory uncached, so we hit the same problem.
>>
>> Scrubbing only prevents repeated error reporting of correctable errors
>> which only repeat on a cache miss. Perhaps we should just add an empty
>> version that is a nop.
>
> Can the error be cleared by a cache clean&invalidate by VA?

Yes, but only if the line is in fact dirty. If the line is clean, it
will just make the error repeat on every access. You can't make the
line dirty for the same reasons you've mentioned.

Rob

EDAC on arm64

[Catalin Marinas]

On Mon, Mar 02, 2015 at 04:57:23PM -0600, Rob Herring wrote:
> On Mon, Mar 2, 2015 at 4:27 PM, Catalin Marinas <catalin.marinas@arm.com> wrote:
> > On Mon, Mar 02, 2015 at 10:34:16AM -0600, Rob Herring wrote:
> >> On Mon, Mar 2, 2015 at 8:58 AM, Catalin Marinas <catalin.marinas@arm.com> wrote:
> >> > On Mon, Mar 02, 2015 at 10:59:32AM +0000, Will Deacon wrote:
> >> >> On Sat, Feb 28, 2015 at 12:52:03AM +0000, Jon Masters wrote:
> >> >> > Have you considered reviving the patch you posted previously for EDAC
> >> >> > support (the atomic_scrub read/write test piece dependency)?
> >> >> >
> >> >> > http://lists.infradead.org/pipermail/linux-arm-kernel/2014-April/249039.html
> >> >>
> >> >> Well, we'd need a way to handle the non-coherent DMA case and it's really
> >> >> not clear how to fix that.
> >> >
> >> > I agree, that's where the discussions stopped. Basically the EDAC memory
> >> > writing is racy with any non-cacheable memory accesses (by CPU or
> >> > device). The only way we could safely use this is only if all the
> >> > devices are coherent *and* KVM is disabled. With KVM, guests may access
> >> > the memory uncached, so we hit the same problem.
> >>
> >> Scrubbing only prevents repeated error reporting of correctable errors
> >> which only repeat on a cache miss. Perhaps we should just add an empty
> >> version that is a nop.
> >
> > Can the error be cleared by a cache clean&invalidate by VA?
> 
> Yes, but only if the line is in fact dirty. If the line is clean, it
> will just make the error repeat on every access.

So is the error reported even if the line is removed from the cache
entirely via a (clean _and_) invalidate operation? Once removed from the
cache, an access cannot hit the line as it's no longer there.

-- 
Catalin

EDAC on arm64

[Rob Herring]

On Tue, Mar 3, 2015 at 5:01 AM, Catalin Marinas <catalin.marinas@arm.com> wrote:
> On Mon, Mar 02, 2015 at 04:57:23PM -0600, Rob Herring wrote:
>> On Mon, Mar 2, 2015 at 4:27 PM, Catalin Marinas <catalin.marinas@arm.com> wrote:
>> > On Mon, Mar 02, 2015 at 10:34:16AM -0600, Rob Herring wrote:
>> >> On Mon, Mar 2, 2015 at 8:58 AM, Catalin Marinas <catalin.marinas@arm.com> wrote:
>> >> > On Mon, Mar 02, 2015 at 10:59:32AM +0000, Will Deacon wrote:
>> >> >> On Sat, Feb 28, 2015 at 12:52:03AM +0000, Jon Masters wrote:
>> >> >> > Have you considered reviving the patch you posted previously for EDAC
>> >> >> > support (the atomic_scrub read/write test piece dependency)?
>> >> >> >
>> >> >> > http://lists.infradead.org/pipermail/linux-arm-kernel/2014-April/249039.html
>> >> >>
>> >> >> Well, we'd need a way to handle the non-coherent DMA case and it's really
>> >> >> not clear how to fix that.
>> >> >
>> >> > I agree, that's where the discussions stopped. Basically the EDAC memory
>> >> > writing is racy with any non-cacheable memory accesses (by CPU or
>> >> > device). The only way we could safely use this is only if all the
>> >> > devices are coherent *and* KVM is disabled. With KVM, guests may access
>> >> > the memory uncached, so we hit the same problem.
>> >>
>> >> Scrubbing only prevents repeated error reporting of correctable errors
>> >> which only repeat on a cache miss. Perhaps we should just add an empty
>> >> version that is a nop.
>> >
>> > Can the error be cleared by a cache clean&invalidate by VA?
>>
>> Yes, but only if the line is in fact dirty. If the line is clean, it
>> will just make the error repeat on every access.
>
> So is the error reported even if the line is removed from the cache
> entirely via a (clean _and_) invalidate operation? Once removed from the
> cache, an access cannot hit the line as it's no longer there.

But when there is a miss and you fetch the line from memory again, the
error will be triggered again. Invalidating the cache would work would
work if you are doing ECC checks in the caches, but I'm thinking more
of the scenario where the memory controller is doing the checking. An
error is going to be generated whenever the memory controller fetches
a word (64-bits) with an error. The error could be induced either on
the read in which case it is one time transient error or on the write
in which case it will repeat until you write that location again. I
don't think you typically know which case it is. Assuming that
location is only read, then you will get errors reported each time
there is a cache miss (or DMA reads the location).

BTW, some processors do ECC calc and checks in the caches and maintain
that all the way to memory. Then the memory contents are protected in
the caches and internal buses and you're not doing ECC twice (in the
L2/L3 and memory ctrlr). Of course, you can't have non-coherent DMA in
that case.

Rob

>
> --
> Catalin

EDAC on arm64

[Arnd Bergmann]

On Monday 02 March 2015 14:58:41 Catalin Marinas wrote:
> On Mon, Mar 02, 2015 at 10:59:32AM +0000, Will Deacon wrote:
> > On Sat, Feb 28, 2015 at 12:52:03AM +0000, Jon Masters wrote:
> > > Have you considered reviving the patch you posted previously for EDAC
> > > support (the atomic_scrub read/write test piece dependency)?
> > > 
> > > http://lists.infradead.org/pipermail/linux-arm-kernel/2014-April/249039.html
> > 
> > Well, we'd need a way to handle the non-coherent DMA case and it's really
> > not clear how to fix that.
> 
> I agree, that's where the discussions stopped. Basically the EDAC memory
> writing is racy with any non-cacheable memory accesses (by CPU or
> device). The only way we could safely use this is only if all the
> devices are coherent *and* KVM is disabled. With KVM, guests may access
> the memory uncached, so we hit the same problem.

Is this a setting of the host, or does the guest always have this capability?

If a guest can influence the caching of a page it has access to, I can
imagine all sorts of security problems with malicious guests regardless
of EDAC.

	Arnd

EDAC on arm64

[Catalin Marinas]

On Mon, Mar 02, 2015 at 08:40:16PM +0100, Arnd Bergmann wrote:
> On Monday 02 March 2015 14:58:41 Catalin Marinas wrote:
> > On Mon, Mar 02, 2015 at 10:59:32AM +0000, Will Deacon wrote:
> > > On Sat, Feb 28, 2015 at 12:52:03AM +0000, Jon Masters wrote:
> > > > Have you considered reviving the patch you posted previously for EDAC
> > > > support (the atomic_scrub read/write test piece dependency)?
> > > > 
> > > > http://lists.infradead.org/pipermail/linux-arm-kernel/2014-April/249039.html
> > > 
> > > Well, we'd need a way to handle the non-coherent DMA case and it's really
> > > not clear how to fix that.
> > 
> > I agree, that's where the discussions stopped. Basically the EDAC memory
> > writing is racy with any non-cacheable memory accesses (by CPU or
> > device). The only way we could safely use this is only if all the
> > devices are coherent *and* KVM is disabled. With KVM, guests may access
> > the memory uncached, so we hit the same problem.
> 
> Is this a setting of the host, or does the guest always have this capability?

The guest can always make it stricter than what the host set in stage 2
(i.e. from Normal Cacheable -> NonCacheable -> Device) but never in the
other direction.

> If a guest can influence the caching of a page it has access to, I can
> imagine all sorts of security problems with malicious guests regardless
> of EDAC.

Not as long as the host is aware of this. Basically it needs to flush
the cache on a page when it is mapped into the guest address space (IPA)
and flush it again when reading a page from guest.

-- 
Catalin

EDAC on arm64

[Arnd Bergmann]

On Monday 02 March 2015 22:25:16 Catalin Marinas wrote:
> On Mon, Mar 02, 2015 at 08:40:16PM +0100, Arnd Bergmann wrote:
> > On Monday 02 March 2015 14:58:41 Catalin Marinas wrote:
> > > On Mon, Mar 02, 2015 at 10:59:32AM +0000, Will Deacon wrote:
> > > > On Sat, Feb 28, 2015 at 12:52:03AM +0000, Jon Masters wrote:
> > > > > Have you considered reviving the patch you posted previously for EDAC
> > > > > support (the atomic_scrub read/write test piece dependency)?
> > > > > 
> > > > > http://lists.infradead.org/pipermail/linux-arm-kernel/2014-April/249039.html
> > > > 
> > > > Well, we'd need a way to handle the non-coherent DMA case and it's really
> > > > not clear how to fix that.
> > > 
> > > I agree, that's where the discussions stopped. Basically the EDAC memory
> > > writing is racy with any non-cacheable memory accesses (by CPU or
> > > device). The only way we could safely use this is only if all the
> > > devices are coherent *and* KVM is disabled. With KVM, guests may access
> > > the memory uncached, so we hit the same problem.
> > 
> > Is this a setting of the host, or does the guest always have this capability?
> 
> The guest can always make it stricter than what the host set in stage 2
> (i.e. from Normal Cacheable -> NonCacheable -> Device) but never in the
> other direction.

Do you have an idea what the purpose of this is? Why would a guest
even want to mark pages as noncachable that are mapped into the
host as cachable and that might have dirty cache lines?

> > If a guest can influence the caching of a page it has access to, I can
> > imagine all sorts of security problems with malicious guests regardless
> > of EDAC.
> 
> Not as long as the host is aware of this. Basically it needs to flush
> the cache on a page when it is mapped into the guest address space (IPA)
> and flush it again when reading a page from guest.

You have to flush and invalidate the cache line, but of course nobody
wants to do that because it totally destroys performance.

	Arnd

EDAC on arm64

[Catalin Marinas]

On Tue, Mar 03, 2015 at 10:23:06AM +0100, Arnd Bergmann wrote:
> On Monday 02 March 2015 22:25:16 Catalin Marinas wrote:
> > On Mon, Mar 02, 2015 at 08:40:16PM +0100, Arnd Bergmann wrote:
> > > On Monday 02 March 2015 14:58:41 Catalin Marinas wrote:
> > > > On Mon, Mar 02, 2015 at 10:59:32AM +0000, Will Deacon wrote:
> > > > > On Sat, Feb 28, 2015 at 12:52:03AM +0000, Jon Masters wrote:
> > > > > > Have you considered reviving the patch you posted previously for EDAC
> > > > > > support (the atomic_scrub read/write test piece dependency)?
> > > > > > 
> > > > > > http://lists.infradead.org/pipermail/linux-arm-kernel/2014-April/249039.html
> > > > > 
> > > > > Well, we'd need a way to handle the non-coherent DMA case and it's really
> > > > > not clear how to fix that.
> > > > 
> > > > I agree, that's where the discussions stopped. Basically the EDAC memory
> > > > writing is racy with any non-cacheable memory accesses (by CPU or
> > > > device). The only way we could safely use this is only if all the
> > > > devices are coherent *and* KVM is disabled. With KVM, guests may access
> > > > the memory uncached, so we hit the same problem.
> > > 
> > > Is this a setting of the host, or does the guest always have this capability?
> > 
> > The guest can always make it stricter than what the host set in stage 2
> > (i.e. from Normal Cacheable -> NonCacheable -> Device) but never in the
> > other direction.
> 
> Do you have an idea what the purpose of this is? Why would a guest
> even want to mark pages as noncachable that are mapped into the
> host as cachable and that might have dirty cache lines?

The stage 1 / stage 2 attributes combination works such that the
hypervisor can impose more stricter attributes or none at all (in which
case it is up to the guest to decide what it needs). So for example
devices mapped into the guest address space (e.g. the GIC) are marked as
Device memory in stage 2 so that the guest could never map them as
Normal Cacheable memory (with some bad consequences).

The other direction is that the guest may want to create a stricter
mapping than what the host wants. A possible reason is some frame
buffer or anything else where the guest assumes that by creating a
non-cacheable mapping it won't need to do cache maintenance. That's why
when KVM maps a page into the guest address space, it flushes the cache
so there are no dirty lines. Since the host would not write to such page
again, it won't dirty the cache (and if the guest does, it needs to deal
with it itself).

There are some scenarios where this does not work well: a virtual frame
buffer emulated by Qemu where the guest thinks it is non-cacheable and
Qemu maps it as cacheable. The only sane solution here is to tell the
guest that the (virtual) frame buffer device is DMA coherent and that it
should use a cacheable mapping. But I don't think the host should
somehow (transparently) upgrade the cacheability that the guest thinks
it has.

> > > If a guest can influence the caching of a page it has access to, I can
> > > imagine all sorts of security problems with malicious guests regardless
> > > of EDAC.
> > 
> > Not as long as the host is aware of this. Basically it needs to flush
> > the cache on a page when it is mapped into the guest address space (IPA)
> > and flush it again when reading a page from guest.
> 
> You have to flush and invalidate the cache line, but of course nobody
> wants to do that because it totally destroys performance.

There are two cases when the cache needs flushing: (1) when a page is
mapped into the guest address space (done lazily via the page faulting
mechanism) and (2) when the host reads a page already mapped in the
guest address space (e.g. swapping out). They indeed take some time but
none of them are on a critical path.

(and maybe at some point we'll get fully transparent caches on ARM as
well, so we don't have to worry about cache maintenance)

-- 
Catalin