Highmem issues with MMC filesystem

Highmem issues with MMC filesystem

[Hemanth V]

Hi All,

We are trying to enable highmem support for OMAP based boards, while doing 
so we are
unable to boot poky UI if the filesystem is in a MMC card.

The poky UI comes up fine if the filesystem is in NAND or NFS,
while with MMC it gives errors like segmentation faults/syntax error while 
there are none.
Seems like read from MMC is getting corrupted.

I tried  the 4 tests which are available as part of mmc_test for highmem, 
all of them seem to PASS.
Wanted to get inputs if there are any known issues with MMC block layer 
handling highmem pages.

Thanks
Hemanth 

Re: Highmem issues with MMC filesystem

[Nicolas Pitre]

On Wed, 17 Mar 2010, Hemanth V wrote:

> Hi All,
> 
> We are trying to enable highmem support for OMAP based boards, while doing so
> we are
> unable to boot poky UI if the filesystem is in a MMC card.
> 
> The poky UI comes up fine if the filesystem is in NAND or NFS,
> while with MMC it gives errors like segmentation faults/syntax error while
> there are none.
> Seems like read from MMC is getting corrupted.

Yes, this is a known issue with highmem on ARM architectures v6 and 
later, related to some cache mishandling.  This is not a MMC issue as 
the problem occurs when booting from SATA too. The problem is still 
investigated.


Nicolas

Re: Highmem issues with MMC filesystem

[Hemanth V]

----- Original Message ----- 
From: "Nicolas Pitre" <nico@fluxnic.net>
To: "Hemanth V" <hemanthv@ti.com>
Cc: <linux-mmc@vger.kernel.org>; <pierre@ossman.eu>
Sent: Wednesday, March 17, 2010 7:19 PM
Subject: Re: Highmem issues with MMC filesystem


> On Wed, 17 Mar 2010, Hemanth V wrote:
>
>> Hi All,
>>
>> We are trying to enable highmem support for OMAP based boards, while 
>> doing so
>> we are
>> unable to boot poky UI if the filesystem is in a MMC card.
>>
>> The poky UI comes up fine if the filesystem is in NAND or NFS,
>> while with MMC it gives errors like segmentation faults/syntax error 
>> while
>> there are none.
>> Seems like read from MMC is getting corrupted.
>
> Yes, this is a known issue with highmem on ARM architectures v6 and
> later, related to some cache mishandling.  This is not a MMC issue as
> the problem occurs when booting from SATA too. The problem is still
> investigated.
>

Nicolas, Russel are there any more details available which might help
in investigating this further.

Thanks
Hemanth

>
> Nicolas
> 

Highmem issues with MMC filesystem

[Hemanth V]

----- Original Message ----- 
From: "Nicolas Pitre" <nico@fluxnic.net>
To: "Hemanth V" <hemanthv@ti.com>
Cc: <linux-mmc@vger.kernel.org>; <pierre@ossman.eu>
Sent: Wednesday, March 17, 2010 7:19 PM
Subject: Re: Highmem issues with MMC filesystem


> On Wed, 17 Mar 2010, Hemanth V wrote:
>
>> Hi All,
>>
>> We are trying to enable highmem support for OMAP based boards, while 
>> doing so
>> we are
>> unable to boot poky UI if the filesystem is in a MMC card.
>>
>> The poky UI comes up fine if the filesystem is in NAND or NFS,
>> while with MMC it gives errors like segmentation faults/syntax error 
>> while
>> there are none.
>> Seems like read from MMC is getting corrupted.
>
> Yes, this is a known issue with highmem on ARM architectures v6 and
> later, related to some cache mishandling.  This is not a MMC issue as
> the problem occurs when booting from SATA too. The problem is still
> investigated.
>

Nicolas, Russel are there any more details available which might help
in investigating this further.

Thanks
Hemanth

>
> Nicolas
> 

Re: Highmem issues with MMC filesystem

[Nicolas Pitre]

On Wed, 17 Mar 2010, Hemanth V wrote:

> 
> ----- Original Message ----- From: "Nicolas Pitre" <nico@fluxnic.net>
> To: "Hemanth V" <hemanthv@ti.com>
> Cc: <linux-mmc@vger.kernel.org>; <pierre@ossman.eu>
> Sent: Wednesday, March 17, 2010 7:19 PM
> Subject: Re: Highmem issues with MMC filesystem
> 
> 
> > On Wed, 17 Mar 2010, Hemanth V wrote:
> > 
> > > Hi All,
> > > 
> > > We are trying to enable highmem support for OMAP based boards, while doing
> > > so
> > > we are
> > > unable to boot poky UI if the filesystem is in a MMC card.
> > > 
> > > The poky UI comes up fine if the filesystem is in NAND or NFS,
> > > while with MMC it gives errors like segmentation faults/syntax error while
> > > there are none.
> > > Seems like read from MMC is getting corrupted.
> > 
> > Yes, this is a known issue with highmem on ARM architectures v6 and
> > later, related to some cache mishandling.  This is not a MMC issue as
> > the problem occurs when booting from SATA too. The problem is still
> > investigated.
> > 
> 
> Nicolas, Russel are there any more details available which might help
> in investigating this further.

The only conclusion I came to so far is that ARMv5 where highmem works 
just fine in all cases has VIVT cache whereas ARMv6 has VIPT cache.  
And the problem with VIPT caches occurs when direct DMA is involved, 
otherwise there is no problem if PIO or NFS is used.  Sprinkling some 
flush_cache_all() in a few places makes things work, but this is not a 
satisfactory solution.


Nicolas

Highmem issues with MMC filesystem

[Nicolas Pitre]

On Wed, 17 Mar 2010, Hemanth V wrote:

> 
> ----- Original Message ----- From: "Nicolas Pitre" <nico@fluxnic.net>
> To: "Hemanth V" <hemanthv@ti.com>
> Cc: <linux-mmc@vger.kernel.org>; <pierre@ossman.eu>
> Sent: Wednesday, March 17, 2010 7:19 PM
> Subject: Re: Highmem issues with MMC filesystem
> 
> 
> > On Wed, 17 Mar 2010, Hemanth V wrote:
> > 
> > > Hi All,
> > > 
> > > We are trying to enable highmem support for OMAP based boards, while doing
> > > so
> > > we are
> > > unable to boot poky UI if the filesystem is in a MMC card.
> > > 
> > > The poky UI comes up fine if the filesystem is in NAND or NFS,
> > > while with MMC it gives errors like segmentation faults/syntax error while
> > > there are none.
> > > Seems like read from MMC is getting corrupted.
> > 
> > Yes, this is a known issue with highmem on ARM architectures v6 and
> > later, related to some cache mishandling.  This is not a MMC issue as
> > the problem occurs when booting from SATA too. The problem is still
> > investigated.
> > 
> 
> Nicolas, Russel are there any more details available which might help
> in investigating this further.

The only conclusion I came to so far is that ARMv5 where highmem works 
just fine in all cases has VIVT cache whereas ARMv6 has VIPT cache.  
And the problem with VIPT caches occurs when direct DMA is involved, 
otherwise there is no problem if PIO or NFS is used.  Sprinkling some 
flush_cache_all() in a few places makes things work, but this is not a 
satisfactory solution.


Nicolas

Re: Highmem issues with MMC filesystem

[Russell King - ARM Linux]

On Wed, Mar 17, 2010 at 10:58:48AM -0400, Nicolas Pitre wrote:
> On Wed, 17 Mar 2010, Hemanth V wrote:
> > Nicolas, Russel are there any more details available which might help
> > in investigating this further.
> 
> The only conclusion I came to so far is that ARMv5 where highmem works 
> just fine in all cases has VIVT cache whereas ARMv6 has VIPT cache.  
> And the problem with VIPT caches occurs when direct DMA is involved, 
> otherwise there is no problem if PIO or NFS is used.  Sprinkling some 
> flush_cache_all() in a few places makes things work, but this is not a 
> satisfactory solution.

This sounds like the problem we had with the DMA API.  Since that's now
fixed, there shouldn't be a problem with the latest (-rc) kernels, or
a kernel with my old streaming DMA patches applied.

Highmem issues with MMC filesystem

[Russell King - ARM Linux]

On Wed, Mar 17, 2010 at 10:58:48AM -0400, Nicolas Pitre wrote:
> On Wed, 17 Mar 2010, Hemanth V wrote:
> > Nicolas, Russel are there any more details available which might help
> > in investigating this further.
> 
> The only conclusion I came to so far is that ARMv5 where highmem works 
> just fine in all cases has VIVT cache whereas ARMv6 has VIPT cache.  
> And the problem with VIPT caches occurs when direct DMA is involved, 
> otherwise there is no problem if PIO or NFS is used.  Sprinkling some 
> flush_cache_all() in a few places makes things work, but this is not a 
> satisfactory solution.

This sounds like the problem we had with the DMA API.  Since that's now
fixed, there shouldn't be a problem with the latest (-rc) kernels, or
a kernel with my old streaming DMA patches applied.

Re: Highmem issues with MMC filesystem

[saeed bishara]

>> The only conclusion I came to so far is that ARMv5 where highmem works
>> just fine in all cases has VIVT cache whereas ARMv6 has VIPT cache.
>> And the problem with VIPT caches occurs when direct DMA is involved,
>> otherwise there is no problem if PIO or NFS is used.  Sprinkling some
>> flush_cache_all() in a few places makes things work, but this is not a
>> satisfactory solution.
>
> This sounds like the problem we had with the DMA API.  Since that's now
> fixed, there shouldn't be a problem with the latest (-rc) kernels, or
> a kernel with my old streaming DMA patches applied.
The failure happens also on 2.6.34.rc1,  as Nico said, it looks like
that buffers that are subject to DMA remain dirty, as I understand it,
for vipt nonaliasing cpu's, the kernel doesn't clean user space cache
lines. if I force kmap_atomic/kunmap_atomic to highmem pages that are
not mapped by the kernel (kmap_high_get returns null), then the issue
disappears.
saeed

Highmem issues with MMC filesystem

[saeed bishara]

>> The only conclusion I came to so far is that ARMv5 where highmem works
>> just fine in all cases has VIVT cache whereas ARMv6 has VIPT cache.
>> And the problem with VIPT caches occurs when direct DMA is involved,
>> otherwise there is no problem if PIO or NFS is used. ?Sprinkling some
>> flush_cache_all() in a few places makes things work, but this is not a
>> satisfactory solution.
>
> This sounds like the problem we had with the DMA API. ?Since that's now
> fixed, there shouldn't be a problem with the latest (-rc) kernels, or
> a kernel with my old streaming DMA patches applied.
The failure happens also on 2.6.34.rc1,  as Nico said, it looks like
that buffers that are subject to DMA remain dirty, as I understand it,
for vipt nonaliasing cpu's, the kernel doesn't clean user space cache
lines. if I force kmap_atomic/kunmap_atomic to highmem pages that are
not mapped by the kernel (kmap_high_get returns null), then the issue
disappears.
saeed

Re: Highmem issues with MMC filesystem

[Russell King - ARM Linux]

On Thu, Mar 18, 2010 at 01:15:58PM +0200, saeed bishara wrote:
> >> The only conclusion I came to so far is that ARMv5 where highmem works
> >> just fine in all cases has VIVT cache whereas ARMv6 has VIPT cache.
> >> And the problem with VIPT caches occurs when direct DMA is involved,
> >> otherwise there is no problem if PIO or NFS is used.  Sprinkling some
> >> flush_cache_all() in a few places makes things work, but this is not a
> >> satisfactory solution.
> >
> > This sounds like the problem we had with the DMA API.  Since that's now
> > fixed, there shouldn't be a problem with the latest (-rc) kernels, or
> > a kernel with my old streaming DMA patches applied.
> The failure happens also on 2.6.34.rc1,  as Nico said, it looks like
> that buffers that are subject to DMA remain dirty, as I understand it,
> for vipt nonaliasing cpu's, the kernel doesn't clean user space cache
> lines. if I force kmap_atomic/kunmap_atomic to highmem pages that are
> not mapped by the kernel (kmap_high_get returns null), then the issue
> disappears.

In no case does the kernel ever clean user space cache lines for DMA;
that's not the responsibility of the DMA API.

Highmem issues with MMC filesystem

[Russell King - ARM Linux]

On Thu, Mar 18, 2010 at 01:15:58PM +0200, saeed bishara wrote:
> >> The only conclusion I came to so far is that ARMv5 where highmem works
> >> just fine in all cases has VIVT cache whereas ARMv6 has VIPT cache.
> >> And the problem with VIPT caches occurs when direct DMA is involved,
> >> otherwise there is no problem if PIO or NFS is used. ?Sprinkling some
> >> flush_cache_all() in a few places makes things work, but this is not a
> >> satisfactory solution.
> >
> > This sounds like the problem we had with the DMA API. ?Since that's now
> > fixed, there shouldn't be a problem with the latest (-rc) kernels, or
> > a kernel with my old streaming DMA patches applied.
> The failure happens also on 2.6.34.rc1,  as Nico said, it looks like
> that buffers that are subject to DMA remain dirty, as I understand it,
> for vipt nonaliasing cpu's, the kernel doesn't clean user space cache
> lines. if I force kmap_atomic/kunmap_atomic to highmem pages that are
> not mapped by the kernel (kmap_high_get returns null), then the issue
> disappears.

In no case does the kernel ever clean user space cache lines for DMA;
that's not the responsibility of the DMA API.

Re: Highmem issues with MMC filesystem

[Nicolas Pitre]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 2151 bytes --]

On Thu, 18 Mar 2010, Russell King - ARM Linux wrote:

> On Thu, Mar 18, 2010 at 01:15:58PM +0200, saeed bishara wrote:
> > >> The only conclusion I came to so far is that ARMv5 where highmem works
> > >> just fine in all cases has VIVT cache whereas ARMv6 has VIPT cache.
> > >> And the problem with VIPT caches occurs when direct DMA is involved,
> > >> otherwise there is no problem if PIO or NFS is used.  Sprinkling some
> > >> flush_cache_all() in a few places makes things work, but this is not a
> > >> satisfactory solution.
> > >
> > > This sounds like the problem we had with the DMA API.  Since that's now
> > > fixed, there shouldn't be a problem with the latest (-rc) kernels, or
> > > a kernel with my old streaming DMA patches applied.
> > The failure happens also on 2.6.34.rc1,  as Nico said, it looks like
> > that buffers that are subject to DMA remain dirty, as I understand it,
> > for vipt nonaliasing cpu's, the kernel doesn't clean user space cache
> > lines. if I force kmap_atomic/kunmap_atomic to highmem pages that are
> > not mapped by the kernel (kmap_high_get returns null), then the issue
> > disappears.
> 
> In no case does the kernel ever clean user space cache lines for DMA;
> that's not the responsibility of the DMA API.

Let's forget about user space.  Even some kernel space memory is 
affected too.

The issue as I see it is that highmem pages being DMA'd to may be cached 
even when they're unmapped on VIPT machines.  And the DMA code performs 
L1 cache maintenance only on pages which are virtually mapped.

Contrary to VIVT caches which have to be flushed all the time, VIPT 
caches may avoid cache flushing in some cases, which may lead to some 
highmem pages not being mapped but still cached somehow.  But so far I 
just can't find how that could happen.

The only way a highmem page can be unmapped is through kunmap_atomic() 
where an explicit __cpuc_flush_dcache_area() is performed, or through 
flush_all_zero_pkmaps() where flush_cache_kmaps() translates into 
flush_cache_all().

So something allows for some highmem pages to escape cache flushing when 
unmapped.  But I can't find it.


Nicolas

Highmem issues with MMC filesystem

[Nicolas Pitre]

On Thu, 18 Mar 2010, Russell King - ARM Linux wrote:

> On Thu, Mar 18, 2010 at 01:15:58PM +0200, saeed bishara wrote:
> > >> The only conclusion I came to so far is that ARMv5 where highmem works
> > >> just fine in all cases has VIVT cache whereas ARMv6 has VIPT cache.
> > >> And the problem with VIPT caches occurs when direct DMA is involved,
> > >> otherwise there is no problem if PIO or NFS is used. ?Sprinkling some
> > >> flush_cache_all() in a few places makes things work, but this is not a
> > >> satisfactory solution.
> > >
> > > This sounds like the problem we had with the DMA API. ?Since that's now
> > > fixed, there shouldn't be a problem with the latest (-rc) kernels, or
> > > a kernel with my old streaming DMA patches applied.
> > The failure happens also on 2.6.34.rc1,  as Nico said, it looks like
> > that buffers that are subject to DMA remain dirty, as I understand it,
> > for vipt nonaliasing cpu's, the kernel doesn't clean user space cache
> > lines. if I force kmap_atomic/kunmap_atomic to highmem pages that are
> > not mapped by the kernel (kmap_high_get returns null), then the issue
> > disappears.
> 
> In no case does the kernel ever clean user space cache lines for DMA;
> that's not the responsibility of the DMA API.

Let's forget about user space.  Even some kernel space memory is 
affected too.

The issue as I see it is that highmem pages being DMA'd to may be cached 
even when they're unmapped on VIPT machines.  And the DMA code performs 
L1 cache maintenance only on pages which are virtually mapped.

Contrary to VIVT caches which have to be flushed all the time, VIPT 
caches may avoid cache flushing in some cases, which may lead to some 
highmem pages not being mapped but still cached somehow.  But so far I 
just can't find how that could happen.

The only way a highmem page can be unmapped is through kunmap_atomic() 
where an explicit __cpuc_flush_dcache_area() is performed, or through 
flush_all_zero_pkmaps() where flush_cache_kmaps() translates into 
flush_cache_all().

So something allows for some highmem pages to escape cache flushing when 
unmapped.  But I can't find it.


Nicolas

RE: Highmem issues with MMC filesystem

[Shilimkar, Santosh]

> -----Original Message-----
> From: linux-arm-kernel-bounces@lists.infradead.org [mailto:linux-arm-kernel-
> bounces@lists.infradead.org] On Behalf Of Nicolas Pitre
> Sent: Thursday, March 18, 2010 6:50 PM
> To: Russell King - ARM Linux
> Cc: linux-mmc@vger.kernel.org; V, Hemanth; saeed bishara; pierre@ossman.eu; linux-arm-
> kernel@lists.infradead.org
> Subject: Re: Highmem issues with MMC filesystem
> 
> On Thu, 18 Mar 2010, Russell King - ARM Linux wrote:
> 
> > On Thu, Mar 18, 2010 at 01:15:58PM +0200, saeed bishara wrote:
> > > >> The only conclusion I came to so far is that ARMv5 where highmem works
> > > >> just fine in all cases has VIVT cache whereas ARMv6 has VIPT cache.
> > > >> And the problem with VIPT caches occurs when direct DMA is involved,
> > > >> otherwise there is no problem if PIO or NFS is used.  Sprinkling some
> > > >> flush_cache_all() in a few places makes things work, but this is not a
> > > >> satisfactory solution.
> > > >
> > > > This sounds like the problem we had with the DMA API.  Since that's now
> > > > fixed, there shouldn't be a problem with the latest (-rc) kernels, or
> > > > a kernel with my old streaming DMA patches applied.
> > > The failure happens also on 2.6.34.rc1,  as Nico said, it looks like
> > > that buffers that are subject to DMA remain dirty, as I understand it,
> > > for vipt nonaliasing cpu's, the kernel doesn't clean user space cache
> > > lines. if I force kmap_atomic/kunmap_atomic to highmem pages that are
> > > not mapped by the kernel (kmap_high_get returns null), then the issue
> > > disappears.
> >
> > In no case does the kernel ever clean user space cache lines for DMA;
> > that's not the responsibility of the DMA API.
> 
> Let's forget about user space.  Even some kernel space memory is
> affected too.
> 
> The issue as I see it is that highmem pages being DMA'd to may be cached
> even when they're unmapped on VIPT machines.  And the DMA code performs
> L1 cache maintenance only on pages which are virtually mapped.
> 
> Contrary to VIVT caches which have to be flushed all the time, VIPT
> caches may avoid cache flushing in some cases, which may lead to some
> highmem pages not being mapped but still cached somehow.  But so far I
> just can't find how that could happen.
> 
> The only way a highmem page can be unmapped is through kunmap_atomic()
> where an explicit __cpuc_flush_dcache_area() is performed, or through
> flush_all_zero_pkmaps() where flush_cache_kmaps() translates into
> flush_cache_all().
> 
> So something allows for some highmem pages to escape cache flushing when
> unmapped.  But I can't find it.
> 
Or could it be that there is appropriate cacheflush happening but data gets
stuck in CPU writebuffers instead of reaching to main memory. In this case
too DMA won't see the contents and a barrier (dsb) is necessary to ensure
that write buffer is drained before DMA takes over the buffer.

Regards,
Santosh

Highmem issues with MMC filesystem

[Shilimkar, Santosh]

> -----Original Message-----
> From: linux-arm-kernel-bounces at lists.infradead.org [mailto:linux-arm-kernel-
> bounces at lists.infradead.org] On Behalf Of Nicolas Pitre
> Sent: Thursday, March 18, 2010 6:50 PM
> To: Russell King - ARM Linux
> Cc: linux-mmc at vger.kernel.org; V, Hemanth; saeed bishara; pierre at ossman.eu; linux-arm-
> kernel at lists.infradead.org
> Subject: Re: Highmem issues with MMC filesystem
> 
> On Thu, 18 Mar 2010, Russell King - ARM Linux wrote:
> 
> > On Thu, Mar 18, 2010 at 01:15:58PM +0200, saeed bishara wrote:
> > > >> The only conclusion I came to so far is that ARMv5 where highmem works
> > > >> just fine in all cases has VIVT cache whereas ARMv6 has VIPT cache.
> > > >> And the problem with VIPT caches occurs when direct DMA is involved,
> > > >> otherwise there is no problem if PIO or NFS is used. ?Sprinkling some
> > > >> flush_cache_all() in a few places makes things work, but this is not a
> > > >> satisfactory solution.
> > > >
> > > > This sounds like the problem we had with the DMA API. ?Since that's now
> > > > fixed, there shouldn't be a problem with the latest (-rc) kernels, or
> > > > a kernel with my old streaming DMA patches applied.
> > > The failure happens also on 2.6.34.rc1,  as Nico said, it looks like
> > > that buffers that are subject to DMA remain dirty, as I understand it,
> > > for vipt nonaliasing cpu's, the kernel doesn't clean user space cache
> > > lines. if I force kmap_atomic/kunmap_atomic to highmem pages that are
> > > not mapped by the kernel (kmap_high_get returns null), then the issue
> > > disappears.
> >
> > In no case does the kernel ever clean user space cache lines for DMA;
> > that's not the responsibility of the DMA API.
> 
> Let's forget about user space.  Even some kernel space memory is
> affected too.
> 
> The issue as I see it is that highmem pages being DMA'd to may be cached
> even when they're unmapped on VIPT machines.  And the DMA code performs
> L1 cache maintenance only on pages which are virtually mapped.
> 
> Contrary to VIVT caches which have to be flushed all the time, VIPT
> caches may avoid cache flushing in some cases, which may lead to some
> highmem pages not being mapped but still cached somehow.  But so far I
> just can't find how that could happen.
> 
> The only way a highmem page can be unmapped is through kunmap_atomic()
> where an explicit __cpuc_flush_dcache_area() is performed, or through
> flush_all_zero_pkmaps() where flush_cache_kmaps() translates into
> flush_cache_all().
> 
> So something allows for some highmem pages to escape cache flushing when
> unmapped.  But I can't find it.
> 
Or could it be that there is appropriate cacheflush happening but data gets
stuck in CPU writebuffers instead of reaching to main memory. In this case
too DMA won't see the contents and a barrier (dsb) is necessary to ensure
that write buffer is drained before DMA takes over the buffer.

Regards,
Santosh

Re: Highmem issues with MMC filesystem

[saeed bishara]

> In no case does the kernel ever clean user space cache lines for DMA;
> that's not the responsibility of the DMA API.
I understand that it's not the DMA API's responsibility to flush user
space cache lines, but other parts of the kernel should do that right?
otherwise DIRECT_IO won't work (though the issue here has nothing to
do with DIRECT_IO).

saeed

Highmem issues with MMC filesystem

[saeed bishara]

> In no case does the kernel ever clean user space cache lines for DMA;
> that's not the responsibility of the DMA API.
I understand that it's not the DMA API's responsibility to flush user
space cache lines, but other parts of the kernel should do that right?
otherwise DIRECT_IO won't work (though the issue here has nothing to
do with DIRECT_IO).

saeed

Re: Highmem issues with MMC filesystem

[Russell King - ARM Linux]

On Thu, Mar 18, 2010 at 07:00:06PM +0530, Shilimkar, Santosh wrote:
> Or could it be that there is appropriate cacheflush happening but data gets
> stuck in CPU writebuffers instead of reaching to main memory. In this case
> too DMA won't see the contents and a barrier (dsb) is necessary to ensure
> that write buffer is drained before DMA takes over the buffer.

That would imply that the data on the device is becoming corrupted.

What exactly is the problem that we're discussing?  Is it that the data
on the block device is becoming corrupted, or is the data being read off
the block device corrupted?

Highmem issues with MMC filesystem

[Russell King - ARM Linux]

On Thu, Mar 18, 2010 at 07:00:06PM +0530, Shilimkar, Santosh wrote:
> Or could it be that there is appropriate cacheflush happening but data gets
> stuck in CPU writebuffers instead of reaching to main memory. In this case
> too DMA won't see the contents and a barrier (dsb) is necessary to ensure
> that write buffer is drained before DMA takes over the buffer.

That would imply that the data on the device is becoming corrupted.

What exactly is the problem that we're discussing?  Is it that the data
on the block device is becoming corrupted, or is the data being read off
the block device corrupted?

Re: Highmem issues with MMC filesystem

[Hemanth V]

----- Original Message ----- 
From: "Russell King - ARM Linux" <linux@arm.linux.org.uk>
To: "Shilimkar, Santosh" <santosh.shilimkar@ti.com>
Cc: "Nicolas Pitre" <nico@fluxnic.net>; <linux-mmc@vger.kernel.org>; "V, 
Hemanth" <hemanthv@ti.com>; "saeed bishara" <saeed.bishara@gmail.com>; 
<pierre@ossman.eu>; <linux-arm-kernel@lists.infradead.org>
Sent: Thursday, March 18, 2010 7:49 PM
Subject: Re: Highmem issues with MMC filesystem


> On Thu, Mar 18, 2010 at 07:00:06PM +0530, Shilimkar, Santosh wrote:
>> Or could it be that there is appropriate cacheflush happening but data 
>> gets
>> stuck in CPU writebuffers instead of reaching to main memory. In this 
>> case
>> too DMA won't see the contents and a barrier (dsb) is necessary to ensure
>> that write buffer is drained before DMA takes over the buffer.
>
> That would imply that the data on the device is becoming corrupted.
>
> What exactly is the problem that we're discussing?  Is it that the data
> on the block device is becoming corrupted, or is the data being read off
> the block device corrupted?
>

It seems like data being read off the block device is being corrupted,
since if we replace highmem kernel with a non-highmem kernel
the filesystem is able to bootup fine. 

Highmem issues with MMC filesystem

[Hemanth V]

----- Original Message ----- 
From: "Russell King - ARM Linux" <linux@arm.linux.org.uk>
To: "Shilimkar, Santosh" <santosh.shilimkar@ti.com>
Cc: "Nicolas Pitre" <nico@fluxnic.net>; <linux-mmc@vger.kernel.org>; "V, 
Hemanth" <hemanthv@ti.com>; "saeed bishara" <saeed.bishara@gmail.com>; 
<pierre@ossman.eu>; <linux-arm-kernel@lists.infradead.org>
Sent: Thursday, March 18, 2010 7:49 PM
Subject: Re: Highmem issues with MMC filesystem


> On Thu, Mar 18, 2010 at 07:00:06PM +0530, Shilimkar, Santosh wrote:
>> Or could it be that there is appropriate cacheflush happening but data 
>> gets
>> stuck in CPU writebuffers instead of reaching to main memory. In this 
>> case
>> too DMA won't see the contents and a barrier (dsb) is necessary to ensure
>> that write buffer is drained before DMA takes over the buffer.
>
> That would imply that the data on the device is becoming corrupted.
>
> What exactly is the problem that we're discussing?  Is it that the data
> on the block device is becoming corrupted, or is the data being read off
> the block device corrupted?
>

It seems like data being read off the block device is being corrupted,
since if we replace highmem kernel with a non-highmem kernel
the filesystem is able to bootup fine. 

Re: Highmem issues with MMC filesystem

[Nicolas Pitre]

On Thu, 18 Mar 2010, Russell King - ARM Linux wrote:

> On Thu, Mar 18, 2010 at 07:00:06PM +0530, Shilimkar, Santosh wrote:
> > Or could it be that there is appropriate cacheflush happening but data gets
> > stuck in CPU writebuffers instead of reaching to main memory. In this case
> > too DMA won't see the contents and a barrier (dsb) is necessary to ensure
> > that write buffer is drained before DMA takes over the buffer.

No, this is not the issue.

> What exactly is the problem that we're discussing?  Is it that the data
> on the block device is becoming corrupted, or is the data being read off
> the block device corrupted?

Data read from a DMA based block device is corrupted in memory and 
causing all sorts of misbehavior such as segmentation faults in user 
space, or EXT2 complaining about broken filesystem metadata.  This is 
serious enough to prevent a successful boot to a login prompt.  Of 
course the filesystem is just fine as using the same kernel and limiting 
the memory size so highmem doesn't get involved "fixes" the issue.  
Also keeping highmem pages active and adding a couple flush_cache_all() 
in some places also apparently "fixes" the issue.

Given that systems with VIVT caches have a full cache flush on each task 
switch, I added such a cache flush in the ARMv6 case as well to see what 
would happen.  And this _almost_ "fixed" the issue as long as there is 
enough running tasks at once on the system to enforce that cache flush 
often enough.  But with a single task running, such as gcc, then the 
segmentation faults come back.

See my previous email for my current hypothesis about the actual 
problem.


Nicolas

Highmem issues with MMC filesystem

[Nicolas Pitre]

On Thu, 18 Mar 2010, Russell King - ARM Linux wrote:

> On Thu, Mar 18, 2010 at 07:00:06PM +0530, Shilimkar, Santosh wrote:
> > Or could it be that there is appropriate cacheflush happening but data gets
> > stuck in CPU writebuffers instead of reaching to main memory. In this case
> > too DMA won't see the contents and a barrier (dsb) is necessary to ensure
> > that write buffer is drained before DMA takes over the buffer.

No, this is not the issue.

> What exactly is the problem that we're discussing?  Is it that the data
> on the block device is becoming corrupted, or is the data being read off
> the block device corrupted?

Data read from a DMA based block device is corrupted in memory and 
causing all sorts of misbehavior such as segmentation faults in user 
space, or EXT2 complaining about broken filesystem metadata.  This is 
serious enough to prevent a successful boot to a login prompt.  Of 
course the filesystem is just fine as using the same kernel and limiting 
the memory size so highmem doesn't get involved "fixes" the issue.  
Also keeping highmem pages active and adding a couple flush_cache_all() 
in some places also apparently "fixes" the issue.

Given that systems with VIVT caches have a full cache flush on each task 
switch, I added such a cache flush in the ARMv6 case as well to see what 
would happen.  And this _almost_ "fixed" the issue as long as there is 
enough running tasks at once on the system to enforce that cache flush 
often enough.  But with a single task running, such as gcc, then the 
segmentation faults come back.

See my previous email for my current hypothesis about the actual 
problem.


Nicolas

Re: Highmem issues with MMC filesystem

[Russell King - ARM Linux]

On Thu, Mar 18, 2010 at 12:17:50PM -0400, Nicolas Pitre wrote:
> On Thu, 18 Mar 2010, Russell King - ARM Linux wrote:
> > What exactly is the problem that we're discussing?  Is it that the data
> > on the block device is becoming corrupted, or is the data being read off
> > the block device corrupted?
> 
> Data read from a DMA based block device is corrupted in memory and 
> causing all sorts of misbehavior such as segmentation faults in user 
> space, or EXT2 complaining about broken filesystem metadata.

Isn't ext2 metadata read into lowmem pages?

The ext2 bitmap code sources its backing store from the old fs buffercache
code, which can only use lowmem - so the ext2 bitmaps are in lowmem (see
set_bh_page).  The superblock comes from the buffercache (grep for sb_bread)
as well, so that's lowmem, and it looks to me like inode reading also
comes via the buffercache.

It seems that all ext2 metadata is in lowmem, so I don't get the highmem
interaction causing ext2 fs metadata corruption.

Highmem issues with MMC filesystem

[Russell King - ARM Linux]

On Thu, Mar 18, 2010 at 12:17:50PM -0400, Nicolas Pitre wrote:
> On Thu, 18 Mar 2010, Russell King - ARM Linux wrote:
> > What exactly is the problem that we're discussing?  Is it that the data
> > on the block device is becoming corrupted, or is the data being read off
> > the block device corrupted?
> 
> Data read from a DMA based block device is corrupted in memory and 
> causing all sorts of misbehavior such as segmentation faults in user 
> space, or EXT2 complaining about broken filesystem metadata.

Isn't ext2 metadata read into lowmem pages?

The ext2 bitmap code sources its backing store from the old fs buffercache
code, which can only use lowmem - so the ext2 bitmaps are in lowmem (see
set_bh_page).  The superblock comes from the buffercache (grep for sb_bread)
as well, so that's lowmem, and it looks to me like inode reading also
comes via the buffercache.

It seems that all ext2 metadata is in lowmem, so I don't get the highmem
interaction causing ext2 fs metadata corruption.

Re: Highmem issues with MMC filesystem

[Nicolas Pitre]

On Thu, 18 Mar 2010, Russell King - ARM Linux wrote:

> On Thu, Mar 18, 2010 at 12:17:50PM -0400, Nicolas Pitre wrote:
> > On Thu, 18 Mar 2010, Russell King - ARM Linux wrote:
> > > What exactly is the problem that we're discussing?  Is it that the data
> > > on the block device is becoming corrupted, or is the data being read off
> > > the block device corrupted?
> > 
> > Data read from a DMA based block device is corrupted in memory and 
> > causing all sorts of misbehavior such as segmentation faults in user 
> > space, or EXT2 complaining about broken filesystem metadata.
> 
> Isn't ext2 metadata read into lowmem pages?

Possible.  I've not looked at it closely enough to know for sure.

> The ext2 bitmap code sources its backing store from the old fs buffercache
> code, which can only use lowmem - so the ext2 bitmaps are in lowmem (see
> set_bh_page).  The superblock comes from the buffercache (grep for sb_bread)
> as well, so that's lowmem, and it looks to me like inode reading also
> comes via the buffercache.
> 
> It seems that all ext2 metadata is in lowmem, so I don't get the highmem
> interaction causing ext2 fs metadata corruption.

Well, some EXT2 errors appear randomly through the kernel console output 
and get mixed with all the avalenge of user space processes crashing due 
to strange errors or simply segfaulting during a boot.  Luckily when 
that happens the user space wasn't even able to remount the root fs 
read-write, so performing a fsck after a reboot (without highmem pages) 
reveals a sane and clean filesystem.


Nicolas

Highmem issues with MMC filesystem

[Nicolas Pitre]

On Thu, 18 Mar 2010, Russell King - ARM Linux wrote:

> On Thu, Mar 18, 2010 at 12:17:50PM -0400, Nicolas Pitre wrote:
> > On Thu, 18 Mar 2010, Russell King - ARM Linux wrote:
> > > What exactly is the problem that we're discussing?  Is it that the data
> > > on the block device is becoming corrupted, or is the data being read off
> > > the block device corrupted?
> > 
> > Data read from a DMA based block device is corrupted in memory and 
> > causing all sorts of misbehavior such as segmentation faults in user 
> > space, or EXT2 complaining about broken filesystem metadata.
> 
> Isn't ext2 metadata read into lowmem pages?

Possible.  I've not looked at it closely enough to know for sure.

> The ext2 bitmap code sources its backing store from the old fs buffercache
> code, which can only use lowmem - so the ext2 bitmaps are in lowmem (see
> set_bh_page).  The superblock comes from the buffercache (grep for sb_bread)
> as well, so that's lowmem, and it looks to me like inode reading also
> comes via the buffercache.
> 
> It seems that all ext2 metadata is in lowmem, so I don't get the highmem
> interaction causing ext2 fs metadata corruption.

Well, some EXT2 errors appear randomly through the kernel console output 
and get mixed with all the avalenge of user space processes crashing due 
to strange errors or simply segfaulting during a boot.  Luckily when 
that happens the user space wasn't even able to remount the root fs 
read-write, so performing a fsck after a reboot (without highmem pages) 
reveals a sane and clean filesystem.


Nicolas

Re: Highmem issues with MMC filesystem

[Jamie Lokier]

Nicolas Pitre wrote:
> Data read from a DMA based block device is corrupted in memory and 
> causing all sorts of misbehavior such as segmentation faults in user 
> space, or EXT2 complaining about broken filesystem metadata.

I doubt it is related, but just in case: I've seen similar on a
no-doubt completely unrelated ARM9 system on now-ancient kernels:
occasional ext3 metadata corruption.

In my case it was due to a bug in the chip's IDE driver, which did:

   1. Setup DMA address register.
   2. Appropriate cache flushes.
   3. Write to remaining DMA registers to start DMA.

It turned out step 1 causes the DMA controller to begin reading from
that address into an internal 128-byte FIFO, before officially
starting DMA.  Moving step 2 before step 1 fixed that.

-- Jamie

Highmem issues with MMC filesystem

[Jamie Lokier]

Nicolas Pitre wrote:
> Data read from a DMA based block device is corrupted in memory and 
> causing all sorts of misbehavior such as segmentation faults in user 
> space, or EXT2 complaining about broken filesystem metadata.

I doubt it is related, but just in case: I've seen similar on a
no-doubt completely unrelated ARM9 system on now-ancient kernels:
occasional ext3 metadata corruption.

In my case it was due to a bug in the chip's IDE driver, which did:

   1. Setup DMA address register.
   2. Appropriate cache flushes.
   3. Write to remaining DMA registers to start DMA.

It turned out step 1 causes the DMA controller to begin reading from
that address into an internal 128-byte FIFO, before officially
starting DMA.  Moving step 2 before step 1 fixed that.

-- Jamie

Re: Highmem issues with MMC filesystem

[Nicolas Pitre]

On Fri, 19 Mar 2010, Jamie Lokier wrote:

> Nicolas Pitre wrote:
> > Data read from a DMA based block device is corrupted in memory and 
> > causing all sorts of misbehavior such as segmentation faults in user 
> > space, or EXT2 complaining about broken filesystem metadata.
> 
> I doubt it is related, but just in case: I've seen similar on a
> no-doubt completely unrelated ARM9 system on now-ancient kernels:
> occasional ext3 metadata corruption.
> 
> In my case it was due to a bug in the chip's IDE driver, which did:
[...]

Driver bugs would tend to be visible with or without highmem in play.

And in this case, the same SATA driver is used on two different system, 
one being ARMv5 with absolutely no issues with highmem, and the other 
being ARMv6 with highmem problems.  So that pretty much rules out IDE 
driver bugs.

Furthermore the same highmem issues were reported to occur on an OMAP 
system with the filesystem on a MMC card i.e. different CPU manufacturer 
with a different block device driver.

So the issue really seems to be about ARMv5 (works) vs ARMv6 (doesn't 
work), which makes some sense given the huge difference between both 
cache models.


Nicolas

Highmem issues with MMC filesystem

[Nicolas Pitre]

On Fri, 19 Mar 2010, Jamie Lokier wrote:

> Nicolas Pitre wrote:
> > Data read from a DMA based block device is corrupted in memory and 
> > causing all sorts of misbehavior such as segmentation faults in user 
> > space, or EXT2 complaining about broken filesystem metadata.
> 
> I doubt it is related, but just in case: I've seen similar on a
> no-doubt completely unrelated ARM9 system on now-ancient kernels:
> occasional ext3 metadata corruption.
> 
> In my case it was due to a bug in the chip's IDE driver, which did:
[...]

Driver bugs would tend to be visible with or without highmem in play.

And in this case, the same SATA driver is used on two different system, 
one being ARMv5 with absolutely no issues with highmem, and the other 
being ARMv6 with highmem problems.  So that pretty much rules out IDE 
driver bugs.

Furthermore the same highmem issues were reported to occur on an OMAP 
system with the filesystem on a MMC card i.e. different CPU manufacturer 
with a different block device driver.

So the issue really seems to be about ARMv5 (works) vs ARMv6 (doesn't 
work), which makes some sense given the huge difference between both 
cache models.


Nicolas

Re: Highmem issues with MMC filesystem

[Russell King - ARM Linux]

On Fri, Mar 19, 2010 at 09:17:19AM -0400, Nicolas Pitre wrote:
> And in this case, the same SATA driver is used on two different system, 
> one being ARMv5 with absolutely no issues with highmem, and the other 
> being ARMv6 with highmem problems.  So that pretty much rules out IDE 
> driver bugs.

No it doesn't - there's more changes between ARMv5 and ARMv6 than just
the cache model.  There's weak memory ordering effects too.

A missing barrier or a badly implemented barrier can result in ARMv5
and earlier working perfectly, but ARMv6+ randomly failing.

Eg, we know that ARMv6+ CPUs with L2 cache, the wmb() implementation is
not sufficient, and DMA can be started before the write to
dma_alloc_coherent memory has become visible to the DMA controller, and
that's a topic of discussion at the present time.

Highmem issues with MMC filesystem

[Russell King - ARM Linux]

On Fri, Mar 19, 2010 at 09:17:19AM -0400, Nicolas Pitre wrote:
> And in this case, the same SATA driver is used on two different system, 
> one being ARMv5 with absolutely no issues with highmem, and the other 
> being ARMv6 with highmem problems.  So that pretty much rules out IDE 
> driver bugs.

No it doesn't - there's more changes between ARMv5 and ARMv6 than just
the cache model.  There's weak memory ordering effects too.

A missing barrier or a badly implemented barrier can result in ARMv5
and earlier working perfectly, but ARMv6+ randomly failing.

Eg, we know that ARMv6+ CPUs with L2 cache, the wmb() implementation is
not sufficient, and DMA can be started before the write to
dma_alloc_coherent memory has become visible to the DMA controller, and
that's a topic of discussion at the present time.

Re: Highmem issues with MMC filesystem

[Hemanth V]

> Nicolas Pitre wrote:
>
> Furthermore the same highmem issues were reported to occur on an OMAP
> system with the filesystem on a MMC card i.e. different CPU manufacturer
> with a different block device driver.
>
> So the issue really seems to be about ARMv5 (works) vs ARMv6 (doesn't
> work), which makes some sense given the huge difference between both
> cache models.
>

Adding a cache flush before starting MMC DMA solves the problem for me,
so dumping cache contents can give some clues to the problem. Does anyone
have access to a debugger which can dump cache contents.

Thanks
Hemanth

Highmem issues with MMC filesystem

[Hemanth V]

> Nicolas Pitre wrote:
>
> Furthermore the same highmem issues were reported to occur on an OMAP
> system with the filesystem on a MMC card i.e. different CPU manufacturer
> with a different block device driver.
>
> So the issue really seems to be about ARMv5 (works) vs ARMv6 (doesn't
> work), which makes some sense given the huge difference between both
> cache models.
>

Adding a cache flush before starting MMC DMA solves the problem for me,
so dumping cache contents can give some clues to the problem. Does anyone
have access to a debugger which can dump cache contents.

Thanks
Hemanth

RE: Highmem issues with MMC filesystem

[Shilimkar, Santosh]

> -----Original Message-----
> From: V, Hemanth
> Sent: Friday, March 19, 2010 7:59 PM
> To: Nicolas Pitre
> Cc: Jamie Lokier; Russell King - ARM Linux; linux-mmc@vger.kernel.org; Shilimkar, Santosh;
> pierre@ossman.eu; saeed bishara; linux-arm-kernel@lists.infradead.org
> Subject: Re: Highmem issues with MMC filesystem
> 
> > Nicolas Pitre wrote:
> >
> > Furthermore the same highmem issues were reported to occur on an OMAP
> > system with the filesystem on a MMC card i.e. different CPU manufacturer
> > with a different block device driver.
> >
> > So the issue really seems to be about ARMv5 (works) vs ARMv6 (doesn't
> > work), which makes some sense given the huge difference between both
> > cache models.
> >
> 
> Adding a cache flush before starting MMC DMA solves the problem for me,
> so dumping cache contents can give some clues to the problem. Does anyone
> have access to a debugger which can dump cache contents.
> 
Since the cache_flush is helping and we are talking about DMA_FROM_DEVICE case,
Russells recent DMA mapping series does have fix for this.

Do you have Russell's DMA mapping series integrated in your kernel you are trying ?
It's already in the 2.6.34-rc1

Regards,
Santosh

Highmem issues with MMC filesystem

[Shilimkar, Santosh]

> -----Original Message-----
> From: V, Hemanth
> Sent: Friday, March 19, 2010 7:59 PM
> To: Nicolas Pitre
> Cc: Jamie Lokier; Russell King - ARM Linux; linux-mmc at vger.kernel.org; Shilimkar, Santosh;
> pierre at ossman.eu; saeed bishara; linux-arm-kernel at lists.infradead.org
> Subject: Re: Highmem issues with MMC filesystem
> 
> > Nicolas Pitre wrote:
> >
> > Furthermore the same highmem issues were reported to occur on an OMAP
> > system with the filesystem on a MMC card i.e. different CPU manufacturer
> > with a different block device driver.
> >
> > So the issue really seems to be about ARMv5 (works) vs ARMv6 (doesn't
> > work), which makes some sense given the huge difference between both
> > cache models.
> >
> 
> Adding a cache flush before starting MMC DMA solves the problem for me,
> so dumping cache contents can give some clues to the problem. Does anyone
> have access to a debugger which can dump cache contents.
> 
Since the cache_flush is helping and we are talking about DMA_FROM_DEVICE case,
Russells recent DMA mapping series does have fix for this.

Do you have Russell's DMA mapping series integrated in your kernel you are trying ?
It's already in the 2.6.34-rc1

Regards,
Santosh

Re: Highmem issues with MMC filesystem

[Catalin Marinas]

On Thu, 2010-03-18 at 13:20 +0000, Nicolas Pitre wrote:
> The only way a highmem page can be unmapped is through kunmap_atomic()
> where an explicit __cpuc_flush_dcache_area() is performed, or through
> flush_all_zero_pkmaps() where flush_cache_kmaps() translates into
> flush_cache_all().

The thing that I couldn't fully understand with the kunmap_atomic()
function is that there is a path (when kvaddr < FIXADDR_START) where no
cache flushing occurs. Can this not happen?

-- 
Catalin

Highmem issues with MMC filesystem

[Catalin Marinas]

On Thu, 2010-03-18 at 13:20 +0000, Nicolas Pitre wrote:
> The only way a highmem page can be unmapped is through kunmap_atomic()
> where an explicit __cpuc_flush_dcache_area() is performed, or through
> flush_all_zero_pkmaps() where flush_cache_kmaps() translates into
> flush_cache_all().

The thing that I couldn't fully understand with the kunmap_atomic()
function is that there is a path (when kvaddr < FIXADDR_START) where no
cache flushing occurs. Can this not happen?

-- 
Catalin

Re: Highmem issues with MMC filesystem

[Russell King - ARM Linux]

On Fri, Mar 19, 2010 at 08:06:12PM +0530, Shilimkar, Santosh wrote:
> Since the cache_flush is helping and we are talking about DMA_FROM_DEVICE
> case, Russells recent DMA mapping series does have fix for this.
> 
> Do you have Russell's DMA mapping series integrated in your kernel you are
> trying ? It's already in the 2.6.34-rc1

I wonder whether we can get _one_ email from each person who has
encountered the problem giving full details of what kernels they're
running and the problem is that they've experienced.

At the moment, that information is spread throughout this thread,
with some clues from one person and others from another person -
there's no way to tie this all together, and I certainly can't follow
it.

So, let's end this thread here (I'm going to start ignoring it), and
start a new one with some *proper* and *full* bug reports from *each*
person with a problem.

Highmem issues with MMC filesystem

[Russell King - ARM Linux]

On Fri, Mar 19, 2010 at 08:06:12PM +0530, Shilimkar, Santosh wrote:
> Since the cache_flush is helping and we are talking about DMA_FROM_DEVICE
> case, Russells recent DMA mapping series does have fix for this.
> 
> Do you have Russell's DMA mapping series integrated in your kernel you are
> trying ? It's already in the 2.6.34-rc1

I wonder whether we can get _one_ email from each person who has
encountered the problem giving full details of what kernels they're
running and the problem is that they've experienced.

At the moment, that information is spread throughout this thread,
with some clues from one person and others from another person -
there's no way to tie this all together, and I certainly can't follow
it.

So, let's end this thread here (I'm going to start ignoring it), and
start a new one with some *proper* and *full* bug reports from *each*
person with a problem.

Re: Highmem issues with MMC filesystem

[Russell King - ARM Linux]

On Fri, Mar 19, 2010 at 02:41:17PM +0000, Catalin Marinas wrote:
> On Thu, 2010-03-18 at 13:20 +0000, Nicolas Pitre wrote:
> > The only way a highmem page can be unmapped is through kunmap_atomic()
> > where an explicit __cpuc_flush_dcache_area() is performed, or through
> > flush_all_zero_pkmaps() where flush_cache_kmaps() translates into
> > flush_cache_all().
> 
> The thing that I couldn't fully understand with the kunmap_atomic()
> function is that there is a path (when kvaddr < FIXADDR_START) where no
> cache flushing occurs. Can this not happen?

kunmap interfaces are not for cache flushing; the cache flushing is
only there to ensure consistency when unmapping a mapping on VIVT CPUs.

If VIVT CPUs could do writebacks without a valid mapping in place,
then the cache flush for highmem pages would not be required.

Highmem issues with MMC filesystem

[Russell King - ARM Linux]

On Fri, Mar 19, 2010 at 02:41:17PM +0000, Catalin Marinas wrote:
> On Thu, 2010-03-18 at 13:20 +0000, Nicolas Pitre wrote:
> > The only way a highmem page can be unmapped is through kunmap_atomic()
> > where an explicit __cpuc_flush_dcache_area() is performed, or through
> > flush_all_zero_pkmaps() where flush_cache_kmaps() translates into
> > flush_cache_all().
> 
> The thing that I couldn't fully understand with the kunmap_atomic()
> function is that there is a path (when kvaddr < FIXADDR_START) where no
> cache flushing occurs. Can this not happen?

kunmap interfaces are not for cache flushing; the cache flushing is
only there to ensure consistency when unmapping a mapping on VIVT CPUs.

If VIVT CPUs could do writebacks without a valid mapping in place,
then the cache flush for highmem pages would not be required.

Re: Highmem issues with MMC filesystem

[Catalin Marinas]

> On Fri, Mar 19, 2010 at 02:41:17PM +0000, Catalin Marinas wrote:
> > On Thu, 2010-03-18 at 13:20 +0000, Nicolas Pitre wrote:
> > > The only way a highmem page can be unmapped is through kunmap_atomic()
> > > where an explicit __cpuc_flush_dcache_area() is performed, or through
> > > flush_all_zero_pkmaps() where flush_cache_kmaps() translates into
> > > flush_cache_all().
> >
> > The thing that I couldn't fully understand with the kunmap_atomic()
> > function is that there is a path (when kvaddr < FIXADDR_START) where no
> > cache flushing occurs. Can this not happen?
> 
> kunmap interfaces are not for cache flushing; the cache flushing is
> only there to ensure consistency when unmapping a mapping on VIVT CPUs.

I agree, but then why don't we conditionally call
__cpuc_flush_dcache_area() in kunmap_atomic() so that we avoid this
flush on non-aliasing VIPT?

-- 
Catalin

Highmem issues with MMC filesystem

[Catalin Marinas]

> On Fri, Mar 19, 2010 at 02:41:17PM +0000, Catalin Marinas wrote:
> > On Thu, 2010-03-18 at 13:20 +0000, Nicolas Pitre wrote:
> > > The only way a highmem page can be unmapped is through kunmap_atomic()
> > > where an explicit __cpuc_flush_dcache_area() is performed, or through
> > > flush_all_zero_pkmaps() where flush_cache_kmaps() translates into
> > > flush_cache_all().
> >
> > The thing that I couldn't fully understand with the kunmap_atomic()
> > function is that there is a path (when kvaddr < FIXADDR_START) where no
> > cache flushing occurs. Can this not happen?
> 
> kunmap interfaces are not for cache flushing; the cache flushing is
> only there to ensure consistency when unmapping a mapping on VIVT CPUs.

I agree, but then why don't we conditionally call
__cpuc_flush_dcache_area() in kunmap_atomic() so that we avoid this
flush on non-aliasing VIPT?

-- 
Catalin

Re: Highmem issues with MMC filesystem

[Russell King - ARM Linux]

On Fri, Mar 19, 2010 at 02:52:00PM +0000, Catalin Marinas wrote:
> > On Fri, Mar 19, 2010 at 02:41:17PM +0000, Catalin Marinas wrote:
> > > On Thu, 2010-03-18 at 13:20 +0000, Nicolas Pitre wrote:
> > > > The only way a highmem page can be unmapped is through kunmap_atomic()
> > > > where an explicit __cpuc_flush_dcache_area() is performed, or through
> > > > flush_all_zero_pkmaps() where flush_cache_kmaps() translates into
> > > > flush_cache_all().
> > >
> > > The thing that I couldn't fully understand with the kunmap_atomic()
> > > function is that there is a path (when kvaddr < FIXADDR_START) where no
> > > cache flushing occurs. Can this not happen?
> > 
> > kunmap interfaces are not for cache flushing; the cache flushing is
> > only there to ensure consistency when unmapping a mapping on VIVT CPUs.
> 
> I agree, but then why don't we conditionally call
> __cpuc_flush_dcache_area() in kunmap_atomic() so that we avoid this
> flush on non-aliasing VIPT?

Probably because highmem was written at the time for VIVT CPUs.  I'm sure
Nicolas will accept patches to improve performance of highmem for VIPT.

Highmem issues with MMC filesystem

[Russell King - ARM Linux]

On Fri, Mar 19, 2010 at 02:52:00PM +0000, Catalin Marinas wrote:
> > On Fri, Mar 19, 2010 at 02:41:17PM +0000, Catalin Marinas wrote:
> > > On Thu, 2010-03-18 at 13:20 +0000, Nicolas Pitre wrote:
> > > > The only way a highmem page can be unmapped is through kunmap_atomic()
> > > > where an explicit __cpuc_flush_dcache_area() is performed, or through
> > > > flush_all_zero_pkmaps() where flush_cache_kmaps() translates into
> > > > flush_cache_all().
> > >
> > > The thing that I couldn't fully understand with the kunmap_atomic()
> > > function is that there is a path (when kvaddr < FIXADDR_START) where no
> > > cache flushing occurs. Can this not happen?
> > 
> > kunmap interfaces are not for cache flushing; the cache flushing is
> > only there to ensure consistency when unmapping a mapping on VIVT CPUs.
> 
> I agree, but then why don't we conditionally call
> __cpuc_flush_dcache_area() in kunmap_atomic() so that we avoid this
> flush on non-aliasing VIPT?

Probably because highmem was written at the time for VIVT CPUs.  I'm sure
Nicolas will accept patches to improve performance of highmem for VIPT.

Re: Highmem issues with MMC filesystem

[Catalin Marinas]

On Fri, 2010-03-19 at 14:54 +0000, Russell King - ARM Linux wrote:
> On Fri, Mar 19, 2010 at 02:52:00PM +0000, Catalin Marinas wrote:
> > > On Fri, Mar 19, 2010 at 02:41:17PM +0000, Catalin Marinas wrote:
> > > > On Thu, 2010-03-18 at 13:20 +0000, Nicolas Pitre wrote:
> > > > > The only way a highmem page can be unmapped is through kunmap_atomic()
> > > > > where an explicit __cpuc_flush_dcache_area() is performed, or through
> > > > > flush_all_zero_pkmaps() where flush_cache_kmaps() translates into
> > > > > flush_cache_all().
> > > >
> > > > The thing that I couldn't fully understand with the kunmap_atomic()
> > > > function is that there is a path (when kvaddr < FIXADDR_START) where no
> > > > cache flushing occurs. Can this not happen?
> > >
> > > kunmap interfaces are not for cache flushing; the cache flushing is
> > > only there to ensure consistency when unmapping a mapping on VIVT CPUs.
> >
> > I agree, but then why don't we conditionally call
> > __cpuc_flush_dcache_area() in kunmap_atomic() so that we avoid this
> > flush on non-aliasing VIPT?
> 
> Probably because highmem was written at the time for VIVT CPUs.  I'm sure
> Nicolas will accept patches to improve performance of highmem for VIPT.

See below for the VIPT case. But my initial question still remains for
VIVT caches - are all the cases covered?


ARM: Do not flush the cache in kunmap_atomic if non-aliasing VIPT

From: Catalin Marinas <catalin.marinas@arm.com>

If the cache is non-aliasing VIPT, there is no need to flush the D-cache
during kunmap_atomic().

Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
Cc: Nicolas Pitre <nico@fluxnic.net>
---
 arch/arm/mm/highmem.c |    3 ++-
 1 files changed, 2 insertions(+), 1 deletions(-)

diff --git a/arch/arm/mm/highmem.c b/arch/arm/mm/highmem.c
index 2be1ec7..51c01a9 100644
--- a/arch/arm/mm/highmem.c
+++ b/arch/arm/mm/highmem.c
@@ -79,7 +79,8 @@ void kunmap_atomic(void *kvaddr, enum km_type type)
 	unsigned int idx = type + KM_TYPE_NR * smp_processor_id();
 
 	if (kvaddr >= (void *)FIXADDR_START) {
-		__cpuc_flush_dcache_area((void *)vaddr, PAGE_SIZE);
+		if (!cache_is_vipt_nonaliasing())
+			__cpuc_flush_dcache_area((void *)vaddr, PAGE_SIZE);
 #ifdef CONFIG_DEBUG_HIGHMEM
 		BUG_ON(vaddr != __fix_to_virt(FIX_KMAP_BEGIN + idx));
 		set_pte_ext(TOP_PTE(vaddr), __pte(0), 0);


-- 
Catalin

Highmem issues with MMC filesystem

[Catalin Marinas]

On Fri, 2010-03-19 at 14:54 +0000, Russell King - ARM Linux wrote:
> On Fri, Mar 19, 2010 at 02:52:00PM +0000, Catalin Marinas wrote:
> > > On Fri, Mar 19, 2010 at 02:41:17PM +0000, Catalin Marinas wrote:
> > > > On Thu, 2010-03-18 at 13:20 +0000, Nicolas Pitre wrote:
> > > > > The only way a highmem page can be unmapped is through kunmap_atomic()
> > > > > where an explicit __cpuc_flush_dcache_area() is performed, or through
> > > > > flush_all_zero_pkmaps() where flush_cache_kmaps() translates into
> > > > > flush_cache_all().
> > > >
> > > > The thing that I couldn't fully understand with the kunmap_atomic()
> > > > function is that there is a path (when kvaddr < FIXADDR_START) where no
> > > > cache flushing occurs. Can this not happen?
> > >
> > > kunmap interfaces are not for cache flushing; the cache flushing is
> > > only there to ensure consistency when unmapping a mapping on VIVT CPUs.
> >
> > I agree, but then why don't we conditionally call
> > __cpuc_flush_dcache_area() in kunmap_atomic() so that we avoid this
> > flush on non-aliasing VIPT?
> 
> Probably because highmem was written at the time for VIVT CPUs.  I'm sure
> Nicolas will accept patches to improve performance of highmem for VIPT.

See below for the VIPT case. But my initial question still remains for
VIVT caches - are all the cases covered?


ARM: Do not flush the cache in kunmap_atomic if non-aliasing VIPT

From: Catalin Marinas <catalin.marinas@arm.com>

If the cache is non-aliasing VIPT, there is no need to flush the D-cache
during kunmap_atomic().

Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
Cc: Nicolas Pitre <nico@fluxnic.net>
---
 arch/arm/mm/highmem.c |    3 ++-
 1 files changed, 2 insertions(+), 1 deletions(-)

diff --git a/arch/arm/mm/highmem.c b/arch/arm/mm/highmem.c
index 2be1ec7..51c01a9 100644
--- a/arch/arm/mm/highmem.c
+++ b/arch/arm/mm/highmem.c
@@ -79,7 +79,8 @@ void kunmap_atomic(void *kvaddr, enum km_type type)
 	unsigned int idx = type + KM_TYPE_NR * smp_processor_id();
 
 	if (kvaddr >= (void *)FIXADDR_START) {
-		__cpuc_flush_dcache_area((void *)vaddr, PAGE_SIZE);
+		if (!cache_is_vipt_nonaliasing())
+			__cpuc_flush_dcache_area((void *)vaddr, PAGE_SIZE);
 #ifdef CONFIG_DEBUG_HIGHMEM
 		BUG_ON(vaddr != __fix_to_virt(FIX_KMAP_BEGIN + idx));
 		set_pte_ext(TOP_PTE(vaddr), __pte(0), 0);


-- 
Catalin

Re: Highmem issues with MMC filesystem

[Russell King - ARM Linux]

On Fri, Mar 19, 2010 at 04:46:54PM +0000, Catalin Marinas wrote:
> On Fri, 2010-03-19 at 14:54 +0000, Russell King - ARM Linux wrote:
> > On Fri, Mar 19, 2010 at 02:52:00PM +0000, Catalin Marinas wrote:
> > > > On Fri, Mar 19, 2010 at 02:41:17PM +0000, Catalin Marinas wrote:
> > > > > On Thu, 2010-03-18 at 13:20 +0000, Nicolas Pitre wrote:
> > > > > > The only way a highmem page can be unmapped is through kunmap_atomic()
> > > > > > where an explicit __cpuc_flush_dcache_area() is performed, or through
> > > > > > flush_all_zero_pkmaps() where flush_cache_kmaps() translates into
> > > > > > flush_cache_all().
> > > > >
> > > > > The thing that I couldn't fully understand with the kunmap_atomic()
> > > > > function is that there is a path (when kvaddr < FIXADDR_START) where no
> > > > > cache flushing occurs. Can this not happen?
> > > >
> > > > kunmap interfaces are not for cache flushing; the cache flushing is
> > > > only there to ensure consistency when unmapping a mapping on VIVT CPUs.
> > >
> > > I agree, but then why don't we conditionally call
> > > __cpuc_flush_dcache_area() in kunmap_atomic() so that we avoid this
> > > flush on non-aliasing VIPT?
> > 
> > Probably because highmem was written at the time for VIVT CPUs.  I'm sure
> > Nicolas will accept patches to improve performance of highmem for VIPT.
> 
> See below for the VIPT case. But my initial question still remains for
> VIVT caches - are all the cases covered?

On non-highmem, no cache flushing is expected on kunmap_atomic; it
becomes (almost) a no-op.

As I've already said, the _only_ reason for flushing the caches in
kunmap_atomic() is to ensure consistency when removing the mapping.

Highmem issues with MMC filesystem

[Russell King - ARM Linux]

On Fri, Mar 19, 2010 at 04:46:54PM +0000, Catalin Marinas wrote:
> On Fri, 2010-03-19 at 14:54 +0000, Russell King - ARM Linux wrote:
> > On Fri, Mar 19, 2010 at 02:52:00PM +0000, Catalin Marinas wrote:
> > > > On Fri, Mar 19, 2010 at 02:41:17PM +0000, Catalin Marinas wrote:
> > > > > On Thu, 2010-03-18 at 13:20 +0000, Nicolas Pitre wrote:
> > > > > > The only way a highmem page can be unmapped is through kunmap_atomic()
> > > > > > where an explicit __cpuc_flush_dcache_area() is performed, or through
> > > > > > flush_all_zero_pkmaps() where flush_cache_kmaps() translates into
> > > > > > flush_cache_all().
> > > > >
> > > > > The thing that I couldn't fully understand with the kunmap_atomic()
> > > > > function is that there is a path (when kvaddr < FIXADDR_START) where no
> > > > > cache flushing occurs. Can this not happen?
> > > >
> > > > kunmap interfaces are not for cache flushing; the cache flushing is
> > > > only there to ensure consistency when unmapping a mapping on VIVT CPUs.
> > >
> > > I agree, but then why don't we conditionally call
> > > __cpuc_flush_dcache_area() in kunmap_atomic() so that we avoid this
> > > flush on non-aliasing VIPT?
> > 
> > Probably because highmem was written at the time for VIVT CPUs.  I'm sure
> > Nicolas will accept patches to improve performance of highmem for VIPT.
> 
> See below for the VIPT case. But my initial question still remains for
> VIVT caches - are all the cases covered?

On non-highmem, no cache flushing is expected on kunmap_atomic; it
becomes (almost) a no-op.

As I've already said, the _only_ reason for flushing the caches in
kunmap_atomic() is to ensure consistency when removing the mapping.

Re: Highmem issues with MMC filesystem

[Nicolas Pitre]

On Fri, 19 Mar 2010, Catalin Marinas wrote:

> > On Fri, Mar 19, 2010 at 02:41:17PM +0000, Catalin Marinas wrote:
> > > On Thu, 2010-03-18 at 13:20 +0000, Nicolas Pitre wrote:
> > > > The only way a highmem page can be unmapped is through kunmap_atomic()
> > > > where an explicit __cpuc_flush_dcache_area() is performed, or through
> > > > flush_all_zero_pkmaps() where flush_cache_kmaps() translates into
> > > > flush_cache_all().
> > >
> > > The thing that I couldn't fully understand with the kunmap_atomic()
> > > function is that there is a path (when kvaddr < FIXADDR_START) where no
> > > cache flushing occurs. Can this not happen?
> > 
> > kunmap interfaces are not for cache flushing; the cache flushing is
> > only there to ensure consistency when unmapping a mapping on VIVT CPUs.
> 
> I agree, but then why don't we conditionally call
> __cpuc_flush_dcache_area() in kunmap_atomic() so that we avoid this
> flush on non-aliasing VIPT?

We should indeed.


Nicolas

Highmem issues with MMC filesystem

[Nicolas Pitre]

On Fri, 19 Mar 2010, Catalin Marinas wrote:

> > On Fri, Mar 19, 2010 at 02:41:17PM +0000, Catalin Marinas wrote:
> > > On Thu, 2010-03-18 at 13:20 +0000, Nicolas Pitre wrote:
> > > > The only way a highmem page can be unmapped is through kunmap_atomic()
> > > > where an explicit __cpuc_flush_dcache_area() is performed, or through
> > > > flush_all_zero_pkmaps() where flush_cache_kmaps() translates into
> > > > flush_cache_all().
> > >
> > > The thing that I couldn't fully understand with the kunmap_atomic()
> > > function is that there is a path (when kvaddr < FIXADDR_START) where no
> > > cache flushing occurs. Can this not happen?
> > 
> > kunmap interfaces are not for cache flushing; the cache flushing is
> > only there to ensure consistency when unmapping a mapping on VIVT CPUs.
> 
> I agree, but then why don't we conditionally call
> __cpuc_flush_dcache_area() in kunmap_atomic() so that we avoid this
> flush on non-aliasing VIPT?

We should indeed.


Nicolas

Re: Highmem issues with MMC filesystem

[Nicolas Pitre]

On Fri, 19 Mar 2010, Russell King - ARM Linux wrote:

> On Fri, Mar 19, 2010 at 09:17:19AM -0400, Nicolas Pitre wrote:
> > And in this case, the same SATA driver is used on two different system, 
> > one being ARMv5 with absolutely no issues with highmem, and the other 
> > being ARMv6 with highmem problems.  So that pretty much rules out IDE 
> > driver bugs.
> 
> No it doesn't - there's more changes between ARMv5 and ARMv6 than just
> the cache model.  There's weak memory ordering effects too.

Sure.  But at the moment we have:

 - ARMv5 without highmem -> works

 - ARMv5 with highmem -> works

 - ARMv6 without highmem -> works

 - ARMv6 with highmem -> fails

In all four cases the SATA driver, the hard disk and the filesystem on 
it are identical.  In the two ARMv5 cases the system and the kernel are 
identical.  Ditto for the two ARMv6 cases.

And the kernel used is the very latest i.e. v2.6.34-rc1-1642-gfc7f99c.


Nicolas

Highmem issues with MMC filesystem

[Nicolas Pitre]

On Fri, 19 Mar 2010, Russell King - ARM Linux wrote:

> On Fri, Mar 19, 2010 at 09:17:19AM -0400, Nicolas Pitre wrote:
> > And in this case, the same SATA driver is used on two different system, 
> > one being ARMv5 with absolutely no issues with highmem, and the other 
> > being ARMv6 with highmem problems.  So that pretty much rules out IDE 
> > driver bugs.
> 
> No it doesn't - there's more changes between ARMv5 and ARMv6 than just
> the cache model.  There's weak memory ordering effects too.

Sure.  But at the moment we have:

 - ARMv5 without highmem -> works

 - ARMv5 with highmem -> works

 - ARMv6 without highmem -> works

 - ARMv6 with highmem -> fails

In all four cases the SATA driver, the hard disk and the filesystem on 
it are identical.  In the two ARMv5 cases the system and the kernel are 
identical.  Ditto for the two ARMv6 cases.

And the kernel used is the very latest i.e. v2.6.34-rc1-1642-gfc7f99c.


Nicolas

Re: Highmem issues with MMC filesystem

[Catalin Marinas]

On Fri, 2010-03-19 at 17:59 +0000, Nicolas Pitre wrote:
> On Fri, 19 Mar 2010, Russell King - ARM Linux wrote:
> 
> > On Fri, Mar 19, 2010 at 09:17:19AM -0400, Nicolas Pitre wrote:
> > > And in this case, the same SATA driver is used on two different system,
> > > one being ARMv5 with absolutely no issues with highmem, and the other
> > > being ARMv6 with highmem problems.  So that pretty much rules out IDE
> > > driver bugs.
> >
> > No it doesn't - there's more changes between ARMv5 and ARMv6 than just
> > the cache model.  There's weak memory ordering effects too.
> 
> Sure.  But at the moment we have:
> 
>  - ARMv5 without highmem -> works
> 
>  - ARMv5 with highmem -> works
> 
>  - ARMv6 without highmem -> works
> 
>  - ARMv6 with highmem -> fails
> 
> In all four cases the SATA driver, the hard disk and the filesystem on
> it are identical.  In the two ARMv5 cases the system and the kernel are
> identical.  Ditto for the two ARMv6 cases.

Which ARMv6 processor is this?

Is the SATA driver using DMA (I think I read this in the thread but it
started as MMC issues and got confused)?

Thanks.

-- 
Catalin

Highmem issues with MMC filesystem

[Catalin Marinas]

On Fri, 2010-03-19 at 17:59 +0000, Nicolas Pitre wrote:
> On Fri, 19 Mar 2010, Russell King - ARM Linux wrote:
> 
> > On Fri, Mar 19, 2010 at 09:17:19AM -0400, Nicolas Pitre wrote:
> > > And in this case, the same SATA driver is used on two different system,
> > > one being ARMv5 with absolutely no issues with highmem, and the other
> > > being ARMv6 with highmem problems.  So that pretty much rules out IDE
> > > driver bugs.
> >
> > No it doesn't - there's more changes between ARMv5 and ARMv6 than just
> > the cache model.  There's weak memory ordering effects too.
> 
> Sure.  But at the moment we have:
> 
>  - ARMv5 without highmem -> works
> 
>  - ARMv5 with highmem -> works
> 
>  - ARMv6 without highmem -> works
> 
>  - ARMv6 with highmem -> fails
> 
> In all four cases the SATA driver, the hard disk and the filesystem on
> it are identical.  In the two ARMv5 cases the system and the kernel are
> identical.  Ditto for the two ARMv6 cases.

Which ARMv6 processor is this?

Is the SATA driver using DMA (I think I read this in the thread but it
started as MMC issues and got confused)?

Thanks.

-- 
Catalin

Re: Highmem issues with MMC filesystem

[Nicolas Pitre]

On Fri, 19 Mar 2010, Nicolas Pitre wrote:

> On Fri, 19 Mar 2010, Catalin Marinas wrote:
> 
> > > On Fri, Mar 19, 2010 at 02:41:17PM +0000, Catalin Marinas wrote:
> > > > On Thu, 2010-03-18 at 13:20 +0000, Nicolas Pitre wrote:
> > > > > The only way a highmem page can be unmapped is through kunmap_atomic()
> > > > > where an explicit __cpuc_flush_dcache_area() is performed, or through
> > > > > flush_all_zero_pkmaps() where flush_cache_kmaps() translates into
> > > > > flush_cache_all().
> > > >
> > > > The thing that I couldn't fully understand with the kunmap_atomic()
> > > > function is that there is a path (when kvaddr < FIXADDR_START) where no
> > > > cache flushing occurs. Can this not happen?
> > > 
> > > kunmap interfaces are not for cache flushing; the cache flushing is
> > > only there to ensure consistency when unmapping a mapping on VIVT CPUs.
> > 
> > I agree, but then why don't we conditionally call
> > __cpuc_flush_dcache_area() in kunmap_atomic() so that we avoid this
> > flush on non-aliasing VIPT?
> 
> We should indeed.

Wait...  This is actually going to make the issue even worse.

Not that this isn't a good idea, but here's how the system works at the 
moment with highmem.

A highmem page can have 2 states: virtually mapped in the pkmap area, or 
not mapped at all.  When it is mapped then page_address() returns a 
valid virtual address for it.  In that case the cache for that mapping 
can be valid, even dirty.  So the DMA API will perform cache handling 
before/after the DMA operation.

However, before the page is unmapped, the VIVT cache has to be flushed 
for that page.  This is why the DMA code currently doesn't bother doing 
any L1 cache handling when a highmem page is not mapped -- the cache 
just can't refer to such a page.

But on ARMv6 this is different.  The L1 cache is VIPT and it therefore 
doesn't have to be flushed as often as a VIVT cache.  Still, as far as I 
know, the highmem code currently always flush any page to be unmapped.  
But somewhere somehow an unmapped highmem page becomes subject to DMA 
and apparently can still be L1 cached.  But the DMA code doesn't flush 
its cache due to the page not being mapped and then problems occur.

Two solutions:

1) We find the remaining places where highmem pages may get unmapped and 
   make sure the cache for them is always flushed at that point.  This 
   is most likely when some highmem pages are removed from a user space 
   process or the like.

2) We stop flushing the cache for highmem pages when they get unmapped 
   on VIPT systems. This includes kunmap_atomic() and flush_cache_kmaps().
   However that means that we need a way to flush the cache for unmapped
   highmem pages, preferably using physical addresses since by vertue of 
   not being mapped they don't have any kernel virtual address attached 
   to them (is that possible?).


Nicolas

Highmem issues with MMC filesystem

[Nicolas Pitre]

On Fri, 19 Mar 2010, Nicolas Pitre wrote:

> On Fri, 19 Mar 2010, Catalin Marinas wrote:
> 
> > > On Fri, Mar 19, 2010 at 02:41:17PM +0000, Catalin Marinas wrote:
> > > > On Thu, 2010-03-18 at 13:20 +0000, Nicolas Pitre wrote:
> > > > > The only way a highmem page can be unmapped is through kunmap_atomic()
> > > > > where an explicit __cpuc_flush_dcache_area() is performed, or through
> > > > > flush_all_zero_pkmaps() where flush_cache_kmaps() translates into
> > > > > flush_cache_all().
> > > >
> > > > The thing that I couldn't fully understand with the kunmap_atomic()
> > > > function is that there is a path (when kvaddr < FIXADDR_START) where no
> > > > cache flushing occurs. Can this not happen?
> > > 
> > > kunmap interfaces are not for cache flushing; the cache flushing is
> > > only there to ensure consistency when unmapping a mapping on VIVT CPUs.
> > 
> > I agree, but then why don't we conditionally call
> > __cpuc_flush_dcache_area() in kunmap_atomic() so that we avoid this
> > flush on non-aliasing VIPT?
> 
> We should indeed.

Wait...  This is actually going to make the issue even worse.

Not that this isn't a good idea, but here's how the system works at the 
moment with highmem.

A highmem page can have 2 states: virtually mapped in the pkmap area, or 
not mapped at all.  When it is mapped then page_address() returns a 
valid virtual address for it.  In that case the cache for that mapping 
can be valid, even dirty.  So the DMA API will perform cache handling 
before/after the DMA operation.

However, before the page is unmapped, the VIVT cache has to be flushed 
for that page.  This is why the DMA code currently doesn't bother doing 
any L1 cache handling when a highmem page is not mapped -- the cache 
just can't refer to such a page.

But on ARMv6 this is different.  The L1 cache is VIPT and it therefore 
doesn't have to be flushed as often as a VIVT cache.  Still, as far as I 
know, the highmem code currently always flush any page to be unmapped.  
But somewhere somehow an unmapped highmem page becomes subject to DMA 
and apparently can still be L1 cached.  But the DMA code doesn't flush 
its cache due to the page not being mapped and then problems occur.

Two solutions:

1) We find the remaining places where highmem pages may get unmapped and 
   make sure the cache for them is always flushed at that point.  This 
   is most likely when some highmem pages are removed from a user space 
   process or the like.

2) We stop flushing the cache for highmem pages when they get unmapped 
   on VIPT systems. This includes kunmap_atomic() and flush_cache_kmaps().
   However that means that we need a way to flush the cache for unmapped
   highmem pages, preferably using physical addresses since by vertue of 
   not being mapped they don't have any kernel virtual address attached 
   to them (is that possible?).


Nicolas

Re: Highmem issues with MMC filesystem

[Russell King - ARM Linux]

On Fri, Mar 19, 2010 at 02:27:24PM -0400, Nicolas Pitre wrote:
> But on ARMv6 this is different.  The L1 cache is VIPT and it therefore 
> doesn't have to be flushed as often as a VIVT cache.  Still, as far as I 
> know, the highmem code currently always flush any page to be unmapped.  
> But somewhere somehow an unmapped highmem page becomes subject to DMA 
> and apparently can still be L1 cached.

This does not explain the corrupted ext2 metadata, which is in lowmem
pages.

Let's get a fresh set of clear bug reports giving all relevant information.
Let's not continue to scatter small little bits of information in multiple
emails.

>From now on, I'm ignoring this thread completely; it's a waste of time
to continue with it.

Highmem issues with MMC filesystem

[Russell King - ARM Linux]

On Fri, Mar 19, 2010 at 02:27:24PM -0400, Nicolas Pitre wrote:
> But on ARMv6 this is different.  The L1 cache is VIPT and it therefore 
> doesn't have to be flushed as often as a VIVT cache.  Still, as far as I 
> know, the highmem code currently always flush any page to be unmapped.  
> But somewhere somehow an unmapped highmem page becomes subject to DMA 
> and apparently can still be L1 cached.

This does not explain the corrupted ext2 metadata, which is in lowmem
pages.

Let's get a fresh set of clear bug reports giving all relevant information.
Let's not continue to scatter small little bits of information in multiple
emails.

Re: Highmem issues with MMC filesystem

[Nicolas Pitre]

On Fri, 19 Mar 2010, Russell King - ARM Linux wrote:

> On Fri, Mar 19, 2010 at 02:27:24PM -0400, Nicolas Pitre wrote:
> > But on ARMv6 this is different.  The L1 cache is VIPT and it therefore 
> > doesn't have to be flushed as often as a VIVT cache.  Still, as far as I 
> > know, the highmem code currently always flush any page to be unmapped.  
> > But somewhere somehow an unmapped highmem page becomes subject to DMA 
> > and apparently can still be L1 cached.
> 
> This does not explain the corrupted ext2 metadata, which is in lowmem
> pages.

Please could you forget about EXT2 for now?  Let's pretend I never 
mentioned it.  As a matter of fact I can,t see EXT2 errors anymore in 
the flood of user space segfaults.


Nicolas
> Let's get a fresh set of clear bug reports giving all relevant information.
> Let's not continue to scatter small little bits of information in multiple
> emails.

I think I did, less than 30 minutes ago.


Nicolas

Highmem issues with MMC filesystem

[Nicolas Pitre]

On Fri, 19 Mar 2010, Russell King - ARM Linux wrote:

> On Fri, Mar 19, 2010 at 02:27:24PM -0400, Nicolas Pitre wrote:
> > But on ARMv6 this is different.  The L1 cache is VIPT and it therefore 
> > doesn't have to be flushed as often as a VIVT cache.  Still, as far as I 
> > know, the highmem code currently always flush any page to be unmapped.  
> > But somewhere somehow an unmapped highmem page becomes subject to DMA 
> > and apparently can still be L1 cached.
> 
> This does not explain the corrupted ext2 metadata, which is in lowmem
> pages.

Please could you forget about EXT2 for now?  Let's pretend I never 
mentioned it.  As a matter of fact I can,t see EXT2 errors anymore in 
the flood of user space segfaults.


Nicolas
> Let's get a fresh set of clear bug reports giving all relevant information.
> Let's not continue to scatter small little bits of information in multiple
> emails.

I think I did, less than 30 minutes ago.


Nicolas

Re: Highmem issues with MMC filesystem

[Nicolas Pitre]

On Fri, 19 Mar 2010, Catalin Marinas wrote:

> On Fri, 2010-03-19 at 17:59 +0000, Nicolas Pitre wrote:
> > On Fri, 19 Mar 2010, Russell King - ARM Linux wrote:
> > 
> > > On Fri, Mar 19, 2010 at 09:17:19AM -0400, Nicolas Pitre wrote:
> > > > And in this case, the same SATA driver is used on two different system,
> > > > one being ARMv5 with absolutely no issues with highmem, and the other
> > > > being ARMv6 with highmem problems.  So that pretty much rules out IDE
> > > > driver bugs.
> > >
> > > No it doesn't - there's more changes between ARMv5 and ARMv6 than just
> > > the cache model.  There's weak memory ordering effects too.
> > 
> > Sure.  But at the moment we have:
> > 
> >  - ARMv5 without highmem -> works
> > 
> >  - ARMv5 with highmem -> works
> > 
> >  - ARMv6 without highmem -> works
> > 
> >  - ARMv6 with highmem -> fails
> > 
> > In all four cases the SATA driver, the hard disk and the filesystem on
> > it are identical.  In the two ARMv5 cases the system and the kernel are
> > identical.  Ditto for the two ARMv6 cases.
> 
> Which ARMv6 processor is this?

Marvell 88AP510 aka Dove.  But the same issue was reported on OMAP.

> Is the SATA driver using DMA (I think I read this in the thread but it
> started as MMC issues and got confused)?

Any block device using DMA is affected.  Drivers using PIO are not, 
neither is NFS.  And again, using the _same_ kernel with mem=512m so to 
be sure highmem doesn't kick in exhibits no issue even with DMA.


Nicolas

Highmem issues with MMC filesystem

[Nicolas Pitre]

On Fri, 19 Mar 2010, Catalin Marinas wrote:

> On Fri, 2010-03-19 at 17:59 +0000, Nicolas Pitre wrote:
> > On Fri, 19 Mar 2010, Russell King - ARM Linux wrote:
> > 
> > > On Fri, Mar 19, 2010 at 09:17:19AM -0400, Nicolas Pitre wrote:
> > > > And in this case, the same SATA driver is used on two different system,
> > > > one being ARMv5 with absolutely no issues with highmem, and the other
> > > > being ARMv6 with highmem problems.  So that pretty much rules out IDE
> > > > driver bugs.
> > >
> > > No it doesn't - there's more changes between ARMv5 and ARMv6 than just
> > > the cache model.  There's weak memory ordering effects too.
> > 
> > Sure.  But at the moment we have:
> > 
> >  - ARMv5 without highmem -> works
> > 
> >  - ARMv5 with highmem -> works
> > 
> >  - ARMv6 without highmem -> works
> > 
> >  - ARMv6 with highmem -> fails
> > 
> > In all four cases the SATA driver, the hard disk and the filesystem on
> > it are identical.  In the two ARMv5 cases the system and the kernel are
> > identical.  Ditto for the two ARMv6 cases.
> 
> Which ARMv6 processor is this?

Marvell 88AP510 aka Dove.  But the same issue was reported on OMAP.

> Is the SATA driver using DMA (I think I read this in the thread but it
> started as MMC issues and got confused)?

Any block device using DMA is affected.  Drivers using PIO are not, 
neither is NFS.  And again, using the _same_ kernel with mem=512m so to 
be sure highmem doesn't kick in exhibits no issue even with DMA.


Nicolas

Re: Highmem issues with MMC filesystem

[Nicolas Pitre]

On Fri, 19 Mar 2010, Nicolas Pitre wrote:

> A highmem page can have 2 states: virtually mapped in the pkmap area, or 
> not mapped at all.  When it is mapped then page_address() returns a 
> valid virtual address for it.  In that case the cache for that mapping 
> can be valid, even dirty.  So the DMA API will perform cache handling 
> before/after the DMA operation.
> 
> However, before the page is unmapped, the VIVT cache has to be flushed 
> for that page.  This is why the DMA code currently doesn't bother doing 
> any L1 cache handling when a highmem page is not mapped -- the cache 
> just can't refer to such a page.
> 
> But on ARMv6 this is different.  The L1 cache is VIPT and it therefore 
> doesn't have to be flushed as often as a VIVT cache.  Still, as far as I 
> know, the highmem code currently always flush any page to be unmapped.  
> But somewhere somehow an unmapped highmem page becomes subject to DMA 
> and apparently can still be L1 cached.  But the DMA code doesn't flush 
> its cache due to the page not being mapped and then problems occur.

And here's a patch to test that hypothesis.  With this patch, all the 
issues with highmem on my ARMv6 system are gone.

Note this is suboptimal.  I still have to open my ARM ARM and see if we 
can do cache maintenance on the L1 cache using physical addresses to 
avoid the IRQ disable, PTE setup and TLB flush altogether.

Also, with this, a couple cache flushes elsewhere could be removed from 
the highmem code for VIPT caches.  But some of them are tricky and 
require more thoughts.

Oh and yes usage of KM_L2_CACHE in this IRQ disabled context is safe.

diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c
index 0da7ecc..79771b0 100644
--- a/arch/arm/mm/dma-mapping.c
+++ b/arch/arm/mm/dma-mapping.c
@@ -22,8 +22,14 @@
 #include <asm/highmem.h>
 #include <asm/cacheflush.h>
 #include <asm/tlbflush.h>
+#include <asm/system.h>
+#include <asm/kmap_types.h>
+#include <asm/fixmap.h>
+#include <asm/pgtable.h>
 #include <asm/sizes.h>
 
+#include "mm.h"
+
 /* Sanity check size */
 #if (CONSISTENT_DMA_SIZE % SZ_2M)
 #error "CONSISTENT_DMA_SIZE must be multiple of 2MiB"
@@ -464,6 +470,18 @@ static void dma_cache_maint_page(struct page *page, unsigned long offset,
 				vaddr += offset;
 				op(vaddr, len, dir);
 				kunmap_high(page);
+			} else if (cache_is_vipt_nonaliasing()) {
+				unsigned long va, idx, pte, flags;
+				unsigned long pa = page_to_phys(page);
+				idx = KM_L2_CACHE + KM_TYPE_NR * smp_processor_id();
+				va = __fix_to_virt(FIX_KMAP_BEGIN + idx);
+				pte = pfn_pte(pa >> PAGE_SHIFT, PAGE_KERNEL);
+				local_irq_save(flags);
+				set_pte_ext(TOP_PTE(va), pte, 0);
+				local_flush_tlb_kernel_page(va);
+				vaddr = (void*)va + offset;
+				op(vaddr, len, dir);
+				local_irq_restore(flags);
 			}
 		} else {
 			vaddr = page_address(page) + offset;

Highmem issues with MMC filesystem

[Nicolas Pitre]

On Fri, 19 Mar 2010, Nicolas Pitre wrote:

> A highmem page can have 2 states: virtually mapped in the pkmap area, or 
> not mapped at all.  When it is mapped then page_address() returns a 
> valid virtual address for it.  In that case the cache for that mapping 
> can be valid, even dirty.  So the DMA API will perform cache handling 
> before/after the DMA operation.
> 
> However, before the page is unmapped, the VIVT cache has to be flushed 
> for that page.  This is why the DMA code currently doesn't bother doing 
> any L1 cache handling when a highmem page is not mapped -- the cache 
> just can't refer to such a page.
> 
> But on ARMv6 this is different.  The L1 cache is VIPT and it therefore 
> doesn't have to be flushed as often as a VIVT cache.  Still, as far as I 
> know, the highmem code currently always flush any page to be unmapped.  
> But somewhere somehow an unmapped highmem page becomes subject to DMA 
> and apparently can still be L1 cached.  But the DMA code doesn't flush 
> its cache due to the page not being mapped and then problems occur.

And here's a patch to test that hypothesis.  With this patch, all the 
issues with highmem on my ARMv6 system are gone.

Note this is suboptimal.  I still have to open my ARM ARM and see if we 
can do cache maintenance on the L1 cache using physical addresses to 
avoid the IRQ disable, PTE setup and TLB flush altogether.

Also, with this, a couple cache flushes elsewhere could be removed from 
the highmem code for VIPT caches.  But some of them are tricky and 
require more thoughts.

Oh and yes usage of KM_L2_CACHE in this IRQ disabled context is safe.

diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c
index 0da7ecc..79771b0 100644
--- a/arch/arm/mm/dma-mapping.c
+++ b/arch/arm/mm/dma-mapping.c
@@ -22,8 +22,14 @@
 #include <asm/highmem.h>
 #include <asm/cacheflush.h>
 #include <asm/tlbflush.h>
+#include <asm/system.h>
+#include <asm/kmap_types.h>
+#include <asm/fixmap.h>
+#include <asm/pgtable.h>
 #include <asm/sizes.h>
 
+#include "mm.h"
+
 /* Sanity check size */
 #if (CONSISTENT_DMA_SIZE % SZ_2M)
 #error "CONSISTENT_DMA_SIZE must be multiple of 2MiB"
@@ -464,6 +470,18 @@ static void dma_cache_maint_page(struct page *page, unsigned long offset,
 				vaddr += offset;
 				op(vaddr, len, dir);
 				kunmap_high(page);
+			} else if (cache_is_vipt_nonaliasing()) {
+				unsigned long va, idx, pte, flags;
+				unsigned long pa = page_to_phys(page);
+				idx = KM_L2_CACHE + KM_TYPE_NR * smp_processor_id();
+				va = __fix_to_virt(FIX_KMAP_BEGIN + idx);
+				pte = pfn_pte(pa >> PAGE_SHIFT, PAGE_KERNEL);
+				local_irq_save(flags);
+				set_pte_ext(TOP_PTE(va), pte, 0);
+				local_flush_tlb_kernel_page(va);
+				vaddr = (void*)va + offset;
+				op(vaddr, len, dir);
+				local_irq_restore(flags);
 			}
 		} else {
 			vaddr = page_address(page) + offset;

Re: Highmem issues with MMC filesystem

[Hemanth V]

> On Fri, 19 Mar 2010, Nicolas Pitre wrote:
>
>> A highmem page can have 2 states: virtually mapped in the pkmap area, or
>> not mapped at all.  When it is mapped then page_address() returns a
>> valid virtual address for it.  In that case the cache for that mapping
>> can be valid, even dirty.  So the DMA API will perform cache handling
>> before/after the DMA operation.
>>
>> However, before the page is unmapped, the VIVT cache has to be flushed
>> for that page.  This is why the DMA code currently doesn't bother doing
>> any L1 cache handling when a highmem page is not mapped -- the cache
>> just can't refer to such a page.
>>
>> But on ARMv6 this is different.  The L1 cache is VIPT and it therefore
>> doesn't have to be flushed as often as a VIVT cache.  Still, as far as I
>> know, the highmem code currently always flush any page to be unmapped.
>> But somewhere somehow an unmapped highmem page becomes subject to DMA
>> and apparently can still be L1 cached.  But the DMA code doesn't flush
>> its cache due to the page not being mapped and then problems occur.
>
> And here's a patch to test that hypothesis.  With this patch, all the
> issues with highmem on my ARMv6 system are gone.
>

Modified version of this patch also solves the problem for 2.6.32-rc7
on ARM v7.

Thanks
Hemanth

Highmem issues with MMC filesystem

[Hemanth V]

> On Fri, 19 Mar 2010, Nicolas Pitre wrote:
>
>> A highmem page can have 2 states: virtually mapped in the pkmap area, or
>> not mapped at all.  When it is mapped then page_address() returns a
>> valid virtual address for it.  In that case the cache for that mapping
>> can be valid, even dirty.  So the DMA API will perform cache handling
>> before/after the DMA operation.
>>
>> However, before the page is unmapped, the VIVT cache has to be flushed
>> for that page.  This is why the DMA code currently doesn't bother doing
>> any L1 cache handling when a highmem page is not mapped -- the cache
>> just can't refer to such a page.
>>
>> But on ARMv6 this is different.  The L1 cache is VIPT and it therefore
>> doesn't have to be flushed as often as a VIVT cache.  Still, as far as I
>> know, the highmem code currently always flush any page to be unmapped.
>> But somewhere somehow an unmapped highmem page becomes subject to DMA
>> and apparently can still be L1 cached.  But the DMA code doesn't flush
>> its cache due to the page not being mapped and then problems occur.
>
> And here's a patch to test that hypothesis.  With this patch, all the
> issues with highmem on my ARMv6 system are gone.
>

Modified version of this patch also solves the problem for 2.6.32-rc7
on ARM v7.

Thanks
Hemanth

Re: Highmem issues with MMC filesystem

[Russell King - ARM Linux]

On Fri, Mar 19, 2010 at 02:51:39PM -0400, Nicolas Pitre wrote:
> Nicolas
> > Let's get a fresh set of clear bug reports giving all relevant information.
> > Let's not continue to scatter small little bits of information in multiple
> > emails.
> 
> I think I did, less than 30 minutes ago.

If you don't want to involve me in a discussion that I can clearly
understand, that's fine, just don't expect me to be quick about
applying any resulting patch.

I've tried to make reasonable requests, but it seems that as per
normal, reasonable requests get ignored.

Highmem issues with MMC filesystem

[Russell King - ARM Linux]

On Fri, Mar 19, 2010 at 02:51:39PM -0400, Nicolas Pitre wrote:
> Nicolas
> > Let's get a fresh set of clear bug reports giving all relevant information.
> > Let's not continue to scatter small little bits of information in multiple
> > emails.
> 
> I think I did, less than 30 minutes ago.

If you don't want to involve me in a discussion that I can clearly
understand, that's fine, just don't expect me to be quick about
applying any resulting patch.

I've tried to make reasonable requests, but it seems that as per
normal, reasonable requests get ignored.

Re: Highmem issues with MMC filesystem

[Nicolas Pitre]

On Mon, 22 Mar 2010, Russell King - ARM Linux wrote:

> On Fri, Mar 19, 2010 at 02:51:39PM -0400, Nicolas Pitre wrote:
> > Nicolas
> > > Let's get a fresh set of clear bug reports giving all relevant information.
> > > Let's not continue to scatter small little bits of information in multiple
> > > emails.
> > 
> > I think I did, less than 30 minutes ago.
> 
> If you don't want to involve me in a discussion that I can clearly
> understand, that's fine, just don't expect me to be quick about
> applying any resulting patch.

There are no patches to be applied at this point.  When I have a proper 
patch to submit I'll summarize the issues along with the patch.


Nicolas

Highmem issues with MMC filesystem

[Nicolas Pitre]

On Mon, 22 Mar 2010, Russell King - ARM Linux wrote:

> On Fri, Mar 19, 2010 at 02:51:39PM -0400, Nicolas Pitre wrote:
> > Nicolas
> > > Let's get a fresh set of clear bug reports giving all relevant information.
> > > Let's not continue to scatter small little bits of information in multiple
> > > emails.
> > 
> > I think I did, less than 30 minutes ago.
> 
> If you don't want to involve me in a discussion that I can clearly
> understand, that's fine, just don't expect me to be quick about
> applying any resulting patch.

There are no patches to be applied at this point.  When I have a proper 
patch to submit I'll summarize the issues along with the patch.


Nicolas

Re: Highmem issues with MMC filesystem

[Nicolas Pitre]

On Mon, 22 Mar 2010, Hemanth V wrote:

> > On Fri, 19 Mar 2010, Nicolas Pitre wrote:
> >
> >> A highmem page can have 2 states: virtually mapped in the pkmap area, or
> >> not mapped at all.  When it is mapped then page_address() returns a
> >> valid virtual address for it.  In that case the cache for that mapping
> >> can be valid, even dirty.  So the DMA API will perform cache handling
> >> before/after the DMA operation.
> >>
> >> However, before the page is unmapped, the VIVT cache has to be flushed
> >> for that page.  This is why the DMA code currently doesn't bother doing
> >> any L1 cache handling when a highmem page is not mapped -- the cache
> >> just can't refer to such a page.
> >>
> >> But on ARMv6 this is different.  The L1 cache is VIPT and it therefore
> >> doesn't have to be flushed as often as a VIVT cache.  Still, as far as I
> >> know, the highmem code currently always flush any page to be unmapped.
> >> But somewhere somehow an unmapped highmem page becomes subject to DMA
> >> and apparently can still be L1 cached.  But the DMA code doesn't flush
> >> its cache due to the page not being mapped and then problems occur.
> >
> > And here's a patch to test that hypothesis.  With this patch, all the
> > issues with highmem on my ARMv6 system are gone.
> >
> 
> Modified version of this patch also solves the problem for 2.6.32-rc7
> on ARM v7.

OK thanks for testing.  I'll work on a proper patch now.


Nicolas

Highmem issues with MMC filesystem

[Nicolas Pitre]

On Mon, 22 Mar 2010, Hemanth V wrote:

> > On Fri, 19 Mar 2010, Nicolas Pitre wrote:
> >
> >> A highmem page can have 2 states: virtually mapped in the pkmap area, or
> >> not mapped at all.  When it is mapped then page_address() returns a
> >> valid virtual address for it.  In that case the cache for that mapping
> >> can be valid, even dirty.  So the DMA API will perform cache handling
> >> before/after the DMA operation.
> >>
> >> However, before the page is unmapped, the VIVT cache has to be flushed
> >> for that page.  This is why the DMA code currently doesn't bother doing
> >> any L1 cache handling when a highmem page is not mapped -- the cache
> >> just can't refer to such a page.
> >>
> >> But on ARMv6 this is different.  The L1 cache is VIPT and it therefore
> >> doesn't have to be flushed as often as a VIVT cache.  Still, as far as I
> >> know, the highmem code currently always flush any page to be unmapped.
> >> But somewhere somehow an unmapped highmem page becomes subject to DMA
> >> and apparently can still be L1 cached.  But the DMA code doesn't flush
> >> its cache due to the page not being mapped and then problems occur.
> >
> > And here's a patch to test that hypothesis.  With this patch, all the
> > issues with highmem on my ARMv6 system are gone.
> >
> 
> Modified version of this patch also solves the problem for 2.6.32-rc7
> on ARM v7.

OK thanks for testing.  I'll work on a proper patch now.


Nicolas