[RFC] mmaping with VIVT cache

[RFC] mmaping with VIVT cache

[P F]

Hi,

Please excuse the pseudonym.

I have recently encountered a problem in the context of the UVC (webcam)
driver.  The driver maintainer, Laurent, suggested that we bring the
thread to linux-arm-kernel for wider discussion.  See here for reference:
http://lists.berlios.de/pipermail/linux-uvc-devel/2010-August/005864.html

The problem I encountered was this: using an ARM926EJ-S with OHCI USB host,
I noticed intermittent graphical corruption of frames captured by several
different webcams, both UVC and non-UVC.

I was able to isolate the corruption down to one specific instance.  In
this instance, I collected three images:
1A.jpg - collected from host, prior to corruption
2A.jpg - collected from host, demonstrating corruption
2B.jpg - collected from inline USB analyzer, same image as 2A

Images 1A and 2A are in sequence, i.e., 2A immediately follows 1A.  2B is
not corrupted, and a binary diff between 2A and 2B shows two 32-byte
chunks have been changed.  A binary diff between 2A and 1A shows that
these same two chunks match.  Therefore, it seems there is some
contamination of the second image by the first.

As I described in the linked message, I found a workaround for this issue
by adding
  vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot)
in the UVC driver's mmap() function, but Laurent explained that this was
invalid.

Does anybody have ideas for a proper fix?  I can reproduce this corruption
easily, and would be happy to test proposed patches.

Best regards


      
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 1A.jpg
Type: image/jpeg
Size: 10109 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20100913/147ad402/attachment-0003.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 2A.jpg
Type: image/jpeg
Size: 11660 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20100913/147ad402/attachment-0004.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 2B.jpg
Type: image/jpeg
Size: 11660 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20100913/147ad402/attachment-0005.jpg>

[RFC] mmaping with VIVT cache

[Russell King - ARM Linux]

On Mon, Sep 13, 2010 at 12:58:41PM -0700, P F wrote:
> Hi,
> 
> Please excuse the pseudonym.
> 
> I have recently encountered a problem in the context of the UVC (webcam)
> driver.  The driver maintainer, Laurent, suggested that we bring the
> thread to linux-arm-kernel for wider discussion.  See here for reference:
> http://lists.berlios.de/pipermail/linux-uvc-devel/2010-August/005864.html
> 
> The problem I encountered was this: using an ARM926EJ-S with OHCI USB host,
> I noticed intermittent graphical corruption of frames captured by several
> different webcams, both UVC and non-UVC.
> 
> I was able to isolate the corruption down to one specific instance.  In
> this instance, I collected three images:
> 1A.jpg - collected from host, prior to corruption
> 2A.jpg - collected from host, demonstrating corruption
> 2B.jpg - collected from inline USB analyzer, same image as 2A
> 
> Images 1A and 2A are in sequence, i.e., 2A immediately follows 1A.  2B is
> not corrupted, and a binary diff between 2A and 2B shows two 32-byte
> chunks have been changed.  A binary diff between 2A and 1A shows that
> these same two chunks match.  Therefore, it seems there is some
> contamination of the second image by the first.
> 
> As I described in the linked message, I found a workaround for this issue
> by adding
>   vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot)
> in the UVC driver's mmap() function, but Laurent explained that this was
> invalid.
> 
> Does anybody have ideas for a proper fix?  I can reproduce this corruption
> easily, and would be happy to test proposed patches.

The fact is that mapping buffers as cacheable into userspace on non-
coherent architectures _will_ result in the cache contents getting in
the way of updates to the DMA'd buffer.  It's a fundamental property
of such hardware implementations.

I've tried to introduce an API for mapping "coherent" DMA buffers
into userspace, and it got nowhere (well, only ARM implements it,
and hardly anyone uses it) - hence why we still have these problems.

So I don't have an answer to solve your problem - afaiac, it's up to
the device driver authors to sort it out, and apply pressure to other
parts of the kernel community (those who resist a way to make this
kind of thing work) to sort out how to cleanly map buffers used for
DMA into userspace.

[RFC] mmaping with VIVT cache

[Eric Miao]

On Wed, Sep 15, 2010 at 7:50 AM, Russell King - ARM Linux
<linux@arm.linux.org.uk> wrote:
> On Mon, Sep 13, 2010 at 12:58:41PM -0700, P F wrote:
>> Hi,
>>
>> Please excuse the pseudonym.
>>
>> I have recently encountered a problem in the context of the UVC (webcam)
>> driver. ?The driver maintainer, Laurent, suggested that we bring the
>> thread to linux-arm-kernel for wider discussion. ?See here for reference:
>> http://lists.berlios.de/pipermail/linux-uvc-devel/2010-August/005864.html
>>
>> The problem I encountered was this: using an ARM926EJ-S with OHCI USB host,
>> I noticed intermittent graphical corruption of frames captured by several
>> different webcams, both UVC and non-UVC.
>>
>> I was able to isolate the corruption down to one specific instance. ?In
>> this instance, I collected three images:
>> 1A.jpg - collected from host, prior to corruption
>> 2A.jpg - collected from host, demonstrating corruption
>> 2B.jpg - collected from inline USB analyzer, same image as 2A
>>
>> Images 1A and 2A are in sequence, i.e., 2A immediately follows 1A. ?2B is
>> not corrupted, and a binary diff between 2A and 2B shows two 32-byte
>> chunks have been changed. ?A binary diff between 2A and 1A shows that
>> these same two chunks match. ?Therefore, it seems there is some
>> contamination of the second image by the first.
>>
>> As I described in the linked message, I found a workaround for this issue
>> by adding
>> ? vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot)
>> in the UVC driver's mmap() function, but Laurent explained that this was
>> invalid.
>>
>> Does anybody have ideas for a proper fix? ?I can reproduce this corruption
>> easily, and would be happy to test proposed patches.
>
> The fact is that mapping buffers as cacheable into userspace on non-
> coherent architectures _will_ result in the cache contents getting in
> the way of updates to the DMA'd buffer. ?It's a fundamental property
> of such hardware implementations.
>

Maybe we can just detect if the buffer is mapped in alias, and modify the
PTE, just as what we did to inconsistent write-combine? So to make this
transparent to other consistent architectures?

> I've tried to introduce an API for mapping "coherent" DMA buffers
> into userspace, and it got nowhere (well, only ARM implements it,
> and hardly anyone uses it) - hence why we still have these problems.
>
> So I don't have an answer to solve your problem - afaiac, it's up to
> the device driver authors to sort it out, and apply pressure to other
> parts of the kernel community (those who resist a way to make this
> kind of thing work) to sort out how to cleanly map buffers used for
> DMA into userspace.
>
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel at lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
>

[RFC] mmaping with VIVT cache

[Russell King - ARM Linux]

On Wed, Sep 15, 2010 at 09:06:02AM +0800, Eric Miao wrote:
> On Wed, Sep 15, 2010 at 7:50 AM, Russell King - ARM Linux
> <linux@arm.linux.org.uk> wrote:
> > On Mon, Sep 13, 2010 at 12:58:41PM -0700, P F wrote:
> >> Hi,
> >>
> >> Please excuse the pseudonym.
> >>
> >> I have recently encountered a problem in the context of the UVC (webcam)
> >> driver. ?The driver maintainer, Laurent, suggested that we bring the
> >> thread to linux-arm-kernel for wider discussion. ?See here for reference:
> >> http://lists.berlios.de/pipermail/linux-uvc-devel/2010-August/005864.html
> >>
> >> The problem I encountered was this: using an ARM926EJ-S with OHCI USB host,
> >> I noticed intermittent graphical corruption of frames captured by several
> >> different webcams, both UVC and non-UVC.
> >>
> >> I was able to isolate the corruption down to one specific instance. ?In
> >> this instance, I collected three images:
> >> 1A.jpg - collected from host, prior to corruption
> >> 2A.jpg - collected from host, demonstrating corruption
> >> 2B.jpg - collected from inline USB analyzer, same image as 2A
> >>
> >> Images 1A and 2A are in sequence, i.e., 2A immediately follows 1A. ?2B is
> >> not corrupted, and a binary diff between 2A and 2B shows two 32-byte
> >> chunks have been changed. ?A binary diff between 2A and 1A shows that
> >> these same two chunks match. ?Therefore, it seems there is some
> >> contamination of the second image by the first.
> >>
> >> As I described in the linked message, I found a workaround for this issue
> >> by adding
> >> ? vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot)
> >> in the UVC driver's mmap() function, but Laurent explained that this was
> >> invalid.
> >>
> >> Does anybody have ideas for a proper fix? ?I can reproduce this corruption
> >> easily, and would be happy to test proposed patches.
> >
> > The fact is that mapping buffers as cacheable into userspace on non-
> > coherent architectures _will_ result in the cache contents getting in
> > the way of updates to the DMA'd buffer. ?It's a fundamental property
> > of such hardware implementations.
> >
> 
> Maybe we can just detect if the buffer is mapped in alias, and modify the
> PTE, just as what we did to inconsistent write-combine? So to make this
> transparent to other consistent architectures?

All pages which are mapped into userspace have at least one alias.

[RFC] mmaping with VIVT cache

[Eric Miao]

On Wed, Sep 15, 2010 at 3:31 PM, Russell King - ARM Linux
<linux@arm.linux.org.uk> wrote:
> On Wed, Sep 15, 2010 at 09:06:02AM +0800, Eric Miao wrote:
>> On Wed, Sep 15, 2010 at 7:50 AM, Russell King - ARM Linux
>> <linux@arm.linux.org.uk> wrote:
>> > On Mon, Sep 13, 2010 at 12:58:41PM -0700, P F wrote:
>> >> Hi,
>> >>
>> >> Please excuse the pseudonym.
>> >>
>> >> I have recently encountered a problem in the context of the UVC (webcam)
>> >> driver. ?The driver maintainer, Laurent, suggested that we bring the
>> >> thread to linux-arm-kernel for wider discussion. ?See here for reference:
>> >> http://lists.berlios.de/pipermail/linux-uvc-devel/2010-August/005864.html
>> >>
>> >> The problem I encountered was this: using an ARM926EJ-S with OHCI USB host,
>> >> I noticed intermittent graphical corruption of frames captured by several
>> >> different webcams, both UVC and non-UVC.
>> >>
>> >> I was able to isolate the corruption down to one specific instance. ?In
>> >> this instance, I collected three images:
>> >> 1A.jpg - collected from host, prior to corruption
>> >> 2A.jpg - collected from host, demonstrating corruption
>> >> 2B.jpg - collected from inline USB analyzer, same image as 2A
>> >>
>> >> Images 1A and 2A are in sequence, i.e., 2A immediately follows 1A. ?2B is
>> >> not corrupted, and a binary diff between 2A and 2B shows two 32-byte
>> >> chunks have been changed. ?A binary diff between 2A and 1A shows that
>> >> these same two chunks match. ?Therefore, it seems there is some
>> >> contamination of the second image by the first.
>> >>
>> >> As I described in the linked message, I found a workaround for this issue
>> >> by adding
>> >> ? vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot)
>> >> in the UVC driver's mmap() function, but Laurent explained that this was
>> >> invalid.
>> >>
>> >> Does anybody have ideas for a proper fix? ?I can reproduce this corruption
>> >> easily, and would be happy to test proposed patches.
>> >
>> > The fact is that mapping buffers as cacheable into userspace on non-
>> > coherent architectures _will_ result in the cache contents getting in
>> > the way of updates to the DMA'd buffer. ?It's a fundamental property
>> > of such hardware implementations.
>> >
>>
>> Maybe we can just detect if the buffer is mapped in alias, and modify the
>> PTE, just as what we did to inconsistent write-combine? So to make this
>> transparent to other consistent architectures?
>
> All pages which are mapped into userspace have at least one alias.
>

Hrm... stumped.

[RFC] mmaping with VIVT cache

[Pawel Moll]

> pgprot_noncached will be unsafe on ARMv6 and ARMv7 architectures when
> used for mapping existing memory, as you'll be violating an architectural
> restriction (having aliasing mappings of memory as 'Strongly ordered'
> when there is a pre-existing 'Memory' type mapping.)
> adead.org/mailman/listinfo/linux-arm-kernel

Oh, I'm not concerned about it - it was a custom hack in order to get a
"sh-customer" running. I've just mentioned it as an example that the
problem is real and encountered by different people in different
places...

Pawe?

[RFC] mmaping with VIVT cache

[Russell King - ARM Linux]

On Mon, Sep 20, 2010 at 11:38:44AM +0100, Pawel Moll wrote:
> > However, in this case, architectures using VIPT aliasing caches will
> > also trip over it - has anyone used this webcam on something like
> > Sparc or powerpc?  I'm guessing that it'll fail there too...
> 
> It wasn't a webcam but USB audio card, not sparc nor powerpc but sh but
> the I believe problem stays the same - vmalloc-ed buffer mapped into the
> user-space...
> 
> http://thread.gmane.org/gmane.linux.alsa.devel/55882/focus=7160
> 
> I've ended up with pgprot_noncached(), as there was no "alias-safe API"
> available, and apparently it's still the case? ;-)

pgprot_noncached will be unsafe on ARMv6 and ARMv7 architectures when
used for mapping existing memory, as you'll be violating an architectural
restriction (having aliasing mappings of memory as 'Strongly ordered'
when there is a pre-existing 'Memory' type mapping.)

[RFC] mmaping with VIVT cache

[Pawel Moll]

> However, in this case, architectures using VIPT aliasing caches will
> also trip over it - has anyone used this webcam on something like
> Sparc or powerpc?  I'm guessing that it'll fail there too...

It wasn't a webcam but USB audio card, not sparc nor powerpc but sh but
the I believe problem stays the same - vmalloc-ed buffer mapped into the
user-space...

http://thread.gmane.org/gmane.linux.alsa.devel/55882/focus=7160

I've ended up with pgprot_noncached(), as there was no "alias-safe API"
available, and apparently it's still the case? ;-)

Cheers!

Pawe?

[RFC] mmaping with VIVT cache

[Russell King - ARM Linux]

On Fri, Sep 17, 2010 at 12:07:50PM -0700, P F wrote:
> From your comment, I take it that there is no kernel API to invalidate the
> cache of a range in a VMA.  Do you think linux-mm would be a more
> appropriate forum to continue investigating?  Any input would be
> appreciated.

That's not really the point.  To implement an API in terms of "it
invalidates a range in a VMA" is absurd when you have a kernel
supporting VIVT, VIVT with an address space tag, VIPT, VIPT with
address space tag, PIPT, etc.  Does it mean you have to invalidate
for every different kind of cache?  If you define it in terms of
"invalidates a range in a VMA" then yes it does - even if a cache
architecture doesn't need it.

It's much better to define APIs in terms of their purpose - such as
"we're tearing down MMU mappings, so flush this range in the VMA".
That allows implementations to say "do I need to do anything when
this operation occurs?  No - I can optimize it away."

The problem we face is that you wouldn't port a Unix-like OS to a
CPU with VIVT caches if you were in a sane state of mind...  It
is the worst for aliasing issues, such as the one you've found.  As
ARM is the only Linux arch which has VIVT caches, we trip over many
of these issues.

However, in this case, architectures using VIPT aliasing caches will
also trip over it - has anyone used this webcam on something like
Sparc or powerpc?  I'm guessing that it'll fail there too...

[RFC] mmaping with VIVT cache

[P F]

--- On Thu, 9/16/10, Russell King - ARM Linux <linux@arm.linux.org.uk> wrote:
> On Wed, Sep 15, 2010 at 05:25:38PM-0700, P F wrote:
> > The mmap'd buffer is allocated by vmalloc_32(), and later mapped with
> > vmalloc_to_page()/vm_insert_page().? The DMA'dbuffer is allocated by
> > usb_buffer_alloc().? At the appropriate time, the DMA'd buffer is
> > memcpy()'d into the mmap'd buffer, and userspace is allowed to handle
> > it, at which point it is corrupt.
> 
> Ah, that's slightly different as you're not talking about
> the DMA buffers themselves.
> 
> However, anyone who is not familiar with VIVT or aliasing
> VIPT caches is going to be caught out by this sort of thing (which
> basically means most ARMs, and at least some PA-RISC.)

Again, thank you for the explanation, Russell.? For some reason I did not
quite grok the root cause of this issue before, but it is now clear to me.

> I don't think we have any APIs in the kernel to handle
> aliasing between the vmalloc space and userspace...

Perhaps I am revealing my ignorance of the driver ecosystem, but I find
this somewhat surprising.? I would think that this sort of construction
would not be uncommon.

For this issue, I can see two solutions:
1.) somehow make the vmalloc'd space non-cacheable so both the user- and
kernelspace mappings match, and there is no cache to alias
2.) invalidate the cache of the userspace mapping before handing the
buffer to userspace (after memcpy() above)

As Laurent noted earlier, (1) is a nonstarter as it has a negative
performance impact.? (2) I propose is a valid and correct solution.? This
is driver-specific, but each buffer corresponds to an image frame (e.g.,
from a webcam), so they are independent, and therefore it does not make
sense to share the cache between frames.? As long as userspace holds the
buffer, the cache is valid, but once it relinquishes the buffer, the cache
holds only stale data and should be purged.

>From your comment, I take it that there is no kernel API to invalidate the
cache of a range in a VMA.  Do you think linux-mm would be a more
appropriate forum to continue investigating?  Any input would be
appreciated.


      

[RFC] mmaping with VIVT cache

[Russell King - ARM Linux]

On Wed, Sep 15, 2010 at 05:25:38PM -0700, P F wrote:
> The mmap'd buffer is allocated by vmalloc_32(), and later mapped with
> vmalloc_to_page()/vm_insert_page().  The DMA'd buffer is allocated by
> usb_buffer_alloc().  At the appropriate time, the DMA'd buffer is
> memcpy()'d into the mmap'd buffer, and userspace is allowed to handle it,
> at which point it is corrupt.

Ah, that's slightly different as you're not talking about the DMA buffers
themselves.

However, anyone who is not familiar with VIVT or aliasing VIPT caches is
going to be caught out by this sort of thing (which basically means most
ARMs, and at least some PA-RISC.)

I don't think we have any APIs in the kernel to handle aliasing between
the vmalloc space and userspace...

[RFC] mmaping with VIVT cache

[P F]

Hi Russell,

--- On Tue, 9/14/10, Russell King - ARM Linux <linux@arm.linux.org.uk> wrote:
> 
> The fact is that mapping buffers as cacheable into
> userspace on non-
> coherent architectures _will_ result in the cache contents
> getting in
> the way of updates to the DMA'd buffer.? It's a
> fundamental property
> of such hardware implementations.

Thank you for the explanation.

I realize this is driver specific, but I would like to confirm that your
explanation matches my problem.  Unless I misunderstand the UVC driver,
the buffer which is mapped into userspace is distinct from the one which
is updated by DMA.

The mmap'd buffer is allocated by vmalloc_32(), and later mapped with
vmalloc_to_page()/vm_insert_page().  The DMA'd buffer is allocated by
usb_buffer_alloc().  At the appropriate time, the DMA'd buffer is
memcpy()'d into the mmap'd buffer, and userspace is allowed to handle it,
at which point it is corrupt.

Is this scenario consistent with your above explanation, or does your explanation require that the mmap'd buffer and DMA'd buffer be one and
the same?