Re: [PATCH 06/17] arm: mmu_gather rework

[Russell King <rmk@arm.linux.org.uk>]

On Mon, Feb 28, 2011 at 12:44:47PM +0100, Peter Zijlstra wrote:
> Right, so the normal case is:
> 
>   unmap_region()
>     tlb_gather_mmu()

The fullmm argument is important here as it specifies the mode.
      tlb_gather_mmu(, 0)

>     unmap_vmas()
>       for (; vma; vma = vma->vm_next)
>         unmao_page_range()
>           tlb_start_vma() -> flush cache range
>           zap_*_range()
>             ptep_get_and_clear_full() -> batch/track external tlbs
>             tlb_remove_tlb_entry() -> batch/track external tlbs
>             tlb_remove_page() -> track range/batch page
>           tlb_end_vma() -> flush tlb range

       tlb_finish_mmu() -> nothing

> 
>  [ for architectures that have hardware page table walkers
>    concurrent faults can still load the page tables ]
> 
>     free_pgtables()

        tlb_gather_mmu(, 1)

>       while (vma)
>         unlink_*_vma()
>         free_*_range()
>           *_free_tlb()
>     tlb_finish_mmu()

      tlb_finish_mmu() -> flush tlb mm

> 
>   free vmas

So this is all fine.  Note that we *don't* use the range stuff here.

> Now, if we want to track ranges _and_ have hardware page table walkers
> (ARM seems to be one such), we must flush TLBs at tlb_end_vma() because
> flush_tlb_range() requires a vma pointer (ARM and TILE actually use more
> than ->vm_mm), and on tlb_finish_mmu() issue a full mm wide invalidate
> because the hardware walker could have re-populated the cache after
> clearing the PTEs but before freeing the page tables.

No.  The hardware walker won't re-populate the TLB after the page table
entries have been cleared - where would it get this information from if
not from the page tables?

> What ARM does is it retains the last vma pointer and tracks
> pte_free_tlb() range and uses that in tlb_finish_mmu(), which is a tad
> hacky.

It may be hacky but then the TLB shootdown interface is hacky too.  We
don't keep the vma around to re-use after tlb_end_vma() - if you think
that then you misunderstand what's going on.  The vma pointer is kept
around as a cheap way of allowing tlb_finish_mmu() to distinguish
between the unmap_region() mode and the shift_arg_pages() mode.

> Mostly because of shift_arg_pages(), where we have:
> 
>   shift_arg_pages()
>     tlb_gather_mmu()

      tlb_gather_mmu(, 0)

>     free_*_range()
>     tlb_finish_mmu()

      tlb_finish_mmu() does nothing without the ARM change.
      tlb_finish_mmu() -> flush_tlb_mm() with the ARM change.

And this is where the bug was - these page table entries could find
their way into the TLB and persist after they've been torn down.

> For which ARM now punts and does a full tlb invalidate (no vma pointer).
> But also because it drags along that vma pointer, which might not at all
> match the range its actually going to invalidate (and hence its vm_flags
> might not accurately reflect things -- at worst more expensive than
> needed).

Where do you get that from?  Where exactly in the above code would the
VMA pointer get set?  In this case, it will be NULL, so we do a
flush_tlb_mm() for this case.  We have to - we don't have any VMA to
deal with at this point.

> The reason I wanted flush_tlb_range() to take an mm_struct and not the
> current vm_area_struct is because we can avoid doing the
> flush_tlb_range() from tlb_end_vma() and delay the thing until
> tlb_finish_mmu() without having to resort to such games as above. We
> could simply track the full range over all VMAs and free'd page-tables
> and do one range invalidate.

No.  That's stupid.  Consider the case where you have to loop one page
at a time over the range (we do on ARM.)  If we ended up with your
suggestion above, that means we could potentially have to loop 4K at a
time over 3GB of address space.  That's idiotic when we have an
instruction which can flush the entire TLB for a particular thread.

-- 
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of: