[PATCH 0/2] powerpc/64s/radix: avoid unnecessary TLB flushes on fault

[PATCH 0/2] powerpc/64s/radix: avoid unnecessary TLB flushes on fault

[Nicholas Piggin]

These two patches make radix match hash and not flush the TLB after
fixing up faults unnecessarily.

There was some concern that accelerators need to have this flush, but
nothing is documented or commented, so it should just be removed. We
have a coprocessor count in the mm context now, and that can easily
be special cased if neccesary.

This and a few other changes reduce our broadcast tlbie rates by 10x
on a kernel compile benchmark, so it would be good to get it in.

Thanks,
Nick

Nicholas Piggin (2):
  powerpc/64s/radix: do not flush TLB when relaxing access
  powerpc/64s/radix: do not flush TLB on spurious fault

 arch/powerpc/include/asm/book3s/64/tlbflush.h | 7 +++++++
 arch/powerpc/mm/pgtable-book3s64.c            | 1 -
 arch/powerpc/mm/pgtable.c                     | 3 ++-
 3 files changed, 9 insertions(+), 2 deletions(-)

-- 
2.17.0

[PATCH 1/2] powerpc/64s/radix: do not flush TLB when relaxing access

[Nicholas Piggin]

Radix flushes the TLB when updating ptes to increase permissiveness
of protection (increase access authority). Book3S does not require
TLB flushing in this case, and it is not done on hash. This patch
avoids the flush for radix.

>From Power ISA v3.0B, p.1090:

    Setting a Reference or Change Bit or Upgrading Access Authority
    (PTE Subject to Atomic Hardware Updates)

    If the only change being made to a valid PTE that is subject to
    atomic hardware updates is to set the Reference or Change bit to 1
    or to add access authorities, a simpler sequence suffices because
    the translation hardware will refetch the PTE if an access is
    attempted for which the only problems were reference and/or change
    bits needing to be set or insufficient access authority.

Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
 arch/powerpc/mm/pgtable-book3s64.c | 1 -
 arch/powerpc/mm/pgtable.c          | 3 ++-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/arch/powerpc/mm/pgtable-book3s64.c b/arch/powerpc/mm/pgtable-book3s64.c
index 518518fb7c45..6e991eaccab4 100644
--- a/arch/powerpc/mm/pgtable-book3s64.c
+++ b/arch/powerpc/mm/pgtable-book3s64.c
@@ -40,7 +40,6 @@ int pmdp_set_access_flags(struct vm_area_struct *vma, unsigned long address,
 	if (changed) {
 		__ptep_set_access_flags(vma->vm_mm, pmdp_ptep(pmdp),
 					pmd_pte(entry), address);
-		flush_pmd_tlb_range(vma, address, address + HPAGE_PMD_SIZE);
 	}
 	return changed;
 }
diff --git a/arch/powerpc/mm/pgtable.c b/arch/powerpc/mm/pgtable.c
index 9f361ae571e9..5b07a626df5b 100644
--- a/arch/powerpc/mm/pgtable.c
+++ b/arch/powerpc/mm/pgtable.c
@@ -224,7 +224,8 @@ int ptep_set_access_flags(struct vm_area_struct *vma, unsigned long address,
 		if (!is_vm_hugetlb_page(vma))
 			assert_pte_locked(vma->vm_mm, address);
 		__ptep_set_access_flags(vma->vm_mm, ptep, entry, address);
-		flush_tlb_page(vma, address);
+		if (!IS_ENABLED(CONFIG_PPC_BOOK3S_64))
+			flush_tlb_page(vma, address);
 	}
 	return changed;
 }
-- 
2.17.0

Re: [PATCH 1/2] powerpc/64s/radix: do not flush TLB when relaxing access

[Balbir Singh]

On Wed, May 9, 2018 at 4:51 PM, Nicholas Piggin <npiggin@gmail.com> wrote:
> Radix flushes the TLB when updating ptes to increase permissiveness
> of protection (increase access authority). Book3S does not require
> TLB flushing in this case, and it is not done on hash. This patch
> avoids the flush for radix.
>
> From Power ISA v3.0B, p.1090:
>
>     Setting a Reference or Change Bit or Upgrading Access Authority
>     (PTE Subject to Atomic Hardware Updates)
>
>     If the only change being made to a valid PTE that is subject to
>     atomic hardware updates is to set the Reference or Change bit to 1
>     or to add access authorities, a simpler sequence suffices because
>     the translation hardware will refetch the PTE if an access is
>     attempted for which the only problems were reference and/or change
>     bits needing to be set or insufficient access authority.
>
> Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
> ---
>  arch/powerpc/mm/pgtable-book3s64.c | 1 -
>  arch/powerpc/mm/pgtable.c          | 3 ++-
>  2 files changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/arch/powerpc/mm/pgtable-book3s64.c b/arch/powerpc/mm/pgtable-book3s64.c
> index 518518fb7c45..6e991eaccab4 100644
> --- a/arch/powerpc/mm/pgtable-book3s64.c
> +++ b/arch/powerpc/mm/pgtable-book3s64.c
> @@ -40,7 +40,6 @@ int pmdp_set_access_flags(struct vm_area_struct *vma, unsigned long address,
>         if (changed) {
>                 __ptep_set_access_flags(vma->vm_mm, pmdp_ptep(pmdp),
>                                         pmd_pte(entry), address);
> -               flush_pmd_tlb_range(vma, address, address + HPAGE_PMD_SIZE);

The comment states that this can be used for missing execution
permissions as well. I am not convinced we can skip a flush in those
cases

>         }
>         return changed;
>  }
> diff --git a/arch/powerpc/mm/pgtable.c b/arch/powerpc/mm/pgtable.c
> index 9f361ae571e9..5b07a626df5b 100644
> --- a/arch/powerpc/mm/pgtable.c
> +++ b/arch/powerpc/mm/pgtable.c
> @@ -224,7 +224,8 @@ int ptep_set_access_flags(struct vm_area_struct *vma, unsigned long address,
>                 if (!is_vm_hugetlb_page(vma))
>                         assert_pte_locked(vma->vm_mm, address);
>                 __ptep_set_access_flags(vma->vm_mm, ptep, entry, address);
> -               flush_tlb_page(vma, address);
> +               if (!IS_ENABLED(CONFIG_PPC_BOOK3S_64))
> +                       flush_tlb_page(vma, address);

Same as above

Balbir Singh.

Re: [PATCH 1/2] powerpc/64s/radix: do not flush TLB when relaxing access

[Nicholas Piggin]

On Wed, 9 May 2018 17:07:47 +1000
Balbir Singh <bsingharora@gmail.com> wrote:

> On Wed, May 9, 2018 at 4:51 PM, Nicholas Piggin <npiggin@gmail.com> wrote:
> > Radix flushes the TLB when updating ptes to increase permissiveness
> > of protection (increase access authority). Book3S does not require
> > TLB flushing in this case, and it is not done on hash. This patch
> > avoids the flush for radix.
> >
> > From Power ISA v3.0B, p.1090:
> >
> >     Setting a Reference or Change Bit or Upgrading Access Authority
> >     (PTE Subject to Atomic Hardware Updates)
> >
> >     If the only change being made to a valid PTE that is subject to
> >     atomic hardware updates is to set the Reference or Change bit to 1
> >     or to add access authorities, a simpler sequence suffices because
> >     the translation hardware will refetch the PTE if an access is
> >     attempted for which the only problems were reference and/or change
> >     bits needing to be set or insufficient access authority.
> >
> > Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
> > ---
> >  arch/powerpc/mm/pgtable-book3s64.c | 1 -
> >  arch/powerpc/mm/pgtable.c          | 3 ++-
> >  2 files changed, 2 insertions(+), 2 deletions(-)
> >
> > diff --git a/arch/powerpc/mm/pgtable-book3s64.c b/arch/powerpc/mm/pgtable-book3s64.c
> > index 518518fb7c45..6e991eaccab4 100644
> > --- a/arch/powerpc/mm/pgtable-book3s64.c
> > +++ b/arch/powerpc/mm/pgtable-book3s64.c
> > @@ -40,7 +40,6 @@ int pmdp_set_access_flags(struct vm_area_struct *vma, unsigned long address,
> >         if (changed) {
> >                 __ptep_set_access_flags(vma->vm_mm, pmdp_ptep(pmdp),
> >                                         pmd_pte(entry), address);
> > -               flush_pmd_tlb_range(vma, address, address + HPAGE_PMD_SIZE);  
> 
> The comment states that this can be used for missing execution
> permissions as well. I am not convinced we can skip a flush in those
> cases

Why not? Execute is part of the access authority. And they're already no
ops on hash. What am I missing?

Thanks,
Nick

Re: [PATCH 1/2] powerpc/64s/radix: do not flush TLB when relaxing access

[Balbir Singh]

On Wed, May 9, 2018 at 5:43 PM, Nicholas Piggin <npiggin@gmail.com> wrote:
> On Wed, 9 May 2018 17:07:47 +1000
> Balbir Singh <bsingharora@gmail.com> wrote:
>
>> On Wed, May 9, 2018 at 4:51 PM, Nicholas Piggin <npiggin@gmail.com> wrote:
>> > Radix flushes the TLB when updating ptes to increase permissiveness
>> > of protection (increase access authority). Book3S does not require
>> > TLB flushing in this case, and it is not done on hash. This patch
>> > avoids the flush for radix.
>> >
>> > From Power ISA v3.0B, p.1090:
>> >
>> >     Setting a Reference or Change Bit or Upgrading Access Authority
>> >     (PTE Subject to Atomic Hardware Updates)
>> >
>> >     If the only change being made to a valid PTE that is subject to
>> >     atomic hardware updates is to set the Reference or Change bit to 1
>> >     or to add access authorities, a simpler sequence suffices because
>> >     the translation hardware will refetch the PTE if an access is
>> >     attempted for which the only problems were reference and/or change
>> >     bits needing to be set or insufficient access authority.
>> >
>> > Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
>> > ---
>> >  arch/powerpc/mm/pgtable-book3s64.c | 1 -
>> >  arch/powerpc/mm/pgtable.c          | 3 ++-
>> >  2 files changed, 2 insertions(+), 2 deletions(-)
>> >
>> > diff --git a/arch/powerpc/mm/pgtable-book3s64.c b/arch/powerpc/mm/pgtable-book3s64.c
>> > index 518518fb7c45..6e991eaccab4 100644
>> > --- a/arch/powerpc/mm/pgtable-book3s64.c
>> > +++ b/arch/powerpc/mm/pgtable-book3s64.c
>> > @@ -40,7 +40,6 @@ int pmdp_set_access_flags(struct vm_area_struct *vma, unsigned long address,
>> >         if (changed) {
>> >                 __ptep_set_access_flags(vma->vm_mm, pmdp_ptep(pmdp),
>> >                                         pmd_pte(entry), address);
>> > -               flush_pmd_tlb_range(vma, address, address + HPAGE_PMD_SIZE);
>>
>> The comment states that this can be used for missing execution
>> permissions as well. I am not convinced we can skip a flush in those
>> cases
>
> Why not? Execute is part of the access authority. And they're already no
> ops on hash. What am I missing?

I have not reviewed the hash code, but if relaxing access means
allowing the code to provide execute permission, won't this result in
spurious faults? A simple test might be to run a JIT workload and see
if the number of faults go up with and without the patch?

Balbir

Re: [PATCH 1/2] powerpc/64s/radix: do not flush TLB when relaxing access

[Nicholas Piggin]

On Wed, 9 May 2018 18:27:07 +1000
Balbir Singh <bsingharora@gmail.com> wrote:

> On Wed, May 9, 2018 at 5:43 PM, Nicholas Piggin <npiggin@gmail.com> wrote:
> > On Wed, 9 May 2018 17:07:47 +1000
> > Balbir Singh <bsingharora@gmail.com> wrote:
> >  
> >> On Wed, May 9, 2018 at 4:51 PM, Nicholas Piggin <npiggin@gmail.com> wrote:  
> >> > Radix flushes the TLB when updating ptes to increase permissiveness
> >> > of protection (increase access authority). Book3S does not require
> >> > TLB flushing in this case, and it is not done on hash. This patch
> >> > avoids the flush for radix.
> >> >
> >> > From Power ISA v3.0B, p.1090:
> >> >
> >> >     Setting a Reference or Change Bit or Upgrading Access Authority
> >> >     (PTE Subject to Atomic Hardware Updates)
> >> >
> >> >     If the only change being made to a valid PTE that is subject to
> >> >     atomic hardware updates is to set the Reference or Change bit to 1
> >> >     or to add access authorities, a simpler sequence suffices because
> >> >     the translation hardware will refetch the PTE if an access is
> >> >     attempted for which the only problems were reference and/or change
> >> >     bits needing to be set or insufficient access authority.
> >> >
> >> > Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
> >> > ---
> >> >  arch/powerpc/mm/pgtable-book3s64.c | 1 -
> >> >  arch/powerpc/mm/pgtable.c          | 3 ++-
> >> >  2 files changed, 2 insertions(+), 2 deletions(-)
> >> >
> >> > diff --git a/arch/powerpc/mm/pgtable-book3s64.c b/arch/powerpc/mm/pgtable-book3s64.c
> >> > index 518518fb7c45..6e991eaccab4 100644
> >> > --- a/arch/powerpc/mm/pgtable-book3s64.c
> >> > +++ b/arch/powerpc/mm/pgtable-book3s64.c
> >> > @@ -40,7 +40,6 @@ int pmdp_set_access_flags(struct vm_area_struct *vma, unsigned long address,
> >> >         if (changed) {
> >> >                 __ptep_set_access_flags(vma->vm_mm, pmdp_ptep(pmdp),
> >> >                                         pmd_pte(entry), address);
> >> > -               flush_pmd_tlb_range(vma, address, address + HPAGE_PMD_SIZE);  
> >>
> >> The comment states that this can be used for missing execution
> >> permissions as well. I am not convinced we can skip a flush in those
> >> cases  
> >
> > Why not? Execute is part of the access authority. And they're already no
> > ops on hash. What am I missing?  
> 
> I have not reviewed the hash code, but if relaxing access means
> allowing the code to provide execute permission,

Yes, adding RWX or RC bits would qualify at least.

> won't this result in
> spurious faults?

Well it gets called as part of page faults, which means the MMU will
already reload the pte when the access is retried, as per the ISA.

A few paths actually don't call it from page fault paths, but the
cost of a superfluous fault occasionally versus always doing a tlbie
makes this the wrong thing to do. x86 and hash are the same, they
don't flush here. Hash mode should behave basically the same way in
terms of what it stores in the TLB and whether it would take more
faults. Radix is the newcomer so it should match hash without some
reason not to.

> A simple test might be to run a JIT workload and see
> if the number of faults go up with and without the patch?

What pattern of accesses are you worried about? Why is execute
different from write, for example?

Thanks,
Nick

[PATCH 2/2] powerpc/64s/radix: do not flush TLB on spurious fault

[Nicholas Piggin]

In the case of a spurious fault (which can happen due to a race with
another thread that changes the page table), the default Linux mm code
calls flush_tlb_page for that address. This is not required because
the pte will be re-fetched. Hash does not wire this up to a hardware
TLB flush for this reason. This patch avoids the flush for radix.

>From Power ISA v3.0B, p.1090:

    Setting a Reference or Change Bit or Upgrading Access Authority
    (PTE Subject to Atomic Hardware Updates)

    If the only change being made to a valid PTE that is subject to
    atomic hardware updates is to set the Refer- ence or Change bit to
    1 or to add access authorities, a simpler sequence suffices
    because the translation hardware will refetch the PTE if an access
    is attempted for which the only problems were reference and/or
    change bits needing to be set or insufficient access authority.

Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
 arch/powerpc/include/asm/book3s/64/tlbflush.h | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/arch/powerpc/include/asm/book3s/64/tlbflush.h b/arch/powerpc/include/asm/book3s/64/tlbflush.h
index 0cac17253513..9c43431e01c5 100644
--- a/arch/powerpc/include/asm/book3s/64/tlbflush.h
+++ b/arch/powerpc/include/asm/book3s/64/tlbflush.h
@@ -137,6 +137,13 @@ static inline void flush_all_mm(struct mm_struct *mm)
 #define flush_tlb_page(vma, addr)	local_flush_tlb_page(vma, addr)
 #define flush_all_mm(mm)		local_flush_all_mm(mm)
 #endif /* CONFIG_SMP */
+
+#define flush_tlb_fix_spurious_fault flush_tlb_fix_spurious_fault
+static inline void flush_tlb_fix_spurious_fault(struct vm_area_struct *vma,
+						unsigned long address)
+{
+}
+
 /*
  * flush the page walk cache for the address
  */
-- 
2.17.0