question about broadcast invalidation and atc invalidation

question about broadcast invalidation and atc invalidation

[Zhongmiao]

Hi Jean:
	Now Soft need to make sure build atc inv cmd to smmu after Execute 
"TLBI" (dvm sync). I don't think it's very reasonable. Why  Smmu The 
smmu automatically sends the inv atc command when smmu receive 
broadcast invalidation ?


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: question about broadcast invalidation and atc invalidation

[Jean-Philippe Brucker]

Hi,

On 21/12/2018 09:31, Zhongmiao wrote:
> Hi Jean:
> 	Now Soft need to make sure build atc inv cmd to smmu after Execute 
> "TLBI" (dvm sync). I don't think it's very reasonable. Why  Smmu The 
> smmu automatically sends the inv atc command when smmu receive 
> broadcast invalidation ?

I'd also like to see this done by hardware, but it's not
straightforward. I see two difficulties with turning broadcast TLBI into
ATC invalidation:

* TLBI uses VMID and ASID to target the TLB entries to invalidate, but
ATS uses RID and PASID. The SMMU would need to do a reverse lookup to
convert VMID:ASID into RID:PASID, and there may be multiple RID:PASID
pairs associated to one VMID:ASID.

* ATS flow control. Each endpoint has a limited number of in-flight ATC
invalidate requests. If the CPU is sending TLBI too fast, the SMMU would
have to buffer them or back-pressure the interconnect, while waiting for
in-flight ATC invalidate to complete.

So for the moment the architecture doesn't support this. We use MMU
notifiers to convert VMID:ASID into RID:PASID and wait for ATC
invalidation to complete.

Thanks,
Jean

Re: question about broadcast invalidation and atc invalidation

[Jean-Philippe Brucker]

Hi,

On 21/12/2018 09:31, Zhongmiao wrote:
> Hi Jean:
> 	Now Soft need to make sure build atc inv cmd to smmu after Execute 
> "TLBI" (dvm sync). I don't think it's very reasonable. Why  Smmu The 
> smmu automatically sends the inv atc command when smmu receive 
> broadcast invalidation ?

I'd also like to see this done by hardware, but it's not
straightforward. I see two difficulties with turning broadcast TLBI into
ATC invalidation:

* TLBI uses VMID and ASID to target the TLB entries to invalidate, but
ATS uses RID and PASID. The SMMU would need to do a reverse lookup to
convert VMID:ASID into RID:PASID, and there may be multiple RID:PASID
pairs associated to one VMID:ASID.

* ATS flow control. Each endpoint has a limited number of in-flight ATC
invalidate requests. If the CPU is sending TLBI too fast, the SMMU would
have to buffer them or back-pressure the interconnect, while waiting for
in-flight ATC invalidate to complete.

So for the moment the architecture doesn't support this. We use MMU
notifiers to convert VMID:ASID into RID:PASID and wait for ATC
invalidation to complete.

Thanks,
Jean

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: question about broadcast invalidation and atc invalidation

[Zhongmiao]

On 2018/12/21 17:31, Zhongmiao wrote:
> Hi Jean:
>     Now Soft need to make sure build atc inv cmd to smmu after Execute
> "TLBI" (dvm sync). I don't think it's very reasonable. Why  Smmu The
> smmu automatically sends the inv atc command when smmu receive broadcast
> invalidation ?

Sorry, The expression is incorrect. The reality should be  "Why not Smmu 
The smmu automatically sends the inv atc command when smmu receive 
broadcast invalidation ?"

>
>
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
>
>


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: question about broadcast invalidation and atc invalidation

[Zhongmiao]

Hi Jean:
On 2018/12/24 14:36, Zhongmiao wrote:
>
>
>
> Hi,
>
> On 21/12/2018 09:31, Zhongmiao wrote:
>> Hi Jean:
>> 	Now Soft need to make sure build atc inv cmd to smmu after Execute
>> "TLBI" (dvm sync). I don't think it's very reasonable. Why  Smmu The
>> smmu automatically sends the inv atc command when smmu receive
>> broadcast invalidation ?
>
> I'd also like to see this done by hardware, but it's not straightforward. I see two difficulties with turning broadcast TLBI into ATC invalidation:
>
> * TLBI uses VMID and ASID to target the TLB entries to invalidate, but ATS uses RID and PASID. The SMMU would need to do a reverse lookup to convert VMID:ASID into RID:PASID, and there may be multiple RID:PASID pairs associated to one VMID:ASID.
>
> * ATS flow control. Each endpoint has a limited number of in-flight ATC invalidate requests. If the CPU is sending TLBI too fast, the SMMU would have to buffer them or back-pressure the interconnect, while waiting for in-flight ATC invalidate to complete.
>
> So for the moment the architecture doesn't support this. We use MMU notifiers to convert VMID:ASID into RID:PASID and wait for ATC invalidation to complete.
>

I got it about this.But How does the software(application) know to go to 
invalidate remote ATC? This entry may not be cached at remote ATC.

> Thanks,
> Jean
>

Thanks,
Miao


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: question about broadcast invalidation and atc invalidation

[Jean-Philippe Brucker]

Hi Miao,

On 24/12/2018 06:44, Zhongmiao wrote:
> Hi Jean:
> On 2018/12/24 14:36, Zhongmiao wrote:
>>
>>
>>
>> Hi,
>>
>> On 21/12/2018 09:31, Zhongmiao wrote:
>>> Hi Jean:
>>> 	Now Soft need to make sure build atc inv cmd to smmu after Execute
>>> "TLBI" (dvm sync). I don't think it's very reasonable. Why  Smmu The
>>> smmu automatically sends the inv atc command when smmu receive
>>> broadcast invalidation ?
>>
>> I'd also like to see this done by hardware, but it's not straightforward. I see two difficulties with turning broadcast TLBI into ATC invalidation:
>>
>> * TLBI uses VMID and ASID to target the TLB entries to invalidate, but ATS uses RID and PASID. The SMMU would need to do a reverse lookup to convert VMID:ASID into RID:PASID, and there may be multiple RID:PASID pairs associated to one VMID:ASID.
>>
>> * ATS flow control. Each endpoint has a limited number of in-flight ATC invalidate requests. If the CPU is sending TLBI too fast, the SMMU would have to buffer them or back-pressure the interconnect, while waiting for in-flight ATC invalidate to complete.>>
>> So for the moment the architecture doesn't support this. We use MMU notifiers to convert VMID:ASID into RID:PASID and wait for ATC invalidation to complete.
>>
> 
> I got it about this.But How does the software(application) know to go to 
> invalidate remote ATC? This entry may not be cached at remote ATC.

Software doesn't know which entry has been cached in the ATC, so it must
invalidate any address that could have been cached. The mm subsystem
calls mmu_notifier_invalidate_range*() or mmu_notifier_change_pte() any
time it changes or deletes a mapping, which means that we'll send plenty
of spurious ATC INV for entries that don't exist.

I don't think we can do much to optimize this. We could theoretically
rely on the PTE access flag to know if an address might have been
cached, and don't send an ATC INV if the access flag was already unset
when clearing the PTE. But the mm subsystem doesn't seem to have any
infrastructure for this at the moment, and I think that adding it would
be too invasive, and the performance improvement too small.

Thanks,
Jean

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel