* [RFC] Use SMMU HTTU for DMA dirty page tracking
@ 2020-05-19 9:42 Xiang Zheng
2020-05-22 17:14 ` Jean-Philippe Brucker
0 siblings, 1 reply; 10+ messages in thread
From: Xiang Zheng @ 2020-05-19 9:42 UTC (permalink / raw)
To: linux-arm-kernel, kvmarm
Cc: jean-philippe, maz, wangzhou1, prime.zeng, Will Deacon
Hi all,
Is there any plan for enabling SMMU HTTU?
I have seen the patch locates in the SVA series patch, which adds
support for HTTU:
https://www.spinics.net/lists/arm-kernel/msg798694.html
HTTU reduces the number of access faults on SMMU fault queue
(permission faults also benifit from it).
Besides reducing the faults, HTTU also helps to track dirty pages for
device DMA. Is it feasible to utilize HTTU to get dirty pages on device
DMA during VFIO live migration?
If SMMU can track dirty pages, devices are not required to implement
additional dirty pages tracking to support VFIO live migration.
--
Thanks,
Xiang
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
^ permalink raw reply [flat|nested] 10+ messages in thread* Re: [RFC] Use SMMU HTTU for DMA dirty page tracking 2020-05-19 9:42 [RFC] Use SMMU HTTU for DMA dirty page tracking Xiang Zheng @ 2020-05-22 17:14 ` Jean-Philippe Brucker 2020-05-25 11:34 ` Xiang Zheng 0 siblings, 1 reply; 10+ messages in thread From: Jean-Philippe Brucker @ 2020-05-22 17:14 UTC (permalink / raw) To: Xiang Zheng Cc: maz, iommu, wangzhou1, prime.zeng, Will Deacon, kvmarm, linux-arm-kernel Hi, On Tue, May 19, 2020 at 05:42:55PM +0800, Xiang Zheng wrote: > Hi all, > > Is there any plan for enabling SMMU HTTU? Not outside of SVA, as far as I know. > I have seen the patch locates in the SVA series patch, which adds > support for HTTU: > https://www.spinics.net/lists/arm-kernel/msg798694.html > > HTTU reduces the number of access faults on SMMU fault queue > (permission faults also benifit from it). > > Besides reducing the faults, HTTU also helps to track dirty pages for > device DMA. Is it feasible to utilize HTTU to get dirty pages on device > DMA during VFIO live migration? As you know there is a VFIO interface for this under discussion: https://lore.kernel.org/kvm/1589781397-28368-1-git-send-email-kwankhede@nvidia.com/ It doesn't implement an internal API to communicate with the IOMMU driver about dirty pages. > If SMMU can track dirty pages, devices are not required to implement > additional dirty pages tracking to support VFIO live migration. It seems feasible, though tracking it in the device might be more efficient. I might have misunderstood but I think for live migration of the Intel NIC they trap guest accesses to the device and introspect its state to figure out which pages it is accessing. With HTTU I suppose (without much knowledge about live migration) that you'd need several new interfaces to the IOMMU drivers: * A way for VFIO to query HTTU support in the SMMU. There are some discussions about communicating more IOMMU capabilities through VFIO but no implementation yet. When HTTU isn't supported the DIRTY_PAGES bitmap would report all pages as they do now. * VFIO_IOMMU_DIRTY_PAGES_FLAG_START/STOP would clear the dirty bit for all VFIO mappings (which is going to take some time). There is a walker in io-pgtable for iova_to_phys() which could be extended. I suppose it's also possible to atomically switch the HA and HD bits in context descriptors. * VFIO_IOMMU_DIRTY_PAGES_FLAG_GET_BITMAP would query the dirty bit for all VFIO mappings. Thanks, Jean _______________________________________________ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [RFC] Use SMMU HTTU for DMA dirty page tracking 2020-05-22 17:14 ` Jean-Philippe Brucker @ 2020-05-25 11:34 ` Xiang Zheng 2020-05-27 3:27 ` Tian, Kevin 2020-05-27 6:46 ` Zengtao (B) 0 siblings, 2 replies; 10+ messages in thread From: Xiang Zheng @ 2020-05-25 11:34 UTC (permalink / raw) To: Jean-Philippe Brucker Cc: alex.williamson, Yan Zhao, maz, iommu, Kirti Wankhede, wangzhou1, prime.zeng, Will Deacon, kvmarm, linux-arm-kernel [+cc Kirti, Yan, Alex] On 2020/5/23 1:14, Jean-Philippe Brucker wrote: > Hi, > > On Tue, May 19, 2020 at 05:42:55PM +0800, Xiang Zheng wrote: >> Hi all, >> >> Is there any plan for enabling SMMU HTTU? > > Not outside of SVA, as far as I know. > >> I have seen the patch locates in the SVA series patch, which adds >> support for HTTU: >> https://www.spinics.net/lists/arm-kernel/msg798694.html >> >> HTTU reduces the number of access faults on SMMU fault queue >> (permission faults also benifit from it). >> >> Besides reducing the faults, HTTU also helps to track dirty pages for >> device DMA. Is it feasible to utilize HTTU to get dirty pages on device >> DMA during VFIO live migration? > > As you know there is a VFIO interface for this under discussion: > https://lore.kernel.org/kvm/1589781397-28368-1-git-send-email-kwankhede@nvidia.com/ > It doesn't implement an internal API to communicate with the IOMMU driver > about dirty pages. > >> If SMMU can track dirty pages, devices are not required to implement >> additional dirty pages tracking to support VFIO live migration. > > It seems feasible, though tracking it in the device might be more > efficient. I might have misunderstood but I think for live migration of > the Intel NIC they trap guest accesses to the device and introspect its > state to figure out which pages it is accessing. > > With HTTU I suppose (without much knowledge about live migration) that > you'd need several new interfaces to the IOMMU drivers: > > * A way for VFIO to query HTTU support in the SMMU. There are some > discussions about communicating more IOMMU capabilities through VFIO but > no implementation yet. When HTTU isn't supported the DIRTY_PAGES bitmap > would report all pages as they do now. > > * VFIO_IOMMU_DIRTY_PAGES_FLAG_START/STOP would clear the dirty bit > for all VFIO mappings (which is going to take some time). There is a > walker in io-pgtable for iova_to_phys() which could be extended. I > suppose it's also possible to atomically switch the HA and HD bits in > context descriptors. Maybe we need not switch HA and HD bits, just turn on them all the time? > > * VFIO_IOMMU_DIRTY_PAGES_FLAG_GET_BITMAP would query the dirty bit for all > VFIO mappings. > I think we need to consider the case of IOMMU dirty pages logging. We want to test Kirti's VFIO migration patches combined with SMMU HTTU, any suggestions? -- Thanks, Xiang _______________________________________________ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm ^ permalink raw reply [flat|nested] 10+ messages in thread
* RE: [RFC] Use SMMU HTTU for DMA dirty page tracking 2020-05-25 11:34 ` Xiang Zheng @ 2020-05-27 3:27 ` Tian, Kevin 2020-05-27 6:45 ` Xiang Zheng 2020-05-27 6:46 ` Zengtao (B) 1 sibling, 1 reply; 10+ messages in thread From: Tian, Kevin @ 2020-05-27 3:27 UTC (permalink / raw) To: Xiang Zheng, Jean-Philippe Brucker Cc: Zhao, Yan Y, maz, iommu, Kirti Wankhede, alex.williamson, linux-arm-kernel, prime.zeng, Will Deacon, kvmarm > From: Xiang Zheng > Sent: Monday, May 25, 2020 7:34 PM > > [+cc Kirti, Yan, Alex] > > On 2020/5/23 1:14, Jean-Philippe Brucker wrote: > > Hi, > > > > On Tue, May 19, 2020 at 05:42:55PM +0800, Xiang Zheng wrote: > >> Hi all, > >> > >> Is there any plan for enabling SMMU HTTU? > > > > Not outside of SVA, as far as I know. > > > > >> I have seen the patch locates in the SVA series patch, which adds > >> support for HTTU: > >> https://www.spinics.net/lists/arm-kernel/msg798694.html > >> > >> HTTU reduces the number of access faults on SMMU fault queue > >> (permission faults also benifit from it). > >> > >> Besides reducing the faults, HTTU also helps to track dirty pages for > >> device DMA. Is it feasible to utilize HTTU to get dirty pages on device > >> DMA during VFIO live migration? > > > > As you know there is a VFIO interface for this under discussion: > > https://lore.kernel.org/kvm/1589781397-28368-1-git-send-email- > kwankhede@nvidia.com/ > > It doesn't implement an internal API to communicate with the IOMMU > driver > > about dirty pages. We plan to add such API later, e.g. to utilize A/D bit in VT-d 2nd-level page tables (Rev 3.0). > > > > >> If SMMU can track dirty pages, devices are not required to implement > >> additional dirty pages tracking to support VFIO live migration. > > > > It seems feasible, though tracking it in the device might be more > > efficient. I might have misunderstood but I think for live migration of > > the Intel NIC they trap guest accesses to the device and introspect its > > state to figure out which pages it is accessing. Does HTTU implement A/D-like mechanism in SMMU page tables, or just report dirty pages in a log buffer? Either way tracking dirty pages in IOMMU side is generic thus doesn't require device-specific tweak like in Intel NIC. Thanks kevin > > > > With HTTU I suppose (without much knowledge about live migration) that > > you'd need several new interfaces to the IOMMU drivers: > > > > * A way for VFIO to query HTTU support in the SMMU. There are some > > discussions about communicating more IOMMU capabilities through VFIO > but > > no implementation yet. When HTTU isn't supported the DIRTY_PAGES > bitmap > > would report all pages as they do now. > > > > * VFIO_IOMMU_DIRTY_PAGES_FLAG_START/STOP would clear the dirty bit > > for all VFIO mappings (which is going to take some time). There is a > > walker in io-pgtable for iova_to_phys() which could be extended. I > > suppose it's also possible to atomically switch the HA and HD bits in > > context descriptors. > > Maybe we need not switch HA and HD bits, just turn on them all the time? > > > > > * VFIO_IOMMU_DIRTY_PAGES_FLAG_GET_BITMAP would query the dirty > bit for all > > VFIO mappings. > > > > I think we need to consider the case of IOMMU dirty pages logging. We want > to test Kirti's VFIO migration patches combined with SMMU HTTU, any > suggestions? > > -- > Thanks, > Xiang > > _______________________________________________ > iommu mailing list > iommu@lists.linux-foundation.org > https://lists.linuxfoundation.org/mailman/listinfo/iommu _______________________________________________ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [RFC] Use SMMU HTTU for DMA dirty page tracking 2020-05-27 3:27 ` Tian, Kevin @ 2020-05-27 6:45 ` Xiang Zheng 2020-05-27 8:40 ` Tian, Kevin 0 siblings, 1 reply; 10+ messages in thread From: Xiang Zheng @ 2020-05-27 6:45 UTC (permalink / raw) To: Tian, Kevin, Jean-Philippe Brucker Cc: Zhao, Yan Y, maz, iommu, Kirti Wankhede, alex.williamson, linux-arm-kernel, prime.zeng, Will Deacon, kvmarm On 2020/5/27 11:27, Tian, Kevin wrote: >> From: Xiang Zheng >> Sent: Monday, May 25, 2020 7:34 PM >> >> [+cc Kirti, Yan, Alex] >> >> On 2020/5/23 1:14, Jean-Philippe Brucker wrote: >>> Hi, >>> >>> On Tue, May 19, 2020 at 05:42:55PM +0800, Xiang Zheng wrote: >>>> Hi all, >>>> >>>> Is there any plan for enabling SMMU HTTU? >>> >>> Not outside of SVA, as far as I know. >>> >> >>>> I have seen the patch locates in the SVA series patch, which adds >>>> support for HTTU: >>>> https://www.spinics.net/lists/arm-kernel/msg798694.html >>>> >>>> HTTU reduces the number of access faults on SMMU fault queue >>>> (permission faults also benifit from it). >>>> >>>> Besides reducing the faults, HTTU also helps to track dirty pages for >>>> device DMA. Is it feasible to utilize HTTU to get dirty pages on device >>>> DMA during VFIO live migration? >>> >>> As you know there is a VFIO interface for this under discussion: >>> https://lore.kernel.org/kvm/1589781397-28368-1-git-send-email- >> kwankhede@nvidia.com/ >>> It doesn't implement an internal API to communicate with the IOMMU >> driver >>> about dirty pages. > > We plan to add such API later, e.g. to utilize A/D bit in VT-d 2nd-level > page tables (Rev 3.0). > Thank you, Kevin. When will you send this series patches? Maybe(Hope) we can also support hardware-based dirty pages tracking via common APIs based on your patches. :) >> >>> >>>> If SMMU can track dirty pages, devices are not required to implement >>>> additional dirty pages tracking to support VFIO live migration. >>> >>> It seems feasible, though tracking it in the device might be more >>> efficient. I might have misunderstood but I think for live migration of >>> the Intel NIC they trap guest accesses to the device and introspect its >>> state to figure out which pages it is accessing. > > Does HTTU implement A/D-like mechanism in SMMU page tables, or just > report dirty pages in a log buffer? Either way tracking dirty pages in IOMMU > side is generic thus doesn't require device-specific tweak like in Intel NIC. > Currently HTTU just implement A/D-like mechanism in SMMU page tables. We certainly expect SMMU can also implement PML-like feature so that we can avoid walking the whole page table to get the dirty pages. By the way, I'm not sure whether HTTU or SLAD can help for mediated deivce. -- Thanks, Xiang _______________________________________________ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm ^ permalink raw reply [flat|nested] 10+ messages in thread
* RE: [RFC] Use SMMU HTTU for DMA dirty page tracking 2020-05-27 6:45 ` Xiang Zheng @ 2020-05-27 8:40 ` Tian, Kevin 2020-05-27 9:14 ` Jean-Philippe Brucker 0 siblings, 1 reply; 10+ messages in thread From: Tian, Kevin @ 2020-05-27 8:40 UTC (permalink / raw) To: Xiang Zheng, Jean-Philippe Brucker Cc: Zhao, Yan Y, maz, iommu, Kirti Wankhede, alex.williamson, linux-arm-kernel, prime.zeng, Will Deacon, kvmarm > From: Xiang Zheng <zhengxiang9@huawei.com> > Sent: Wednesday, May 27, 2020 2:45 PM > > > On 2020/5/27 11:27, Tian, Kevin wrote: > >> From: Xiang Zheng > >> Sent: Monday, May 25, 2020 7:34 PM > >> > >> [+cc Kirti, Yan, Alex] > >> > >> On 2020/5/23 1:14, Jean-Philippe Brucker wrote: > >>> Hi, > >>> > >>> On Tue, May 19, 2020 at 05:42:55PM +0800, Xiang Zheng wrote: > >>>> Hi all, > >>>> > >>>> Is there any plan for enabling SMMU HTTU? > >>> > >>> Not outside of SVA, as far as I know. > >>> > >> > >>>> I have seen the patch locates in the SVA series patch, which adds > >>>> support for HTTU: > >>>> https://www.spinics.net/lists/arm-kernel/msg798694.html > >>>> > >>>> HTTU reduces the number of access faults on SMMU fault queue > >>>> (permission faults also benifit from it). > >>>> > >>>> Besides reducing the faults, HTTU also helps to track dirty pages for > >>>> device DMA. Is it feasible to utilize HTTU to get dirty pages on device > >>>> DMA during VFIO live migration? > >>> > >>> As you know there is a VFIO interface for this under discussion: > >>> https://lore.kernel.org/kvm/1589781397-28368-1-git-send-email- > >> kwankhede@nvidia.com/ > >>> It doesn't implement an internal API to communicate with the IOMMU > >> driver > >>> about dirty pages. > > > > We plan to add such API later, e.g. to utilize A/D bit in VT-d 2nd-level > > page tables (Rev 3.0). > > > > Thank you, Kevin. > > When will you send this series patches? Maybe(Hope) we can also support > hardware-based dirty pages tracking via common APIs based on your > patches. :) Yan is working with Kirti on basic live migration support now. After that part is done, we will start working on A/D bit support. Yes, common APIs are definitely the goal here. > > >> > >>> > >>>> If SMMU can track dirty pages, devices are not required to implement > >>>> additional dirty pages tracking to support VFIO live migration. > >>> > >>> It seems feasible, though tracking it in the device might be more > >>> efficient. I might have misunderstood but I think for live migration of > >>> the Intel NIC they trap guest accesses to the device and introspect its > >>> state to figure out which pages it is accessing. > > > > Does HTTU implement A/D-like mechanism in SMMU page tables, or just > > report dirty pages in a log buffer? Either way tracking dirty pages in IOMMU > > side is generic thus doesn't require device-specific tweak like in Intel NIC. > > > > Currently HTTU just implement A/D-like mechanism in SMMU page tables. > We certainly > expect SMMU can also implement PML-like feature so that we can avoid > walking the > whole page table to get the dirty pages. Is there a link to HTTU introduction? > > By the way, I'm not sure whether HTTU or SLAD can help for mediated deivce. > A/D bit applies to mediated device on VT-d. Thanks Kevin _______________________________________________ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [RFC] Use SMMU HTTU for DMA dirty page tracking 2020-05-27 8:40 ` Tian, Kevin @ 2020-05-27 9:14 ` Jean-Philippe Brucker 2021-02-05 3:30 ` Keqian Zhu 0 siblings, 1 reply; 10+ messages in thread From: Jean-Philippe Brucker @ 2020-05-27 9:14 UTC (permalink / raw) To: Tian, Kevin Cc: Zhao, Yan Y, maz, alex.williamson, iommu, linux-arm-kernel, prime.zeng, Kirti Wankhede, Will Deacon, kvmarm On Wed, May 27, 2020 at 08:40:47AM +0000, Tian, Kevin wrote: > > From: Xiang Zheng <zhengxiang9@huawei.com> > > Sent: Wednesday, May 27, 2020 2:45 PM > > > > > > On 2020/5/27 11:27, Tian, Kevin wrote: > > >> From: Xiang Zheng > > >> Sent: Monday, May 25, 2020 7:34 PM > > >> > > >> [+cc Kirti, Yan, Alex] > > >> > > >> On 2020/5/23 1:14, Jean-Philippe Brucker wrote: > > >>> Hi, > > >>> > > >>> On Tue, May 19, 2020 at 05:42:55PM +0800, Xiang Zheng wrote: > > >>>> Hi all, > > >>>> > > >>>> Is there any plan for enabling SMMU HTTU? > > >>> > > >>> Not outside of SVA, as far as I know. > > >>> > > >> > > >>>> I have seen the patch locates in the SVA series patch, which adds > > >>>> support for HTTU: > > >>>> https://www.spinics.net/lists/arm-kernel/msg798694.html > > >>>> > > >>>> HTTU reduces the number of access faults on SMMU fault queue > > >>>> (permission faults also benifit from it). > > >>>> > > >>>> Besides reducing the faults, HTTU also helps to track dirty pages for > > >>>> device DMA. Is it feasible to utilize HTTU to get dirty pages on device > > >>>> DMA during VFIO live migration? > > >>> > > >>> As you know there is a VFIO interface for this under discussion: > > >>> https://lore.kernel.org/kvm/1589781397-28368-1-git-send-email- > > >> kwankhede@nvidia.com/ > > >>> It doesn't implement an internal API to communicate with the IOMMU > > >> driver > > >>> about dirty pages. > > > > > > We plan to add such API later, e.g. to utilize A/D bit in VT-d 2nd-level > > > page tables (Rev 3.0). > > > > > > > Thank you, Kevin. > > > > When will you send this series patches? Maybe(Hope) we can also support > > hardware-based dirty pages tracking via common APIs based on your > > patches. :) > > Yan is working with Kirti on basic live migration support now. After that > part is done, we will start working on A/D bit support. Yes, common APIs > are definitely the goal here. > > > > > >> > > >>> > > >>>> If SMMU can track dirty pages, devices are not required to implement > > >>>> additional dirty pages tracking to support VFIO live migration. > > >>> > > >>> It seems feasible, though tracking it in the device might be more > > >>> efficient. I might have misunderstood but I think for live migration of > > >>> the Intel NIC they trap guest accesses to the device and introspect its > > >>> state to figure out which pages it is accessing. > > > > > > Does HTTU implement A/D-like mechanism in SMMU page tables, or just > > > report dirty pages in a log buffer? Either way tracking dirty pages in IOMMU > > > side is generic thus doesn't require device-specific tweak like in Intel NIC. > > > > > > > Currently HTTU just implement A/D-like mechanism in SMMU page tables. > > We certainly > > expect SMMU can also implement PML-like feature so that we can avoid > > walking the > > whole page table to get the dirty pages. There is no reporting of dirty pages in log buffer. It might be possible to do software logging based on PRI or Stall, but that requires special support in the endpoint as well as the SMMU. > Is there a link to HTTU introduction? I don't know any gentle introduction, but there are sections D5.4.11 "Hardware management of the Access flag and dirty state" in the ARM Architecture Reference Manual (DDI0487E), and section 3.13 "Translation table entries and Access/Dirty flags" in the SMMU specification (IHI0070C). HTTU stands for "Hardware Translation Table Update". In short, when HTTU is enabled, the SMMU translation performs an atomic read-modify-write on the leaf translation table descriptor, setting some bits depending on the type of memory access. This can be enabled independently on both stage-1 and stage-2 tables (equivalent to your 1st and 2nd page tables levels, I think). Thanks, Jean _______________________________________________ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [RFC] Use SMMU HTTU for DMA dirty page tracking 2020-05-27 9:14 ` Jean-Philippe Brucker @ 2021-02-05 3:30 ` Keqian Zhu 2021-02-05 5:21 ` Tian, Kevin 0 siblings, 1 reply; 10+ messages in thread From: Keqian Zhu @ 2021-02-05 3:30 UTC (permalink / raw) To: Jean-Philippe Brucker, Tian, Kevin Cc: Zhao, Yan Y, maz, iommu, Kirti Wankhede, alex.williamson, prime.zeng, Will Deacon, kvmarm, linux-arm-kernel Hi Jean and Kevin, FYI, I have send out the SMMUv3 HTTU support for DMA dirty tracking[1] a week ago. Thanks, Keqian [1] https://lore.kernel.org/linux-iommu/20210128151742.18840-1-zhukeqian1@huawei.com/ On 2020/5/27 17:14, Jean-Philippe Brucker wrote: > On Wed, May 27, 2020 at 08:40:47AM +0000, Tian, Kevin wrote: >>> From: Xiang Zheng <zhengxiang9@huawei.com> >>> Sent: Wednesday, May 27, 2020 2:45 PM >>> >>> >>> On 2020/5/27 11:27, Tian, Kevin wrote: >>>>> From: Xiang Zheng >>>>> Sent: Monday, May 25, 2020 7:34 PM >>>>> >>>>> [+cc Kirti, Yan, Alex] >>>>> >>>>> On 2020/5/23 1:14, Jean-Philippe Brucker wrote: >>>>>> Hi, >>>>>> >>>>>> On Tue, May 19, 2020 at 05:42:55PM +0800, Xiang Zheng wrote: >>>>>>> Hi all, >>>>>>> >>>>>>> Is there any plan for enabling SMMU HTTU? >>>>>> >>>>>> Not outside of SVA, as far as I know. >>>>>> >>>>> >>>>>>> I have seen the patch locates in the SVA series patch, which adds >>>>>>> support for HTTU: >>>>>>> https://www.spinics.net/lists/arm-kernel/msg798694.html >>>>>>> >>>>>>> HTTU reduces the number of access faults on SMMU fault queue >>>>>>> (permission faults also benifit from it). >>>>>>> >>>>>>> Besides reducing the faults, HTTU also helps to track dirty pages for >>>>>>> device DMA. Is it feasible to utilize HTTU to get dirty pages on device >>>>>>> DMA during VFIO live migration? >>>>>> >>>>>> As you know there is a VFIO interface for this under discussion: >>>>>> https://lore.kernel.org/kvm/1589781397-28368-1-git-send-email- >>>>> kwankhede@nvidia.com/ >>>>>> It doesn't implement an internal API to communicate with the IOMMU >>>>> driver >>>>>> about dirty pages. >>>> >>>> We plan to add such API later, e.g. to utilize A/D bit in VT-d 2nd-level >>>> page tables (Rev 3.0). >>>> >>> >>> Thank you, Kevin. >>> >>> When will you send this series patches? Maybe(Hope) we can also support >>> hardware-based dirty pages tracking via common APIs based on your >>> patches. :) >> >> Yan is working with Kirti on basic live migration support now. After that >> part is done, we will start working on A/D bit support. Yes, common APIs >> are definitely the goal here. >> >>> >>>>> >>>>>> >>>>>>> If SMMU can track dirty pages, devices are not required to implement >>>>>>> additional dirty pages tracking to support VFIO live migration. >>>>>> >>>>>> It seems feasible, though tracking it in the device might be more >>>>>> efficient. I might have misunderstood but I think for live migration of >>>>>> the Intel NIC they trap guest accesses to the device and introspect its >>>>>> state to figure out which pages it is accessing. >>>> >>>> Does HTTU implement A/D-like mechanism in SMMU page tables, or just >>>> report dirty pages in a log buffer? Either way tracking dirty pages in IOMMU >>>> side is generic thus doesn't require device-specific tweak like in Intel NIC. >>>> >>> >>> Currently HTTU just implement A/D-like mechanism in SMMU page tables. >>> We certainly >>> expect SMMU can also implement PML-like feature so that we can avoid >>> walking the >>> whole page table to get the dirty pages. > > There is no reporting of dirty pages in log buffer. It might be possible > to do software logging based on PRI or Stall, but that requires special > support in the endpoint as well as the SMMU. > >> Is there a link to HTTU introduction? > > I don't know any gentle introduction, but there are sections D5.4.11 > "Hardware management of the Access flag and dirty state" in the ARM > Architecture Reference Manual (DDI0487E), and section 3.13 "Translation > table entries and Access/Dirty flags" in the SMMU specification > (IHI0070C). HTTU stands for "Hardware Translation Table Update". > > In short, when HTTU is enabled, the SMMU translation performs an atomic > read-modify-write on the leaf translation table descriptor, setting some > bits depending on the type of memory access. This can be enabled > independently on both stage-1 and stage-2 tables (equivalent to your 1st > and 2nd page tables levels, I think). > > Thanks, > Jean > _______________________________________________ > kvmarm mailing list > kvmarm@lists.cs.columbia.edu > https://lists.cs.columbia.edu/mailman/listinfo/kvmarm > . > _______________________________________________ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm ^ permalink raw reply [flat|nested] 10+ messages in thread
* RE: [RFC] Use SMMU HTTU for DMA dirty page tracking 2021-02-05 3:30 ` Keqian Zhu @ 2021-02-05 5:21 ` Tian, Kevin 0 siblings, 0 replies; 10+ messages in thread From: Tian, Kevin @ 2021-02-05 5:21 UTC (permalink / raw) To: Keqian Zhu, Jean-Philippe Brucker Cc: Sun, Yi Y, Zhao, Yan Y, maz, iommu, Kirti Wankhede, alex.williamson, prime.zeng, Will Deacon, kvmarm, linux-arm-kernel > From: Keqian Zhu <zhukeqian1@huawei.com> > Sent: Friday, February 5, 2021 11:31 AM > > Hi Jean and Kevin, > > FYI, I have send out the SMMUv3 HTTU support for DMA dirty tracking[1] a > week ago. > > Thanks, > Keqian > > [1] https://lore.kernel.org/linux-iommu/20210128151742.18840-1- > zhukeqian1@huawei.com/ Thanks for the pointer. We will take a look. > > On 2020/5/27 17:14, Jean-Philippe Brucker wrote: > > On Wed, May 27, 2020 at 08:40:47AM +0000, Tian, Kevin wrote: > >>> From: Xiang Zheng <zhengxiang9@huawei.com> > >>> Sent: Wednesday, May 27, 2020 2:45 PM > >>> > >>> > >>> On 2020/5/27 11:27, Tian, Kevin wrote: > >>>>> From: Xiang Zheng > >>>>> Sent: Monday, May 25, 2020 7:34 PM > >>>>> > >>>>> [+cc Kirti, Yan, Alex] > >>>>> > >>>>> On 2020/5/23 1:14, Jean-Philippe Brucker wrote: > >>>>>> Hi, > >>>>>> > >>>>>> On Tue, May 19, 2020 at 05:42:55PM +0800, Xiang Zheng wrote: > >>>>>>> Hi all, > >>>>>>> > >>>>>>> Is there any plan for enabling SMMU HTTU? > >>>>>> > >>>>>> Not outside of SVA, as far as I know. > >>>>>> > >>>>> > >>>>>>> I have seen the patch locates in the SVA series patch, which adds > >>>>>>> support for HTTU: > >>>>>>> https://www.spinics.net/lists/arm-kernel/msg798694.html > >>>>>>> > >>>>>>> HTTU reduces the number of access faults on SMMU fault queue > >>>>>>> (permission faults also benifit from it). > >>>>>>> > >>>>>>> Besides reducing the faults, HTTU also helps to track dirty pages for > >>>>>>> device DMA. Is it feasible to utilize HTTU to get dirty pages on device > >>>>>>> DMA during VFIO live migration? > >>>>>> > >>>>>> As you know there is a VFIO interface for this under discussion: > >>>>>> https://lore.kernel.org/kvm/1589781397-28368-1-git-send-email- > >>>>> kwankhede@nvidia.com/ > >>>>>> It doesn't implement an internal API to communicate with the > IOMMU > >>>>> driver > >>>>>> about dirty pages. > >>>> > >>>> We plan to add such API later, e.g. to utilize A/D bit in VT-d 2nd-level > >>>> page tables (Rev 3.0). > >>>> > >>> > >>> Thank you, Kevin. > >>> > >>> When will you send this series patches? Maybe(Hope) we can also > support > >>> hardware-based dirty pages tracking via common APIs based on your > >>> patches. :) > >> > >> Yan is working with Kirti on basic live migration support now. After that > >> part is done, we will start working on A/D bit support. Yes, common APIs > >> are definitely the goal here. > >> > >>> > >>>>> > >>>>>> > >>>>>>> If SMMU can track dirty pages, devices are not required to > implement > >>>>>>> additional dirty pages tracking to support VFIO live migration. > >>>>>> > >>>>>> It seems feasible, though tracking it in the device might be more > >>>>>> efficient. I might have misunderstood but I think for live migration of > >>>>>> the Intel NIC they trap guest accesses to the device and introspect its > >>>>>> state to figure out which pages it is accessing. > >>>> > >>>> Does HTTU implement A/D-like mechanism in SMMU page tables, or > just > >>>> report dirty pages in a log buffer? Either way tracking dirty pages in > IOMMU > >>>> side is generic thus doesn't require device-specific tweak like in Intel NIC. > >>>> > >>> > >>> Currently HTTU just implement A/D-like mechanism in SMMU page > tables. > >>> We certainly > >>> expect SMMU can also implement PML-like feature so that we can avoid > >>> walking the > >>> whole page table to get the dirty pages. > > > > There is no reporting of dirty pages in log buffer. It might be possible > > to do software logging based on PRI or Stall, but that requires special > > support in the endpoint as well as the SMMU. > > > >> Is there a link to HTTU introduction? > > > > I don't know any gentle introduction, but there are sections D5.4.11 > > "Hardware management of the Access flag and dirty state" in the ARM > > Architecture Reference Manual (DDI0487E), and section 3.13 "Translation > > table entries and Access/Dirty flags" in the SMMU specification > > (IHI0070C). HTTU stands for "Hardware Translation Table Update". > > > > In short, when HTTU is enabled, the SMMU translation performs an atomic > > read-modify-write on the leaf translation table descriptor, setting some > > bits depending on the type of memory access. This can be enabled > > independently on both stage-1 and stage-2 tables (equivalent to your 1st > > and 2nd page tables levels, I think). > > > > Thanks, > > Jean > > _______________________________________________ > > kvmarm mailing list > > kvmarm@lists.cs.columbia.edu > > https://lists.cs.columbia.edu/mailman/listinfo/kvmarm > > . > > _______________________________________________ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm ^ permalink raw reply [flat|nested] 10+ messages in thread
* RE: [RFC] Use SMMU HTTU for DMA dirty page tracking 2020-05-25 11:34 ` Xiang Zheng 2020-05-27 3:27 ` Tian, Kevin @ 2020-05-27 6:46 ` Zengtao (B) 1 sibling, 0 replies; 10+ messages in thread From: Zengtao (B) @ 2020-05-27 6:46 UTC (permalink / raw) To: zhengxiang (A), Jean-Philippe Brucker Cc: alex.williamson, Yan Zhao, maz, iommu, Kirti Wankhede, Wangzhou (B), Will Deacon, kvmarm, linux-arm-kernel > -----Original Message----- > From: zhengxiang (A) > Sent: Monday, May 25, 2020 7:34 PM > To: Jean-Philippe Brucker > Cc: linux-arm-kernel@lists.infradead.org; kvmarm@lists.cs.columbia.edu; > Will Deacon; Wanghaibin (D); Zengtao (B); maz@kernel.org; James Morse; > julien.thierry.kdev@gmail.com; Suzuki K Poulose; Wangzhou (B); > iommu@lists.linux-foundation.org; Kirti Wankhede; Yan Zhao; > alex.williamson@redhat.com > Subject: Re: [RFC] Use SMMU HTTU for DMA dirty page tracking > > [+cc Kirti, Yan, Alex] > > On 2020/5/23 1:14, Jean-Philippe Brucker wrote: > > Hi, > > > > On Tue, May 19, 2020 at 05:42:55PM +0800, Xiang Zheng wrote: > >> Hi all, > >> > >> Is there any plan for enabling SMMU HTTU? > > > > Not outside of SVA, as far as I know. > > > > >> I have seen the patch locates in the SVA series patch, which adds > >> support for HTTU: > >> https://www.spinics.net/lists/arm-kernel/msg798694.html > >> > >> HTTU reduces the number of access faults on SMMU fault queue > >> (permission faults also benifit from it). > >> > >> Besides reducing the faults, HTTU also helps to track dirty pages for > >> device DMA. Is it feasible to utilize HTTU to get dirty pages on device > >> DMA during VFIO live migration? > > > > As you know there is a VFIO interface for this under discussion: > > > https://lore.kernel.org/kvm/1589781397-28368-1-git-send-email-kwankhe > de@nvidia.com/ > > It doesn't implement an internal API to communicate with the IOMMU > driver > > about dirty pages. > > > > >> If SMMU can track dirty pages, devices are not required to implement > >> additional dirty pages tracking to support VFIO live migration. > > > > It seems feasible, though tracking it in the device might be more > > efficient. I might have misunderstood but I think for live migration of > > the Intel NIC they trap guest accesses to the device and introspect its > > state to figure out which pages it is accessing. > > > > With HTTU I suppose (without much knowledge about live migration) > that > > you'd need several new interfaces to the IOMMU drivers: > > > > * A way for VFIO to query HTTU support in the SMMU. There are some > > discussions about communicating more IOMMU capabilities through > VFIO but > > no implementation yet. When HTTU isn't supported the DIRTY_PAGES > bitmap > > would report all pages as they do now. > > > > * VFIO_IOMMU_DIRTY_PAGES_FLAG_START/STOP would clear the dirty > bit > > for all VFIO mappings (which is going to take some time). There is a > > walker in io-pgtable for iova_to_phys() which could be extended. I > > suppose it's also possible to atomically switch the HA and HD bits in > > context descriptors. > > Maybe we need not switch HA and HD bits, just turn on them all the time? I think we need START/STOP, start is called when migration is started and STOP called when migration finished. I think you mean that during the migration(between START and STOP), HA and HD functionality is always turned on. > > > > > * VFIO_IOMMU_DIRTY_PAGES_FLAG_GET_BITMAP would query the > dirty bit for all > > VFIO mappings. > > We need a clear during the query which mean we have to reset the dirty status after a query. > > I think we need to consider the case of IOMMU dirty pages logging. We > want > to test Kirti's VFIO migration patches combined with SMMU HTTU, any > suggestions? @kirti @yan @alex As we know, you have worked on this feature for a long time, and it seems that everything is going very well now, thanks very much for your VFIO migration framework, it really helps a lot, and we want to start with SMMU HTTU enabled based on your jobs, it's kind of hardware dirty page tracking, and expected to bring us better performance, any suggestions? Thank you. Zengtao _______________________________________________ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm ^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2021-02-05 5:21 UTC | newest] Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2020-05-19 9:42 [RFC] Use SMMU HTTU for DMA dirty page tracking Xiang Zheng 2020-05-22 17:14 ` Jean-Philippe Brucker 2020-05-25 11:34 ` Xiang Zheng 2020-05-27 3:27 ` Tian, Kevin 2020-05-27 6:45 ` Xiang Zheng 2020-05-27 8:40 ` Tian, Kevin 2020-05-27 9:14 ` Jean-Philippe Brucker 2021-02-05 3:30 ` Keqian Zhu 2021-02-05 5:21 ` Tian, Kevin 2020-05-27 6:46 ` Zengtao (B)
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).