* [PATCH V2 RESEND 0/2] vDPA/ifcvf: implement doorbell mapping feature @ 2021-05-31 7:33 Zhu Lingshan 2021-05-31 7:33 ` [PATCH V2 RESEND 1/2] vDPA/ifcvf: record virtio notify base Zhu Lingshan 2021-05-31 7:33 ` [PATCH V2 RESEND 2/2] vDPA/ifcvf: implement doorbell mapping for ifcvf Zhu Lingshan 0 siblings, 2 replies; 6+ messages in thread From: Zhu Lingshan @ 2021-05-31 7:33 UTC (permalink / raw) To: jasowang, mst; +Cc: virtualization, netdev, kvm, Zhu Lingshan This series implements doorbell mapping feature for ifcvf. Please help review Thanks! Chagnes from V1: calculate the doorbell address per vq than per device(Jason) let upper layer driver decide how to use the non page_size aligned doorbell(Jason) Zhu Lingshan (2): vDPA/ifcvf: record virtio notify base vDPA/ifcvf: implement doorbell mapping for ifcvf drivers/vdpa/ifcvf/ifcvf_base.c | 4 ++++ drivers/vdpa/ifcvf/ifcvf_base.h | 2 ++ drivers/vdpa/ifcvf/ifcvf_main.c | 17 +++++++++++++++++ 3 files changed, 23 insertions(+) -- 2.27.0 ^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH V2 RESEND 1/2] vDPA/ifcvf: record virtio notify base 2021-05-31 7:33 [PATCH V2 RESEND 0/2] vDPA/ifcvf: implement doorbell mapping feature Zhu Lingshan @ 2021-05-31 7:33 ` Zhu Lingshan 2021-05-31 7:33 ` [PATCH V2 RESEND 2/2] vDPA/ifcvf: implement doorbell mapping for ifcvf Zhu Lingshan 1 sibling, 0 replies; 6+ messages in thread From: Zhu Lingshan @ 2021-05-31 7:33 UTC (permalink / raw) To: jasowang, mst; +Cc: virtualization, netdev, kvm, Zhu Lingshan This commit records virtio notify base physical addr and calculate doorbell physical address for vqs. Signed-off-by: Zhu Lingshan <lingshan.zhu@intel.com> --- drivers/vdpa/ifcvf/ifcvf_base.c | 4 ++++ drivers/vdpa/ifcvf/ifcvf_base.h | 2 ++ 2 files changed, 6 insertions(+) diff --git a/drivers/vdpa/ifcvf/ifcvf_base.c b/drivers/vdpa/ifcvf/ifcvf_base.c index 1a661ab45af5..6e197fe0fcf9 100644 --- a/drivers/vdpa/ifcvf/ifcvf_base.c +++ b/drivers/vdpa/ifcvf/ifcvf_base.c @@ -133,6 +133,8 @@ int ifcvf_init_hw(struct ifcvf_hw *hw, struct pci_dev *pdev) &hw->notify_off_multiplier); hw->notify_bar = cap.bar; hw->notify_base = get_cap_addr(hw, &cap); + hw->notify_base_pa = pci_resource_start(pdev, cap.bar) + + le32_to_cpu(cap.offset); IFCVF_DBG(pdev, "hw->notify_base = %p\n", hw->notify_base); break; @@ -161,6 +163,8 @@ int ifcvf_init_hw(struct ifcvf_hw *hw, struct pci_dev *pdev) notify_off = ifc_ioread16(&hw->common_cfg->queue_notify_off); hw->vring[i].notify_addr = hw->notify_base + notify_off * hw->notify_off_multiplier; + hw->vring[i].notify_pa = hw->notify_base_pa + + notify_off * hw->notify_off_multiplier; } hw->lm_cfg = hw->base[IFCVF_LM_BAR]; diff --git a/drivers/vdpa/ifcvf/ifcvf_base.h b/drivers/vdpa/ifcvf/ifcvf_base.h index 0111bfdeb342..447f4ad9c0bf 100644 --- a/drivers/vdpa/ifcvf/ifcvf_base.h +++ b/drivers/vdpa/ifcvf/ifcvf_base.h @@ -73,6 +73,7 @@ struct vring_info { u16 last_avail_idx; bool ready; void __iomem *notify_addr; + phys_addr_t notify_pa; u32 irq; struct vdpa_callback cb; char msix_name[256]; @@ -87,6 +88,7 @@ struct ifcvf_hw { u8 notify_bar; /* Notificaiton bar address */ void __iomem *notify_base; + phys_addr_t notify_base_pa; u32 notify_off_multiplier; u64 req_features; u64 hw_features; -- 2.27.0 ^ permalink raw reply related [flat|nested] 6+ messages in thread
* [PATCH V2 RESEND 2/2] vDPA/ifcvf: implement doorbell mapping for ifcvf 2021-05-31 7:33 [PATCH V2 RESEND 0/2] vDPA/ifcvf: implement doorbell mapping feature Zhu Lingshan 2021-05-31 7:33 ` [PATCH V2 RESEND 1/2] vDPA/ifcvf: record virtio notify base Zhu Lingshan @ 2021-05-31 7:33 ` Zhu Lingshan 2021-05-31 7:56 ` Jason Wang 1 sibling, 1 reply; 6+ messages in thread From: Zhu Lingshan @ 2021-05-31 7:33 UTC (permalink / raw) To: jasowang, mst; +Cc: virtualization, netdev, kvm, Zhu Lingshan This commit implements doorbell mapping feature for ifcvf. This feature maps the notify page to userspace, to eliminate vmexit when kick a vq. Signed-off-by: Zhu Lingshan <lingshan.zhu@intel.com> --- drivers/vdpa/ifcvf/ifcvf_main.c | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/drivers/vdpa/ifcvf/ifcvf_main.c b/drivers/vdpa/ifcvf/ifcvf_main.c index ab0ab5cf0f6e..effb0e549135 100644 --- a/drivers/vdpa/ifcvf/ifcvf_main.c +++ b/drivers/vdpa/ifcvf/ifcvf_main.c @@ -413,6 +413,22 @@ static int ifcvf_vdpa_get_vq_irq(struct vdpa_device *vdpa_dev, return vf->vring[qid].irq; } +static struct vdpa_notification_area ifcvf_get_vq_notification(struct vdpa_device *vdpa_dev, + u16 idx) +{ + struct ifcvf_adapter *adapter = vdpa_to_adapter(vdpa_dev); + struct ifcvf_hw *vf = vdpa_to_vf(vdpa_dev); + struct pci_dev *pdev = adapter->pdev; + struct vdpa_notification_area area; + + area.addr = vf->vring[idx].notify_pa; + area.size = PAGE_SIZE; + if (area.addr % PAGE_SIZE) + IFCVF_DBG(pdev, "vq %u doorbell address is not PAGE_SIZE aligned\n", idx); + + return area; +} + /* * IFCVF currently does't have on-chip IOMMU, so not * implemented set_map()/dma_map()/dma_unmap() @@ -440,6 +456,7 @@ static const struct vdpa_config_ops ifc_vdpa_ops = { .get_config = ifcvf_vdpa_get_config, .set_config = ifcvf_vdpa_set_config, .set_config_cb = ifcvf_vdpa_set_config_cb, + .get_vq_notification = ifcvf_get_vq_notification, }; static int ifcvf_probe(struct pci_dev *pdev, const struct pci_device_id *id) -- 2.27.0 ^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH V2 RESEND 2/2] vDPA/ifcvf: implement doorbell mapping for ifcvf 2021-05-31 7:33 ` [PATCH V2 RESEND 2/2] vDPA/ifcvf: implement doorbell mapping for ifcvf Zhu Lingshan @ 2021-05-31 7:56 ` Jason Wang 2021-06-01 3:27 ` Zhu, Lingshan 0 siblings, 1 reply; 6+ messages in thread From: Jason Wang @ 2021-05-31 7:56 UTC (permalink / raw) To: Zhu Lingshan, mst; +Cc: virtualization, netdev, kvm 在 2021/5/31 下午3:33, Zhu Lingshan 写道: > This commit implements doorbell mapping feature for ifcvf. > This feature maps the notify page to userspace, to eliminate > vmexit when kick a vq. > > Signed-off-by: Zhu Lingshan <lingshan.zhu@intel.com> > --- > drivers/vdpa/ifcvf/ifcvf_main.c | 17 +++++++++++++++++ > 1 file changed, 17 insertions(+) > > diff --git a/drivers/vdpa/ifcvf/ifcvf_main.c b/drivers/vdpa/ifcvf/ifcvf_main.c > index ab0ab5cf0f6e..effb0e549135 100644 > --- a/drivers/vdpa/ifcvf/ifcvf_main.c > +++ b/drivers/vdpa/ifcvf/ifcvf_main.c > @@ -413,6 +413,22 @@ static int ifcvf_vdpa_get_vq_irq(struct vdpa_device *vdpa_dev, > return vf->vring[qid].irq; > } > > +static struct vdpa_notification_area ifcvf_get_vq_notification(struct vdpa_device *vdpa_dev, > + u16 idx) > +{ > + struct ifcvf_adapter *adapter = vdpa_to_adapter(vdpa_dev); > + struct ifcvf_hw *vf = vdpa_to_vf(vdpa_dev); > + struct pci_dev *pdev = adapter->pdev; > + struct vdpa_notification_area area; > + > + area.addr = vf->vring[idx].notify_pa; > + area.size = PAGE_SIZE; > + if (area.addr % PAGE_SIZE) > + IFCVF_DBG(pdev, "vq %u doorbell address is not PAGE_SIZE aligned\n", idx); Let's leave the decision to upper layer by: (see vp_vdpa_get_vq_notification) area.addr = notify_pa; area.size = notify_offset_multiplier; Thanks > + > + return area; > +} > + > /* > * IFCVF currently does't have on-chip IOMMU, so not > * implemented set_map()/dma_map()/dma_unmap() > @@ -440,6 +456,7 @@ static const struct vdpa_config_ops ifc_vdpa_ops = { > .get_config = ifcvf_vdpa_get_config, > .set_config = ifcvf_vdpa_set_config, > .set_config_cb = ifcvf_vdpa_set_config_cb, > + .get_vq_notification = ifcvf_get_vq_notification, > }; > > static int ifcvf_probe(struct pci_dev *pdev, const struct pci_device_id *id) ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH V2 RESEND 2/2] vDPA/ifcvf: implement doorbell mapping for ifcvf 2021-05-31 7:56 ` Jason Wang @ 2021-06-01 3:27 ` Zhu, Lingshan 2021-06-01 3:31 ` Jason Wang 0 siblings, 1 reply; 6+ messages in thread From: Zhu, Lingshan @ 2021-06-01 3:27 UTC (permalink / raw) To: Jason Wang, mst; +Cc: virtualization, netdev, kvm On 5/31/2021 3:56 PM, Jason Wang wrote: > > 在 2021/5/31 下午3:33, Zhu Lingshan 写道: >> This commit implements doorbell mapping feature for ifcvf. >> This feature maps the notify page to userspace, to eliminate >> vmexit when kick a vq. >> >> Signed-off-by: Zhu Lingshan <lingshan.zhu@intel.com> >> --- >> drivers/vdpa/ifcvf/ifcvf_main.c | 17 +++++++++++++++++ >> 1 file changed, 17 insertions(+) >> >> diff --git a/drivers/vdpa/ifcvf/ifcvf_main.c >> b/drivers/vdpa/ifcvf/ifcvf_main.c >> index ab0ab5cf0f6e..effb0e549135 100644 >> --- a/drivers/vdpa/ifcvf/ifcvf_main.c >> +++ b/drivers/vdpa/ifcvf/ifcvf_main.c >> @@ -413,6 +413,22 @@ static int ifcvf_vdpa_get_vq_irq(struct >> vdpa_device *vdpa_dev, >> return vf->vring[qid].irq; >> } >> +static struct vdpa_notification_area >> ifcvf_get_vq_notification(struct vdpa_device *vdpa_dev, >> + u16 idx) >> +{ >> + struct ifcvf_adapter *adapter = vdpa_to_adapter(vdpa_dev); >> + struct ifcvf_hw *vf = vdpa_to_vf(vdpa_dev); >> + struct pci_dev *pdev = adapter->pdev; >> + struct vdpa_notification_area area; >> + >> + area.addr = vf->vring[idx].notify_pa; >> + area.size = PAGE_SIZE; >> + if (area.addr % PAGE_SIZE) >> + IFCVF_DBG(pdev, "vq %u doorbell address is not PAGE_SIZE >> aligned\n", idx); > > > Let's leave the decision to upper layer by: (see > vp_vdpa_get_vq_notification) > > area.addr = notify_pa; > area.size = notify_offset_multiplier; > > Thanks Hi Jason, notify_offset_multiplier can be zero, means vqs share the same doorbell address, distinguished by qid. and in vdpa.c: if (vma->vm_end - vma->vm_start != notify.size) return -ENOTSUPP; so a zero size would cause this feature failure. mmap should work on at least a page, so if we really want "area.size = notify_offset_multiplier;" I think we should add some code in vdpa.c, like: if(!notify.size) notify.size = PAGE_SIZE; sounds good? Thanks Zhu Lingshan > > >> + >> + return area; >> +} >> + >> /* >> * IFCVF currently does't have on-chip IOMMU, so not >> * implemented set_map()/dma_map()/dma_unmap() >> @@ -440,6 +456,7 @@ static const struct vdpa_config_ops ifc_vdpa_ops = { >> .get_config = ifcvf_vdpa_get_config, >> .set_config = ifcvf_vdpa_set_config, >> .set_config_cb = ifcvf_vdpa_set_config_cb, >> + .get_vq_notification = ifcvf_get_vq_notification, >> }; >> static int ifcvf_probe(struct pci_dev *pdev, const struct >> pci_device_id *id) > ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH V2 RESEND 2/2] vDPA/ifcvf: implement doorbell mapping for ifcvf 2021-06-01 3:27 ` Zhu, Lingshan @ 2021-06-01 3:31 ` Jason Wang 0 siblings, 0 replies; 6+ messages in thread From: Jason Wang @ 2021-06-01 3:31 UTC (permalink / raw) To: Zhu, Lingshan, mst; +Cc: virtualization, netdev, kvm 在 2021/6/1 上午11:27, Zhu, Lingshan 写道: > > > On 5/31/2021 3:56 PM, Jason Wang wrote: >> >> 在 2021/5/31 下午3:33, Zhu Lingshan 写道: >>> This commit implements doorbell mapping feature for ifcvf. >>> This feature maps the notify page to userspace, to eliminate >>> vmexit when kick a vq. >>> >>> Signed-off-by: Zhu Lingshan <lingshan.zhu@intel.com> >>> --- >>> drivers/vdpa/ifcvf/ifcvf_main.c | 17 +++++++++++++++++ >>> 1 file changed, 17 insertions(+) >>> >>> diff --git a/drivers/vdpa/ifcvf/ifcvf_main.c >>> b/drivers/vdpa/ifcvf/ifcvf_main.c >>> index ab0ab5cf0f6e..effb0e549135 100644 >>> --- a/drivers/vdpa/ifcvf/ifcvf_main.c >>> +++ b/drivers/vdpa/ifcvf/ifcvf_main.c >>> @@ -413,6 +413,22 @@ static int ifcvf_vdpa_get_vq_irq(struct >>> vdpa_device *vdpa_dev, >>> return vf->vring[qid].irq; >>> } >>> +static struct vdpa_notification_area >>> ifcvf_get_vq_notification(struct vdpa_device *vdpa_dev, >>> + u16 idx) >>> +{ >>> + struct ifcvf_adapter *adapter = vdpa_to_adapter(vdpa_dev); >>> + struct ifcvf_hw *vf = vdpa_to_vf(vdpa_dev); >>> + struct pci_dev *pdev = adapter->pdev; >>> + struct vdpa_notification_area area; >>> + >>> + area.addr = vf->vring[idx].notify_pa; >>> + area.size = PAGE_SIZE; >>> + if (area.addr % PAGE_SIZE) >>> + IFCVF_DBG(pdev, "vq %u doorbell address is not PAGE_SIZE >>> aligned\n", idx); >> >> >> Let's leave the decision to upper layer by: (see >> vp_vdpa_get_vq_notification) >> >> area.addr = notify_pa; >> area.size = notify_offset_multiplier; >> >> Thanks > > Hi Jason, > > notify_offset_multiplier can be zero, means vqs share the same > doorbell address, distinguished by qid. > and in vdpa.c: > > if (vma->vm_end - vma->vm_start != notify.size) > return -ENOTSUPP; > > so a zero size would cause this feature failure. > mmap should work on at least a page, so if we really want "area.size = > notify_offset_multiplier;" > I think we should add some code in vdpa.c, like: > > if(!notify.size) > notify.size = PAGE_SIZE; > > sounds good? It's the responsibility of the driver to report a correct one. So I think it's better to tweak it as: area.size = notify_offset_multiplier ?: PAGE_SIZE; Thanks > > Thanks > Zhu Lingshan >> >> >>> + >>> + return area; >>> +} >>> + >>> /* >>> * IFCVF currently does't have on-chip IOMMU, so not >>> * implemented set_map()/dma_map()/dma_unmap() >>> @@ -440,6 +456,7 @@ static const struct vdpa_config_ops ifc_vdpa_ops >>> = { >>> .get_config = ifcvf_vdpa_get_config, >>> .set_config = ifcvf_vdpa_set_config, >>> .set_config_cb = ifcvf_vdpa_set_config_cb, >>> + .get_vq_notification = ifcvf_get_vq_notification, >>> }; >>> static int ifcvf_probe(struct pci_dev *pdev, const struct >>> pci_device_id *id) >> > ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2021-06-01 3:32 UTC | newest] Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-05-31 7:33 [PATCH V2 RESEND 0/2] vDPA/ifcvf: implement doorbell mapping feature Zhu Lingshan 2021-05-31 7:33 ` [PATCH V2 RESEND 1/2] vDPA/ifcvf: record virtio notify base Zhu Lingshan 2021-05-31 7:33 ` [PATCH V2 RESEND 2/2] vDPA/ifcvf: implement doorbell mapping for ifcvf Zhu Lingshan 2021-05-31 7:56 ` Jason Wang 2021-06-01 3:27 ` Zhu, Lingshan 2021-06-01 3:31 ` Jason Wang
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).