This series enabled Intel FGPA SmartNIC C5000X-PL virtio-blk for vDPA. This series requires: Stefano's vdpa block patchset: https://lkml.org/lkml/2021/3/15/2113 my patchset to enable Intel FGPA SmartNIC C5000X-PL virtio-net for vDPA: https://lkml.org/lkml/2021/3/17/432 changes from V3: remove (pdev->device < 0x1000 || pdev->device > 0x107f) checks in probe(), because id_table already cut them off(Jason) changes from V2: both get_features() and get_config_size() use switch code block now(Stefano) changes from V1: (1)add comments to explain this driver drives virtio modern devices and transitional devices in modern mode.(Jason) (2)remove IFCVF_BLK_SUPPORTED_FEATURES, use hardware feature bits directly(Jason) (3)add error handling and message in get_config_size(Stefano) Thanks! Zhu Lingshan (3): vDPA/ifcvf: deduce VIRTIO device ID when probe vDPA/ifcvf: enable Intel C5000X-PL virtio-block for vDPA vDPA/ifcvf: get_config_size should return dev specific config size drivers/vdpa/ifcvf/ifcvf_base.h | 9 ++++- drivers/vdpa/ifcvf/ifcvf_main.c | 65 ++++++++++++++++++++++++++------- 2 files changed, 59 insertions(+), 15 deletions(-) -- 2.27.0
This commit deduces VIRTIO device ID as device type when probe, then ifcvf_vdpa_get_device_id() can simply return the ID. ifcvf_vdpa_get_features() and ifcvf_vdpa_get_config_size() can work properly based on the device ID. Signed-off-by: Zhu Lingshan <lingshan.zhu@intel.com> --- drivers/vdpa/ifcvf/ifcvf_base.h | 1 + drivers/vdpa/ifcvf/ifcvf_main.c | 27 +++++++++++++++------------ 2 files changed, 16 insertions(+), 12 deletions(-) diff --git a/drivers/vdpa/ifcvf/ifcvf_base.h b/drivers/vdpa/ifcvf/ifcvf_base.h index b2eeb16b9c2c..1c04cd256fa7 100644 --- a/drivers/vdpa/ifcvf/ifcvf_base.h +++ b/drivers/vdpa/ifcvf/ifcvf_base.h @@ -84,6 +84,7 @@ struct ifcvf_hw { u32 notify_off_multiplier; u64 req_features; u64 hw_features; + u32 dev_type; struct virtio_pci_common_cfg __iomem *common_cfg; void __iomem *net_cfg; struct vring_info vring[IFCVF_MAX_QUEUE_PAIRS * 2]; diff --git a/drivers/vdpa/ifcvf/ifcvf_main.c b/drivers/vdpa/ifcvf/ifcvf_main.c index 44d7586019da..66927ec81fa5 100644 --- a/drivers/vdpa/ifcvf/ifcvf_main.c +++ b/drivers/vdpa/ifcvf/ifcvf_main.c @@ -323,19 +323,9 @@ static u32 ifcvf_vdpa_get_generation(struct vdpa_device *vdpa_dev) static u32 ifcvf_vdpa_get_device_id(struct vdpa_device *vdpa_dev) { - struct ifcvf_adapter *adapter = vdpa_to_adapter(vdpa_dev); - struct pci_dev *pdev = adapter->pdev; - u32 ret = -ENODEV; - - if (pdev->device < 0x1000 || pdev->device > 0x107f) - return ret; - - if (pdev->device < 0x1040) - ret = pdev->subsystem_device; - else - ret = pdev->device - 0x1040; + struct ifcvf_hw *vf = vdpa_to_vf(vdpa_dev); - return ret; + return vf->dev_type; } static u32 ifcvf_vdpa_get_vendor_id(struct vdpa_device *vdpa_dev) @@ -466,6 +456,19 @@ static int ifcvf_probe(struct pci_dev *pdev, const struct pci_device_id *id) pci_set_drvdata(pdev, adapter); vf = &adapter->vf; + + /* This drirver drives both modern virtio devices and transitional + * devices in modern mode. + * vDPA requires feature bit VIRTIO_F_ACCESS_PLATFORM, + * so legacy devices and transitional devices in legacy + * mode will not work for vDPA, this driver will not + * drive devices with legacy interface. + */ + if (pdev->device < 0x1040) + vf->dev_type = pdev->subsystem_device; + else + vf->dev_type = pdev->device - 0x1040; + vf->base = pcim_iomap_table(pdev); adapter->pdev = pdev; -- 2.27.0
This commit enabled Intel FPGA SmartNIC C5000X-PL virtio-block for vDPA. Signed-off-by: Zhu Lingshan <lingshan.zhu@intel.com> Reviewed-by: Stefano Garzarella <sgarzare@redhat.com> Acked-by: Jason Wang <jasowang@redhat.com> --- drivers/vdpa/ifcvf/ifcvf_base.h | 8 +++++++- drivers/vdpa/ifcvf/ifcvf_main.c | 19 ++++++++++++++++++- 2 files changed, 25 insertions(+), 2 deletions(-) diff --git a/drivers/vdpa/ifcvf/ifcvf_base.h b/drivers/vdpa/ifcvf/ifcvf_base.h index 1c04cd256fa7..0111bfdeb342 100644 --- a/drivers/vdpa/ifcvf/ifcvf_base.h +++ b/drivers/vdpa/ifcvf/ifcvf_base.h @@ -15,6 +15,7 @@ #include <linux/pci_regs.h> #include <linux/vdpa.h> #include <uapi/linux/virtio_net.h> +#include <uapi/linux/virtio_blk.h> #include <uapi/linux/virtio_config.h> #include <uapi/linux/virtio_pci.h> @@ -28,7 +29,12 @@ #define C5000X_PL_SUBSYS_VENDOR_ID 0x8086 #define C5000X_PL_SUBSYS_DEVICE_ID 0x0001 -#define IFCVF_SUPPORTED_FEATURES \ +#define C5000X_PL_BLK_VENDOR_ID 0x1AF4 +#define C5000X_PL_BLK_DEVICE_ID 0x1001 +#define C5000X_PL_BLK_SUBSYS_VENDOR_ID 0x8086 +#define C5000X_PL_BLK_SUBSYS_DEVICE_ID 0x0002 + +#define IFCVF_NET_SUPPORTED_FEATURES \ ((1ULL << VIRTIO_NET_F_MAC) | \ (1ULL << VIRTIO_F_ANY_LAYOUT) | \ (1ULL << VIRTIO_F_VERSION_1) | \ diff --git a/drivers/vdpa/ifcvf/ifcvf_main.c b/drivers/vdpa/ifcvf/ifcvf_main.c index 66927ec81fa5..9a4a6df91f08 100644 --- a/drivers/vdpa/ifcvf/ifcvf_main.c +++ b/drivers/vdpa/ifcvf/ifcvf_main.c @@ -168,10 +168,23 @@ static struct ifcvf_hw *vdpa_to_vf(struct vdpa_device *vdpa_dev) static u64 ifcvf_vdpa_get_features(struct vdpa_device *vdpa_dev) { + struct ifcvf_adapter *adapter = vdpa_to_adapter(vdpa_dev); struct ifcvf_hw *vf = vdpa_to_vf(vdpa_dev); + struct pci_dev *pdev = adapter->pdev; + u64 features; - features = ifcvf_get_features(vf) & IFCVF_SUPPORTED_FEATURES; + switch (vf->dev_type) { + case VIRTIO_ID_NET: + features = ifcvf_get_features(vf) & IFCVF_NET_SUPPORTED_FEATURES; + break; + case VIRTIO_ID_BLOCK: + features = ifcvf_get_features(vf); + break; + default: + features = 0; + IFCVF_ERR(pdev, "VIRTIO ID %u not supported\n", vf->dev_type); + } return features; } @@ -514,6 +527,10 @@ static struct pci_device_id ifcvf_pci_ids[] = { C5000X_PL_DEVICE_ID, C5000X_PL_SUBSYS_VENDOR_ID, C5000X_PL_SUBSYS_DEVICE_ID) }, + { PCI_DEVICE_SUB(C5000X_PL_BLK_VENDOR_ID, + C5000X_PL_BLK_DEVICE_ID, + C5000X_PL_BLK_SUBSYS_VENDOR_ID, + C5000X_PL_BLK_SUBSYS_DEVICE_ID) }, { 0 }, }; -- 2.27.0
get_config_size() should return the size based on the decected device type. Signed-off-by: Zhu Lingshan <lingshan.zhu@intel.com> Reviewed-by: Stefano Garzarella <sgarzare@redhat.com> Acked-by: Jason Wang <jasowang@redhat.com> --- drivers/vdpa/ifcvf/ifcvf_main.c | 19 ++++++++++++++++++- 1 file changed, 18 insertions(+), 1 deletion(-) diff --git a/drivers/vdpa/ifcvf/ifcvf_main.c b/drivers/vdpa/ifcvf/ifcvf_main.c index 9a4a6df91f08..e48e6b74fe2e 100644 --- a/drivers/vdpa/ifcvf/ifcvf_main.c +++ b/drivers/vdpa/ifcvf/ifcvf_main.c @@ -356,7 +356,24 @@ static u32 ifcvf_vdpa_get_vq_align(struct vdpa_device *vdpa_dev) static size_t ifcvf_vdpa_get_config_size(struct vdpa_device *vdpa_dev) { - return sizeof(struct virtio_net_config); + struct ifcvf_adapter *adapter = vdpa_to_adapter(vdpa_dev); + struct ifcvf_hw *vf = vdpa_to_vf(vdpa_dev); + struct pci_dev *pdev = adapter->pdev; + size_t size; + + switch (vf->dev_type) { + case VIRTIO_ID_NET: + size = sizeof(struct virtio_net_config); + break; + case VIRTIO_ID_BLOCK: + size = sizeof(struct virtio_blk_config); + break; + default: + size = 0; + IFCVF_ERR(pdev, "VIRTIO ID %u not supported\n", vf->dev_type); + } + + return size; } static void ifcvf_vdpa_get_config(struct vdpa_device *vdpa_dev, -- 2.27.0
在 2021/4/19 下午2:33, Zhu Lingshan 写道: > This commit deduces VIRTIO device ID as device type when probe, > then ifcvf_vdpa_get_device_id() can simply return the ID. > ifcvf_vdpa_get_features() and ifcvf_vdpa_get_config_size() > can work properly based on the device ID. > > Signed-off-by: Zhu Lingshan <lingshan.zhu@intel.com> Acked-by: Jason Wang <jasowang@redhat.com> > --- > drivers/vdpa/ifcvf/ifcvf_base.h | 1 + > drivers/vdpa/ifcvf/ifcvf_main.c | 27 +++++++++++++++------------ > 2 files changed, 16 insertions(+), 12 deletions(-) > > diff --git a/drivers/vdpa/ifcvf/ifcvf_base.h b/drivers/vdpa/ifcvf/ifcvf_base.h > index b2eeb16b9c2c..1c04cd256fa7 100644 > --- a/drivers/vdpa/ifcvf/ifcvf_base.h > +++ b/drivers/vdpa/ifcvf/ifcvf_base.h > @@ -84,6 +84,7 @@ struct ifcvf_hw { > u32 notify_off_multiplier; > u64 req_features; > u64 hw_features; > + u32 dev_type; > struct virtio_pci_common_cfg __iomem *common_cfg; > void __iomem *net_cfg; > struct vring_info vring[IFCVF_MAX_QUEUE_PAIRS * 2]; > diff --git a/drivers/vdpa/ifcvf/ifcvf_main.c b/drivers/vdpa/ifcvf/ifcvf_main.c > index 44d7586019da..66927ec81fa5 100644 > --- a/drivers/vdpa/ifcvf/ifcvf_main.c > +++ b/drivers/vdpa/ifcvf/ifcvf_main.c > @@ -323,19 +323,9 @@ static u32 ifcvf_vdpa_get_generation(struct vdpa_device *vdpa_dev) > > static u32 ifcvf_vdpa_get_device_id(struct vdpa_device *vdpa_dev) > { > - struct ifcvf_adapter *adapter = vdpa_to_adapter(vdpa_dev); > - struct pci_dev *pdev = adapter->pdev; > - u32 ret = -ENODEV; > - > - if (pdev->device < 0x1000 || pdev->device > 0x107f) > - return ret; > - > - if (pdev->device < 0x1040) > - ret = pdev->subsystem_device; > - else > - ret = pdev->device - 0x1040; > + struct ifcvf_hw *vf = vdpa_to_vf(vdpa_dev); > > - return ret; > + return vf->dev_type; > } > > static u32 ifcvf_vdpa_get_vendor_id(struct vdpa_device *vdpa_dev) > @@ -466,6 +456,19 @@ static int ifcvf_probe(struct pci_dev *pdev, const struct pci_device_id *id) > pci_set_drvdata(pdev, adapter); > > vf = &adapter->vf; > + > + /* This drirver drives both modern virtio devices and transitional > + * devices in modern mode. > + * vDPA requires feature bit VIRTIO_F_ACCESS_PLATFORM, > + * so legacy devices and transitional devices in legacy > + * mode will not work for vDPA, this driver will not > + * drive devices with legacy interface. > + */ > + if (pdev->device < 0x1040) > + vf->dev_type = pdev->subsystem_device; > + else > + vf->dev_type = pdev->device - 0x1040; > + > vf->base = pcim_iomap_table(pdev); > > adapter->pdev = pdev;
On Mon, Apr 19, 2021 at 02:33:25PM +0800, Zhu Lingshan wrote: > This commit enabled Intel FPGA SmartNIC C5000X-PL virtio-block > for vDPA. > > Signed-off-by: Zhu Lingshan <lingshan.zhu@intel.com> > Reviewed-by: Stefano Garzarella <sgarzare@redhat.com> > Acked-by: Jason Wang <jasowang@redhat.com> > --- > drivers/vdpa/ifcvf/ifcvf_base.h | 8 +++++++- > drivers/vdpa/ifcvf/ifcvf_main.c | 19 ++++++++++++++++++- > 2 files changed, 25 insertions(+), 2 deletions(-) > > diff --git a/drivers/vdpa/ifcvf/ifcvf_base.h b/drivers/vdpa/ifcvf/ifcvf_base.h > index 1c04cd256fa7..0111bfdeb342 100644 > --- a/drivers/vdpa/ifcvf/ifcvf_base.h > +++ b/drivers/vdpa/ifcvf/ifcvf_base.h > @@ -15,6 +15,7 @@ > #include <linux/pci_regs.h> > #include <linux/vdpa.h> > #include <uapi/linux/virtio_net.h> > +#include <uapi/linux/virtio_blk.h> > #include <uapi/linux/virtio_config.h> > #include <uapi/linux/virtio_pci.h> > > @@ -28,7 +29,12 @@ > #define C5000X_PL_SUBSYS_VENDOR_ID 0x8086 > #define C5000X_PL_SUBSYS_DEVICE_ID 0x0001 > > -#define IFCVF_SUPPORTED_FEATURES \ > +#define C5000X_PL_BLK_VENDOR_ID 0x1AF4 Come on this is just PCI_VENDOR_ID_REDHAT_QUMRANET right? > +#define C5000X_PL_BLK_DEVICE_ID 0x1001 0x1001 is a transitional blk device from virtio spec too right? Let's add these to virtio_ids.h? > +#define C5000X_PL_BLK_SUBSYS_VENDOR_ID 0x8086 > +#define C5000X_PL_BLK_SUBSYS_DEVICE_ID 0x0002 VIRTIO_ID_BLOCK? > + > +#define IFCVF_NET_SUPPORTED_FEATURES \ > ((1ULL << VIRTIO_NET_F_MAC) | \ > (1ULL << VIRTIO_F_ANY_LAYOUT) | \ > (1ULL << VIRTIO_F_VERSION_1) | \ > diff --git a/drivers/vdpa/ifcvf/ifcvf_main.c b/drivers/vdpa/ifcvf/ifcvf_main.c > index 66927ec81fa5..9a4a6df91f08 100644 > --- a/drivers/vdpa/ifcvf/ifcvf_main.c > +++ b/drivers/vdpa/ifcvf/ifcvf_main.c > @@ -168,10 +168,23 @@ static struct ifcvf_hw *vdpa_to_vf(struct vdpa_device *vdpa_dev) > > static u64 ifcvf_vdpa_get_features(struct vdpa_device *vdpa_dev) > { > + struct ifcvf_adapter *adapter = vdpa_to_adapter(vdpa_dev); > struct ifcvf_hw *vf = vdpa_to_vf(vdpa_dev); > + struct pci_dev *pdev = adapter->pdev; > + > u64 features; > > - features = ifcvf_get_features(vf) & IFCVF_SUPPORTED_FEATURES; > + switch (vf->dev_type) { > + case VIRTIO_ID_NET: > + features = ifcvf_get_features(vf) & IFCVF_NET_SUPPORTED_FEATURES; > + break; > + case VIRTIO_ID_BLOCK: > + features = ifcvf_get_features(vf); > + break; > + default: > + features = 0; > + IFCVF_ERR(pdev, "VIRTIO ID %u not supported\n", vf->dev_type); > + } > > return features; > } > @@ -514,6 +527,10 @@ static struct pci_device_id ifcvf_pci_ids[] = { > C5000X_PL_DEVICE_ID, > C5000X_PL_SUBSYS_VENDOR_ID, > C5000X_PL_SUBSYS_DEVICE_ID) }, > + { PCI_DEVICE_SUB(C5000X_PL_BLK_VENDOR_ID, > + C5000X_PL_BLK_DEVICE_ID, > + C5000X_PL_BLK_SUBSYS_VENDOR_ID, > + C5000X_PL_BLK_SUBSYS_DEVICE_ID) }, > > { 0 }, > }; > -- > 2.27.0
On 5/3/2021 4:47 PM, Michael S. Tsirkin wrote: > On Mon, Apr 19, 2021 at 02:33:25PM +0800, Zhu Lingshan wrote: >> This commit enabled Intel FPGA SmartNIC C5000X-PL virtio-block >> for vDPA. >> >> Signed-off-by: Zhu Lingshan <lingshan.zhu@intel.com> >> Reviewed-by: Stefano Garzarella <sgarzare@redhat.com> >> Acked-by: Jason Wang <jasowang@redhat.com> >> --- >> drivers/vdpa/ifcvf/ifcvf_base.h | 8 +++++++- >> drivers/vdpa/ifcvf/ifcvf_main.c | 19 ++++++++++++++++++- >> 2 files changed, 25 insertions(+), 2 deletions(-) >> >> diff --git a/drivers/vdpa/ifcvf/ifcvf_base.h b/drivers/vdpa/ifcvf/ifcvf_base.h >> index 1c04cd256fa7..0111bfdeb342 100644 >> --- a/drivers/vdpa/ifcvf/ifcvf_base.h >> +++ b/drivers/vdpa/ifcvf/ifcvf_base.h >> @@ -15,6 +15,7 @@ >> #include <linux/pci_regs.h> >> #include <linux/vdpa.h> >> #include <uapi/linux/virtio_net.h> >> +#include <uapi/linux/virtio_blk.h> >> #include <uapi/linux/virtio_config.h> >> #include <uapi/linux/virtio_pci.h> >> >> @@ -28,7 +29,12 @@ >> #define C5000X_PL_SUBSYS_VENDOR_ID 0x8086 >> #define C5000X_PL_SUBSYS_DEVICE_ID 0x0001 >> >> -#define IFCVF_SUPPORTED_FEATURES \ >> +#define C5000X_PL_BLK_VENDOR_ID 0x1AF4 > > Come on this is just PCI_VENDOR_ID_REDHAT_QUMRANET right? Hi Michael, I will re-use the predefined macro in next patches #define C5000X_PL_BLK_VENDOR_ID PCI_VENDOR_ID_REDHAT_QUMRANET #define C5000X_PL_BLK_SUBSYS_DEVICE_ID VIRTIO_ID_BLOCK? to keep the readabilities in probe(). > > > >> +#define C5000X_PL_BLK_DEVICE_ID 0x1001 > 0x1001 is a transitional blk device from virtio spec too right? Let's add these to virtio_ids.h? will update virtio_ids.h in next patchset. Thanks Zhu Lingshan > >> +#define C5000X_PL_BLK_SUBSYS_VENDOR_ID 0x8086 >> +#define C5000X_PL_BLK_SUBSYS_DEVICE_ID 0x0002 > VIRTIO_ID_BLOCK? > >> + >> +#define IFCVF_NET_SUPPORTED_FEATURES \ >> ((1ULL << VIRTIO_NET_F_MAC) | \ >> (1ULL << VIRTIO_F_ANY_LAYOUT) | \ >> (1ULL << VIRTIO_F_VERSION_1) | \ >> diff --git a/drivers/vdpa/ifcvf/ifcvf_main.c b/drivers/vdpa/ifcvf/ifcvf_main.c >> index 66927ec81fa5..9a4a6df91f08 100644 >> --- a/drivers/vdpa/ifcvf/ifcvf_main.c >> +++ b/drivers/vdpa/ifcvf/ifcvf_main.c >> @@ -168,10 +168,23 @@ static struct ifcvf_hw *vdpa_to_vf(struct vdpa_device *vdpa_dev) >> >> static u64 ifcvf_vdpa_get_features(struct vdpa_device *vdpa_dev) >> { >> + struct ifcvf_adapter *adapter = vdpa_to_adapter(vdpa_dev); >> struct ifcvf_hw *vf = vdpa_to_vf(vdpa_dev); >> + struct pci_dev *pdev = adapter->pdev; >> + >> u64 features; >> >> - features = ifcvf_get_features(vf) & IFCVF_SUPPORTED_FEATURES; >> + switch (vf->dev_type) { >> + case VIRTIO_ID_NET: >> + features = ifcvf_get_features(vf) & IFCVF_NET_SUPPORTED_FEATURES; >> + break; >> + case VIRTIO_ID_BLOCK: >> + features = ifcvf_get_features(vf); >> + break; >> + default: >> + features = 0; >> + IFCVF_ERR(pdev, "VIRTIO ID %u not supported\n", vf->dev_type); >> + } >> >> return features; >> } >> @@ -514,6 +527,10 @@ static struct pci_device_id ifcvf_pci_ids[] = { >> C5000X_PL_DEVICE_ID, >> C5000X_PL_SUBSYS_VENDOR_ID, >> C5000X_PL_SUBSYS_DEVICE_ID) }, >> + { PCI_DEVICE_SUB(C5000X_PL_BLK_VENDOR_ID, >> + C5000X_PL_BLK_DEVICE_ID, >> + C5000X_PL_BLK_SUBSYS_VENDOR_ID, >> + C5000X_PL_BLK_SUBSYS_DEVICE_ID) }, >> >> { 0 }, >> }; >> -- >> 2.27.0
Hi Zhu,
On Mon, Apr 19, 2021 at 8:39 AM Zhu Lingshan <lingshan.zhu@intel.com> wrote:
>
> This series enabled Intel FGPA SmartNIC C5000X-PL virtio-blk for vDPA.
Looking at the IFCVF upstream vDPA driver (with this series applied), it
seems that there is still some cleaning to be done to support virtio-blk
devices:
- ifcvf_vdpa_get_config() and ifcvf_vdpa_set_config() use
`sizeof(struct virtio_net_config)` to check the inputs.
This seems wrong for a virtio-blk device. Maybe we can set the config
size for each device in ifcvf_vdpa_dev_add() and use that field to
check the inputs. We can reuse the same field also in
ifcvf_vdpa_get_config_size().
- Just for make the code more readable we should rename `net_cfg` field
to `device_cfg`in `struct ifcvf_hw`.
What do you think?
Thanks,
Stefano
Hello Stefano,
Thanks for your review and advices, I will fix them
Thanks,
Zhu Lingshan
On 11/24/2021 11:39 PM, Stefano Garzarella wrote:
> Hi Zhu,
>
> On Mon, Apr 19, 2021 at 8:39 AM Zhu Lingshan <lingshan.zhu@intel.com> wrote:
>> This series enabled Intel FGPA SmartNIC C5000X-PL virtio-blk for vDPA.
> Looking at the IFCVF upstream vDPA driver (with this series applied), it
> seems that there is still some cleaning to be done to support virtio-blk
> devices:
>
> - ifcvf_vdpa_get_config() and ifcvf_vdpa_set_config() use
> `sizeof(struct virtio_net_config)` to check the inputs.
> This seems wrong for a virtio-blk device. Maybe we can set the config
> size for each device in ifcvf_vdpa_dev_add() and use that field to
> check the inputs. We can reuse the same field also in
> ifcvf_vdpa_get_config_size().
>
> - Just for make the code more readable we should rename `net_cfg` field
> to `device_cfg`in `struct ifcvf_hw`.
>
> What do you think?
>
> Thanks,
> Stefano
>