Re: [PATCH v3] pci_vfio: Support 64KB kernel page_size with vfio-pci driver

["Tone Zhang (Arm Technology China)" <Tone.Zhang@arm.com>]

Hi Anatoly,

Thanks for the comments.

> -----Original Message-----
> From: Burakov, Anatoly <anatoly.burakov@intel.com>
> Sent: Monday, December 10, 2018 7:46 PM
> To: Tone Zhang (Arm Technology China) <Tone.Zhang@arm.com>;
> dev@dpdk.org
> Cc: Gavin Hu (Arm Technology China) <Gavin.Hu@arm.com>; Honnappa
> Nagarahalli <Honnappa.Nagarahalli@arm.com>; Steve Capper
> <Steve.Capper@arm.com>; nd <nd@arm.com>
> Subject: Re: [PATCH v3] pci_vfio: Support 64KB kernel page_size with vfio-pci
> driver
> 
> On 19-Nov-18 2:37 AM, tone.zhang wrote:
> > With a larger PAGE_SIZE it is possible for the MSI table to very close
> > to the end of the BAR s.t. when we align the start and end of the MSI
> > table to the PAGE_SIZE, the end offset of the MSI table is out of the
> > PCI BAR boundary.
> >
> > This patch addresses the issue by comparing both the start and the end
> > offset of the MSI table with the BAR size, and skip the mapping if it
> > is out of Bar scope.
> >
> > The patch fixes the debug log as below:
> > EAL: Skipping BAR0
> >
> > Signed-off-by: tone.zhang <tone.zhang@arm.com>
> > Reviewed-by: Gavin Hu <Gavin.Hu@arm.com>
> > Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
> > Reviewed-by: Steve Capper <Steve.Capper@arm.com>
> > Reviewed-by: Anatoly Burakov <anatoly.burakov@intel.com>
> > ---
> >   drivers/bus/pci/linux/pci_vfio.c | 27 ++++++++++++++++++++++-----
> >   1 file changed, 22 insertions(+), 5 deletions(-)
> >
> > diff --git a/drivers/bus/pci/linux/pci_vfio.c
> > b/drivers/bus/pci/linux/pci_vfio.c
> > index ffd26f1..0e7bfd7 100644
> > --- a/drivers/bus/pci/linux/pci_vfio.c
> > +++ b/drivers/bus/pci/linux/pci_vfio.c
> > @@ -457,9 +457,11 @@ pci_vfio_mmap_bar(int vfio_dev_fd, struct
> mapped_pci_resource *vfio_res,
> >   	struct pci_msix_table *msix_table = &vfio_res->msix_table;
> >   	struct pci_map *bar = &vfio_res->maps[bar_index];
> >
> > -	if (bar->size == 0)
> > +	if (bar->size == 0) {
> >   		/* Skip this BAR */
> > +		RTE_LOG(INFO, EAL, "Skipping BAR%d\n", bar_index);
> >   		return 0;
> 
> I would perhaps make it a DEBUG rather than INFO.

Will update in v4 version soon.

> 
> > +	}
> >
> >   	if (msix_table->bar_index == bar_index) {
> >   		/*
> > @@ -468,8 +470,14 @@ pci_vfio_mmap_bar(int vfio_dev_fd, struct
> mapped_pci_resource *vfio_res,
> >   		 */
> >   		uint32_t table_start = msix_table->offset;
> >   		uint32_t table_end = table_start + msix_table->size;
> > -		table_end = (table_end + ~PAGE_MASK) & PAGE_MASK;
> > -		table_start &= PAGE_MASK;
> > +		table_end = RTE_ALIGN(table_end, PAGE_SIZE);
> > +		table_start = RTE_ALIGN(table_start, PAGE_SIZE);
> 
> I believe table_start &= PAGE_MASK is equivalent to table_start =
> RTE_ALIGN_FLOOR(), not RTE_ALIGN() (which is an alias for RTE_ALIGN_CEIL()).

Will change to RTE_ALIGN_FLOOR. Then table_start will be less than bar->size, the below code will be updated further in v4.

> 
> > +
> > +		/* If page-aligned start of MSI-X table is beyond BAR size,
> > +		 * shrink the mapping size to MSI-X table start address.
> > +		 */
> > +		if (table_start >= bar->size)
> > +			table_start = msix_table->offset;
> >
> >   		if (table_start == 0 && table_end >= bar->size) {
> >   			/* Cannot map this BAR */
> > @@ -481,8 +489,17 @@ pci_vfio_mmap_bar(int vfio_dev_fd, struct
> > mapped_pci_resource *vfio_res,
> >
> >   		memreg[0].offset = bar->offset;
> >   		memreg[0].size = table_start;
> > -		memreg[1].offset = bar->offset + table_end;
> > -		memreg[1].size = bar->size - table_end;
> > +		if (bar->size < table_end) {
> > +			/*
> > +			 * If MSI-X table end is beyond BAR end, don't attempt
> > +			 * to perform second mapping.
> > +			 */
> > +			memreg[1].offset = 0;
> > +			memreg[1].size = 0;
> > +		} else {
> > +			memreg[1].offset = bar->offset + table_end;
> > +			memreg[1].size = bar->size - table_end;
> > +		}
> >
> >   		RTE_LOG(DEBUG, EAL,
> >   			"Trying to map BAR%d that contains the MSI-X "
> >
> 
> 
> --
> Thanks,
> Anatoly

Thnaks!

Br,
Tone