Re: [PATCH] eal: map io resources for non x86 architectures

[Santosh Shukla <sshukla@mvista.com>]

On Tue, Dec 29, 2015 at 8:21 PM, Santosh Shukla <sshukla@mvista.com> wrote:
> On Tue, Dec 29, 2015 at 7:34 PM, Alex Williamson
> <alex.williamson@redhat.com> wrote:
>> On Tue, 2015-12-29 at 16:17 +0530, Santosh Shukla wrote:
>>> On Tue, Dec 29, 2015 at 3:26 PM, Burakov, Anatoly
>>> <anatoly.burakov@intel.com> wrote:
>>> > Hi Santosh,
>>> >
>>> > > On Fri, Dec 18, 2015 at 6:25 PM, Santosh Shukla <sshukla@mvista.c
>>> > > om>
>>> > > wrote:
>>> > > > On Fri, Dec 18, 2015 at 1:51 PM, Yuanhan Liu
>>> > > > <yuanhan.liu@linux.intel.com> wrote:
>>> > > > > On Fri, Dec 18, 2015 at 01:24:41PM +0530, Santosh Shukla
>>> > > > > wrote:
>>> > > > > > > > I guess we have done enough evaluation / investigation
>>> > > > > > > > that
>>> > > > > > > > suggest - so to map iopci region to userspace in arch
>>> > > > > > > > agnostic-way -
>>> > > > > > > >
>>> > > > > > > > # either we need to modify kernel
>>> > > > > > > >                - Make sure all the non-x86 arch to
>>> > > > > > > > support
>>> > > > > > > > mapping for iopci region (i.e. pci_mmap_page_range). I
>>> > > > > > > > don;t
>>> > > > > > > > think its a correct approach though.
>>> > > > > > > >             or
>>> > > > > > > >                - include /dev/ioport char-mem device
>>> > > > > > > > file who
>>> > > > > > > > could do more than byte operation, Note that this
>>> > > > > > > > implementation
>>> > > > > > > > does not exist in kernel.  I could send an RFC to lkml.
>>> > > > > > >
>>> > > > > > > Maybe you could propose the two to lkml, to get some
>>> > > > > > > feedbacks
>>> > > > > > > from those kernel/ARM gurus? Please cc me if you do so.
>>> > > > > > >
>>> > > > > >
>>> > > > > > The latter one I already shared old lkml thread, Pl.
>>> > > > > > revisit my v1
>>> > > > > > 0/6 patch [1] and in that refer [2].
>>> > > > >
>>> > > > > Oops, sorry, a bit busy, that I didn't look at it carefully.
>>> > > > > My bad,
>>> > > > > anyway.
>>> > > > >
>>> > > > > > Josh has already proposed to lkml but for some reason
>>> > > > > > thread didn't
>>> > > > > > went far. I can restart that discussion giving dpdk use-
>>> > > > > > case as an
>>> > > > > > example/ requirement.
>>> > > > >
>>> > > > > I had a quick go through of the discussion. Both hpa and Arnd
>>> > > > > seem to
>>> > > > > be fine with the ioctl interface on /dev/port. Have you tried
>>> > > > > that?
>>> > > > > And if you want to restart it, ioctl might be a better option
>>> > > > > than
>>> > > > > /dev/ioport, judging from the discussion.
>>> > > > >
>>> > > >
>>> > > > I tried legacy patch and re-writing with ioctl-way; doing
>>> > > > changes in
>>> > > > dpdk port as well in kernel, had to test on quite other hw not
>>> > > > only
>>> > > > arm64 though! so it will take time for me, I am travelling
>>> > > > tomorrow so
>>> > > > bit delayed, We'll post patch to lkml and share dpdk-virtio
>>> > > > feedback
>>> > > > perhaps by Monday.
>>> > > >
>>> > >
>>> > > I posted a query about /dev/ioports approach in lkml thread [1],
>>> > > and Arnd
>>> > > suggested to use vfio framework but it looks like vfio too does
>>> > > not map
>>> > > ioresource_io region. Same communicated to Arnd and waiting for
>>> > > his reply.
>>> > >
>>> > > In mean time I like to ask general question;
>>> > > - Has anyone tried vfio/non-iommu approach for virtio pmd driver?
>>> > > If not
>>> > > then Is there any plan? Someone planning to take up.
>>> > > [1] https://lkml.org/lkml/2015/12/23/145
>>> >
>>> > I have submitted a patch to support no-iommu mode, but I'm not
>>> > aware of anyone trying VFIO-noiommu at all. That's probably
>>> > expected since it's Christmas/New Year in a lot of places, and
>>> > everyone is on a break.
>>> >
>>> > That said, I'm not sure I completely understand what is it that
>>> > you're asking about. The code you're referring to (in vfio_pci.c,
>>> > line 854 as of kernel 4.3) is checking if a PCI BAR is available
>>> > for IO (hence checking if the IORESOURCE_MEM
>>>
>>>
>>> Thanks for reply! You comment might help to move this discuss to next
>>> level.
>>>
>>> Look at kernel/resource.c, it exports two symbol ioport_resource and
>>> iomem_resource and sets appropriate flag type i.e.. IORESOURCE_IO and
>>> IORESOURCE_MEM. In virtio-net case; it creates both pci region i.e..
>>> _io bar and _mem bar. dpdk virtio pmd driver (<= 0.95 virtio spec)
>>> uses pci _io bar region for device initialization as virtio headers
>>> are locate at pci _io bar region. Since it uses pci _iobar region so
>>> likely it update pci_resource.[index].flag = IORESOURCE_IO.  and vfio
>>> mmap function wont handle ioresource_io (i guess). And that is why I
>>> asked same to lkml thread.
>>>
>>>
>>> bit is set). There isn't any "ioresource_mem" region as far as VFIO
>>> is
>>> concerned, there are only BARs, ROM, VGA and PCI
>>> config regions (linux/vfio.h, line 314 as of kernel 4.3). So if
>>> you're
>>> missing some PCI regions for VFIO to map, they would first need to be
>>> added to VFIO kernel implementation before they can be used by DPDK.
>>> That is, unless I'm misunderstanding something :)
>>> >
>>>
>>> Agree. As mentioned above, I guess ioresource_io pci bar
>>> implementation missing in vfio kernel? but before adding code support
>>> in kernel I would like to hear from experts example, You, Alex!
>>> (looping alex)
>>
>> MMIO and I/O port space are simply regions as far as VFIO is concerned.
>>  The userspace API supports the concept of I/O port regions that can be
>> mmap'd by userspace, but the implementation does not since I/O port
>> regions cannot be mmap'd on x86.  Someone needs to propose some code
>> for vfio-pci to support it.  Thanks,
>>
>
> I will work on this and post an RFC soon to LKML, That RFC targetted
> to map IOport region for non-x86 platform in particular. Thanks for
> confirming the vfio behaviour.
>

Some update:

Patches used for setup -
- Used Alex(s) vfio-noiommu linux-next patch
- then used Anatoly(s) vfio-noiommu dpdk patch [1]
- Added ioport pci bar map code support in kernel
- Did similar change at dpdk(s) eal_pci_vfio.c file

I could able to run testpmd on virtio-net dpdk's pmd driver..
I am cleaning the patches / changes in kernel and dpdk, soon posting
the v3 patchset..

Thanks for the input!

[1] http://dpdk.org/dev/patchwork/patch/9619/

>> Alex