Requirement to get BAR pci_bus_address in user space

Requirement to get BAR pci_bus_address in user space

[Srinath Mannam]

Hi Bjorn,


We observed an issue in NVMe CMD cards in SPDK NVMe libraries.
Similar issue fixed with "nvme-pci: Use PCI bus address for
data/queues in CMB" commit in nvme kernel driver.
As per the fix, pci_bus_address of CMB bar is required to program to NVMe cards.

We need to implement similar fix for SPDK NVMe libraries also.

User space libraries get pci device resource information (resource phy
address, resource size) from resource file in sysfs.
But no mechanism available to get  pci_bus_address of corresponding
resource physical address in sysfs.

Please suggest me if any alternate mechanism available to get
pci_bus_address in user space.

Thank you.

Regards,
Srinath.

Re: Requirement to get BAR pci_bus_address in user space

[Christoph Hellwig]

The only safe way to use PCI(e) devices in userspace is through vfio.
I think that is where you need to take your inquiries.

Re: Requirement to get BAR pci_bus_address in user space

[Srinath Mannam]

++ Alex Williamson, kvm,

Hi Christoph,

Thank you for quick reply.

If we want to add this in vfio then I think we need to do the same in
uio case also.

As I mentioned in previous mail, in the current implementation
resource information (address and size) is gathering from resource
named file created in /sys directory.
So I expect it would be better to have similar method as existing in sysfs.


Regards,
Srinath.

On Thu, Jun 14, 2018 at 3:50 PM, Christoph Hellwig <hch@lst.de> wrote:
> The only safe way to use PCI(e) devices in userspace is through vfio.
> I think that is where you need to take your inquiries.

Re: Requirement to get BAR pci_bus_address in user space

[okaya]

On 2018-06-14 06:29, Srinath Mannam wrote:
> ++ Alex Williamson, kvm,
> 
> Hi Christoph,
> 
> Thank you for quick reply.
> 
> If we want to add this in vfio then I think we need to do the same in
> uio case also.
> 
> As I mentioned in previous mail, in the current implementation
> resource information (address and size) is gathering from resource
> named file created in /sys directory.
> So I expect it would be better to have similar method as existing in 
> sysfs.
> 

Can you give some info on why you need the actual bar address value?

> 
> Regards,
> Srinath.
> 
> On Thu, Jun 14, 2018 at 3:50 PM, Christoph Hellwig <hch@lst.de> wrote:
>> The only safe way to use PCI(e) devices in userspace is through vfio.
>> I think that is where you need to take your inquiries.

Re: Requirement to get BAR pci_bus_address in user space

[Srinath Mannam]

Hi Sinan Kaya,

Here are the details,

The issue is, For CMB cards SQs are allocated inside device BAR memory
which is different from normal cards.
In Normal cards SQ memory allocated at host side.
In both the cases physical address of CQ memory is programmed in NVMe
controller register.
This method works for normal cards because CQ memory is at host side.
But in CMB cards pci bus address equivalent to CQ memory needs to program.

More details are in the patch: nvme-pci: Use PCI bus address for
data/queues in CMB.

With the above patch issue is fixed in the NVMe kernel driver, But
similar fix is required in SPDK library also.
So, We need a mechanism to get pci_bus_address in user space libraries
to address this issue.

Regards,
Srinath.


On Thu, Jun 14, 2018 at 4:03 PM,  <okaya@codeaurora.org> wrote:
> On 2018-06-14 06:29, Srinath Mannam wrote:
>>
>> ++ Alex Williamson, kvm,
>>
>> Hi Christoph,
>>
>> Thank you for quick reply.
>>
>> If we want to add this in vfio then I think we need to do the same in
>> uio case also.
>>
>> As I mentioned in previous mail, in the current implementation
>> resource information (address and size) is gathering from resource
>> named file created in /sys directory.
>> So I expect it would be better to have similar method as existing in
>> sysfs.
>>
>
> Can you give some info on why you need the actual bar address value?
>
>>
>> Regards,
>> Srinath.
>>
>> On Thu, Jun 14, 2018 at 3:50 PM, Christoph Hellwig <hch@lst.de> wrote:
>>>
>>> The only safe way to use PCI(e) devices in userspace is through vfio.
>>> I think that is where you need to take your inquiries.

Re: Requirement to get BAR pci_bus_address in user space

[Keith Busch]

On Thu, Jun 14, 2018 at 04:18:15PM +0530, Srinath Mannam wrote:
> The issue is, For CMB cards SQs are allocated inside device BAR memory
> which is different from normal cards.
> In Normal cards SQ memory allocated at host side.
> In both the cases physical address of CQ memory is programmed in NVMe
> controller register.
> This method works for normal cards because CQ memory is at host side.
> But in CMB cards pci bus address equivalent to CQ memory needs to program.

How do you know if the address that you can see is even the real bus
address? You could be running against a direct assigned device in a
guest where you don't get to see the actual bus address.

Re: Requirement to get BAR pci_bus_address in user space

[Alex Williamson]

On Thu, 14 Jun 2018 16:18:15 +0530
Srinath Mannam <srinath.mannam@broadcom.com> wrote:

> Hi Sinan Kaya,
> 
> Here are the details,
> 
> The issue is, For CMB cards SQs are allocated inside device BAR memory
> which is different from normal cards.
> In Normal cards SQ memory allocated at host side.
> In both the cases physical address of CQ memory is programmed in NVMe
> controller register.
> This method works for normal cards because CQ memory is at host side.
> But in CMB cards pci bus address equivalent to CQ memory needs to program.
> 
> More details are in the patch: nvme-pci: Use PCI bus address for
> data/queues in CMB.
> 
> With the above patch issue is fixed in the NVMe kernel driver, But
> similar fix is required in SPDK library also.
> So, We need a mechanism to get pci_bus_address in user space libraries
> to address this issue.

I don't understand the CQ vs CMB, but I think I gather that there's some
sort of buffer that's allocated from within the devices MMIO BAR and
some programming of the device needs to reference that buffer.
Wouldn't you therefore use the vfio type1 IOMMU MAP_DMA ioctl to map
the BAR into the IOVA address space and you can then use the IOVA +
offset into the BAR for the device to reference the buffer?  It seems
this is the same way we'd setup a peer-to-peer mapping, but we're using
it for the device to reference itself effectively.  Thanks,

Alex

Re: Requirement to get BAR pci_bus_address in user space

[Christoph Hellwig]

On Thu, Jun 14, 2018 at 08:50:15AM -0600, Alex Williamson wrote:
> I don't understand the CQ vs CMB, but I think I gather that there's some
> sort of buffer that's allocated from within the devices MMIO BAR and
> some programming of the device needs to reference that buffer.
> Wouldn't you therefore use the vfio type1 IOMMU MAP_DMA ioctl to map
> the BAR into the IOVA address space and you can then use the IOVA +
> offset into the BAR for the device to reference the buffer?  It seems
> this is the same way we'd setup a peer-to-peer mapping, but we're using
> it for the device to reference itself effectively.  Thanks,

That's exactly what I meant..

Re: Requirement to get BAR pci_bus_address in user space

[Walker, Benjamin]

On Thu, 2018-06-14 at 08:50 -0600, Alex Williamson wrote:
> On Thu, 14 Jun 2018 16:18:15 +0530
> Srinath Mannam <srinath.mannam@broadcom.com> wrote:
> 
> > Hi Sinan Kaya,
> > 
> > Here are the details,
> > 
> > The issue is, For CMB cards SQs are allocated inside device BAR memory
> > which is different from normal cards.
> > In Normal cards SQ memory allocated at host side.
> > In both the cases physical address of CQ memory is programmed in NVMe
> > controller register.
> > This method works for normal cards because CQ memory is at host side.
> > But in CMB cards pci bus address equivalent to CQ memory needs to program.
> > 
> > More details are in the patch: nvme-pci: Use PCI bus address for
> > data/queues in CMB.
> > 
> > With the above patch issue is fixed in the NVMe kernel driver, But
> > similar fix is required in SPDK library also.
> > So, We need a mechanism to get pci_bus_address in user space libraries
> > to address this issue.
> 
> I don't understand the CQ vs CMB, but I think I gather that there's some
> sort of buffer that's allocated from within the devices MMIO BAR and
> some programming of the device needs to reference that buffer.
> Wouldn't you therefore use the vfio type1 IOMMU MAP_DMA ioctl to map
> the BAR into the IOVA address space and you can then use the IOVA +
> offset into the BAR for the device to reference the buffer?  It seems
> this is the same way we'd setup a peer-to-peer mapping, but we're using
> it for the device to reference itself effectively.  Thanks,

SPDK already does this today. It's capable of placing submission and completion
queues in the device's controller memory buffer (CMB) and will do all of the
right things for both uio (look up physical address) and vfio (send
IOMMU_MAP_DMA ioctl to set a iova and use that). It additionally already
supports using the CMB as a target for peer-to-peer data transfers.

This was just confirmed to work on the most recent SPDK release. If you are not
on the most recent release, I recommend trying that first. If there is still
some issue, please do take it to the SPDK mailing list.

Thanks,
Ben

Re: Requirement to get BAR pci_bus_address in user space

[Srinath Mannam]

Thank you very much Alex and Ben for all the details.

Please find my comments in lined..


On Fri, Jun 15, 2018 at 1:34 AM, Walker, Benjamin
<benjamin.walker@intel.com> wrote:
> On Thu, 2018-06-14 at 08:50 -0600, Alex Williamson wrote:
>> On Thu, 14 Jun 2018 16:18:15 +0530
>> Srinath Mannam <srinath.mannam@broadcom.com> wrote:
>>
>> > Hi Sinan Kaya,
>> >
>> > Here are the details,
>> >
>> > The issue is, For CMB cards SQs are allocated inside device BAR memory
>> > which is different from normal cards.
>> > In Normal cards SQ memory allocated at host side.
>> > In both the cases physical address of CQ memory is programmed in NVMe
>> > controller register.
>> > This method works for normal cards because CQ memory is at host side.
>> > But in CMB cards pci bus address equivalent to CQ memory needs to program.
>> >
>> > More details are in the patch: nvme-pci: Use PCI bus address for
>> > data/queues in CMB.
>> >
>> > With the above patch issue is fixed in the NVMe kernel driver, But
>> > similar fix is required in SPDK library also.
>> > So, We need a mechanism to get pci_bus_address in user space libraries
>> > to address this issue.
>>
>> I don't understand the CQ vs CMB, but I think I gather that there's some
Sorry, It was mistake, I meant SQ not CQ.
>> sort of buffer that's allocated from within the devices MMIO BAR and
>> some programming of the device needs to reference that buffer.
>> Wouldn't you therefore use the vfio type1 IOMMU MAP_DMA ioctl to map
>> the BAR into the IOVA address space and you can then use the IOVA +
>> offset into the BAR for the device to reference the buffer?  It seems
>> this is the same way we'd setup a peer-to-peer mapping, but we're using
>> it for the device to reference itself effectively.  Thanks,
>
> SPDK already does this today. It's capable of placing submission and completion
> queues in the device's controller memory buffer (CMB) and will do all of the
> right things for both uio (look up physical address) and vfio (send
> IOMMU_MAP_DMA ioctl to set a iova and use that). It additionally already
> supports using the CMB as a target for peer-to-peer data transfers.
>
> This was just confirmed to work on the most recent SPDK release. If you are not
> on the most recent release, I recommend trying that first. If there is still
> some issue, please do take it to the SPDK mailing list.
>
We are using SPDK version 18.04.
Is fix available in 18.04? Please point me to the patch with which fix
was added.

> Thanks,
> Ben