[SPDK] Re: sysfs-pci and SPDK IOAT

[SPDK] Re: sysfs-pci and SPDK IOAT

[wiz.chen]

[-- Attachment #1: Type: text/plain, Size: 434 bytes --]

Hi,

we change to use SPDK pci APIs to access NTB.

After spdk_pci_device_map_bar(), it can get the "mapped address" for accessing remote memory.

It's okay that using memcpy to copy "data" to "mapped address":
   memcpy(mapped_addr,  data_buf, size);

However, when using SPDK IOAT:
   spdk_ioat_build_copy(ioat, cb_arg, cb_fn, mapped_addr, data_buf, size);
 
    It will fail to  translate mapped_addr by spdk_vtophys().

[SPDK] Re: sysfs-pci and SPDK IOAT

[Yufan Chen]

[-- Attachment #1: Type: text/plain, Size: 575 bytes --]

Hi,

  Because changing ntb bar size, It needs to upgrade EEPROM.
  Thus, I use the patch: https://staging.gerrithub.io/c/spdk/dpdk/+/441070
        to check if spdk_mem_register can work or not.

   However, if I create a nvmf target, it will cause segfault.

   After investigation,  create nvmf target will call mem_map_notify_walk() to register all
         mapped memory.

  It seems that the mapped memory of NTB PCI bar cannot be registered into RDMA
         (Note: I use 'mlx5_dv' provider.)
        
  Do you have any suggestion? 
  Thanks.

-yufan

[SPDK] Re: sysfs-pci and SPDK IOAT

[Harris, James R]

[-- Attachment #1: Type: text/plain, Size: 886 bytes --]

Hi,

You'll need to do the alignment yourself for now.  I cannot vouch for the correctness of the vfio patch.

-Jim


On 3/22/21, 8:25 PM, "wiz.chen(a)gmail.com" <wiz.chen(a)gmail.com> wrote:

    hi,

      I check spdk_mem_register(), it needs 2-MB aligned address.

      After some investigation:

      1.  nvme cmb --> nvme_pcie_ctrlr_map_io_cmb():
              it do 2MB alignment here.

      2. vfio: map all PCI BARs in 2MB-aligned virtual addresses
          https://staging.gerrithub.io/c/spdk/dpdk/+/441070

         Does the patch: "vfio: map all PCI BARs in 2MB-aligned virtual addresses" have any side effect?

          Or  I do the 2MB alignment by myself is a batter choice?
    _______________________________________________
    SPDK mailing list -- spdk(a)lists.01.org
    To unsubscribe send an email to spdk-leave(a)lists.01.org

[SPDK] Re: sysfs-pci and SPDK IOAT

[wiz.chen]

[-- Attachment #1: Type: text/plain, Size: 479 bytes --]

hi,

  I check spdk_mem_register(), it needs 2-MB aligned address.

  After some investigation:

  1.  nvme cmb --> nvme_pcie_ctrlr_map_io_cmb():
          it do 2MB alignment here.

  2. vfio: map all PCI BARs in 2MB-aligned virtual addresses
      https://staging.gerrithub.io/c/spdk/dpdk/+/441070

     Does the patch: "vfio: map all PCI BARs in 2MB-aligned virtual addresses" have any side effect?

      Or  I do the 2MB alignment by myself is a batter choice?

[SPDK] Re: sysfs-pci and SPDK IOAT

[Harris, James R]

[-- Attachment #1: Type: text/plain, Size: 818 bytes --]

Hi,

Can you try using spdk_mem_register() after you have called spdk_pci_device_map_bar()?

-Jim


On 3/22/21, 7:05 PM, "wiz.chen(a)gmail.com" <wiz.chen(a)gmail.com> wrote:

    Hi,

    we change to use SPDK pci APIs to access NTB.

    After spdk_pci_device_map_bar(), it can get the "mapped address" for accessing remote memory.

    It's okay that using memcpy to copy "data" to "mapped address":
       memcpy(mapped_addr,  data_buf, size);

    However, when using SPDK IOAT:
       spdk_ioat_build_copy(ioat, cb_arg, cb_fn, mapped_addr, data_buf, size);

        It will fail to  translate mapped_addr by spdk_vtophys().
    _______________________________________________
    SPDK mailing list -- spdk(a)lists.01.org
    To unsubscribe send an email to spdk-leave(a)lists.01.org