[SPDK] Re: Using cmb_copy in QEMU environment

[SPDK] Re: Using cmb_copy in QEMU environment

[Harris, James R]

[-- Attachment #1: Type: text/plain, Size: 878 bytes --]



On 4/20/20, 8:38 AM, "Stephen  Bates" <sbates(a)raithlin.com> wrote:

    Hi All

    > Thanks for reporting, could you submit a github issue? Thanks.

    Can you point me to the issue when created. I added the cmb_copy example a while back and it was working in QEMU at that time. I suspect this is a memory allocation regression and would like to track the issue. Note the Linux kernel equivalent  (p2pdma) *does* still work inside QEMUwith fake NVMe SSDs. So this seems to be an SPDK specific thing...

Hi Archie and Changpeng,

Could you also try commit 3429f97?  This commit is just before some CMB-related patches that were recently merged.

-Jim


    Cheers

    Stephen 




    _______________________________________________
    SPDK mailing list -- spdk(a)lists.01.org
    To unsubscribe send an email to spdk-leave(a)lists.01.org

[SPDK] Re: Using cmb_copy in QEMU environment

[archiemorningstar88]

[-- Attachment #1: Type: text/plain, Size: 918 bytes --]

Hi Jim,

I see similar error with 3429f97.

Archie

> On 4/20/20, 8:38 AM, "Stephen  Bates" <sbates(a)raithlin.com&gt; wrote:
> 
>     Hi All
> 
> 
>     Can you point me to the issue when created. I added the cmb_copy example a while back
> and it was working in QEMU at that time. I suspect this is a memory allocation regression
> and would like to track the issue. Note the Linux kernel equivalent  (p2pdma) *does* still
> work inside QEMUwith fake NVMe SSDs. So this seems to be an SPDK specific thing...
> 
> Hi Archie and Changpeng,
> 
> Could you also try commit 3429f97?  This commit is just before some CMB-related patches
> that were recently merged.
> 
> -Jim
> 
> 
>     Cheers
> 
>     Stephen 
> 
> 
> 
> 
>     _______________________________________________
>     SPDK mailing list -- spdk(a)lists.01.org
>     To unsubscribe send an email to spdk-leave(a)lists.01.org

[SPDK] Re: Using cmb_copy in QEMU environment

[archiemorningstar88]

[-- Attachment #1: Type: text/plain, Size: 2509 bytes --]

Issue created:

https://github.com/spdk/spdk/issues/1359

Archie
> Hi Archie,
> 
> Thanks for reporting, could you submit a github issue? Thanks.
> 
> I used the latest SPDK code with the submission queue in the CMB, it also got failure with
> CMB, I will take a look for the issue.
> 
> 
> > -----Original Message-----
> > From: archiemorningstar88(a)gmail.com <archiemorningstar88(a)gmail.com&gt;
> > Sent: Sunday, April 19, 2020 4:58 AM
> > To: spdk(a)lists.01.org
> > Subject: [SPDK] Using cmb_copy in QEMU environment
> > 
> > Hi,
> > 
> > I've been trying to get cmb_copy (examples/nvme/cmb_copy) to work using an
> > emulated SSD with CMB in a QEMU environment, but cmb_copy cannot allocate
> > a CMB buffer. The identify (examples/nvme/identify) reports the two emulated
> > SSD's I created have 128MB of CMB. I'm running QEMU v4.2.0 with the guest OS
> > Ubuntu 18.04. I've also tried various revisions of SPDK (20.01.1, 19.10.1,
> 19.04.1,
> > 18.04.1) with similar results.
> > 
> > I see these two errors print out which I believe couldn't register the memory so
> > the call to spdk_nvme_ctrlr_alloc_cmb_io_buffer() fails (pctrlr-
> > >cmb_bar_virt_addr is NULL):
> > could not get phys addr for 0x1088200000
> > nvme_pcie.c: 554:nvme_pcie_ctrlr_map_cmb: *ERROR*: spdk_mem_register()
> > failed
> > 
> > I tried to trace the code and see that when it searches for the physical address
> > g_vtophys_pci_devices list is empty when trying to register the CMB memory
> > space. It only gets later populated with the two SSDs when cleanup_pci_devices()
> > is called at the end of spdk_pci_device_attach().
> > 
> > My qemu startup:
> > x86_64-softmmu/qemu-system-x86_64 \
> > -cpu host \
> > -machine q35,accel=kvm,kernel-irqchip=split \
> > -smp 8 \
> > -m 32768 \
> > -enable-kvm \
> > -drive if=virtio,file=test1.qcow2,cache=none \
> > -vnc :1 \
> > -device e1000,netdev=net0 \
> > -netdev user,id=net0,hostfwd=tcp::5555-:22 \
> > -drive file=nvme0.img,if=none,id=D22,format=raw \
> > -device nvme,drive=D22,serial=1234,cmb_size_mb=128 \
> > -drive file=nvme1.img,if=none,id=D23,format=raw \
> > -device nvme,drive=D23,serial=5678,cmb_size_mb=128 \
> > -device intel-iommu,intremap=on \
> > 
> > If anyone has any suggestions it would be greatly appreciated.
> > 
> > Archie
> > _______________________________________________
> > SPDK mailing list -- spdk(a)lists.01.org
> > To unsubscribe send an email to spdk-leave(a)lists.01.org

[SPDK] Re: Using cmb_copy in QEMU environment

[Stephen Bates]

[-- Attachment #1: Type: text/plain, Size: 463 bytes --]

Hi All

> Thanks for reporting, could you submit a github issue? Thanks.

Can you point me to the issue when created. I added the cmb_copy example a while back and it was working in QEMU at that time. I suspect this is a memory allocation regression and would like to track the issue. Note the Linux kernel equivalent  (p2pdma) *does* still work inside QEMUwith fake NVMe SSDs. So this seems to be an SPDK specific thing...

Cheers
 
Stephen 



[SPDK] Re: Using cmb_copy in QEMU environment

[Liu, Changpeng]

[-- Attachment #1: Type: text/plain, Size: 2316 bytes --]

Hi Archie,

Thanks for reporting, could you submit a github issue? Thanks.

I used the latest SPDK code with the submission queue in the CMB, it also got failure with CMB, I will take a look for the issue.


> -----Original Message-----
> From: archiemorningstar88(a)gmail.com <archiemorningstar88(a)gmail.com>
> Sent: Sunday, April 19, 2020 4:58 AM
> To: spdk(a)lists.01.org
> Subject: [SPDK] Using cmb_copy in QEMU environment
> 
> Hi,
> 
> I've been trying to get cmb_copy (examples/nvme/cmb_copy) to work using an
> emulated SSD with CMB in a QEMU environment, but cmb_copy cannot allocate
> a CMB buffer. The identify (examples/nvme/identify) reports the two emulated
> SSD's I created have 128MB of CMB. I'm running QEMU v4.2.0 with the guest OS
> Ubuntu 18.04. I've also tried various revisions of SPDK (20.01.1, 19.10.1, 19.04.1,
> 18.04.1) with similar results.
> 
> I see these two errors print out which I believe couldn't register the memory so
> the call to spdk_nvme_ctrlr_alloc_cmb_io_buffer() fails (pctrlr-
> >cmb_bar_virt_addr is NULL):
> could not get phys addr for 0x1088200000
> nvme_pcie.c: 554:nvme_pcie_ctrlr_map_cmb: *ERROR*: spdk_mem_register()
> failed
> 
> I tried to trace the code and see that when it searches for the physical address
> g_vtophys_pci_devices list is empty when trying to register the CMB memory
> space. It only gets later populated with the two SSDs when cleanup_pci_devices()
> is called at the end of spdk_pci_device_attach().
> 
> My qemu startup:
> x86_64-softmmu/qemu-system-x86_64 \
> -cpu host \
> -machine q35,accel=kvm,kernel-irqchip=split \
> -smp 8 \
> -m 32768 \
> -enable-kvm \
> -drive if=virtio,file=test1.qcow2,cache=none \
> -vnc :1 \
> -device e1000,netdev=net0 \
> -netdev user,id=net0,hostfwd=tcp::5555-:22 \
> -drive file=nvme0.img,if=none,id=D22,format=raw \
> -device nvme,drive=D22,serial=1234,cmb_size_mb=128 \
> -drive file=nvme1.img,if=none,id=D23,format=raw \
> -device nvme,drive=D23,serial=5678,cmb_size_mb=128 \
> -device intel-iommu,intremap=on \
> 
> If anyone has any suggestions it would be greatly appreciated.
> 
> Archie
> _______________________________________________
> SPDK mailing list -- spdk(a)lists.01.org
> To unsubscribe send an email to spdk-leave(a)lists.01.org