[SPDK] Re: NVMe devices with CMB

[SPDK] Re: NVMe devices with CMB

[Kit Chow]

[-- Attachment #1: Type: text/plain, Size: 1827 bytes --]

Hi Benjamin,

I found the following qemu nvme blog - http://blog.frankenmichl.de/2018/02/13/add-nvme-device-to-vm/

Following the instructions, I get the NVMe device listed below with a bar4 size of 4K. Is this the CMB and is it large enough for small p2p copies?

Thanks
Kit

----

00:03.0 Non-Volatile memory controller: Intel Corporation QEMU NVM Express Controller (rev 02) (prog-if 02 [NVM Express])
        Subsystem: Red Hat, Inc. QEMU Virtual Machine
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin A routed to IRQ 23
        NUMA node: 0
        Region 0: Memory at fc070000 (64-bit, non-prefetchable) [size=8K]
        Region 4: Memory at fc073000 (32-bit, non-prefetchable) [size=4K]
        Capabilities: [40] MSI-X: Enable+ Count=64 Masked-
                Vector table: BAR=4 offset=00000000
                PBA: BAR=4 offset=00000800
        Capabilities: [80] Express (v2) Root Complex Integrated Endpoint, MSI 00
                DevCap: MaxPayload 128 bytes, PhantFunc 0
                        ExtTag- RBE+
                DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
                        RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
                        MaxPayload 128 bytes, MaxReadReq 128 bytes
                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
                DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
        Kernel driver in use: nvme
        Kernel modules: nvme

[SPDK] Re: NVMe devices with CMB

[Klaus Jensen]

[-- Attachment #1: Type: text/plain, Size: 687 bytes --]

On Nov  4 22:39, Kit Chow wrote:
> Hi Benjamin,
> 
> I found the following qemu nvme blog - http://blog.frankenmichl.de/2018/02/13/add-nvme-device-to-vm/
> 
> Following the instructions, I get the NVMe device listed below with a bar4 size of 4K. Is this the CMB and is it large enough for small p2p copies?
> 

Hi Kit,

You need to specifically enable the CMB with the `cmb_size_mb`
parameters (argument in MBs).

  -device "nvme,id=nvme0,serial=deadbeef,cmb_size_mb=64"

Looks like you are using a pretty old version of QEMU. In current
versions the CMB is exclusively using BAR 4 - I think what you are
seeing is an older version where the MSI-X tables were in BAR4.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[SPDK] Re: NVMe devices with CMB

[Walker, Benjamin]

[-- Attachment #1: Type: text/plain, Size: 482 bytes --]


> From: Kit Chow <kchow(a)maxlinear.com>
> 
> Hi,
> 
> Do you know of any NVMe SSDs that have CMBs?  I've asked Samsung but the
> rep did not seem to know and asked me to send my question to the samsung
> support email. I have not heard back yet.
> 
> If NVMe SSDs w/ CMBs aren't currently/readily available, what NVMe devices
> are being used by SPDK to do p2p copies.

I believe you can use the QEMU emulated NVMe SSD to test CMB. That's probably the easiest way.