Re: [SPDK] Bluestore device’s device selector for Samsung NVMe

[Yang, Ziye <ziye.yang at intel.com>]

[-- Attachment #1: Type: text/plain, Size: 6817 bytes --]

Hi  Yanko,

Some additional info (copied from SPDK mailing list archive: https://lists.01.org/pipermail/spdk/2018-April/001770.html), which I answered the questions from other before. You may work with Tushar to fix the document in Ceph community.


You need to bind the NVMe driver in UIO or VFIO first. If you want to bind to it to uio, make sure that you disable iommu in grub during the boot, and you need to have uio_pci_generic in loaded kernel module.  Currently, setup.sh in SPDK first tries to bind VFIO first then UIO, you can just slightly modify your script and bind to uio first.  In my platform, I used uio to bind NVMe SSD device.

To make sure that you can enable  Ceph + SPDK,  you can do the following work first after you bind your NVMe SSD device to uio driver.

0: Make sure that the hugepages are allocated via scripts/setup.sh
1: use examples/nvme/identify/identify in SPDK folder to find the  sn of the NVMe device. This is used to find the NVMe SSDs.
2  Use vstart.sh (The single machine model) to test enable Ceph + SPDK, e.g.,  you can use the following commands:

MON=3 OSD=1 MDS=1 MGR=1 RGW=1 ../src/vstart.sh -n -x -l -b

You need to modify vstart.sh to generate the ceph.conf

The following contents should be put in the configuration of [osd] part
bluestore_block_db_path = ""
bluestore_block_db_size = 0
bluestore_block_db_create = false
bluestore_block_wal_path = ""
bluestore_block_wal_size = 0
bluestore_block_wal_create = false
bluestore_spdk_mem = 2048    #allocate 2048 pages, you can change it.
bluestore_block_path = spdk:55cd2e404be053be   # This should be your SN.

3  For the cluster configuration, You need to follow the configuration described in step 2. Currently only bluestore can use SPDK user space NVMe driver.



Best Regards
Ziye Yang 


-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Tushar Gohad
Sent: Tuesday, January 15, 2019 12:45 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] Bluestore device’s device selector for Samsung NVMe

Hi Yanko,

Acknowledging what Jim said, the SPDK backend for Bluestore is not widely in use due to limited benefit today (the bottlenecks are elsewhere, outside of the Bluestore blockdev layer), which is why the documentation at [1] is only minimal.  That being said, we should get the documentation fixed.  I see that you posted this message to ceph-users.  I will work with you there to get this addressed.

Thanks,
Tushar

On Mon, Jan 14, 2019 at 11:49 AM Harris, James R <james.r.harris(a)intel.com>
wrote:

> Hi Yanko,
>
> I'm not sure if such a manual exists - one of the Ceph mailing lists 
> may be a better forum to get pointers for that.
>
> Regards,
>
> -Jim
>
>
> On 1/14/19, 10:41 AM, "SPDK on behalf of Yanko Davila" < 
> spdk-bounces(a)lists.01.org on behalf of davila(a)cognitec.com> wrote:
>
>     Hi Jim
>
>     Thank you for your answer. I was able to find the device selector. 
> Now I have an issue understanding the steps to activate the osd. Once 
> I setup spdk the device disappears from lsblk as expected. So the ceph 
> manual is not very helpful after spdk is enabled. Is there any manual 
> that walks you through the steps to add an spdk nvme to ceph ?? Thanks 
> again for your time.
>
>     Yanko.
>
>     > On Jan 14, 2019, at 4:18 PM, Harris, James R < 
> james.r.harris(a)intel.com> wrote:
>     >
>     > Hi Yanko,
>     >
>     > I think the Ceph documentation is not quite accurate.  Not all 
> Intel NVMe SSDs have device ID 0x0953.  Similarly, Samsung NVMe SSDs 
> likely have several different device IDs depending on the model.
>     >
>     > I would suggest just doing "lspci | grep Volatile" to find the 
> NVMe SSD bus/device/functions on your system.
>     >
>     > -Jim
>     >
>     >
>     > On 1/14/19, 7:12 AM, "SPDK on behalf of Yanko Davila" < 
> spdk-bounces(a)lists.01.org on behalf of davila(a)cognitec.com> wrote:
>     >
>     >    Hello
>     >
>     >    My name is Yanko Davila, I´m new to ceph so please pardon my
> ignorance.
>     >    I have a question about Bluestore and SPDK.
>     >
>     >    I´m currently running ceph version:
>     >
>     >    ceph version 12.2.10 (177915764b752804194937482a39e95e0ca3de94)
> luminous
>     >    (stable)
>     >
>     >    on Debian:
>     >
>     >    Linux <hostname> 4.9.0-8-amd64 #1 SMP Debian 4.9.110-3+deb9u4
>     >    (2018-08-21) x86_64 GNU/Linux
>     >
>     >    Distributor ID:    Debian
>     >    Description:    Debian GNU/Linux 9.5 (stretch)
>     >    Release:    9.5
>     >    Codename:    stretch
>     >
>     >    I´m trying to add an NVMe osd using bluestore but I´m struggling
> to find
>     >    the device selector for that NVMe. So far I´ve been  able to
> compile
>     >    spdk and succesfully run the setup.sh script. I can also
> succesfully run
>     >    the identify example which leads me to think that spdk is working
> as
>     >    expected.
>     >
>     >    When I read the online manual (
>     >
> http://docs.ceph.com/docs/master/rados/configuration/bluestore-config-ref/#spdk-usage
>     >    ) it gives an example for an Intel PCIe SSD:
>     >
>     >    <quote>
>     >
>     >    For example, users can find the device selector of an Intel PCIe
> SSD with:
>     >
>     >    $ lspci -mm -n -D -d 8086:0953
>     >
>     >    <quote />
>     >
>     >    When I try the same command adjusting for my Samsung SSD it
> returns
>     >    nothing or the return is just blank here is what I tried:
>     >
>     >    $ lspci -mm -n -D -d 144d:a801
>     >
>     >
>     >    Assuming that I gave you enough information. Can anyone spot what
> I´m
>     >    doing wrong? Does spdk only works on Intel SSDs ? Any comment is
> highly
>     >    appreciated. Thank You for your time.
>     >
>     >    Yanko.
>     >
>     >
>     >    _______________________________________________
>     >    SPDK mailing list
>     >    SPDK(a)lists.01.org
>     >    https://lists.01.org/mailman/listinfo/spdk
>     >
>     >
>     > _______________________________________________
>     > SPDK mailing list
>     > SPDK(a)lists.01.org
>     > https://lists.01.org/mailman/listinfo/spdk
>     _______________________________________________
>     SPDK mailing list
>     SPDK(a)lists.01.org
>     https://lists.01.org/mailman/listinfo/spdk
>
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk