From mboxrd@z Thu Jan 1 00:00:00 1970 Content-Type: multipart/mixed; boundary="===============3804463484387721605==" MIME-Version: 1.0 From: Yang, Ziye Subject: =?utf-8?q?Re=3A_=5BSPDK=5D_Bluestore_device=E2=80=99s_device_selector_for?= =?utf-8?q?_Samsung_NVMe?= Date: Tue, 15 Jan 2019 05:03:20 +0000 Message-ID: In-Reply-To: CAJYz097VgB6+CCqFZvw1zZ3S2501YOA5zaB61GGOti6CG+7uOw@mail.gmail.com List-ID: To: spdk@lists.01.org --===============3804463484387721605== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Hi Yanko, Some additional info (copied from SPDK mailing list archive: https://lists.= 01.org/pipermail/spdk/2018-April/001770.html), which I answered the questio= ns from other before. You may work with Tushar to fix the document in Ceph = community. You need to bind the NVMe driver in UIO or VFIO first. If you want to bind = to it to uio, make sure that you disable iommu in grub during the boot, and= you need to have uio_pci_generic in loaded kernel module. Currently, setu= p.sh in SPDK first tries to bind VFIO first then UIO, you can just slightly= modify your script and bind to uio first. In my platform, I used uio to b= ind NVMe SSD device. To make sure that you can enable Ceph + SPDK, you can do the following wo= rk first after you bind your NVMe SSD device to uio driver. 0: Make sure that the hugepages are allocated via scripts/setup.sh 1: use examples/nvme/identify/identify in SPDK folder to find the sn of th= e NVMe device. This is used to find the NVMe SSDs. 2 Use vstart.sh (The single machine model) to test enable Ceph + SPDK, e.g= ., you can use the following commands: MON=3D3 OSD=3D1 MDS=3D1 MGR=3D1 RGW=3D1 ../src/vstart.sh -n -x -l -b You need to modify vstart.sh to generate the ceph.conf The following contents should be put in the configuration of [osd] part bluestore_block_db_path =3D "" bluestore_block_db_size =3D 0 bluestore_block_db_create =3D false bluestore_block_wal_path =3D "" bluestore_block_wal_size =3D 0 bluestore_block_wal_create =3D false bluestore_spdk_mem =3D 2048 #allocate 2048 pages, you can change it. bluestore_block_path =3D spdk:55cd2e404be053be # This should be your SN. 3 For the cluster configuration, You need to follow the configuration desc= ribed in step 2. Currently only bluestore can use SPDK user space NVMe driv= er. Best Regards Ziye Yang = -----Original Message----- From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Tushar Gohad Sent: Tuesday, January 15, 2019 12:45 PM To: Storage Performance Development Kit Subject: Re: [SPDK] Bluestore device=E2=80=99s device selector for Samsung = NVMe Hi Yanko, Acknowledging what Jim said, the SPDK backend for Bluestore is not widely i= n use due to limited benefit today (the bottlenecks are elsewhere, outside = of the Bluestore blockdev layer), which is why the documentation at [1] is = only minimal. That being said, we should get the documentation fixed. I s= ee that you posted this message to ceph-users. I will work with you there = to get this addressed. Thanks, Tushar On Mon, Jan 14, 2019 at 11:49 AM Harris, James R wrote: > Hi Yanko, > > I'm not sure if such a manual exists - one of the Ceph mailing lists = > may be a better forum to get pointers for that. > > Regards, > > -Jim > > > =EF=BB=BFOn 1/14/19, 10:41 AM, "SPDK on behalf of Yanko Davila" < = > spdk-bounces(a)lists.01.org on behalf of davila(a)cognitec.com> wrote: > > Hi Jim > > Thank you for your answer. I was able to find the device selector. = > Now I have an issue understanding the steps to activate the osd. Once = > I setup spdk the device disappears from lsblk as expected. So the ceph = > manual is not very helpful after spdk is enabled. Is there any manual = > that walks you through the steps to add an spdk nvme to ceph ?? Thanks = > again for your time. > > Yanko. > > > On Jan 14, 2019, at 4:18 PM, Harris, James R < = > james.r.harris(a)intel.com> wrote: > > > > Hi Yanko, > > > > I think the Ceph documentation is not quite accurate. Not all = > Intel NVMe SSDs have device ID 0x0953. Similarly, Samsung NVMe SSDs = > likely have several different device IDs depending on the model. > > > > I would suggest just doing "lspci | grep Volatile" to find the = > NVMe SSD bus/device/functions on your system. > > > > -Jim > > > > > > On 1/14/19, 7:12 AM, "SPDK on behalf of Yanko Davila" < = > spdk-bounces(a)lists.01.org on behalf of davila(a)cognitec.com> wrote: > > > > Hello > > > > My name is Yanko Davila, I=C2=B4m new to ceph so please pardon my > ignorance. > > I have a question about Bluestore and SPDK. > > > > I=C2=B4m currently running ceph version: > > > > ceph version 12.2.10 (177915764b752804194937482a39e95e0ca3de94) > luminous > > (stable) > > > > on Debian: > > > > Linux 4.9.0-8-amd64 #1 SMP Debian 4.9.110-3+deb9u4 > > (2018-08-21) x86_64 GNU/Linux > > > > Distributor ID: Debian > > Description: Debian GNU/Linux 9.5 (stretch) > > Release: 9.5 > > Codename: stretch > > > > I=C2=B4m trying to add an NVMe osd using bluestore but I=C2=B4m = struggling > to find > > the device selector for that NVMe. So far I=C2=B4ve been able to > compile > > spdk and succesfully run the setup.sh script. I can also > succesfully run > > the identify example which leads me to think that spdk is working > as > > expected. > > > > When I read the online manual ( > > > http://docs.ceph.com/docs/master/rados/configuration/bluestore-config-ref= /#spdk-usage > > ) it gives an example for an Intel PCIe SSD: > > > > > > > > For example, users can find the device selector of an Intel PCIe > SSD with: > > > > $ lspci -mm -n -D -d 8086:0953 > > > > > > > > When I try the same command adjusting for my Samsung SSD it > returns > > nothing or the return is just blank here is what I tried: > > > > $ lspci -mm -n -D -d 144d:a801 > > > > > > Assuming that I gave you enough information. Can anyone spot what > I=C2=B4m > > doing wrong? Does spdk only works on Intel SSDs ? Any comment is > highly > > appreciated. Thank You for your time. > > > > Yanko. > > > > > > _______________________________________________ > > SPDK mailing list > > SPDK(a)lists.01.org > > https://lists.01.org/mailman/listinfo/spdk > > > > > > _______________________________________________ > > SPDK mailing list > > SPDK(a)lists.01.org > > https://lists.01.org/mailman/listinfo/spdk > _______________________________________________ > SPDK mailing list > SPDK(a)lists.01.org > https://lists.01.org/mailman/listinfo/spdk > > > _______________________________________________ > SPDK mailing list > SPDK(a)lists.01.org > https://lists.01.org/mailman/listinfo/spdk > _______________________________________________ SPDK mailing list SPDK(a)lists.01.org https://lists.01.org/mailman/listinfo/spdk --===============3804463484387721605==--