All of lore.kernel.org
 help / color / mirror / Atom feed
* [SPDK] Re: For Dummies guide to setting up ramdisk NVMe/TCP target
@ 2020-10-14  8:50 Curtin, Eric
  0 siblings, 0 replies; 8+ messages in thread
From: Curtin, Eric @ 2020-10-14  8:50 UTC (permalink / raw)
  To: spdk

[-- Attachment #1: Type: text/plain, Size: 4194 bytes --]

How would I get my CPU mask on Linux? I'm running on a Virtualbox VM. I also used 0xf as a CPU mask and this was my output:

./build/bin/nvmf_tgt -m 0xf
[2020-10-14 09:47:07.650399] Starting SPDK v20.10-pre git sha1 779a6bdf8 / DPDK 20.08.0 initialization...
[2020-10-14 09:47:07.651075] [ DPDK EAL parameters: [2020-10-14 09:47:07.651092] nvmf [2020-10-14 09:47:07.651126] --no-shconf [2020-10-14 09:47:07.651132] -c 0xf [2020-10-14 09:47:07.651138] --log-level=lib.eal:6 [2020-10-14 09:47:07.651144] --log-level=lib.cryptodev:5 [2020-10-14 09:47:07.651150] --log-level=user1:6 [2020-10-14 09:47:07.651160] --iova-mode=pa [2020-10-14 09:47:07.651166] --base-virtaddr=0x200000000000 [2020-10-14 09:47:07.651172] --match-allocations [2020-10-14 09:47:07.651181] --file-prefix=spdk_pid702834 [2020-10-14 09:47:07.651187] ]
EAL: FATAL: Cannot use IOVA as 'PA' since physical addresses are not available
EAL: Cannot use IOVA as 'PA' since physical addresses are not available
[2020-10-14 09:47:07.659984] init.c: 577:spdk_env_init: *ERROR*: Failed to initialize DPDK
[2020-10-14 09:47:07.659999] app.c: 535:app_setup_env: *ERROR*: Unable to initialize SPDK env

Regards,

Eric Curtin
Software Engineer

Ovens Campus,
Cork,
Ireland

Dell EMC

Next planned absences: NA
________________________________
From: Rui Chang <Rui.Chang(a)arm.com>
Sent: Wednesday 14 October 2020 08:50
To: Curtin, Eric <Eric.Curtin(a)dell.com>
Cc: spdk(a)lists.01.org <spdk(a)lists.01.org>
Subject: RE: [SPDK] For Dummies guide to setting up ramdisk NVMe/TCP target


[EXTERNAL EMAIL]

Hi Eric



Refer to this page for HOWTO:

https://spdk.io/doc/nvmf.html



You said you want to run on one machine, suppose your machine’s IP is 8.8.8.6



  1.  Build your SPDK

git clone https://github.com/spdk/spdk spdk

cd spdk

git submodule update --init

./configure

make



  1.  Run:



Start target:

                ./build/bin/nvmf_tgt -m 0xf          (-m gives the CPU mask. Normally you can make it on the same CPU socket as network card to get better performance)



On another console, run rpc comamnds to do configuration:

./scripts/rpc.py nvmf_create_transport -t TCP      (there are lots of additional options you can use if you wish)

./scripts/rpc.py bdev_malloc_create -b Malloc0 200 512

./scripts/rpc.py nvmf_create_subsystem nqn.2016-06.io.spdk:cnode1 -a -s SPDK00000000000001 -d SPDK_Controller1

./scripts/rpc.py nvmf_subsystem_add_ns nqn.2016-06.io.spdk:cnode1 Malloc0

./scripts/rpc.py nvmf_subsystem_add_listener nqn.2016-06.io.spdk:cnode1 -t TCP -a 8.8.8.6 -s 4420                 (suppose 8.8.8.6 is the IP you want the target to listen on)



Run io:

./build/examples/perf -r 'trtype:TCP adrfam:IPv4 traddr:8.8.8.6 trsvcid:4420 subnqn:nqn.2016-06.io.spdk:cnode1'  -q 128 -o 4096  -w write -t 100 -c 0xf0  (-c give the cpu mask for perf tool, you’d better make it different with nvmf_tgt to get better performance)



Regards,

Rui





-----Original Message-----

From: eric.curtin(a)dell.com<mailto:eric.curtin(a)dell.com> <eric.curtin(a)dell.com<mailto:eric.curtin(a)dell.com>>

Sent: Thursday, October 1, 2020 3:23 PM

To: spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>

Subject: [SPDK] For Dummies guide to setting up ramdisk NVMe/TCP target



Hi Guys,



I'm just getting started with SPDK and wanted to create a fake ramdisk NVMe/TCP target block device, just so I could simulate NVMe/TCP traffic on a single Linux instance for development purposes.



Anybody know how to set up a machine like this for NVMe/TCP development?

_______________________________________________

SPDK mailing list -- spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>

To unsubscribe send an email to spdk-leave(a)lists.01.org<mailto:spdk-leave(a)lists.01.org>

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

^ permalink raw reply	[flat|nested] 8+ messages in thread
* [SPDK] Re: For Dummies guide to setting up ramdisk NVMe/TCP target
@ 2020-10-16 14:20 Harris, James R
  0 siblings, 0 replies; 8+ messages in thread
From: Harris, James R @ 2020-10-16 14:20 UTC (permalink / raw)
  To: spdk

[-- Attachment #1: Type: text/plain, Size: 9962 bytes --]

Hi Eric,

SPDK will start polled-mode threads on each of the cores specified in your coremask.  So for coremask 0xF0, it would start polled-mode threads on cores 4, 5, 6 and 7.  The number of cores in your coremask really depends on your system and what you want to test - if you give the application more cores to run on, you'll be able to get better overall performance and throughput.

If you don't specify a coremask, it will just use core 0 only.  That's fine for just getting something up and running to play around with.

-Jim


On 10/16/20, 7:13 AM, "Curtin, Eric" <Eric.Curtin(a)dell.com> wrote:

    Thanks guys, I got it working. You were right I was not running as root. The other steps I had to perform in case anyone else ever gets curious (I used 127.0.0.1). The only think I don't understand is the coremasks I used 0xf0 and 0x10, I just guessed those based on random examples I saw online. How do I know what coremasks I want to pick?

    mkdir -p /dev/hugepages
    mountpoint -q /dev/hugepages || mount -t hugetlbfs nodev /dev/hugepages
    echo 1024 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
    build/bin/nvmf_tgt -m 0xf

    In another terminal:

    scripts/rpc.py nvmf_create_transport -t TCP
    scripts/rpc.py bdev_malloc_create -b Malloc0 200 512
    scripts/rpc.py nvmf_create_subsystem nqn.2016-06.io.spdk:cnode1 -a -s SPDK00000000000001 -d SPDK_Controller1
    scripts/rpc.py nvmf_subsystem_add_ns nqn.2016-06.io.spdk:cnode1 Malloc0
    scripts/rpc.py nvmf_subsystem_add_listener nqn.2016-06.io.spdk:cnode1 -t TCP -a 127.0.0.1 -s 4420
    build/examples/perf -r 'trtype:TCP adrfam:IPv4 traddr:127.0.0.1 trsvcid:4420 subnqn:nqn.2016-06.io.spdk:cnode1' -q 128 -o 4096 -w write -t 100 -c 0x10

    Mise le meas/Regards,

    Eric Curtin
    Software Engineer

    Ovens Campus,
    Cork,
    Ireland

    Dell EMC

    Next planned absences: NA
    ________________________________
    From: Rui Chang <Rui.Chang(a)arm.com>
    Sent: Thursday 15 October 2020 02:49
    To: Curtin, Eric <Eric.Curtin(a)dell.com>
    Cc: Storage Performance Development Kit <spdk(a)lists.01.org>
    Subject: RE: [SPDK] Re: For Dummies guide to setting up ramdisk NVMe/TCP target


    [EXTERNAL EMAIL]

    Yeah, it also works on my virtualbox 6.1 with Win10/X86_64 host and Ubuntu 20.4 guest.

    Regards,
    Rui

    -----Original Message-----
    From: Luse, Paul E <paul.e.luse(a)intel.com>
    Sent: Thursday, October 15, 2020 07:48
    To: Storage Performance Development Kit <spdk(a)lists.01.org>; Rui Chang <Rui.Chang(a)arm.com>
    Cc: spdk(a)lists.01.org
    Subject: [SPDK] Re: For Dummies guide to setting up ramdisk NVMe/TCP target

    Hi Eric,

    Yeah, run with sudo.  I get the same thing when I run w/o, it's DPDK unable to get the PA with privilege.

    Thx
    Paul

    From: Luse, Paul E <paul.e.luse(a)intel.com>
    Date: Wednesday, October 14, 2020 at 4:43 PM
    To: Storage Performance Development Kit <spdk(a)lists.01.org>, Rui Chang <Rui.Chang(a)arm.com>
    Cc: spdk(a)lists.01.org <spdk(a)lists.01.org>
    Subject: Re: [SPDK] Re: For Dummies guide to setting up ramdisk NVMe/TCP target Hi Eric,

    Bummer that's not working.  It works OK for my with vbox 6.1 on a mac and ubuntu 1901 and the output says iova-mode=pa. Did you change any settings on your VM and try simple stuf like adding memory and/or running with one core (how many cores did you give your VM?).  Also are you running privelaged?

    Thx
    Paul

    peluse(a)pels:~/spdk$ sudo ./build/bin/nvmf_tgt
    [2020-10-14 23:38:50.539538] Starting SPDK v20.10-pre git sha1 779a6bdf8 / DPDK 20.08.0 initialization...
    [2020-10-14 23:38:50.540128] [ DPDK EAL parameters: [2020-10-14 23:38:50.540346] nvmf [2020-10-14 23:38:50.540556] --no-shconf [2020-10-14 23:38:50.540755] -c 0x1 [2020-10-14 23:38:50.540957] --log-level=lib.eal:6 [2020-10-14 23:38:50.541153] --log-level=lib.cryptodev:5 [2020-10-14 23:38:50.541342] --log-level=user1:6 [2020-10-14 23:38:50.541566] --iova-mode=pa [2020-10-14 23:38:50.541763] --base-virtaddr=0x200000000000 [2020-10-14 23:38:50.541957] --match-allocations [2020-10-14 23:38:50.542290] --file-prefix=spdk_pid918 [2020-10-14 23:38:50.542544] ]
    EAL: No legacy callbacks, legacy socket not created
    [2020-10-14 23:38:50.552172] app.c: 667:spdk_app_start: *NOTICE*: Total cores available: 1

    From: Curtin, Eric <Eric.Curtin(a)dell.com>
    Date: Wednesday, October 14, 2020 at 1:56 AM
    To: Rui Chang <Rui.Chang(a)arm.com>
    Cc: spdk(a)lists.01.org <spdk(a)lists.01.org>
    Subject: [SPDK] Re: For Dummies guide to setting up ramdisk NVMe/TCP target How would I get my CPU mask on Linux? I'm running on a Virtualbox VM. I also used 0xf as a CPU mask and this was my output:

    ./build/bin/nvmf_tgt -m 0xf
    [2020-10-14 09:47:07.650399] Starting SPDK v20.10-pre git sha1 779a6bdf8 / DPDK 20.08.0 initialization...
    [2020-10-14 09:47:07.651075] [ DPDK EAL parameters: [2020-10-14 09:47:07.651092] nvmf [2020-10-14 09:47:07.651126] --no-shconf [2020-10-14 09:47:07.651132] -c 0xf [2020-10-14 09:47:07.651138] --log-level=lib.eal:6 [2020-10-14 09:47:07.651144] --log-level=lib.cryptodev:5 [2020-10-14 09:47:07.651150] --log-level=user1:6 [2020-10-14 09:47:07.651160] --iova-mode=pa [2020-10-14 09:47:07.651166] --base-virtaddr=0x200000000000 [2020-10-14 09:47:07.651172] --match-allocations [2020-10-14 09:47:07.651181] --file-prefix=spdk_pid702834 [2020-10-14 09:47:07.651187] ]
    EAL: FATAL: Cannot use IOVA as 'PA' since physical addresses are not available
    EAL: Cannot use IOVA as 'PA' since physical addresses are not available
    [2020-10-14 09:47:07.659984] init.c: 577:spdk_env_init: *ERROR*: Failed to initialize DPDK
    [2020-10-14 09:47:07.659999] app.c: 535:app_setup_env: *ERROR*: Unable to initialize SPDK env

    Regards,

    Eric Curtin
    Software Engineer

    Ovens Campus,
    Cork,
    Ireland

    Dell EMC

    Next planned absences: NA
    ________________________________
    From: Rui Chang <Rui.Chang(a)arm.com>
    Sent: Wednesday 14 October 2020 08:50
    To: Curtin, Eric <Eric.Curtin(a)dell.com>
    Cc: spdk(a)lists.01.org <spdk(a)lists.01.org>
    Subject: RE: [SPDK] For Dummies guide to setting up ramdisk NVMe/TCP target


    [EXTERNAL EMAIL]

    Hi Eric



    Refer to this page for HOWTO:

    https://spdk.io/doc/nvmf.html



    You said you want to run on one machine, suppose your machine's IP is 8.8.8.6



      1.  Build your SPDK

    git clone https://github.com/spdk/spdk spdk

    cd spdk

    git submodule update --init

    ./configure

    make



      1.  Run:



    Start target:

                    ./build/bin/nvmf_tgt -m 0xf          (-m gives the CPU mask. Normally you can make it on the same CPU socket as network card to get better performance)



    On another console, run rpc comamnds to do configuration:

    ./scripts/rpc.py nvmf_create_transport -t TCP      (there are lots of additional options you can use if you wish)

    ./scripts/rpc.py bdev_malloc_create -b Malloc0 200 512

    ./scripts/rpc.py nvmf_create_subsystem nqn.2016-06.io.spdk:cnode1 -a -s SPDK00000000000001 -d SPDK_Controller1

    ./scripts/rpc.py nvmf_subsystem_add_ns nqn.2016-06.io.spdk:cnode1 Malloc0

    ./scripts/rpc.py nvmf_subsystem_add_listener nqn.2016-06.io.spdk:cnode1 -t TCP -a 8.8.8.6 -s 4420                 (suppose 8.8.8.6 is the IP you want the target to listen on)



    Run io:

    ./build/examples/perf -r 'trtype:TCP adrfam:IPv4 traddr:8.8.8.6 trsvcid:4420 subnqn:nqn.2016-06.io.spdk:cnode1'  -q 128 -o 4096  -w write -t 100 -c 0xf0  (-c give the cpu mask for perf tool, you'd better make it different with nvmf_tgt to get better performance)



    Regards,

    Rui





    -----Original Message-----

    From: eric.curtin(a)dell.com<mailto:eric.curtin(a)dell.com> <eric.curtin(a)dell.com<mailto:eric.curtin(a)dell.com>>

    Sent: Thursday, October 1, 2020 3:23 PM

    To: spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>

    Subject: [SPDK] For Dummies guide to setting up ramdisk NVMe/TCP target



    Hi Guys,



    I'm just getting started with SPDK and wanted to create a fake ramdisk NVMe/TCP target block device, just so I could simulate NVMe/TCP traffic on a single Linux instance for development purposes.



    Anybody know how to set up a machine like this for NVMe/TCP development?

    _______________________________________________

    SPDK mailing list -- spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>

    To unsubscribe send an email to spdk-leave(a)lists.01.org<mailto:spdk-leave(a)lists.01.org>

    IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
    _______________________________________________
    SPDK mailing list -- spdk(a)lists.01.org
    To unsubscribe send an email to spdk-leave(a)lists.01.org _______________________________________________
    SPDK mailing list -- spdk(a)lists.01.org
    To unsubscribe send an email to spdk-leave(a)lists.01.org
    IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
    _______________________________________________
    SPDK mailing list -- spdk(a)lists.01.org
    To unsubscribe send an email to spdk-leave(a)lists.01.org


^ permalink raw reply	[flat|nested] 8+ messages in thread
* [SPDK] Re: For Dummies guide to setting up ramdisk NVMe/TCP target
@ 2020-10-16 14:12 Curtin, Eric
  0 siblings, 0 replies; 8+ messages in thread
From: Curtin, Eric @ 2020-10-16 14:12 UTC (permalink / raw)
  To: spdk

[-- Attachment #1: Type: text/plain, Size: 8694 bytes --]

Thanks guys, I got it working. You were right I was not running as root. The other steps I had to perform in case anyone else ever gets curious (I used 127.0.0.1). The only think I don't understand is the coremasks I used 0xf0 and 0x10, I just guessed those based on random examples I saw online. How do I know what coremasks I want to pick?

mkdir -p /dev/hugepages
mountpoint -q /dev/hugepages || mount -t hugetlbfs nodev /dev/hugepages
echo 1024 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
build/bin/nvmf_tgt -m 0xf

In another terminal:

scripts/rpc.py nvmf_create_transport -t TCP
scripts/rpc.py bdev_malloc_create -b Malloc0 200 512
scripts/rpc.py nvmf_create_subsystem nqn.2016-06.io.spdk:cnode1 -a -s SPDK00000000000001 -d SPDK_Controller1
scripts/rpc.py nvmf_subsystem_add_ns nqn.2016-06.io.spdk:cnode1 Malloc0
scripts/rpc.py nvmf_subsystem_add_listener nqn.2016-06.io.spdk:cnode1 -t TCP -a 127.0.0.1 -s 4420
build/examples/perf -r 'trtype:TCP adrfam:IPv4 traddr:127.0.0.1 trsvcid:4420 subnqn:nqn.2016-06.io.spdk:cnode1' -q 128 -o 4096 -w write -t 100 -c 0x10

Mise le meas/Regards,

Eric Curtin
Software Engineer

Ovens Campus,
Cork,
Ireland

Dell EMC

Next planned absences: NA
________________________________
From: Rui Chang <Rui.Chang(a)arm.com>
Sent: Thursday 15 October 2020 02:49
To: Curtin, Eric <Eric.Curtin(a)dell.com>
Cc: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: RE: [SPDK] Re: For Dummies guide to setting up ramdisk NVMe/TCP target


[EXTERNAL EMAIL]

Yeah, it also works on my virtualbox 6.1 with Win10/X86_64 host and Ubuntu 20.4 guest.

Regards,
Rui

-----Original Message-----
From: Luse, Paul E <paul.e.luse(a)intel.com>
Sent: Thursday, October 15, 2020 07:48
To: Storage Performance Development Kit <spdk(a)lists.01.org>; Rui Chang <Rui.Chang(a)arm.com>
Cc: spdk(a)lists.01.org
Subject: [SPDK] Re: For Dummies guide to setting up ramdisk NVMe/TCP target

Hi Eric,

Yeah, run with sudo.  I get the same thing when I run w/o, it's DPDK unable to get the PA with privilege.

Thx
Paul

From: Luse, Paul E <paul.e.luse(a)intel.com>
Date: Wednesday, October 14, 2020 at 4:43 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>, Rui Chang <Rui.Chang(a)arm.com>
Cc: spdk(a)lists.01.org <spdk(a)lists.01.org>
Subject: Re: [SPDK] Re: For Dummies guide to setting up ramdisk NVMe/TCP target Hi Eric,

Bummer that's not working.  It works OK for my with vbox 6.1 on a mac and ubuntu 1901 and the output says iova-mode=pa. Did you change any settings on your VM and try simple stuf like adding memory and/or running with one core (how many cores did you give your VM?).  Also are you running privelaged?

Thx
Paul

peluse(a)pels:~/spdk$ sudo ./build/bin/nvmf_tgt
[2020-10-14 23:38:50.539538] Starting SPDK v20.10-pre git sha1 779a6bdf8 / DPDK 20.08.0 initialization...
[2020-10-14 23:38:50.540128] [ DPDK EAL parameters: [2020-10-14 23:38:50.540346] nvmf [2020-10-14 23:38:50.540556] --no-shconf [2020-10-14 23:38:50.540755] -c 0x1 [2020-10-14 23:38:50.540957] --log-level=lib.eal:6 [2020-10-14 23:38:50.541153] --log-level=lib.cryptodev:5 [2020-10-14 23:38:50.541342] --log-level=user1:6 [2020-10-14 23:38:50.541566] --iova-mode=pa [2020-10-14 23:38:50.541763] --base-virtaddr=0x200000000000 [2020-10-14 23:38:50.541957] --match-allocations [2020-10-14 23:38:50.542290] --file-prefix=spdk_pid918 [2020-10-14 23:38:50.542544] ]
EAL: No legacy callbacks, legacy socket not created
[2020-10-14 23:38:50.552172] app.c: 667:spdk_app_start: *NOTICE*: Total cores available: 1

From: Curtin, Eric <Eric.Curtin(a)dell.com>
Date: Wednesday, October 14, 2020 at 1:56 AM
To: Rui Chang <Rui.Chang(a)arm.com>
Cc: spdk(a)lists.01.org <spdk(a)lists.01.org>
Subject: [SPDK] Re: For Dummies guide to setting up ramdisk NVMe/TCP target How would I get my CPU mask on Linux? I'm running on a Virtualbox VM. I also used 0xf as a CPU mask and this was my output:

./build/bin/nvmf_tgt -m 0xf
[2020-10-14 09:47:07.650399] Starting SPDK v20.10-pre git sha1 779a6bdf8 / DPDK 20.08.0 initialization...
[2020-10-14 09:47:07.651075] [ DPDK EAL parameters: [2020-10-14 09:47:07.651092] nvmf [2020-10-14 09:47:07.651126] --no-shconf [2020-10-14 09:47:07.651132] -c 0xf [2020-10-14 09:47:07.651138] --log-level=lib.eal:6 [2020-10-14 09:47:07.651144] --log-level=lib.cryptodev:5 [2020-10-14 09:47:07.651150] --log-level=user1:6 [2020-10-14 09:47:07.651160] --iova-mode=pa [2020-10-14 09:47:07.651166] --base-virtaddr=0x200000000000 [2020-10-14 09:47:07.651172] --match-allocations [2020-10-14 09:47:07.651181] --file-prefix=spdk_pid702834 [2020-10-14 09:47:07.651187] ]
EAL: FATAL: Cannot use IOVA as 'PA' since physical addresses are not available
EAL: Cannot use IOVA as 'PA' since physical addresses are not available
[2020-10-14 09:47:07.659984] init.c: 577:spdk_env_init: *ERROR*: Failed to initialize DPDK
[2020-10-14 09:47:07.659999] app.c: 535:app_setup_env: *ERROR*: Unable to initialize SPDK env

Regards,

Eric Curtin
Software Engineer

Ovens Campus,
Cork,
Ireland

Dell EMC

Next planned absences: NA
________________________________
From: Rui Chang <Rui.Chang(a)arm.com>
Sent: Wednesday 14 October 2020 08:50
To: Curtin, Eric <Eric.Curtin(a)dell.com>
Cc: spdk(a)lists.01.org <spdk(a)lists.01.org>
Subject: RE: [SPDK] For Dummies guide to setting up ramdisk NVMe/TCP target


[EXTERNAL EMAIL]

Hi Eric



Refer to this page for HOWTO:

https://spdk.io/doc/nvmf.html



You said you want to run on one machine, suppose your machine's IP is 8.8.8.6



  1.  Build your SPDK

git clone https://github.com/spdk/spdk spdk

cd spdk

git submodule update --init

./configure

make



  1.  Run:



Start target:

                ./build/bin/nvmf_tgt -m 0xf          (-m gives the CPU mask. Normally you can make it on the same CPU socket as network card to get better performance)



On another console, run rpc comamnds to do configuration:

./scripts/rpc.py nvmf_create_transport -t TCP      (there are lots of additional options you can use if you wish)

./scripts/rpc.py bdev_malloc_create -b Malloc0 200 512

./scripts/rpc.py nvmf_create_subsystem nqn.2016-06.io.spdk:cnode1 -a -s SPDK00000000000001 -d SPDK_Controller1

./scripts/rpc.py nvmf_subsystem_add_ns nqn.2016-06.io.spdk:cnode1 Malloc0

./scripts/rpc.py nvmf_subsystem_add_listener nqn.2016-06.io.spdk:cnode1 -t TCP -a 8.8.8.6 -s 4420                 (suppose 8.8.8.6 is the IP you want the target to listen on)



Run io:

./build/examples/perf -r 'trtype:TCP adrfam:IPv4 traddr:8.8.8.6 trsvcid:4420 subnqn:nqn.2016-06.io.spdk:cnode1'  -q 128 -o 4096  -w write -t 100 -c 0xf0  (-c give the cpu mask for perf tool, you'd better make it different with nvmf_tgt to get better performance)



Regards,

Rui





-----Original Message-----

From: eric.curtin(a)dell.com<mailto:eric.curtin(a)dell.com> <eric.curtin(a)dell.com<mailto:eric.curtin(a)dell.com>>

Sent: Thursday, October 1, 2020 3:23 PM

To: spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>

Subject: [SPDK] For Dummies guide to setting up ramdisk NVMe/TCP target



Hi Guys,



I'm just getting started with SPDK and wanted to create a fake ramdisk NVMe/TCP target block device, just so I could simulate NVMe/TCP traffic on a single Linux instance for development purposes.



Anybody know how to set up a machine like this for NVMe/TCP development?

_______________________________________________

SPDK mailing list -- spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>

To unsubscribe send an email to spdk-leave(a)lists.01.org<mailto:spdk-leave(a)lists.01.org>

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org _______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

^ permalink raw reply	[flat|nested] 8+ messages in thread
* [SPDK] Re: For Dummies guide to setting up ramdisk NVMe/TCP target
@ 2020-10-15  1:49 Rui Chang
  0 siblings, 0 replies; 8+ messages in thread
From: Rui Chang @ 2020-10-15  1:49 UTC (permalink / raw)
  To: spdk

[-- Attachment #1: Type: text/plain, Size: 7124 bytes --]

Yeah, it also works on my virtualbox 6.1 with Win10/X86_64 host and Ubuntu 20.4 guest.

Regards,
Rui

-----Original Message-----
From: Luse, Paul E <paul.e.luse(a)intel.com>
Sent: Thursday, October 15, 2020 07:48
To: Storage Performance Development Kit <spdk(a)lists.01.org>; Rui Chang <Rui.Chang(a)arm.com>
Cc: spdk(a)lists.01.org
Subject: [SPDK] Re: For Dummies guide to setting up ramdisk NVMe/TCP target

Hi Eric,

Yeah, run with sudo.  I get the same thing when I run w/o, it's DPDK unable to get the PA with privilege.

Thx
Paul

From: Luse, Paul E <paul.e.luse(a)intel.com>
Date: Wednesday, October 14, 2020 at 4:43 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>, Rui Chang <Rui.Chang(a)arm.com>
Cc: spdk(a)lists.01.org <spdk(a)lists.01.org>
Subject: Re: [SPDK] Re: For Dummies guide to setting up ramdisk NVMe/TCP target Hi Eric,

Bummer that's not working.  It works OK for my with vbox 6.1 on a mac and ubuntu 1901 and the output says iova-mode=pa. Did you change any settings on your VM and try simple stuf like adding memory and/or running with one core (how many cores did you give your VM?).  Also are you running privelaged?

Thx
Paul

peluse(a)pels:~/spdk$ sudo ./build/bin/nvmf_tgt
[2020-10-14 23:38:50.539538] Starting SPDK v20.10-pre git sha1 779a6bdf8 / DPDK 20.08.0 initialization...
[2020-10-14 23:38:50.540128] [ DPDK EAL parameters: [2020-10-14 23:38:50.540346] nvmf [2020-10-14 23:38:50.540556] --no-shconf [2020-10-14 23:38:50.540755] -c 0x1 [2020-10-14 23:38:50.540957] --log-level=lib.eal:6 [2020-10-14 23:38:50.541153] --log-level=lib.cryptodev:5 [2020-10-14 23:38:50.541342] --log-level=user1:6 [2020-10-14 23:38:50.541566] --iova-mode=pa [2020-10-14 23:38:50.541763] --base-virtaddr=0x200000000000 [2020-10-14 23:38:50.541957] --match-allocations [2020-10-14 23:38:50.542290] --file-prefix=spdk_pid918 [2020-10-14 23:38:50.542544] ]
EAL: No legacy callbacks, legacy socket not created
[2020-10-14 23:38:50.552172] app.c: 667:spdk_app_start: *NOTICE*: Total cores available: 1

From: Curtin, Eric <Eric.Curtin(a)dell.com>
Date: Wednesday, October 14, 2020 at 1:56 AM
To: Rui Chang <Rui.Chang(a)arm.com>
Cc: spdk(a)lists.01.org <spdk(a)lists.01.org>
Subject: [SPDK] Re: For Dummies guide to setting up ramdisk NVMe/TCP target How would I get my CPU mask on Linux? I'm running on a Virtualbox VM. I also used 0xf as a CPU mask and this was my output:

./build/bin/nvmf_tgt -m 0xf
[2020-10-14 09:47:07.650399] Starting SPDK v20.10-pre git sha1 779a6bdf8 / DPDK 20.08.0 initialization...
[2020-10-14 09:47:07.651075] [ DPDK EAL parameters: [2020-10-14 09:47:07.651092] nvmf [2020-10-14 09:47:07.651126] --no-shconf [2020-10-14 09:47:07.651132] -c 0xf [2020-10-14 09:47:07.651138] --log-level=lib.eal:6 [2020-10-14 09:47:07.651144] --log-level=lib.cryptodev:5 [2020-10-14 09:47:07.651150] --log-level=user1:6 [2020-10-14 09:47:07.651160] --iova-mode=pa [2020-10-14 09:47:07.651166] --base-virtaddr=0x200000000000 [2020-10-14 09:47:07.651172] --match-allocations [2020-10-14 09:47:07.651181] --file-prefix=spdk_pid702834 [2020-10-14 09:47:07.651187] ]
EAL: FATAL: Cannot use IOVA as 'PA' since physical addresses are not available
EAL: Cannot use IOVA as 'PA' since physical addresses are not available
[2020-10-14 09:47:07.659984] init.c: 577:spdk_env_init: *ERROR*: Failed to initialize DPDK
[2020-10-14 09:47:07.659999] app.c: 535:app_setup_env: *ERROR*: Unable to initialize SPDK env

Regards,

Eric Curtin
Software Engineer

Ovens Campus,
Cork,
Ireland

Dell EMC

Next planned absences: NA
________________________________
From: Rui Chang <Rui.Chang(a)arm.com>
Sent: Wednesday 14 October 2020 08:50
To: Curtin, Eric <Eric.Curtin(a)dell.com>
Cc: spdk(a)lists.01.org <spdk(a)lists.01.org>
Subject: RE: [SPDK] For Dummies guide to setting up ramdisk NVMe/TCP target


[EXTERNAL EMAIL]

Hi Eric



Refer to this page for HOWTO:

https://spdk.io/doc/nvmf.html



You said you want to run on one machine, suppose your machine's IP is 8.8.8.6



  1.  Build your SPDK

git clone https://github.com/spdk/spdk spdk

cd spdk

git submodule update --init

./configure

make



  1.  Run:



Start target:

                ./build/bin/nvmf_tgt -m 0xf          (-m gives the CPU mask. Normally you can make it on the same CPU socket as network card to get better performance)



On another console, run rpc comamnds to do configuration:

./scripts/rpc.py nvmf_create_transport -t TCP      (there are lots of additional options you can use if you wish)

./scripts/rpc.py bdev_malloc_create -b Malloc0 200 512

./scripts/rpc.py nvmf_create_subsystem nqn.2016-06.io.spdk:cnode1 -a -s SPDK00000000000001 -d SPDK_Controller1

./scripts/rpc.py nvmf_subsystem_add_ns nqn.2016-06.io.spdk:cnode1 Malloc0

./scripts/rpc.py nvmf_subsystem_add_listener nqn.2016-06.io.spdk:cnode1 -t TCP -a 8.8.8.6 -s 4420                 (suppose 8.8.8.6 is the IP you want the target to listen on)



Run io:

./build/examples/perf -r 'trtype:TCP adrfam:IPv4 traddr:8.8.8.6 trsvcid:4420 subnqn:nqn.2016-06.io.spdk:cnode1'  -q 128 -o 4096  -w write -t 100 -c 0xf0  (-c give the cpu mask for perf tool, you'd better make it different with nvmf_tgt to get better performance)



Regards,

Rui





-----Original Message-----

From: eric.curtin(a)dell.com<mailto:eric.curtin(a)dell.com> <eric.curtin(a)dell.com<mailto:eric.curtin(a)dell.com>>

Sent: Thursday, October 1, 2020 3:23 PM

To: spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>

Subject: [SPDK] For Dummies guide to setting up ramdisk NVMe/TCP target



Hi Guys,



I'm just getting started with SPDK and wanted to create a fake ramdisk NVMe/TCP target block device, just so I could simulate NVMe/TCP traffic on a single Linux instance for development purposes.



Anybody know how to set up a machine like this for NVMe/TCP development?

_______________________________________________

SPDK mailing list -- spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>

To unsubscribe send an email to spdk-leave(a)lists.01.org<mailto:spdk-leave(a)lists.01.org>

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org _______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

^ permalink raw reply	[flat|nested] 8+ messages in thread
* [SPDK] Re: For Dummies guide to setting up ramdisk NVMe/TCP target
@ 2020-10-14 23:47 Luse, Paul E
  0 siblings, 0 replies; 8+ messages in thread
From: Luse, Paul E @ 2020-10-14 23:47 UTC (permalink / raw)
  To: spdk

[-- Attachment #1: Type: text/plain, Size: 6242 bytes --]

Hi Eric,

Yeah, run with sudo.  I get the same thing when I run w/o, it’s DPDK unable to get the PA with privilege.

Thx
Paul

From: Luse, Paul E <paul.e.luse(a)intel.com>
Date: Wednesday, October 14, 2020 at 4:43 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>, Rui Chang <Rui.Chang(a)arm.com>
Cc: spdk(a)lists.01.org <spdk(a)lists.01.org>
Subject: Re: [SPDK] Re: For Dummies guide to setting up ramdisk NVMe/TCP target
Hi Eric,

Bummer that’s not working.  It works OK for my with vbox 6.1 on a mac and ubuntu 1901 and the output says iova-mode=pa. Did you change any settings on your VM and try simple stuf like adding memory and/or running with one core (how many cores did you give your VM?).  Also are you running privelaged?

Thx
Paul

peluse(a)pels:~/spdk$ sudo ./build/bin/nvmf_tgt
[2020-10-14 23:38:50.539538] Starting SPDK v20.10-pre git sha1 779a6bdf8 / DPDK 20.08.0 initialization...
[2020-10-14 23:38:50.540128] [ DPDK EAL parameters: [2020-10-14 23:38:50.540346] nvmf [2020-10-14 23:38:50.540556] --no-shconf [2020-10-14 23:38:50.540755] -c 0x1 [2020-10-14 23:38:50.540957] --log-level=lib.eal:6 [2020-10-14 23:38:50.541153] --log-level=lib.cryptodev:5 [2020-10-14 23:38:50.541342] --log-level=user1:6 [2020-10-14 23:38:50.541566] --iova-mode=pa [2020-10-14 23:38:50.541763] --base-virtaddr=0x200000000000 [2020-10-14 23:38:50.541957] --match-allocations [2020-10-14 23:38:50.542290] --file-prefix=spdk_pid918 [2020-10-14 23:38:50.542544] ]
EAL: No legacy callbacks, legacy socket not created
[2020-10-14 23:38:50.552172] app.c: 667:spdk_app_start: *NOTICE*: Total cores available: 1

From: Curtin, Eric <Eric.Curtin(a)dell.com>
Date: Wednesday, October 14, 2020 at 1:56 AM
To: Rui Chang <Rui.Chang(a)arm.com>
Cc: spdk(a)lists.01.org <spdk(a)lists.01.org>
Subject: [SPDK] Re: For Dummies guide to setting up ramdisk NVMe/TCP target
How would I get my CPU mask on Linux? I'm running on a Virtualbox VM. I also used 0xf as a CPU mask and this was my output:

./build/bin/nvmf_tgt -m 0xf
[2020-10-14 09:47:07.650399] Starting SPDK v20.10-pre git sha1 779a6bdf8 / DPDK 20.08.0 initialization...
[2020-10-14 09:47:07.651075] [ DPDK EAL parameters: [2020-10-14 09:47:07.651092] nvmf [2020-10-14 09:47:07.651126] --no-shconf [2020-10-14 09:47:07.651132] -c 0xf [2020-10-14 09:47:07.651138] --log-level=lib.eal:6 [2020-10-14 09:47:07.651144] --log-level=lib.cryptodev:5 [2020-10-14 09:47:07.651150] --log-level=user1:6 [2020-10-14 09:47:07.651160] --iova-mode=pa [2020-10-14 09:47:07.651166] --base-virtaddr=0x200000000000 [2020-10-14 09:47:07.651172] --match-allocations [2020-10-14 09:47:07.651181] --file-prefix=spdk_pid702834 [2020-10-14 09:47:07.651187] ]
EAL: FATAL: Cannot use IOVA as 'PA' since physical addresses are not available
EAL: Cannot use IOVA as 'PA' since physical addresses are not available
[2020-10-14 09:47:07.659984] init.c: 577:spdk_env_init: *ERROR*: Failed to initialize DPDK
[2020-10-14 09:47:07.659999] app.c: 535:app_setup_env: *ERROR*: Unable to initialize SPDK env

Regards,

Eric Curtin
Software Engineer

Ovens Campus,
Cork,
Ireland

Dell EMC

Next planned absences: NA
________________________________
From: Rui Chang <Rui.Chang(a)arm.com>
Sent: Wednesday 14 October 2020 08:50
To: Curtin, Eric <Eric.Curtin(a)dell.com>
Cc: spdk(a)lists.01.org <spdk(a)lists.01.org>
Subject: RE: [SPDK] For Dummies guide to setting up ramdisk NVMe/TCP target


[EXTERNAL EMAIL]

Hi Eric



Refer to this page for HOWTO:

https://spdk.io/doc/nvmf.html



You said you want to run on one machine, suppose your machine’s IP is 8.8.8.6



  1.  Build your SPDK

git clone https://github.com/spdk/spdk spdk

cd spdk

git submodule update --init

./configure

make



  1.  Run:



Start target:

                ./build/bin/nvmf_tgt -m 0xf          (-m gives the CPU mask. Normally you can make it on the same CPU socket as network card to get better performance)



On another console, run rpc comamnds to do configuration:

./scripts/rpc.py nvmf_create_transport -t TCP      (there are lots of additional options you can use if you wish)

./scripts/rpc.py bdev_malloc_create -b Malloc0 200 512

./scripts/rpc.py nvmf_create_subsystem nqn.2016-06.io.spdk:cnode1 -a -s SPDK00000000000001 -d SPDK_Controller1

./scripts/rpc.py nvmf_subsystem_add_ns nqn.2016-06.io.spdk:cnode1 Malloc0

./scripts/rpc.py nvmf_subsystem_add_listener nqn.2016-06.io.spdk:cnode1 -t TCP -a 8.8.8.6 -s 4420                 (suppose 8.8.8.6 is the IP you want the target to listen on)



Run io:

./build/examples/perf -r 'trtype:TCP adrfam:IPv4 traddr:8.8.8.6 trsvcid:4420 subnqn:nqn.2016-06.io.spdk:cnode1'  -q 128 -o 4096  -w write -t 100 -c 0xf0  (-c give the cpu mask for perf tool, you’d better make it different with nvmf_tgt to get better performance)



Regards,

Rui





-----Original Message-----

From: eric.curtin(a)dell.com<mailto:eric.curtin(a)dell.com> <eric.curtin(a)dell.com<mailto:eric.curtin(a)dell.com>>

Sent: Thursday, October 1, 2020 3:23 PM

To: spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>

Subject: [SPDK] For Dummies guide to setting up ramdisk NVMe/TCP target



Hi Guys,



I'm just getting started with SPDK and wanted to create a fake ramdisk NVMe/TCP target block device, just so I could simulate NVMe/TCP traffic on a single Linux instance for development purposes.



Anybody know how to set up a machine like this for NVMe/TCP development?

_______________________________________________

SPDK mailing list -- spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>

To unsubscribe send an email to spdk-leave(a)lists.01.org<mailto:spdk-leave(a)lists.01.org>

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org

^ permalink raw reply	[flat|nested] 8+ messages in thread
* [SPDK] Re: For Dummies guide to setting up ramdisk NVMe/TCP target
@ 2020-10-14 23:43 Luse, Paul E
  0 siblings, 0 replies; 8+ messages in thread
From: Luse, Paul E @ 2020-10-14 23:43 UTC (permalink / raw)
  To: spdk

[-- Attachment #1: Type: text/plain, Size: 5790 bytes --]

Hi Eric,

Bummer that’s not working.  It works OK for my with vbox 6.1 on a mac and ubuntu 1901 and the output says iova-mode=pa. Did you change any settings on your VM and try simple stuf like adding memory and/or running with one core (how many cores did you give your VM?).  Also are you running privelaged?

Thx
Paul

peluse(a)pels:~/spdk$ sudo ./build/bin/nvmf_tgt
[2020-10-14 23:38:50.539538] Starting SPDK v20.10-pre git sha1 779a6bdf8 / DPDK 20.08.0 initialization...
[2020-10-14 23:38:50.540128] [ DPDK EAL parameters: [2020-10-14 23:38:50.540346] nvmf [2020-10-14 23:38:50.540556] --no-shconf [2020-10-14 23:38:50.540755] -c 0x1 [2020-10-14 23:38:50.540957] --log-level=lib.eal:6 [2020-10-14 23:38:50.541153] --log-level=lib.cryptodev:5 [2020-10-14 23:38:50.541342] --log-level=user1:6 [2020-10-14 23:38:50.541566] --iova-mode=pa [2020-10-14 23:38:50.541763] --base-virtaddr=0x200000000000 [2020-10-14 23:38:50.541957] --match-allocations [2020-10-14 23:38:50.542290] --file-prefix=spdk_pid918 [2020-10-14 23:38:50.542544] ]
EAL: No legacy callbacks, legacy socket not created
[2020-10-14 23:38:50.552172] app.c: 667:spdk_app_start: *NOTICE*: Total cores available: 1

From: Curtin, Eric <Eric.Curtin(a)dell.com>
Date: Wednesday, October 14, 2020 at 1:56 AM
To: Rui Chang <Rui.Chang(a)arm.com>
Cc: spdk(a)lists.01.org <spdk(a)lists.01.org>
Subject: [SPDK] Re: For Dummies guide to setting up ramdisk NVMe/TCP target
How would I get my CPU mask on Linux? I'm running on a Virtualbox VM. I also used 0xf as a CPU mask and this was my output:

./build/bin/nvmf_tgt -m 0xf
[2020-10-14 09:47:07.650399] Starting SPDK v20.10-pre git sha1 779a6bdf8 / DPDK 20.08.0 initialization...
[2020-10-14 09:47:07.651075] [ DPDK EAL parameters: [2020-10-14 09:47:07.651092] nvmf [2020-10-14 09:47:07.651126] --no-shconf [2020-10-14 09:47:07.651132] -c 0xf [2020-10-14 09:47:07.651138] --log-level=lib.eal:6 [2020-10-14 09:47:07.651144] --log-level=lib.cryptodev:5 [2020-10-14 09:47:07.651150] --log-level=user1:6 [2020-10-14 09:47:07.651160] --iova-mode=pa [2020-10-14 09:47:07.651166] --base-virtaddr=0x200000000000 [2020-10-14 09:47:07.651172] --match-allocations [2020-10-14 09:47:07.651181] --file-prefix=spdk_pid702834 [2020-10-14 09:47:07.651187] ]
EAL: FATAL: Cannot use IOVA as 'PA' since physical addresses are not available
EAL: Cannot use IOVA as 'PA' since physical addresses are not available
[2020-10-14 09:47:07.659984] init.c: 577:spdk_env_init: *ERROR*: Failed to initialize DPDK
[2020-10-14 09:47:07.659999] app.c: 535:app_setup_env: *ERROR*: Unable to initialize SPDK env

Regards,

Eric Curtin
Software Engineer

Ovens Campus,
Cork,
Ireland

Dell EMC

Next planned absences: NA
________________________________
From: Rui Chang <Rui.Chang(a)arm.com>
Sent: Wednesday 14 October 2020 08:50
To: Curtin, Eric <Eric.Curtin(a)dell.com>
Cc: spdk(a)lists.01.org <spdk(a)lists.01.org>
Subject: RE: [SPDK] For Dummies guide to setting up ramdisk NVMe/TCP target


[EXTERNAL EMAIL]

Hi Eric



Refer to this page for HOWTO:

https://spdk.io/doc/nvmf.html



You said you want to run on one machine, suppose your machine’s IP is 8.8.8.6



  1.  Build your SPDK

git clone https://github.com/spdk/spdk spdk

cd spdk

git submodule update --init

./configure

make



  1.  Run:



Start target:

                ./build/bin/nvmf_tgt -m 0xf          (-m gives the CPU mask. Normally you can make it on the same CPU socket as network card to get better performance)



On another console, run rpc comamnds to do configuration:

./scripts/rpc.py nvmf_create_transport -t TCP      (there are lots of additional options you can use if you wish)

./scripts/rpc.py bdev_malloc_create -b Malloc0 200 512

./scripts/rpc.py nvmf_create_subsystem nqn.2016-06.io.spdk:cnode1 -a -s SPDK00000000000001 -d SPDK_Controller1

./scripts/rpc.py nvmf_subsystem_add_ns nqn.2016-06.io.spdk:cnode1 Malloc0

./scripts/rpc.py nvmf_subsystem_add_listener nqn.2016-06.io.spdk:cnode1 -t TCP -a 8.8.8.6 -s 4420                 (suppose 8.8.8.6 is the IP you want the target to listen on)



Run io:

./build/examples/perf -r 'trtype:TCP adrfam:IPv4 traddr:8.8.8.6 trsvcid:4420 subnqn:nqn.2016-06.io.spdk:cnode1'  -q 128 -o 4096  -w write -t 100 -c 0xf0  (-c give the cpu mask for perf tool, you’d better make it different with nvmf_tgt to get better performance)



Regards,

Rui





-----Original Message-----

From: eric.curtin(a)dell.com<mailto:eric.curtin(a)dell.com> <eric.curtin(a)dell.com<mailto:eric.curtin(a)dell.com>>

Sent: Thursday, October 1, 2020 3:23 PM

To: spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>

Subject: [SPDK] For Dummies guide to setting up ramdisk NVMe/TCP target



Hi Guys,



I'm just getting started with SPDK and wanted to create a fake ramdisk NVMe/TCP target block device, just so I could simulate NVMe/TCP traffic on a single Linux instance for development purposes.



Anybody know how to set up a machine like this for NVMe/TCP development?

_______________________________________________

SPDK mailing list -- spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>

To unsubscribe send an email to spdk-leave(a)lists.01.org<mailto:spdk-leave(a)lists.01.org>

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org

^ permalink raw reply	[flat|nested] 8+ messages in thread
* [SPDK] Re: For Dummies guide to setting up ramdisk NVMe/TCP target
@ 2020-10-14  7:50 Rui Chang
  0 siblings, 0 replies; 8+ messages in thread
From: Rui Chang @ 2020-10-14  7:50 UTC (permalink / raw)
  To: spdk

[-- Attachment #1: Type: text/plain, Size: 2605 bytes --]

Hi Eric



Refer to this page for HOWTO:

https://spdk.io/doc/nvmf.html



You said you want to run on one machine, suppose your machine’s IP is 8.8.8.6



  1.  Build your SPDK

git clone https://github.com/spdk/spdk spdk

cd spdk

git submodule update --init

./configure

make



  1.  Run:



Start target:

                ./build/bin/nvmf_tgt -m 0xf          (-m gives the CPU mask. Normally you can make it on the same CPU socket as network card to get better performance)



On another console, run rpc comamnds to do configuration:

./scripts/rpc.py nvmf_create_transport -t TCP      (there are lots of additional options you can use if you wish)

./scripts/rpc.py bdev_malloc_create -b Malloc0 200 512

./scripts/rpc.py nvmf_create_subsystem nqn.2016-06.io.spdk:cnode1 -a -s SPDK00000000000001 -d SPDK_Controller1

./scripts/rpc.py nvmf_subsystem_add_ns nqn.2016-06.io.spdk:cnode1 Malloc0

./scripts/rpc.py nvmf_subsystem_add_listener nqn.2016-06.io.spdk:cnode1 -t TCP -a 8.8.8.6 -s 4420                 (suppose 8.8.8.6 is the IP you want the target to listen on)



Run io:

./build/examples/perf -r 'trtype:TCP adrfam:IPv4 traddr:8.8.8.6 trsvcid:4420 subnqn:nqn.2016-06.io.spdk:cnode1'  -q 128 -o 4096  -w write -t 100 -c 0xf0  (-c give the cpu mask for perf tool, you’d better make it different with nvmf_tgt to get better performance)



Regards,

Rui





-----Original Message-----

From: eric.curtin(a)dell.com<mailto:eric.curtin(a)dell.com> <eric.curtin(a)dell.com<mailto:eric.curtin(a)dell.com>>

Sent: Thursday, October 1, 2020 3:23 PM

To: spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>

Subject: [SPDK] For Dummies guide to setting up ramdisk NVMe/TCP target



Hi Guys,



I'm just getting started with SPDK and wanted to create a fake ramdisk NVMe/TCP target block device, just so I could simulate NVMe/TCP traffic on a single Linux instance for development purposes.



Anybody know how to set up a machine like this for NVMe/TCP development?

_______________________________________________

SPDK mailing list -- spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>

To unsubscribe send an email to spdk-leave(a)lists.01.org<mailto:spdk-leave(a)lists.01.org>

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

^ permalink raw reply	[flat|nested] 8+ messages in thread
* [SPDK] Re: For Dummies guide to setting up ramdisk NVMe/TCP target
@ 2020-10-01 12:54 Zawadzki, Tomasz
  0 siblings, 0 replies; 8+ messages in thread
From: Zawadzki, Tomasz @ 2020-10-01 12:54 UTC (permalink / raw)
  To: spdk

[-- Attachment #1: Type: text/plain, Size: 1520 bytes --]

Hi Eric,

For the scenario that you've described three links would provide best pointers:
- https://spdk.io/doc/getting_started.html for SPDK itself
- https://spdk.io/doc/bdev.html#bdev_config_malloc to create malloc bdev
- https://spdk.io/doc/nvmf.html#nvmf_config to setup the NVMe/TCP target
Above would provide NVMe-oF target ready to connect initiator.

After that you can use another instance of SPDK to connect via NVMe/TCP initiator.
For RPC https://spdk.io/doc/bdev.html#bdev_config_nvme, but other apps like ./build/examples/perf work too.

CI for SPDK runs per-patch testing contained within single machine,
so ./test/nvmf/* scripts can be great examples for development.

The scenario you are describing is part of:
sudo ./test/nvmf/target/nvmf_example.sh --transport=tcp --iso

Thanks,
Tomek

> -----Original Message-----
> From: eric.curtin(a)dell.com <eric.curtin(a)dell.com>
> Sent: Thursday, October 1, 2020 9:23 AM
> To: spdk(a)lists.01.org
> Subject: [SPDK] For Dummies guide to setting up ramdisk NVMe/TCP target
> 
> Hi Guys,
> 
> I'm just getting started with SPDK and wanted to create a fake ramdisk
> NVMe/TCP target block device, just so I could simulate NVMe/TCP traffic on
> a single Linux instance for development purposes.
> 
> Anybody know how to set up a machine like this for NVMe/TCP
> development?
> _______________________________________________
> SPDK mailing list -- spdk(a)lists.01.org
> To unsubscribe send an email to spdk-leave(a)lists.01.org

^ permalink raw reply	[flat|nested] 8+ messages in thread
end of thread, other threads:[~2020-10-16 14:20 UTC | newest]

Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-10-14  8:50 [SPDK] Re: For Dummies guide to setting up ramdisk NVMe/TCP target Curtin, Eric
  -- strict thread matches above, loose matches on Subject: below --
2020-10-16 14:20 Harris, James R
2020-10-16 14:12 Curtin, Eric
2020-10-15  1:49 Rui Chang
2020-10-14 23:47 Luse, Paul E
2020-10-14 23:43 Luse, Paul E
2020-10-14  7:50 Rui Chang
2020-10-01 12:54 Zawadzki, Tomasz

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.