Re: [SPDK] Netlist storage card

* Re: [SPDK] Netlist storage card
@ 2017-01-09 19:37 Dale Corlett
  0 siblings, 0 replies; 24+ messages in thread
From: Dale Corlett @ 2017-01-09 19:37 UTC (permalink / raw)
  To: spdk

[-- Attachment #1: Type: text/plain, Size: 14789 bytes --]

Hi Daniel,

Thanks for your reply.

I have set the log level using rte_set_log_level(RTE_LOG_DEBUG) but the
debug messages still do not display. I used rte_get_log_level() to check
that the log level was set to DEBUG, it returned 8 which corresponds to the
RTE_LOG_DEBUG macro. So it seems like the log level is set to debug, but
the debug messages do not show in the terminal.

Do the EAL messages get written anywhere else? Could I have broken
something when I set the EXTRA_CFLAGS environment variable?

Thanks,
Dale

On Tue, Jan 10, 2017 at 7:38 AM, Verkamp, Daniel <daniel.verkamp(a)intel.com>
wrote:

> Hi Dale,
>
>
>
> This is controlled by the log level; the SPDK example programs don’t
> currently expose this as a command-line parameter, but you can add a call
> to rte_set_log_level() early in the program to change it from the default:
> http://dpdk.org/doc/api/rte__log_8h.html
>
>
>
> -- Daniel
>
>
>
> *From:* SPDK [mailto:spdk-bounces(a)lists.01.org] *On Behalf Of *Dale
> Corlett
> *Sent:* Saturday, January 7, 2017 3:49 PM
>
> *To:* Storage Performance Development Kit <spdk(a)lists.01.org>
> *Subject:* Re: [SPDK] Netlist storage card
>
>
>
> Hi,
>
>
>
> For an SPDK application I am trying to enable the EAL debug message such
> as: "RTE_LOG(DEBUG, EAL, "  Not managed by a supported kernel driver,
> skipped\n");".
>
> I have tried compiling DPDK for debug by setting the environment variable
> "EXTRA_CFLAGS" to "-O0 -g" as described here:
> http://dpdk.readthedocs.io/en/v16.07/prog_guide/dev_kit_
> root_make_help.html#compiling-for-debug, but this does not work.
>
>
>
> Is there any other way to enable these EAL debug messages?
>
>
>
> Thanks,
>
> Dale
>
>
>
> On Mon, Dec 19, 2016 at 5:59 PM, Dileep Sharma <dsharma(a)cloudsimple.com>
> wrote:
>
> Hi Dale,
>
>
>
> SPDK uses the DPDK's PCI generic to get the information about the PCI
> devices. And DPDK in turn uses the uio driver to get the information about
> the underlying PCI devices. I'll suggest you to follow the code flow of
> rte_eal_pci_probe() routine in DPDK to get the feel how DPDK works for PCI
> devices. During code path flow, you'll came to know about various data
> structures that are similar to kernel code and then you can use them
> effectively in your code.
>
>
>
> Thanks,
>
> Dileep
>
>
>
> On Sun, Dec 18, 2016 at 9:27 AM, Dale Corlett <dale.corlett(a)nyriad.com>
> wrote:
>
> Hi,
>
>
>
> I have now moved on to creating a user space driver for the netlist
> storage card. I have written a simple kernel driver that uses the PCI
> driver that just gets the pcie register information from the netlist card.
> I want to port this kernel driver to user space to work with SPDK, but I am
> finding it difficult to workout the steps required to do this. I have
> noticed that some structures like spdk_pci_id, and spdk_pci_device which
> are defined in SPDK are similar to structures that I have used in my kernel
> driver: pci_device_id, and pci_dev. However, SPDK does not seem to have
> structures equivalent to pci_driver which I have used in my kernel driver.
>
> How does SPDK link the PCI driver to the user space driver?
>
> Any assistance would be greatly appreciated.
>
>
>
> My simple kernel driver for the Netlist card:
>
> #include <linux/pci.h>
>
> #include <linux/init.h>
>
> #include <linux/module.h>
>
>
>
>
>
> #define EV_VENDOR_ID 0x1C1B    // Netlist
>
> #define EV_DEVICE_ID_BAR32_WINDOW_32M_16GB 0x0006 // EXPRESSvault
> NVDIMM/DDR3 PCIe GEN3 x8 lane -        For debug use only
>
>
>
> #define PCI_DRIVER_NAME "Netlistev3"
>
>
>
> // Function prototypes:
>
> static int netlist_pci_probe(struct pci_dev *dev, const struct
> pci_device_id *id);
>
>
>
> //PCI_DEVICE(EV_VENDOR_ID, EV_DEVICE_ID_BAR32_WINDOW_32M_16GB);
>
> static const struct pci_device_id netlist_pci_ids [] =
>
> {
>
> {
>
> .vendor = EV_VENDOR_ID,
>
> .device = EV_DEVICE_ID_BAR32_WINDOW_32M_16GB,
>
> .subvendor = PCI_ANY_ID,
>
> .subdevice = PCI_ANY_ID,
>
> },
>
> {
>
> // end: all zeroes
>
> .vendor = 0,
>
> .device = 0,
>
> .subvendor = 0,
>
> .subdevice = 0,
>
> }
>
> };
>
>
>
> static struct pci_driver netlist_pci_driver =
>
> {
>
> .name           = PCI_DRIVER_NAME,
>
> .id_table       = netlist_pci_ids,
>
> .probe          = netlist_pci_probe,
>
> // .remove         = netlist_pci_remove,
>
> // .err_handler    = &pci_err_handler, // This is a structure
>
> };
>
>
>
> static int netlist_pci_probe(struct pci_dev *dev, const struct
> pci_device_id *id)
>
> {
>
> printk("pci_enable_device\n");
>
> if (pci_enable_device(dev) < 0)
>
> {
>
> printk("pci_enable_device FAILED\n");
>
> return -ENODEV;
>
> }
>
> unsigned int venid;
>
> int offset;
>
> for(offset = 0x00; offset <= 0x03C; offset += 4)
>
> {
>
> int retVal = pci_read_config_dword(dev, offset, &venid);
>
> printk ("byte offset 0x%X = 0x%X\n", offset, venid);
>
> }
>
> return 0;
>
> }
>
>
>
>
>
> static int netlist_init(void)
>
> {
>
> int err = pci_register_driver(&netlist_pci_driver);
>
>
>
>     if(err < 0)
>
>     {
>
>     printk("error registering netlist_pci_driver\n");
>
>     }
>
>     else if(err == 0)
>
>     {
>
>     printk("netlist_pci_driver registered successfully\n");
>
>     }
>
>     return err;
>
> }
>
>
>
> static void netlist_cleanup(void)
>
> {
>
> pci_unregister_driver(&netlist_pci_driver);
>
> printk("netlist_pci_driver unregistered\n");
>
> }
>
>
>
> module_init(netlist_init);
>
> module_exit(netlist_cleanup);
>
>
>
> Thanks,
>
> Dale
>
>
>
> On Fri, Dec 2, 2016 at 10:55 AM, Dale Corlett <dale.corlett(a)nyriad.com>
> wrote:
>
> Hi,
>
>
>
> I have the iscsi_tgt program compiled but I do not know how to setup the
> iscsi.conf file. I have removed all of what I think is unnecessary for my
> application from the iscsi.conf.in example so I have the following:
>
>
>
> [iSCSI]
>
>   # node name (not include optional part)
>
>   # Users can optionally change this to fit their environment.
>
>   NodeBase "iqn.2016-06.io.spdk"
>
>
>
>   AuthFile /usr/local/etc/spdk/auth.conf
>
>
>
>   MinConnectionsPerCore 4
>
>   # Power saving related variable, this parameter defines how long an iSCSI
>
>   # connection must be idle before moving it to a state where it will
> consume
>
>   # less power. This variable is defined in terms of microseconds. We set
> default
>
>   # value as 5ms.
>
>   MinConnectionIdleInterval 5000
>
>
>
>   # Socket I/O timeout sec. (0 is infinite)
>
>   Timeout 30
>
>
>
>   # authentication information for discovery session
>
>   DiscoveryAuthMethod Auto
>
>
>
>   #MaxSessions 128
>
>   #MaxConnectionsPerSession 2
>
>
>
>   # iSCSI initial parameters negotiate with initiators
>
>   # NOTE: incorrect values might crash
>
>   DefaultTime2Wait 2
>
>   DefaultTime2Retain 60
>
>
>
>   ImmediateData Yes
>
>   ErrorRecoveryLevel 0
>
>
>
> [AIO]
>
> AIO /dev/ev3mema
>
>
>
> [TargetNode1]
>
>  TargetName disk1
>
>   TargetAlias "Data Disk1"
>
>   Mapping PortalGroup1 InitiatorGroup1
>
>   AuthMethod Auto
>
>   AuthGroup AuthGroup1
>
>   # Enable header and data digest
>
>   # UseDigest Header Data
>
>   UseDigest Auto
>
> # Using the first AIO target
>
>   LUN0 AIO0
>
>
>
> But when I run the iscsi_tgt program like this: sudo ./iscsi_tgt -c
> ../../etc/spdk/iscsi.conf I get the following output:
>
>
>
> Starting Intel(R) DPDK initialization ...
>
> [ DPDK EAL parameters: iscsi -c 1 -n 4 -m 2048 --master-lcore=0
> --file-prefix=rte0 --proc-type=auto ]
>
> EAL: Detected 4 lcore(s)
>
> EAL: Auto-detected process type: PRIMARY
>
> EAL: No free hugepages reported in hugepages-1048576kB
>
> EAL: Probing VFIO support...
>
> done.
>
> Occupied cpu core mask is 0x1
>
> Occupied cpu socket mask is 0x1
>
> Ioat Copy Engine Offload Enabled
>
> tgt_node.c: 590:spdk_iscsi_tgt_node_add_map: ***ERROR***
> iqn.2016-06.io.spdk:disk1: PortalGroup1 not found
>
> tgt_node.c: 727:spdk_iscsi_tgt_node_construct: ***ERROR*** could not add
> map to target
>
> tgt_node.c: 982:spdk_cf_add_iscsi_tgt_node: ***ERROR*** tgt_node1:
> add_iscsi_target_node error
>
> tgt_node.c:1006:spdk_iscsi_init_tgt_nodes: ***ERROR***
> spdk_cf_add_iscsi_tgt_node() failed
>
> iscsi_subsystem.c: 964:spdk_iscsi_subsystem_init: ***ERROR***
> spdk_iscsi_init_tgt_nodes() failed
>
> app.c: 404:spdk_app_init: ***ERROR*** spdk_subsystem_init() failed
>
>
>
> I just want the most basic configuration so that I can just see if the
> Netlist card works with SPDK.
>
>
>
> Thanks,
>
> Dale
>
>
>
> On Thu, Dec 1, 2016 at 5:39 PM, Dale Corlett <dale.corlett(a)nyriad.com>
> wrote:
>
> Hi Param,
>
>
>
> Awesome it worked!
>
>
>
> Thanks,
>
> Dale
>
>
>
> On Thu, Dec 1, 2016 at 5:06 PM, Kumaraparameshwaran Rathnavel <
> krath(a)cloudsimple.com> wrote:
>
> Hi Dale,
>
>
>
> When you compile your SPDK application try giving
> DPDK_DIR=/path/to/dpdk/x86_build folder. This should solve your problem.
> From SPDK directory give make and the DPDK_DIR.
>
>
>
> Regards,
>
> Param.
>
>
>
> On 01-Dec-2016, at 9:23 AM, Dale Corlett <dale.corlett(a)nyriad.com> wrote:
>
>
>
> Hello again,
>
>
>
> When I was trying to run iscsi_tgt (as instructed in
> http://www.spdk.io/spdk/doc/iscsi_getting_started.html) I found that the
> app/iscsi_tgt.c was not compiled, and when I tried to compile it I got the
> following error:
>
> iscsi_tgt.c:38:24: fatal error: rte_config.h: No such file or directory
>
> From an online search I have seen that many other people get a similar
> error to do with missing the rte_config.h file, but there do not seem to be
> any resolutions.
>
> I have found a rte_config.h file in one of the dpdk directories. I tried
> putting this in the /usr/include directory, but this did not fix it.
>
>
>
> Does anyone have any ideas of how to fix this? Has anyone encountered this
> problem before?
>
>
>
> Thanks,
>
> Dale
>
>
>
>
>
>
>
> On Thu, Dec 1, 2016 at 5:46 AM, Walker, Benjamin <
> benjamin.walker(a)intel.com> wrote:
>
> Note also that running the iSCSI target will not create a block device on
> your local system by itself. It’s just a C executable that’s opening up
> some listening sockets. To see a block device on a system, you have to
> connect to the target using an iSCSI initiator. The one we use is iscsiadm,
> which is not part of SPDK but is in wide use and is packaged on all
> mainstream Linux distributions. This is the same iSCSI initiator people use
> with the Linux kernel iSCSI target.
>
>
>
> All of the above is covered in that guide you linked.
>
>
>
> *From:* SPDK [mailto:spdk-bounces(a)lists.01.org] *On Behalf Of *Harris,
> James R
> *Sent:* Tuesday, November 29, 2016 6:07 PM
>
>
> *To:* Storage Performance Development Kit <spdk(a)lists.01.org>
> *Subject:* Re: [SPDK] Netlist storage card
>
>
>
>
>
> Hi Dale,
>
>
>
> You can pass the name of your config file to the iscsi_tgt app using the
> –c option.
>
>
>
> -Jim
>
>
>
>
>
> *From: *SPDK <spdk-bounces(a)lists.01.org> on behalf of Dale Corlett <
> dale.corlett(a)nyriad.com>
> *Reply-To: *Storage Performance Development Kit <spdk(a)lists.01.org>
> *Date: *Tuesday, November 29, 2016 at 3:38 PM
> *To: *Storage Performance Development Kit <spdk(a)lists.01.org>
> *Subject: *Re: [SPDK] Netlist storage card
>
>
>
> Hi,
>
>
>
> I have added the [AIO] section to the config file: /home/dale/spdk/etc/
> spdk/iscsi.conf.in but I cannot figure out how to "run" SPDK to create
> the SPDK block device called AIO0.
>
>
>
> I have tried following the guid for the iscsi block device:
> http://www.spdk.io/spdk/doc/iscsi_getting_started.html but I am not sure
> if I have used the correct config file.
>
>
>
> Thanks,
>
> Dale
>
>
>
> On Wed, Nov 30, 2016 at 10:41 AM, Daniel Verkamp <daniel.verkamp(a)intel.com>
> wrote:
>
> On 11/29/2016 02:15 PM, Dale Corlett wrote:
> > Hi,
> >
> > Thank you all for your replies.
> >
> > Jim, I think that I will first try the AIO first, and then try
> > writing the userspace driver and bdev module. I am not sure of what I
> > should do with the config files, or how to get the .ko file for the
> > AIO module. Is there a makefile that gets the .ko module file or do I
> > have to create one?
> >
> > Also, when trying to use ./scripts/setup.sh I found that it did not
> > work because I got then message: "logname: no login name". I think
> > that this is a problem with the 16.04 version of Ubuntu:
> > https://bugs.launchpad.net/ubuntu/+source/gnome-terminal/+bug/1537645
> > so I changed line 177 to: username= `echo whoami` and this seems to
> > work.
> >
> > Thanks, Dale
>
> Hi Dale,
>
> You do not need any special kernel modules to use the SPDK AIO bdev -
> it uses the libaio userspace library provided by glibc and the block
> devices that are exposed by the usual kernel drivers for any block
> device that is supported by Linux.
>
> Thanks for the report about logname - I'll prepare a patch to work
> around that issue. The only time the username provided by logname is
> used is to provide access to an unprivileged user to the VFIO device,
> so if you are running the programs using SPDK as root, there should have
> no negative effect.
>
> Thanks,
> -- Daniel
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>
>
>
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>
>
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>
>
>
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>
>
>
>
>
>
>
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>
>
>
>
>
> --
>
> Thanks,
>
> Dileep Sharma
>
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>
>
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>
>

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 35540 bytes --]

^ permalink raw reply	[flat|nested] 24+ messages in thread