[SPDK] Understanding io_channel

[SPDK] Understanding io_channel

[Fenggang Wu]

[-- Attachment #1: Type: text/plain, Size: 2040 bytes --]

Hi,

I am new to SPDK and trying to develop an aggregated virtual block device
module (vbdev_agg.c) that stripes across multiple base devices. I am having
difficulty understanding the general physical meaning of the io_channel.
And particularly in the vbdev_agg case, how can I define the io_channel for
this aggregated device? Or more specifically, what is the right way to
implementing the get_io_channel function in the vbdev_agg module?

My current understanding is each io_device can have many separate
io_channels, each allocated for one thread. However, I/O requests issued to
the bdev_agg will be forwarded to the base device’s io_channel anyway
(after some offset translation), where the io_channel of the vbdev_agg is
not used.

I’ve tried to return NULL in the get_io_channel function of the vbdev_agg
module. It works fine for the read, write, unmap, and flush functions I
implemented in the vbdev_agg module. The vbdev_agg I/O functions (read,
write, unmap, flush) forward the I/O request to the underlying base device
by calling spdk_bdev_{read, write, unmap, flush} again to the corresponding
base device after the offset translation (defined by striping). In the call
back of the completion of the base devices, I call the completion of the io
request for the agg device.

However, other part of the code sometime generate segment fault because of
this NULL channel (e.g. in spdk_put_io_channel(), when accessing
ch->channel, as ch is null pointer). So I am just wondering what is the
right way of defining the vbdev_agg’s io_channel.

Also I found in the comments in struct spdk_io_channel:
 "Modules will allocate extra memory off the end of this structure to store
references to hardware-specific references (i.e. NVMe queue pairs, or
*references
to child device spdk_io_channels* (i.e. virtual bdevs)."
However, I haven't figure out how to use it. Is there any code that
exemplifies this?

Any suggestions/hints will be appreciated. Thank you very much!

Regards,
Fenggang

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 2503 bytes --]

Re: [SPDK] Understanding io_channel

[@ 2017-10-17 7:00 UTC (permalink / raw)]

[-- Attachment #1: Type: text/plain, Size: 1518 bytes --]

Hello Fenggang and all,

	I am new to SPDK and recently I want to write a vbdev device that contain 2 bdev device.

I found in mail-list that vbdev_agg.c written by Fenggang is suitable for me as example.

(Thank you so much ,Fenggang)

And I got the code in  

https://github.com/fgwu/spdk/tree/737ae30c99c80e1bc1d13e535b5a1d9a411ffa8c

and compile it successful.

By using the configure below
============================================
[AIO]
  AIO /dev/ram0 AIO0
  AIO /dev/ram1 AIO1
 [Agg]
        VBDev myraid AIO0 AIO1
……………………………………….
[TargetNode1]
 ……………………
  LUN0 myraid
 …………………..	
==================================================

When I try to connect the iscsi target,

the program get error in conn.c:1212

===================================================
	bytes = spdk_sock_writev(conn->sock, iov, iovec_cnt);
	if (bytes == -1) {
		if (errno == EWOULDBLOCK || errno == EAGAIN) {
			return 0;
		} else {
			perror("writev"); <-------------------------------- get error here
			return -1;
		}
	}
======================================================

I found the iov is null so the variable bytes return by spdk_sock_writev is -1.

I have tried to guess why the iov is error , modify the code and do some experiment.

But the problem still there.

Does anyone can give me some hints/suggestions ??

Any suggestions/hints will be appreciated. Thank you very much!

Regards,
AnNan 

Re: [SPDK] Understanding io_channel

[Harris, James R]

[-- Attachment #1: Type: text/plain, Size: 2313 bytes --]


> On Oct 10, 2017, at 1:53 PM, Fenggang Wu <fenggang(a)cs.umn.edu> wrote:
> 
> Hi Jim,
> 
> Thank you very much for the great answer!  It makes perfect sense to me. This saves so much time.
> 
> 
> On Mon, Oct 9, 2017 at 7:04 PM Harris, James R <james.r.harris(a)intel.com> wrote:
> Hi Fenggang,
> 
> > On Oct 9, 2017, at 12:04 PM, Fenggang Wu <fenggang(a)cs.umn.edu> wrote:
> >
> > Hi,
> >
> > I am new to SPDK and trying to develop an aggregated virtual block device module (vbdev_agg.c) that stripes across multiple base devices. I am having difficulty understanding
> 
> Welcome to SPDK!  An aggregated virtual block device module is interesting - will this do striping and/or concatenation?
> 
> Current I am only considering striping. But I would expect an easy extension from striping to concatenation.

I think striping is the more interesting and common use case.

> <snip>
> 
> Now I have learned from the nvme module and register my agg_disk struct as the void* io_device, or better named, the "unique pointer". Currenly, a space with a size of a array of io_channel pointers is allocated after the io_channel struct. The io_channels pointers of the base devices are kept in the array. They are got (get_io_channel(base_dev)) in the create_cb and put (put_io_channel(base_dev)) in the destroy_cb.

Yes - that sounds right.

>  
> 
> >
> > Any suggestions/hints will be appreciated. Thank you very much!
> 
> If you would like to post your module to GerritHub, I’m sure you’d get some good review feedback from myself and others.  Please note that this is a very active area of development right now.  Your questions are really appreciated and will help us clarify where we need to improve on example code and documentation.
> 
> 
> Personally I would like to share it or even make some contribution to the community if possible. Yet I would have to double check with the industry partner supporting my project to see their opinions.

That would be fantastic.  A striping module would be generally useful to SPDK.  Plus if the module is contributed to SPDK, the SPDK project will make sure it gets tested automatically as part of the per-patch test suite to ensure against regressions.  This might be important to your industry partner.

Thanks,

-Jim

Re: [SPDK] Understanding io_channel

[Fenggang Wu]

[-- Attachment #1: Type: text/plain, Size: 7144 bytes --]

Hi Jim,

Thank you very much for the great answer!  It makes perfect sense to me.
This saves so much time.


On Mon, Oct 9, 2017 at 7:04 PM Harris, James R <james.r.harris(a)intel.com>
wrote:

> Hi Fenggang,
>
> > On Oct 9, 2017, at 12:04 PM, Fenggang Wu <fenggang(a)cs.umn.edu> wrote:
> >
> > Hi,
> >
> > I am new to SPDK and trying to develop an aggregated virtual block
> device module (vbdev_agg.c) that stripes across multiple base devices. I am
> having difficulty understanding
>
> Welcome to SPDK!  An aggregated virtual block device module is interesting
> - will this do striping and/or concatenation?
>
> Current I am only considering striping. But I would expect an easy
extension from striping to concatenation.


> > the general physical meaning of the io_channel. And particularly in the
> vbdev_agg case, how can I define the io_channel for this aggregated device?
> Or more specifically, what is the right way to implementing the
> get_io_channel function in the vbdev_agg module?
> >
> > My current understanding is each io_device can have many separate
> io_channels, each allocated for one thread. However, I/O requests issued to
> the bdev_agg will be forwarded to the base device’s io_channel anyway
> (after some offset translation), where the io_channel of the vbdev_agg is
> not used.
>
> The vbdev_agg I/O channel is basically a place for you to store the I/O
> channels for the base device *for that thread*.
>
> For example, an I/O channel for an nvme block device corresponds to an
> NVMe queue pair.  If this nvme block device is accessed from two different
> threads, those two threads will have two separate I/O channels, each
> channel associated with its own NVMe queue pair.  This ensures that all
> NVMe hardware accesses are done completely lock-free, since only one thread
> operates on any given queue pair.
>
> If you roll this up to your vbdev_agg block device, you may (and likely
> will) have it accessed from two or more different threads.  So you will
> need vbdev_agg I/O channels which will effectively just be a placeholder
> for the underlying base block device I/O channels.  You may have an I/O
> channel on thread 0, which contains pointers to the base bdev channels for
> thread 0.  Another I/O channel on thread 1 will contain pointers to the
> base bdev channels for thread 1.
>
> >
> > I’ve tried to return NULL in the get_io_channel function of the
> vbdev_agg module. It works fine for the read, write, unmap, and flush
> functions I implemented in the vbdev_agg module. The vbdev_agg I/O
> functions (read, write, unmap, flush) forward the I/O request to the
> underlying base device by calling spdk_bdev_{read, write, unmap, flush}
> again to the corresponding base device after the offset translation
> (defined by striping). In the call back of the completion of the base
> devices, I call the completion of the io request for the agg device.
>
> I suspect that NULL is working in some of these cases, because you have
> already allocated I/O channels for the base device and have stored them in
> an internal global data structure.  But this is just a hypothesis.  This
> works OK for single-threaded use cases, but when a vbdev_agg block device
> gets accessed from multiple threads, you will need separate I/O channels
> for the base device too - one for each thread.
>
> >
> > However, other part of the code sometime generate segment fault because
> of this NULL channel (e.g. in spdk_put_io_channel(), when accessing
> ch->channel, as ch is null pointer). So I am just wondering what is the
> right way of defining the vbdev_agg’s io_channel.
> >
> > Also I found in the comments in struct spdk_io_channel:
> >  "Modules will allocate extra memory off the end of this structure to
> store references to hardware-specific references (i.e. NVMe queue pairs, or
> references to child device spdk_io_channels (i.e. virtual bdevs)."
> > However, I haven't figure out how to use it. Is there any code that
> exemplifies this?
>
> nvme is a good example - lib/bdev/nvme/bdev_nvme.c.  Look for struct
> nvme_io_channel.  This is the context buffer for an I/O channel for an nvme
> block device.
>
> Next look for the call to spdk_io_device_register().  The first parameter
> is a unique pointer (it just uses the address of the nvme_ctrlr structure)
> - it just needs to be a pointer value that we know is unique within the
> application.  The next two parameters are function pointers for creating
> and destroying I/O channels for this device.  The last parameter is the
> size of the per-I/O channel context buffer.
>
> The SPDK io_channel module keeps a reference count on all of the I/O
> channels.  If there are multiple requests for an I/O channel for the NVMe
> controller on one thread, we do not need or want to allocate a separate
> NVMe queue pair for each request - we want them to share the same I/O
> channel.  So the create function pointer gets called when the I/O channel
> is first allocated, and the destroy function pointer is not called until
> the last reference to the I/O channel is released.
>
> You will probably also want to look at bdev_nvme_create_cb and
> bdev_nvme_destroy_cb.  The former shows how an I/O channel for this device
> is created - it allocates an I/O qpair for the NVMe controller and then
> starts a poller to poll for completions on that queue pair.  The latter
> frees the I/O queue pair and stops the poller once the I/O channel is
> destroyed.
>

Now I have learned from the nvme module and register my agg_disk struct as
the void* io_device, or better named, the "unique pointer". Currenly, a
space with a size of a array of io_channel pointers is allocated after the
io_channel struct. The io_channels pointers of the base devices are kept in
the array. They are got (get_io_channel(base_dev)) in the create_cb and put
(put_io_channel(base_dev)) in the destroy_cb.


>
> >
> > Any suggestions/hints will be appreciated. Thank you very much!
>
> If you would like to post your module to GerritHub, I’m sure you’d get
> some good review feedback from myself and others.  Please note that this is
> a very active area of development right now.  Your questions are really
> appreciated and will help us clarify where we need to improve on example
> code and documentation.
>
>
Personally I would like to share it or even make some contribution to the
community if possible. Yet I would have to double check with the industry
partner supporting my project to see their opinions.

Besides, I've also got a separate question about the context of the module.
I will ask it in a separate thread soon.

Thanks again!
-Fenggang


Thanks,
>
> -Jim
>
>
> >
> > Regards,
> > Fenggang
> >
> > _______________________________________________
> > SPDK mailing list
> > SPDK(a)lists.01.org
> > https://lists.01.org/mailman/listinfo/spdk
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 8769 bytes --]

Re: [SPDK] Understanding io_channel

[Harris, James R]

[-- Attachment #1: Type: text/plain, Size: 5616 bytes --]

Hi Fenggang,

> On Oct 9, 2017, at 12:04 PM, Fenggang Wu <fenggang(a)cs.umn.edu> wrote:
> 
> Hi,
> 
> I am new to SPDK and trying to develop an aggregated virtual block device module (vbdev_agg.c) that stripes across multiple base devices. I am having difficulty understanding 

Welcome to SPDK!  An aggregated virtual block device module is interesting - will this do striping and/or concatenation?

> the general physical meaning of the io_channel. And particularly in the vbdev_agg case, how can I define the io_channel for this aggregated device? Or more specifically, what is the right way to implementing the get_io_channel function in the vbdev_agg module?
> 
> My current understanding is each io_device can have many separate io_channels, each allocated for one thread. However, I/O requests issued to the bdev_agg will be forwarded to the base device’s io_channel anyway (after some offset translation), where the io_channel of the vbdev_agg is not used.

The vbdev_agg I/O channel is basically a place for you to store the I/O channels for the base device *for that thread*.

For example, an I/O channel for an nvme block device corresponds to an NVMe queue pair.  If this nvme block device is accessed from two different threads, those two threads will have two separate I/O channels, each channel associated with its own NVMe queue pair.  This ensures that all NVMe hardware accesses are done completely lock-free, since only one thread operates on any given queue pair.

If you roll this up to your vbdev_agg block device, you may (and likely will) have it accessed from two or more different threads.  So you will need vbdev_agg I/O channels which will effectively just be a placeholder for the underlying base block device I/O channels.  You may have an I/O channel on thread 0, which contains pointers to the base bdev channels for thread 0.  Another I/O channel on thread 1 will contain pointers to the base bdev channels for thread 1.

> 
> I’ve tried to return NULL in the get_io_channel function of the vbdev_agg module. It works fine for the read, write, unmap, and flush functions I implemented in the vbdev_agg module. The vbdev_agg I/O functions (read, write, unmap, flush) forward the I/O request to the underlying base device by calling spdk_bdev_{read, write, unmap, flush} again to the corresponding base device after the offset translation (defined by striping). In the call back of the completion of the base devices, I call the completion of the io request for the agg device.

I suspect that NULL is working in some of these cases, because you have already allocated I/O channels for the base device and have stored them in an internal global data structure.  But this is just a hypothesis.  This works OK for single-threaded use cases, but when a vbdev_agg block device gets accessed from multiple threads, you will need separate I/O channels for the base device too - one for each thread.

> 
> However, other part of the code sometime generate segment fault because of this NULL channel (e.g. in spdk_put_io_channel(), when accessing ch->channel, as ch is null pointer). So I am just wondering what is the right way of defining the vbdev_agg’s io_channel. 
> 
> Also I found in the comments in struct spdk_io_channel: 
>  "Modules will allocate extra memory off the end of this structure to store references to hardware-specific references (i.e. NVMe queue pairs, or references to child device spdk_io_channels (i.e. virtual bdevs)."
> However, I haven't figure out how to use it. Is there any code that exemplifies this?

nvme is a good example - lib/bdev/nvme/bdev_nvme.c.  Look for struct nvme_io_channel.  This is the context buffer for an I/O channel for an nvme block device.

Next look for the call to spdk_io_device_register().  The first parameter is a unique pointer (it just uses the address of the nvme_ctrlr structure) - it just needs to be a pointer value that we know is unique within the application.  The next two parameters are function pointers for creating and destroying I/O channels for this device.  The last parameter is the size of the per-I/O channel context buffer.

The SPDK io_channel module keeps a reference count on all of the I/O channels.  If there are multiple requests for an I/O channel for the NVMe controller on one thread, we do not need or want to allocate a separate NVMe queue pair for each request - we want them to share the same I/O channel.  So the create function pointer gets called when the I/O channel is first allocated, and the destroy function pointer is not called until the last reference to the I/O channel is released.

You will probably also want to look at bdev_nvme_create_cb and bdev_nvme_destroy_cb.  The former shows how an I/O channel for this device is created - it allocates an I/O qpair for the NVMe controller and then starts a poller to poll for completions on that queue pair.  The latter frees the I/O queue pair and stops the poller once the I/O channel is destroyed.

> 
> Any suggestions/hints will be appreciated. Thank you very much!

If you would like to post your module to GerritHub, I’m sure you’d get some good review feedback from myself and others.  Please note that this is a very active area of development right now.  Your questions are really appreciated and will help us clarify where we need to improve on example code and documentation.

Thanks,

-Jim


> 
> Regards,
> Fenggang
> 
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk