[SPDK] In multi-core environment, which core will execute the IO call back function ??

[SPDK] In multi-core environment, which core will execute the IO call back function ??

[Vincent]

[-- Attachment #1: Type: text/plain, Size: 993 bytes --]

Hello All,

     Recently I am writing a spdk block device  in multi-core environment.
i.e.

initialize  iscsi target by multi-core(./app/iscsi_tgt/iscsi_tgt  -c
./etc/spdk/iscsi.conf  -m 0xFF0).

I use spdk_bdev_write_blocks() to send IO to lower layer of my device, that
is  a nvme device.

But I found the lcore execution the call back of IO is

not the same as the lcore who call  spdk_bdev_write_blocks().

Is this behavior correct ??

Or I make any mistake ??

       I separate the data structure into multi partition and assign each
lcore a partition.

If the lcore execute call back is not the same as the one who send IO, then
I must

use event architecture to send the IO result to correct lcore.  That is a
little troublesome .

      So, does anybody can give me some hints for how to make the  lcore
who send IO

and  the lcore who execution IO call back  the same ??

   Any hints are appreciated

  Thank you very much.

--
Vincent Chang

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 12835 bytes --]

Re: [SPDK] In multi-core environment, which core will execute the IO call back function ??

[Vincent]

[-- Attachment #1: Type: text/plain, Size: 4162 bytes --]

Hello Changpeng,

     Thank you so much for your reply.

    I just check the code of bdev  nvme/aio/null, I found the IO call back
shall call by the same lcore when sending io.

    It doesn't make any sense for my question.


   So I check my code detailly, I found a typo,

   I want call  "spdk_env_get_current_core" , but I typed  to "
spdk_env_get_core_count"....

   That's why I see so strange problem...

  So my problem is solved, all reason is the typo.

Thanks again for your response.

--
Vincent chang














2018-02-01 14:31 GMT+08:00 Liu, Changpeng <changpeng.liu(a)intel.com>:

> Hi Vincent,
>
> I think you can refer to the implementation in directory spdk/lib/bdev,
> such as malloc/aio/null.
> And you can insert your own block driver at this directory.
>
>
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Vincent
> Sent: Thursday, February 1, 2018 2:17 PM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: Re: [SPDK] In multi-core environment, which core will execute the
> IO call back function ??
>
> Hello Changpeng,
>
>     Thank you so much for your quickly response.
>
> There is a nvme device as lower layer for my spdk block device,
>
> and I create several IO channels  for this nvme device by call
> spdk_bdev_get_io_channle()
>
> to let  every  lcore has their owe IO channel,
>
> and I send IO to the nvme device by call   spdk_bdev_write_blocks(ch,.....,
> cb)
>
> ( the "ch"  is the io channel belong to the lcore who call
> spdk_bdev_write_blocks() )
>
> But I found the lcore call the call back function is  NOT the same as the
> lcore who call   spdk_bdev_write_blocks,
>
> as I describe in previous mail.
>
>
>      I just check the menu of API   "spdk_poller_register"   that you
> mentioned in your mail.
>
> This API is register a "poller"  in a lcore to do some thing periodically.
>
> But I still don't known how to use this API to let lcore who send IO and
> lcore who execute call back function the same.
>
> Could you please give some examples to illustrate how to use this API to
> make my purpose happen ??
>
> Thank you so much.
>
> --
> Vincent
>
>
>
>
>
>
>
>
>
> 2018-02-01 13:38 GMT+08:00 Liu, Changpeng <changpeng.liu(a)intel.com>:
> Hi Vincent,
>
> You can try “spdk_bdev_get_io_channel” and “spdk_poller_register” for
> your block device at the lcore you want to do the real I/O processing.
>
>
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Vincent
> Sent: Thursday, February 1, 2018 1:25 PM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: [SPDK] In multi-core environment, which core will execute the IO
> call back function ??
>
> Hello All,
>
>      Recently I am writing a spdk block device  in multi-core environment.
> i.e.
>
> initialize  iscsi target by multi-core(./app/iscsi_tgt/iscsi_tgt  -c
> ./etc/spdk/iscsi.conf  -m 0xFF0).
>
> I use spdk_bdev_write_blocks() to send IO to lower layer of my device,
> that is  a nvme device.
>
> But I found the lcore execution the call back of IO is
>
> not the same as the lcore who call  spdk_bdev_write_blocks().
>
> Is this behavior correct ??
>
> Or I make any mistake ??
>
>        I separate the data structure into multi partition and assign each
> lcore a partition.
>
> If the lcore execute call back is not the same as the one who send IO,
> then I must
>
> use event architecture to send the IO result to correct lcore.  That is a
> little troublesome .
>
>       So, does anybody can give me some hints for how to make the  lcore
> who send IO
>
> and  the lcore who execution IO call back  the same ??
>
>    Any hints are appreciated
>
>   Thank you very much.
>
> --
> Vincent Chang
>
>
>
>
>
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 6924 bytes --]

Re: [SPDK] In multi-core environment, which core will execute the IO call back function ??

[Liu, Changpeng]

[-- Attachment #1: Type: text/plain, Size: 3232 bytes --]

Hi Vincent,

I think you can refer to the implementation in directory spdk/lib/bdev, such as malloc/aio/null.
And you can insert your own block driver at this directory.


From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Vincent
Sent: Thursday, February 1, 2018 2:17 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] In multi-core environment, which core will execute the IO call back function ??

Hello Changpeng,

    Thank you so much for your quickly response.

There is a nvme device as lower layer for my spdk block device,

and I create several IO channels  for this nvme device by call spdk_bdev_get_io_channle() 

to let  every  lcore has their owe IO channel,

and I send IO to the nvme device by call   spdk_bdev_write_blocks(ch,....., cb)  

( the "ch"  is the io channel belong to the lcore who call  spdk_bdev_write_blocks() )

But I found the lcore call the call back function is  NOT the same as the lcore who call   spdk_bdev_write_blocks,

as I describe in previous mail.


     I just check the menu of API   "spdk_poller_register"   that you mentioned in your mail. 

This API is register a "poller"  in a lcore to do some thing periodically.

But I still don't known how to use this API to let lcore who send IO and lcore who execute call back function the same. 

Could you please give some examples to illustrate how to use this API to make my purpose happen ??

Thank you so much.

--
Vincent









2018-02-01 13:38 GMT+08:00 Liu, Changpeng <changpeng.liu(a)intel.com>:
Hi Vincent,

You can try “spdk_bdev_get_io_channel” and “spdk_poller_register” for
your block device at the lcore you want to do the real I/O processing.


From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Vincent
Sent: Thursday, February 1, 2018 1:25 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: [SPDK] In multi-core environment, which core will execute the IO call back function ??

Hello All,

     Recently I am writing a spdk block device  in multi-core environment. i.e. 

initialize  iscsi target by multi-core(./app/iscsi_tgt/iscsi_tgt  -c ./etc/spdk/iscsi.conf  -m 0xFF0).

I use spdk_bdev_write_blocks() to send IO to lower layer of my device, that is  a nvme device. 

But I found the lcore execution the call back of IO is 

not the same as the lcore who call  spdk_bdev_write_blocks().

Is this behavior correct ??

Or I make any mistake ??

       I separate the data structure into multi partition and assign each lcore a partition.

If the lcore execute call back is not the same as the one who send IO, then I must 

use event architecture to send the IO result to correct lcore.  That is a little troublesome .

      So, does anybody can give me some hints for how to make the  lcore who send IO 

and  the lcore who execution IO call back  the same ??

   Any hints are appreciated

  Thank you very much.

--
Vincent Chang





 
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk

Re: [SPDK] In multi-core environment, which core will execute the IO call back function ??

[Vincent]

[-- Attachment #1: Type: text/plain, Size: 2828 bytes --]

Hello Changpeng,

    Thank you so much for your quickly response.

There is a nvme device as lower layer for my spdk block device,

and I create several IO channels  for this nvme device by call
spdk_bdev_get_io_channle()

to let  every  lcore has their owe IO channel,

and I send IO to the nvme device by call   spdk_bdev_write_blocks(ch,.....,
cb)

( the "ch"  is the io channel belong to the lcore who call
spdk_bdev_write_blocks() )

But I found the lcore call the call back function is  *NOT* the same as the
lcore who call   spdk_bdev_write_blocks,

as I describe in previous mail.


     I just check the menu of API   "spdk_poller_register"   that you
mentioned in your mail.

This API is register a "poller"  in a lcore to do some thing periodically.

But I still don't known how to use this API to let lcore who send IO and
lcore who execute call back function the same.

Could you please give some examples to illustrate how to use this API to
make my purpose happen ??

Thank you so much.

--
Vincent









2018-02-01 13:38 GMT+08:00 Liu, Changpeng <changpeng.liu(a)intel.com>:

> Hi Vincent,
>
> You can try “spdk_bdev_get_io_channel” and “spdk_poller_register” for
> your block device at the lcore you want to do the real I/O processing.
>
>
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Vincent
> Sent: Thursday, February 1, 2018 1:25 PM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: [SPDK] In multi-core environment, which core will execute the IO
> call back function ??
>
> Hello All,
>
>      Recently I am writing a spdk block device  in multi-core environment.
> i.e.
>
> initialize  iscsi target by multi-core(./app/iscsi_tgt/iscsi_tgt  -c
> ./etc/spdk/iscsi.conf  -m 0xFF0).
>
> I use spdk_bdev_write_blocks() to send IO to lower layer of my device,
> that is  a nvme device.
>
> But I found the lcore execution the call back of IO is
>
> not the same as the lcore who call  spdk_bdev_write_blocks().
>
> Is this behavior correct ??
>
> Or I make any mistake ??
>
>        I separate the data structure into multi partition and assign each
> lcore a partition.
>
> If the lcore execute call back is not the same as the one who send IO,
> then I must
>
> use event architecture to send the IO result to correct lcore.  That is a
> little troublesome .
>
>       So, does anybody can give me some hints for how to make the  lcore
> who send IO
>
> and  the lcore who execution IO call back  the same ??
>
>    Any hints are appreciated
>
>   Thank you very much.
>
> --
> Vincent Chang
>
>
>
>
>
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 9527 bytes --]

Re: [SPDK] In multi-core environment, which core will execute the IO call back function ??

[Liu, Changpeng]

[-- Attachment #1: Type: text/plain, Size: 1477 bytes --]

Hi Vincent,

You can try “spdk_bdev_get_io_channel” and “spdk_poller_register” for
your block device at the lcore you want to do the real I/O processing.


From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Vincent
Sent: Thursday, February 1, 2018 1:25 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: [SPDK] In multi-core environment, which core will execute the IO call back function ??

Hello All,

     Recently I am writing a spdk block device  in multi-core environment. i.e. 

initialize  iscsi target by multi-core(./app/iscsi_tgt/iscsi_tgt  -c ./etc/spdk/iscsi.conf  -m 0xFF0).

I use spdk_bdev_write_blocks() to send IO to lower layer of my device, that is  a nvme device. 

But I found the lcore execution the call back of IO is 

not the same as the lcore who call  spdk_bdev_write_blocks().

Is this behavior correct ??

Or I make any mistake ??

       I separate the data structure into multi partition and assign each lcore a partition.

If the lcore execute call back is not the same as the one who send IO, then I must 

use event architecture to send the IO result to correct lcore.  That is a little troublesome .

      So, does anybody can give me some hints for how to make the  lcore who send IO 

and  the lcore who execution IO call back  the same ??

   Any hints are appreciated

  Thank you very much.

--
Vincent Chang