* [SPDK] Re: vhost is destroyed many times when starting vhost device
@ 2020-04-26 8:48 Li Feng
0 siblings, 0 replies; 7+ messages in thread
From: Li Feng @ 2020-04-26 8:48 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 9665 bytes --]
I use QEMU: QEMU emulator version 4.2.92 (v5.0.0-rc2-32-g618b649d06-dirty)
Anyway, when there is more than 1 IO queue, the start/stop device will
be called many times.
If there is no way to reduce the start/stop device calls, I think that
reduce the bdev start/stop call is a good option.
Thanks,
Feng Li
Liu, Changpeng <changpeng.liu(a)intel.com> 于2020年4月26日周日 下午3:22写道：
>
> From the virtio specification, it's OK if the driver only use 1 IO queues, even you specified the num-queue to a bigger value.
> BIOS will only use 1 IO queue, so the vhost target still doesn't know when it can start the device.
>
> I tested with QEMU 4.1.50, I don't see the issue you mentioned the amplification factor is 2.
>
>
> > -----Original Message-----
> > From: Li Feng <fengli(a)smartx.com>
> > Sent: Sunday, April 26, 2020 1:17 PM
> > To: Storage Performance Development Kit <spdk(a)lists.01.org>; Kyle Zhang
> > <kyle(a)smartx.com>
> > Subject: [SPDK] Re: vhost is destroyed many times when starting vhost device
> >
> > Comment in-line.
> >
> > Thanks,
> >
> > Feng Li
> >
> > Liu, Changpeng <changpeng.liu(a)intel.com> 于2020年4月26日周日 上午
> > 8:46写道：
> >
> > >
> > > Hi Feng,
> > >
> > > The issue you mentioned does exist, but it's not related with number of disk,
> > but with "num-queues" parameter input from
> > > QEMU, in my test environment, if I used 8 IO queues, the vhost_blk_start() will
> > be called 9 times. Note that the BIOS will
> > > use 1 IO queue so vhost_blk_start() is called 1 time during the BIOS stage.
> >
> > Thanks for your clarification of the relationship with num-queues.
> > In my previous test, the num-queues is equal to 4.
> > The amplification factor is nearly 2.
> >
> > Even set num-queues = 1, the stop/start device and
> > vhost_blk_start/stop is still been called 2 times.
> > The following log is about num-queues =1.
> > At the time of showing grub menu, the event is:
> > start_device
> > vhost_blk_start
> > io_setup
> > start_device
> > stop_device
> > vhost_blk_stop
> > start_device
> > vhost_blk_start
> > io_destroy
> > io_setup
> >
> > After starting the kernel:
> > stop_device
> > vhost_blk_stop
> > stop_device
> > io_destroy
> > start_device
> > vhost_blk_start
> > io_setup
> > stop_device
> > vhost_blk_stop
> > start_device
> > vhost_blk_start
> > io_destroy
> > io_setup
> > start_device
> >
> >
> > My qemu is master branch of upstream.
> >
> > >
> > > In the vhost target side, we don't know the exactly number of queues used for
> > this device, so we will start the device once
> > > got one valid IO queue, and restart the device when got a new IO queue.
> > That's the reason why the start/stop device will
> > > be called multiple times.
> >
> > Could the vhost target have a chance to implement the multi-queue
> > without start/stop the device?
> > Currently, when the io-queue num is up, the call times of the backend
> > bdev constructor/destructor will increase.
> >
> > What do you think my patch works?
> >
> > >
> > > > -----Original Message-----
> > > > From: Li Feng <fengli(a)smartx.com>
> > > > Sent: Thursday, April 23, 2020 8:30 PM
> > > > To: Storage Performance Development Kit <spdk(a)lists.01.org>
> > > > Subject: [SPDK] vhost is destroyed many times when starting vhost device
> > > >
> > > > Hi,
> > > >
> > > > In rte_vhost_compat.c, spdk_extern_vhost_pre_msg_handler will call
> > > > destroy_device when some conditions meet.
> > > >
> > > > I tested and found when a vhost-blk setup, the vhost device will be
> > > > destroyed 9 times.
> > > > And the bdev(no matter which type) will be affected. I'm using the aio
> > > > bdev, and the io_setup/io_destroy will be called 9 times when a disk
> > > > is inserted to a VM.
> > > >
> > > > My concern is:
> > > > Is there any solution to reduce the call times(at least: disks nums * 9)?
> > > > The call will slow down the VM boot time if VM has multiple disks.
> > > > There are some related logs:
> > > >
> > > > Starting SPDK v20.01-pre git sha1 93eecf627 / DPDK 19.11.0 initialization...
> > > > [ DPDK EAL parameters: spdk_tgt --no-shconf -c 1 -m 1
> > > > --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=user1:6
> > > > --iova-mode=pa --base-virtaddr=0x200000000000 --match-allocations
> > > > --file-prefix=spdk_pid2004865 ]
> > > > EAL: No available hugepages reported in hugepages-1048576kB
> > > > app.c: 645:spdk_app_start: *NOTICE*: Total cores available: 1
> > > > reactor.c: 346:_spdk_reactor_run: *NOTICE*: Reactor started on core 0
> > > > bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
> > > > bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
> > > > bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
> > > > rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> > > > destroy device
> > > > rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> > > > bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
> > > > bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
> > > > rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> > > > destroy device
> > > > rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> > > > bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
> > > > bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
> > > > rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> > > > destroy device
> > > > rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> > > > bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
> > > > bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
> > > > rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> > > > destroy device
> > > > rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> > > > bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
> > > > bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
> > > > rte_vhost_compat.c: 174:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> > > > destroy device
> > > > rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> > > > bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
> > > > bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
> > > > rte_vhost_compat.c: 193:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> > > > destroy device
> > > > rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> > > > bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
> > > > bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
> > > > rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> > > > destroy device
> > > > rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> > > > bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
> > > > bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
> > > > rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> > > > destroy device
> > > > rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> > > > bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
> > > > bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
> > > > rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> > > > destroy device
> > > > rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> > > > bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
> > > > bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
> > > >
> > > > Thanks,
> > > >
> > > > Feng Li
> > > >
> > > > --
> > > > The SmartX email address is only for business purpose. Any sent message
> > > > that is not related to the business is not authorized or permitted by
> > > > SmartX.
> > > > 本邮箱为北京志凌海纳科技有限公司（SmartX）工作邮箱. 如本邮箱发出
> > 的
> > > > 邮件与工作无关,该邮件未得到本公司任何的明示或默示的授权.
> > > >
> > > > _______________________________________________
> > > > SPDK mailing list -- spdk(a)lists.01.org
> > > > To unsubscribe send an email to spdk-leave(a)lists.01.org
> > > _______________________________________________
> > > SPDK mailing list -- spdk(a)lists.01.org
> > > To unsubscribe send an email to spdk-leave(a)lists.01.org
> >
> > --
> > The SmartX email address is only for business purpose. Any sent message
> > that is not related to the business is not authorized or permitted by
> > SmartX.
> > 本邮箱为北京志凌海纳科技有限公司（SmartX）工作邮箱. 如本邮箱发出的
> > 邮件与工作无关,该邮件未得到本公司任何的明示或默示的授权.
> >
> > _______________________________________________
> > SPDK mailing list -- spdk(a)lists.01.org
> > To unsubscribe send an email to spdk-leave(a)lists.01.org
> _______________________________________________
> SPDK mailing list -- spdk(a)lists.01.org
> To unsubscribe send an email to spdk-leave(a)lists.01.org
--
The SmartX email address is only for business purpose. Any sent message
that is not related to the business is not authorized or permitted by
SmartX.
本邮箱为北京志凌海纳科技有限公司（SmartX）工作邮箱. 如本邮箱发出的邮件与工作无关,该邮件未得到本公司任何的明示或默示的授权.
^ permalink raw reply [flat|nested] 7+ messages in thread
* [SPDK] Re: vhost is destroyed many times when starting vhost device
@ 2020-04-26 7:22 Liu, Changpeng
0 siblings, 0 replies; 7+ messages in thread
From: Liu, Changpeng @ 2020-04-26 7:22 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 8389 bytes --]
From the virtio specification, it's OK if the driver only use 1 IO queues, even you specified the num-queue to a bigger value.
BIOS will only use 1 IO queue, so the vhost target still doesn't know when it can start the device.
I tested with QEMU 4.1.50, I don't see the issue you mentioned the amplification factor is 2.
> -----Original Message-----
> From: Li Feng <fengli(a)smartx.com>
> Sent: Sunday, April 26, 2020 1:17 PM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>; Kyle Zhang
> <kyle(a)smartx.com>
> Subject: [SPDK] Re: vhost is destroyed many times when starting vhost device
>
> Comment in-line.
>
> Thanks,
>
> Feng Li
>
> Liu, Changpeng <changpeng.liu(a)intel.com> 于2020年4月26日周日 上午
> 8:46写道：
>
> >
> > Hi Feng,
> >
> > The issue you mentioned does exist, but it's not related with number of disk,
> but with "num-queues" parameter input from
> > QEMU, in my test environment, if I used 8 IO queues, the vhost_blk_start() will
> be called 9 times. Note that the BIOS will
> > use 1 IO queue so vhost_blk_start() is called 1 time during the BIOS stage.
>
> Thanks for your clarification of the relationship with num-queues.
> In my previous test, the num-queues is equal to 4.
> The amplification factor is nearly 2.
>
> Even set num-queues = 1, the stop/start device and
> vhost_blk_start/stop is still been called 2 times.
> The following log is about num-queues =1.
> At the time of showing grub menu, the event is:
> start_device
> vhost_blk_start
> io_setup
> start_device
> stop_device
> vhost_blk_stop
> start_device
> vhost_blk_start
> io_destroy
> io_setup
>
> After starting the kernel:
> stop_device
> vhost_blk_stop
> stop_device
> io_destroy
> start_device
> vhost_blk_start
> io_setup
> stop_device
> vhost_blk_stop
> start_device
> vhost_blk_start
> io_destroy
> io_setup
> start_device
>
>
> My qemu is master branch of upstream.
>
> >
> > In the vhost target side, we don't know the exactly number of queues used for
> this device, so we will start the device once
> > got one valid IO queue, and restart the device when got a new IO queue.
> That's the reason why the start/stop device will
> > be called multiple times.
>
> Could the vhost target have a chance to implement the multi-queue
> without start/stop the device?
> Currently, when the io-queue num is up, the call times of the backend
> bdev constructor/destructor will increase.
>
> What do you think my patch works?
>
> >
> > > -----Original Message-----
> > > From: Li Feng <fengli(a)smartx.com>
> > > Sent: Thursday, April 23, 2020 8:30 PM
> > > To: Storage Performance Development Kit <spdk(a)lists.01.org>
> > > Subject: [SPDK] vhost is destroyed many times when starting vhost device
> > >
> > > Hi,
> > >
> > > In rte_vhost_compat.c, spdk_extern_vhost_pre_msg_handler will call
> > > destroy_device when some conditions meet.
> > >
> > > I tested and found when a vhost-blk setup, the vhost device will be
> > > destroyed 9 times.
> > > And the bdev(no matter which type) will be affected. I'm using the aio
> > > bdev, and the io_setup/io_destroy will be called 9 times when a disk
> > > is inserted to a VM.
> > >
> > > My concern is:
> > > Is there any solution to reduce the call times(at least: disks nums * 9)?
> > > The call will slow down the VM boot time if VM has multiple disks.
> > > There are some related logs:
> > >
> > > Starting SPDK v20.01-pre git sha1 93eecf627 / DPDK 19.11.0 initialization...
> > > [ DPDK EAL parameters: spdk_tgt --no-shconf -c 1 -m 1
> > > --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=user1:6
> > > --iova-mode=pa --base-virtaddr=0x200000000000 --match-allocations
> > > --file-prefix=spdk_pid2004865 ]
> > > EAL: No available hugepages reported in hugepages-1048576kB
> > > app.c: 645:spdk_app_start: *NOTICE*: Total cores available: 1
> > > reactor.c: 346:_spdk_reactor_run: *NOTICE*: Reactor started on core 0
> > > bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
> > > bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
> > > bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
> > > rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> > > destroy device
> > > rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> > > bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
> > > bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
> > > rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> > > destroy device
> > > rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> > > bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
> > > bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
> > > rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> > > destroy device
> > > rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> > > bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
> > > bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
> > > rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> > > destroy device
> > > rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> > > bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
> > > bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
> > > rte_vhost_compat.c: 174:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> > > destroy device
> > > rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> > > bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
> > > bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
> > > rte_vhost_compat.c: 193:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> > > destroy device
> > > rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> > > bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
> > > bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
> > > rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> > > destroy device
> > > rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> > > bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
> > > bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
> > > rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> > > destroy device
> > > rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> > > bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
> > > bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
> > > rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> > > destroy device
> > > rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> > > bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
> > > bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
> > >
> > > Thanks,
> > >
> > > Feng Li
> > >
> > > --
> > > The SmartX email address is only for business purpose. Any sent message
> > > that is not related to the business is not authorized or permitted by
> > > SmartX.
> > > 本邮箱为北京志凌海纳科技有限公司（SmartX）工作邮箱. 如本邮箱发出
> 的
> > > 邮件与工作无关,该邮件未得到本公司任何的明示或默示的授权.
> > >
> > > _______________________________________________
> > > SPDK mailing list -- spdk(a)lists.01.org
> > > To unsubscribe send an email to spdk-leave(a)lists.01.org
> > _______________________________________________
> > SPDK mailing list -- spdk(a)lists.01.org
> > To unsubscribe send an email to spdk-leave(a)lists.01.org
>
> --
> The SmartX email address is only for business purpose. Any sent message
> that is not related to the business is not authorized or permitted by
> SmartX.
> 本邮箱为北京志凌海纳科技有限公司（SmartX）工作邮箱. 如本邮箱发出的
> 邮件与工作无关,该邮件未得到本公司任何的明示或默示的授权.
>
> _______________________________________________
> SPDK mailing list -- spdk(a)lists.01.org
> To unsubscribe send an email to spdk-leave(a)lists.01.org
^ permalink raw reply [flat|nested] 7+ messages in thread
* [SPDK] Re: vhost is destroyed many times when starting vhost device
@ 2020-04-26 5:17 Li Feng
0 siblings, 0 replies; 7+ messages in thread
From: Li Feng @ 2020-04-26 5:17 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 7252 bytes --]
Comment in-line.
Thanks,
Feng Li
Liu, Changpeng <changpeng.liu(a)intel.com> 于2020年4月26日周日 上午8:46写道：
>
> Hi Feng,
>
> The issue you mentioned does exist, but it's not related with number of disk, but with "num-queues" parameter input from
> QEMU, in my test environment, if I used 8 IO queues, the vhost_blk_start() will be called 9 times. Note that the BIOS will
> use 1 IO queue so vhost_blk_start() is called 1 time during the BIOS stage.
Thanks for your clarification of the relationship with num-queues.
In my previous test, the num-queues is equal to 4.
The amplification factor is nearly 2.
Even set num-queues = 1, the stop/start device and
vhost_blk_start/stop is still been called 2 times.
The following log is about num-queues =1.
At the time of showing grub menu, the event is:
start_device
vhost_blk_start
io_setup
start_device
stop_device
vhost_blk_stop
start_device
vhost_blk_start
io_destroy
io_setup
After starting the kernel:
stop_device
vhost_blk_stop
stop_device
io_destroy
start_device
vhost_blk_start
io_setup
stop_device
vhost_blk_stop
start_device
vhost_blk_start
io_destroy
io_setup
start_device
My qemu is master branch of upstream.
>
> In the vhost target side, we don't know the exactly number of queues used for this device, so we will start the device once
> got one valid IO queue, and restart the device when got a new IO queue. That's the reason why the start/stop device will
> be called multiple times.
Could the vhost target have a chance to implement the multi-queue
without start/stop the device?
Currently, when the io-queue num is up, the call times of the backend
bdev constructor/destructor will increase.
What do you think my patch works?
>
> > -----Original Message-----
> > From: Li Feng <fengli(a)smartx.com>
> > Sent: Thursday, April 23, 2020 8:30 PM
> > To: Storage Performance Development Kit <spdk(a)lists.01.org>
> > Subject: [SPDK] vhost is destroyed many times when starting vhost device
> >
> > Hi,
> >
> > In rte_vhost_compat.c, spdk_extern_vhost_pre_msg_handler will call
> > destroy_device when some conditions meet.
> >
> > I tested and found when a vhost-blk setup, the vhost device will be
> > destroyed 9 times.
> > And the bdev(no matter which type) will be affected. I'm using the aio
> > bdev, and the io_setup/io_destroy will be called 9 times when a disk
> > is inserted to a VM.
> >
> > My concern is:
> > Is there any solution to reduce the call times(at least: disks nums * 9)?
> > The call will slow down the VM boot time if VM has multiple disks.
> > There are some related logs:
> >
> > Starting SPDK v20.01-pre git sha1 93eecf627 / DPDK 19.11.0 initialization...
> > [ DPDK EAL parameters: spdk_tgt --no-shconf -c 1 -m 1
> > --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=user1:6
> > --iova-mode=pa --base-virtaddr=0x200000000000 --match-allocations
> > --file-prefix=spdk_pid2004865 ]
> > EAL: No available hugepages reported in hugepages-1048576kB
> > app.c: 645:spdk_app_start: *NOTICE*: Total cores available: 1
> > reactor.c: 346:_spdk_reactor_run: *NOTICE*: Reactor started on core 0
> > bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
> > bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
> > bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
> > rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> > destroy device
> > rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> > bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
> > bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
> > rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> > destroy device
> > rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> > bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
> > bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
> > rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> > destroy device
> > rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> > bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
> > bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
> > rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> > destroy device
> > rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> > bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
> > bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
> > rte_vhost_compat.c: 174:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> > destroy device
> > rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> > bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
> > bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
> > rte_vhost_compat.c: 193:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> > destroy device
> > rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> > bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
> > bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
> > rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> > destroy device
> > rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> > bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
> > bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
> > rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> > destroy device
> > rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> > bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
> > bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
> > rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> > destroy device
> > rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> > bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
> > bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
> >
> > Thanks,
> >
> > Feng Li
> >
> > --
> > The SmartX email address is only for business purpose. Any sent message
> > that is not related to the business is not authorized or permitted by
> > SmartX.
> > 本邮箱为北京志凌海纳科技有限公司（SmartX）工作邮箱. 如本邮箱发出的
> > 邮件与工作无关,该邮件未得到本公司任何的明示或默示的授权.
> >
> > _______________________________________________
> > SPDK mailing list -- spdk(a)lists.01.org
> > To unsubscribe send an email to spdk-leave(a)lists.01.org
> _______________________________________________
> SPDK mailing list -- spdk(a)lists.01.org
> To unsubscribe send an email to spdk-leave(a)lists.01.org
--
The SmartX email address is only for business purpose. Any sent message
that is not related to the business is not authorized or permitted by
SmartX.
本邮箱为北京志凌海纳科技有限公司（SmartX）工作邮箱. 如本邮箱发出的邮件与工作无关,该邮件未得到本公司任何的明示或默示的授权.
^ permalink raw reply [flat|nested] 7+ messages in thread
* [SPDK] Re: vhost is destroyed many times when starting vhost device
@ 2020-04-26 0:46 Liu, Changpeng
0 siblings, 0 replies; 7+ messages in thread
From: Liu, Changpeng @ 2020-04-26 0:46 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 5404 bytes --]
Hi Feng,
The issue you mentioned does exist, but it's not related with number of disk, but with "num-queues" parameter input from
QEMU, in my test environment, if I used 8 IO queues, the vhost_blk_start() will be called 9 times. Note that the BIOS will
use 1 IO queue so vhost_blk_start() is called 1 time during the BIOS stage.
In the vhost target side, we don't know the exactly number of queues used for this device, so we will start the device once
got one valid IO queue, and restart the device when got a new IO queue. That's the reason why the start/stop device will
be called multiple times.
> -----Original Message-----
> From: Li Feng <fengli(a)smartx.com>
> Sent: Thursday, April 23, 2020 8:30 PM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: [SPDK] vhost is destroyed many times when starting vhost device
>
> Hi,
>
> In rte_vhost_compat.c, spdk_extern_vhost_pre_msg_handler will call
> destroy_device when some conditions meet.
>
> I tested and found when a vhost-blk setup, the vhost device will be
> destroyed 9 times.
> And the bdev(no matter which type) will be affected. I'm using the aio
> bdev, and the io_setup/io_destroy will be called 9 times when a disk
> is inserted to a VM.
>
> My concern is:
> Is there any solution to reduce the call times(at least: disks nums * 9)?
> The call will slow down the VM boot time if VM has multiple disks.
> There are some related logs:
>
> Starting SPDK v20.01-pre git sha1 93eecf627 / DPDK 19.11.0 initialization...
> [ DPDK EAL parameters: spdk_tgt --no-shconf -c 1 -m 1
> --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=user1:6
> --iova-mode=pa --base-virtaddr=0x200000000000 --match-allocations
> --file-prefix=spdk_pid2004865 ]
> EAL: No available hugepages reported in hugepages-1048576kB
> app.c: 645:spdk_app_start: *NOTICE*: Total cores available: 1
> reactor.c: 346:_spdk_reactor_run: *NOTICE*: Reactor started on core 0
> bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
> bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
> bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
> rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> destroy device
> rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
> bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
> rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> destroy device
> rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
> bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
> rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> destroy device
> rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
> bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
> rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> destroy device
> rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
> bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
> rte_vhost_compat.c: 174:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> destroy device
> rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
> bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
> rte_vhost_compat.c: 193:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> destroy device
> rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
> bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
> rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> destroy device
> rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
> bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
> rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> destroy device
> rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
> bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
> rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> destroy device
> rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
> bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
>
> Thanks,
>
> Feng Li
>
> --
> The SmartX email address is only for business purpose. Any sent message
> that is not related to the business is not authorized or permitted by
> SmartX.
> 本邮箱为北京志凌海纳科技有限公司（SmartX）工作邮箱. 如本邮箱发出的
> 邮件与工作无关,该邮件未得到本公司任何的明示或默示的授权.
>
> _______________________________________________
> SPDK mailing list -- spdk(a)lists.01.org
> To unsubscribe send an email to spdk-leave(a)lists.01.org
^ permalink raw reply [flat|nested] 7+ messages in thread
* [SPDK] Re: vhost is destroyed many times when starting vhost device
@ 2020-04-25 4:28 Li Feng
0 siblings, 0 replies; 7+ messages in thread
From: Li Feng @ 2020-04-25 4:28 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 6347 bytes --]
Hi Xiaodong,
Thanks for your comment.
Add `spdk_put_io_channel` in bdev_remove_cb could resolve the hotplug issue.
I have updated the patch.
Feng Li
Liu, Xiaodong <xiaodong.liu(a)intel.com> 于2020年4月25日周六 上午8:47写道：
>
> Hi, Feng
>
> Thanks for your digging on this scenario. Yes, it is really a problem on which we should consider more.
>
> I looked into your patch. But I just think you missed the situation if the backend bdev is hot removed, seems the dummy_io_channel will prevent the removal process.
>
> --Thanks
> From Xiaodong
>
> -----Original Message-----
> From: Li Feng <fengli(a)smartx.com>
> Sent: Friday, April 24, 2020 10:44 PM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: [SPDK] Re: vhost is destroyed many times when starting vhost device
>
> Hi,
> I have submitted a patch[1] to partially fix this issue.
> Any suggestions?
> After applying this patch, the log of call io_setup/io_destroy will be gone.
>
> [1]: https://review.spdk.io/gerrit/c/spdk/spdk/+/2020
>
> Thanks,
>
> Feng Li
>
> Li Feng <fengli(a)smartx.com> 于2020年4月23日周四 下午8:29写道：
> >
> > Hi,
> >
> > In rte_vhost_compat.c, spdk_extern_vhost_pre_msg_handler will call
> > destroy_device when some conditions meet.
> >
> > I tested and found when a vhost-blk setup, the vhost device will be
> > destroyed 9 times.
> > And the bdev(no matter which type) will be affected. I'm using the aio
> > bdev, and the io_setup/io_destroy will be called 9 times when a disk
> > is inserted to a VM.
> >
> > My concern is:
> > Is there any solution to reduce the call times(at least: disks nums * 9)?
> > The call will slow down the VM boot time if VM has multiple disks.
> > There are some related logs:
> >
> > Starting SPDK v20.01-pre git sha1 93eecf627 / DPDK 19.11.0 initialization...
> > [ DPDK EAL parameters: spdk_tgt --no-shconf -c 1 -m 1
> > --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=user1:6
> > --iova-mode=pa --base-virtaddr=0x200000000000 --match-allocations
> > --file-prefix=spdk_pid2004865 ]
> > EAL: No available hugepages reported in hugepages-1048576kB
> > app.c: 645:spdk_app_start: *NOTICE*: Total cores available: 1
> > reactor.c: 346:_spdk_reactor_run: *NOTICE*: Reactor started on core 0
> > bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
> > bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
> > bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
> > rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> > destroy device
> > rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> > bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
> > bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
> > rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> > destroy device
> > rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> > bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
> > bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
> > rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> > destroy device
> > rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> > bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
> > bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
> > rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> > destroy device
> > rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> > bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
> > bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
> > rte_vhost_compat.c: 174:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> > destroy device
> > rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> > bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
> > bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
> > rte_vhost_compat.c: 193:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> > destroy device
> > rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> > bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
> > bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
> > rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> > destroy device
> > rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> > bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
> > bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
> > rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> > destroy device
> > rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> > bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
> > bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
> > rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> > destroy device
> > rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> > bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
> > bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
> >
> > Thanks,
> >
> > Feng Li
>
> --
> The SmartX email address is only for business purpose. Any sent message that is not related to the business is not authorized or permitted by SmartX.
> 本邮箱为北京志凌海纳科技有限公司（SmartX）工作邮箱. 如本邮箱发出的邮件与工作无关,该邮件未得到本公司任何的明示或默示的授权.
>
> _______________________________________________
> SPDK mailing list -- spdk(a)lists.01.org
> To unsubscribe send an email to spdk-leave(a)lists.01.org
> _______________________________________________
> SPDK mailing list -- spdk(a)lists.01.org
> To unsubscribe send an email to spdk-leave(a)lists.01.org
--
The SmartX email address is only for business purpose. Any sent message
that is not related to the business is not authorized or permitted by
SmartX.
本邮箱为北京志凌海纳科技有限公司（SmartX）工作邮箱. 如本邮箱发出的邮件与工作无关,该邮件未得到本公司任何的明示或默示的授权.
^ permalink raw reply [flat|nested] 7+ messages in thread
* [SPDK] Re: vhost is destroyed many times when starting vhost device
@ 2020-04-25 0:47 Liu, Xiaodong
0 siblings, 0 replies; 7+ messages in thread
From: Liu, Xiaodong @ 2020-04-25 0:47 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 5389 bytes --]
Hi, Feng
Thanks for your digging on this scenario. Yes, it is really a problem on which we should consider more.
I looked into your patch. But I just think you missed the situation if the backend bdev is hot removed, seems the dummy_io_channel will prevent the removal process.
--Thanks
From Xiaodong
-----Original Message-----
From: Li Feng <fengli(a)smartx.com>
Sent: Friday, April 24, 2020 10:44 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: [SPDK] Re: vhost is destroyed many times when starting vhost device
Hi,
I have submitted a patch[1] to partially fix this issue.
Any suggestions?
After applying this patch, the log of call io_setup/io_destroy will be gone.
[1]: https://review.spdk.io/gerrit/c/spdk/spdk/+/2020
Thanks,
Feng Li
Li Feng <fengli(a)smartx.com> 于2020年4月23日周四 下午8:29写道：
>
> Hi,
>
> In rte_vhost_compat.c, spdk_extern_vhost_pre_msg_handler will call
> destroy_device when some conditions meet.
>
> I tested and found when a vhost-blk setup, the vhost device will be
> destroyed 9 times.
> And the bdev(no matter which type) will be affected. I'm using the aio
> bdev, and the io_setup/io_destroy will be called 9 times when a disk
> is inserted to a VM.
>
> My concern is:
> Is there any solution to reduce the call times(at least: disks nums * 9)?
> The call will slow down the VM boot time if VM has multiple disks.
> There are some related logs:
>
> Starting SPDK v20.01-pre git sha1 93eecf627 / DPDK 19.11.0 initialization...
> [ DPDK EAL parameters: spdk_tgt --no-shconf -c 1 -m 1
> --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=user1:6
> --iova-mode=pa --base-virtaddr=0x200000000000 --match-allocations
> --file-prefix=spdk_pid2004865 ]
> EAL: No available hugepages reported in hugepages-1048576kB
> app.c: 645:spdk_app_start: *NOTICE*: Total cores available: 1
> reactor.c: 346:_spdk_reactor_run: *NOTICE*: Reactor started on core 0
> bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
> bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
> bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
> rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> destroy device
> rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
> bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
> rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> destroy device
> rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
> bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
> rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> destroy device
> rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
> bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
> rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> destroy device
> rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
> bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
> rte_vhost_compat.c: 174:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> destroy device
> rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
> bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
> rte_vhost_compat.c: 193:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> destroy device
> rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
> bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
> rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> destroy device
> rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
> bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
> rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> destroy device
> rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
> bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
> rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> destroy device
> rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
> bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
>
> Thanks,
>
> Feng Li
--
The SmartX email address is only for business purpose. Any sent message that is not related to the business is not authorized or permitted by SmartX.
本邮箱为北京志凌海纳科技有限公司（SmartX）工作邮箱. 如本邮箱发出的邮件与工作无关,该邮件未得到本公司任何的明示或默示的授权.
_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org
^ permalink raw reply [flat|nested] 7+ messages in thread
* [SPDK] Re: vhost is destroyed many times when starting vhost device
@ 2020-04-24 14:44 Li Feng
0 siblings, 0 replies; 7+ messages in thread
From: Li Feng @ 2020-04-24 14:44 UTC (permalink / raw)
To: spdk
[-- Attachment #1: Type: text/plain, Size: 4676 bytes --]
Hi,
I have submitted a patch[1] to partially fix this issue.
Any suggestions?
After applying this patch, the log of call io_setup/io_destroy will be gone.
[1]: https://review.spdk.io/gerrit/c/spdk/spdk/+/2020
Thanks,
Feng Li
Li Feng <fengli(a)smartx.com> 于2020年4月23日周四 下午8:29写道：
>
> Hi,
>
> In rte_vhost_compat.c, spdk_extern_vhost_pre_msg_handler will call
> destroy_device when some conditions meet.
>
> I tested and found when a vhost-blk setup, the vhost device will be
> destroyed 9 times.
> And the bdev(no matter which type) will be affected. I'm using the aio
> bdev, and the io_setup/io_destroy will be called 9 times when a disk
> is inserted to a VM.
>
> My concern is:
> Is there any solution to reduce the call times(at least: disks nums * 9)?
> The call will slow down the VM boot time if VM has multiple disks.
> There are some related logs:
>
> Starting SPDK v20.01-pre git sha1 93eecf627 / DPDK 19.11.0 initialization...
> [ DPDK EAL parameters: spdk_tgt --no-shconf -c 1 -m 1
> --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=user1:6
> --iova-mode=pa --base-virtaddr=0x200000000000 --match-allocations
> --file-prefix=spdk_pid2004865 ]
> EAL: No available hugepages reported in hugepages-1048576kB
> app.c: 645:spdk_app_start: *NOTICE*: Total cores available: 1
> reactor.c: 346:_spdk_reactor_run: *NOTICE*: Reactor started on core 0
> bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
> bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
> bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
> rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> destroy device
> rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
> bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
> rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> destroy device
> rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
> bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
> rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> destroy device
> rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
> bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
> rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> destroy device
> rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
> bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
> rte_vhost_compat.c: 174:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> destroy device
> rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
> bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
> rte_vhost_compat.c: 193:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> destroy device
> rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
> bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
> rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> destroy device
> rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
> bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
> rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> destroy device
> rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
> bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
> rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
> destroy device
> rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
> bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
> bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
>
> Thanks,
>
> Feng Li
--
The SmartX email address is only for business purpose. Any sent message
that is not related to the business is not authorized or permitted by
SmartX.
本邮箱为北京志凌海纳科技有限公司（SmartX）工作邮箱. 如本邮箱发出的邮件与工作无关,该邮件未得到本公司任何的明示或默示的授权.
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2020-04-26 8:48 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-04-26 8:48 [SPDK] Re: vhost is destroyed many times when starting vhost device Li Feng
-- strict thread matches above, loose matches on Subject: below --
2020-04-26 7:22 Liu, Changpeng
2020-04-26 5:17 Li Feng
2020-04-26 0:46 Liu, Changpeng
2020-04-25 4:28 Li Feng
2020-04-25 0:47 Liu, Xiaodong
2020-04-24 14:44 Li Feng
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.