[Qemu-devel] [SPDK] qemu process hung at boot-up, no explicit errors or warnings

[Qemu-devel] [SPDK] qemu process hung at boot-up, no explicit errors or warnings

[Bob Chen]

[-- Attachment #1: Type: text/plain, Size: 822 bytes --]

Hi,

I was trying to run qemu with spdk, referring to
http://www.spdk.io/doc/vhost.html#vhost_qemu_config

Everything went well since I had already set up hugepages, vfio, vhost
targets, vhost-scsi device(vhost-block was also tested), etc, without
errors or warnings reported.

But at the last step to run qemu, the process would somehow hang. And its
cpu busy remained 100%.

I used `perf top -p` to monitor the process activity,
  18.28%  [kernel]                      [k] vmx_vcpu_run
   3.06%  [kernel]                      [k] vcpu_enter_guest
   3.05%  [kernel]                      [k] system_call_after_swapgs

Do you have any ideas about what happened here?

Test environment: latest CentOS 7 kernel, nvme ssd, spdk v18.01.x,
dpdk 17.11.1, qemu 2.12.0-rc2. Detail log is attached within this mail.



Thanks,
Bob

[-- Attachment #2: qemu-spdk.log --]
[-- Type: application/octet-stream, Size: 4526 bytes --]

# ./app/vhost/vhost -S /var/tmp -s 4096 -m 0x3 &
[1] 12647
Starting DPDK 17.11.1 initialization...
[ DPDK EAL parameters: vhost -c 0x3 -m 4096 --file-prefix=spdk_pid12647 ]
EAL: Detected 32 lcore(s)
EAL: Probing VFIO support...
EAL: VFIO support initialized
app.c: 377:spdk_app_start: *NOTICE*: Total cores available: 2
reactor.c: 654:spdk_reactors_init: *NOTICE*: Occupied cpu socket mask is 0x3
reactor.c: 426:_spdk_reactor_run: *NOTICE*: Reactor started on core 1 on socket 1
reactor.c: 426:_spdk_reactor_run: *NOTICE*: Reactor started on core 0 on socket 0
copy_engine_ioat.c: 306:copy_engine_ioat_init: *NOTICE*: Ioat Copy Engine Offload Enabled


# ./scripts/rpc.py construct_nvme_bdev -b Nvme0 -t pcie -a 0000:03:00.0
EAL: PCI device 0000:03:00.0 on NUMA socket 0
EAL:   probe driver: 1c5f:540 spdk_nvme
EAL:   using IOMMU type 1 (Type 1)


# ./scripts/rpc.py construct_malloc_bdev 128 4096  -b Malloc0
Malloc0


# ./scripts/rpc.py construct_vhost_scsi_controller --cpumask 0x1 vhost.0
VHOST_CONFIG: vhost-user server: socket created, fd: 26
VHOST_CONFIG: bind to /var/tmp/vhost.0
vhost.c: 728:spdk_vhost_dev_construct: *NOTICE*: Controller vhost.0: new controller added


# ./scripts/rpc.py add_vhost_scsi_lun vhost.0 0 Nvme0n1
vhost_scsi.c: 841:spdk_vhost_scsi_dev_add_tgt: *NOTICE*: Controller vhost.0: defined target 'Target 0' using bdev 'Nvme0n1'


# ./scripts/rpc.py add_vhost_scsi_lun vhost.0 1 Malloc0
vhost_scsi.c: 841:spdk_vhost_scsi_dev_add_tgt: *NOTICE*: Controller vhost.0: defined target 'Target 1' using bdev 'Malloc0'



# qemu-system-x86_64 \
  --enable-kvm \
  -cpu host -smp 2 \
  -m 2G -object memory-backend-file,id=mem0,size=2G,mem-path=/dev/hugepages,share=on -numa node,memdev=mem0 \
  ...
  -chardev socket,id=spdk_vhost_scsi0,path=/var/tmp/vhost.0 \
  -device vhost-user-scsi-pci,id=scsi0,chardev=spdk_vhost_scsi0,num_queues=4 
  ...


# vhost log

VHOST_CONFIG: new vhost user connection is 20
VHOST_CONFIG: new device, handle is 0
VHOST_CONFIG: /var/tmp/vhost.0: read message VHOST_USER_GET_FEATURES
VHOST_CONFIG: /var/tmp/vhost.0: read message VHOST_USER_GET_PROTOCOL_FEATURES
VHOST_CONFIG: /var/tmp/vhost.0: read message VHOST_USER_SET_PROTOCOL_FEATURES
VHOST_CONFIG: /var/tmp/vhost.0: read message VHOST_USER_GET_QUEUE_NUM
VHOST_CONFIG: /var/tmp/vhost.0: read message VHOST_USER_SET_OWNER
VHOST_CONFIG: /var/tmp/vhost.0: read message VHOST_USER_GET_FEATURES
VHOST_CONFIG: /var/tmp/vhost.0: read message VHOST_USER_SET_VRING_CALL
VHOST_CONFIG: vring call idx:0 file:27
VHOST_CONFIG: /var/tmp/vhost.0: read message VHOST_USER_SET_VRING_CALL
VHOST_CONFIG: vring call idx:1 file:28
VHOST_CONFIG: /var/tmp/vhost.0: read message VHOST_USER_SET_VRING_CALL
VHOST_CONFIG: vring call idx:2 file:29
VHOST_CONFIG: /var/tmp/vhost.0: read message VHOST_USER_SET_VRING_CALL
VHOST_CONFIG: vring call idx:3 file:30
VHOST_CONFIG: /var/tmp/vhost.0: read message VHOST_USER_SET_VRING_CALL
VHOST_CONFIG: vring call idx:4 file:31
VHOST_CONFIG: /var/tmp/vhost.0: read message VHOST_USER_SET_VRING_CALL
VHOST_CONFIG: vring call idx:5 file:32
VHOST_CONFIG: /var/tmp/vhost.0: read message VHOST_USER_SET_FEATURES
VHOST_CONFIG: /var/tmp/vhost.0: read message VHOST_USER_SET_MEM_TABLE
VHOST_CONFIG: /var/tmp/vhost.0: read message VHOST_USER_SET_VRING_NUM
VHOST_CONFIG: /var/tmp/vhost.0: read message VHOST_USER_SET_VRING_BASE
VHOST_CONFIG: /var/tmp/vhost.0: read message VHOST_USER_SET_VRING_ADDR
VHOST_CONFIG: guest memory region 0, size: 0x40000000
	 guest physical addr: 0x0
	 guest virtual  addr: 0x7fcfc0000000
	 host  virtual  addr: 0x2aaac0000000
	 mmap addr : 0x2aaac0000000
	 mmap size : 0x40000000
	 mmap align: 0x40000000
	 mmap off  : 0x0
VHOST_CONFIG: reallocate vq from 0 to 1 node
VHOST_CONFIG: reallocate dev from 0 to 1 node
VHOST_CONFIG: /var/tmp/vhost.0: read message VHOST_USER_SET_VRING_KICK
VHOST_CONFIG: vring kick idx:2 file:34
VHOST_CONFIG: /var/tmp/vhost.0: read message VHOST_USER_SET_VRING_CALL
VHOST_CONFIG: vring call idx:0 file:35
VHOST_CONFIG: /var/tmp/vhost.0: read message VHOST_USER_SET_VRING_CALL
VHOST_CONFIG: vring call idx:1 file:27
VHOST_CONFIG: /var/tmp/vhost.0: read message VHOST_USER_SET_VRING_CALL
VHOST_CONFIG: vring call idx:2 file:28
VHOST_CONFIG: /var/tmp/vhost.0: read message VHOST_USER_SET_VRING_CALL
VHOST_CONFIG: vring call idx:3 file:29
VHOST_CONFIG: /var/tmp/vhost.0: read message VHOST_USER_SET_VRING_CALL
VHOST_CONFIG: vring call idx:4 file:30
VHOST_CONFIG: /var/tmp/vhost.0: read message VHOST_USER_SET_VRING_CALL
VHOST_CONFIG: vring call idx:5 file:31 

Re: [Qemu-devel] [SPDK] qemu process hung at boot-up, no explicit errors or warnings

[John Snow]

Forwarding to qemu-block.

On 04/19/2018 06:13 AM, Bob Chen wrote:
> Hi,
> 
> I was trying to run qemu with spdk, referring to
> http://www.spdk.io/doc/vhost.html#vhost_qemu_config
> 
> Everything went well since I had already set up hugepages, vfio, vhost
> targets, vhost-scsi device(vhost-block was also tested), etc, without
> errors or warnings reported.
> 
> But at the last step to run qemu, the process would somehow hang. And its
> cpu busy remained 100%.
> 
> I used `perf top -p` to monitor the process activity,
>   18.28%  [kernel]                      [k] vmx_vcpu_run
>    3.06%  [kernel]                      [k] vcpu_enter_guest
>    3.05%  [kernel]                      [k] system_call_after_swapgs
> 
> Do you have any ideas about what happened here?
> 
> Test environment: latest CentOS 7 kernel, nvme ssd, spdk v18.01.x,
> dpdk 17.11.1, qemu 2.12.0-rc2. Detail log is attached within this mail.
> 

Have you tried any other versions? (2.11.1, or 2.12-rc4?)

> 
> 
> Thanks,
> Bob
> 

Re: [Qemu-devel] [SPDK] qemu process hung at boot-up, no explicit errors or warnings

[Bob Chen]

[-- Attachment #1: Type: text/plain, Size: 1418 bytes --]

2.11.1 could work, qemu is no longer occupying 100% CPU. That's
interesting...

Now I can see the starting screen via vnc, says that not a bootable disk.


Is that because the guest OS's virtio vhost driver is not up to date enough?
​

Thanks,
Bob

2018-04-20 1:17 GMT+08:00 John Snow <jsnow@redhat.com>:

> Forwarding to qemu-block.
>
> On 04/19/2018 06:13 AM, Bob Chen wrote:
> > Hi,
> >
> > I was trying to run qemu with spdk, referring to
> > http://www.spdk.io/doc/vhost.html#vhost_qemu_config
> >
> > Everything went well since I had already set up hugepages, vfio, vhost
> > targets, vhost-scsi device(vhost-block was also tested), etc, without
> > errors or warnings reported.
> >
> > But at the last step to run qemu, the process would somehow hang. And its
> > cpu busy remained 100%.
> >
> > I used `perf top -p` to monitor the process activity,
> >   18.28%  [kernel]                      [k] vmx_vcpu_run
> >    3.06%  [kernel]                      [k] vcpu_enter_guest
> >    3.05%  [kernel]                      [k] system_call_after_swapgs
> >
> > Do you have any ideas about what happened here?
> >
> > Test environment: latest CentOS 7 kernel, nvme ssd, spdk v18.01.x,
> > dpdk 17.11.1, qemu 2.12.0-rc2. Detail log is attached within this mail.
> >
>
> Have you tried any other versions? (2.11.1, or 2.12-rc4?)
>
> >
> >
> > Thanks,
> > Bob
> >
>
>

[-- Attachment #2: booting.png --]
[-- Type: image/png, Size: 105842 bytes --]

Re: [Qemu-devel] [SPDK] qemu process hung at boot-up, no explicit errors or warnings

[John Snow]

On 04/20/2018 07:13 AM, Bob Chen wrote:
> 2.11.1 could work, qemu is no longer occupying 100% CPU. That's
> interesting...
> 

Does 2.12 use 100% even at the firmware menu? Maybe we're not giving
this VM long enough to hit the spot that causes it to use 100%.

> Now I can see the starting screen via vnc, says that not a bootable disk.
> 

If it's stopping at the firmware I think the guest is not being taken
into account, and the firmware is either trying to boot the wrong disk
or your disk is corrupted. (Or it can't see your disk at all -- I'm
personally not very familiar with using SPDK.)

Can you share your command line with us?

> 
> Is that because the guest OS's virtio vhost driver is not up to date enough?
> ​
> 
> Thanks,
> Bob
> 

On the QEMU-devel list and many other technical lists, can you please
reply in-line instead of replying above? ("top posting")

Thank you,
--John

> 2018-04-20 1:17 GMT+08:00 John Snow <jsnow@redhat.com
> <mailto:jsnow@redhat.com>>:
> 
>     Forwarding to qemu-block.
> 
>     On 04/19/2018 06:13 AM, Bob Chen wrote:
>     > Hi,
>     > 
>     > I was trying to run qemu with spdk, referring to
>     > http://www.spdk.io/doc/vhost.html#vhost_qemu_config
>     <http://www.spdk.io/doc/vhost.html#vhost_qemu_config>
>     > 
>     > Everything went well since I had already set up hugepages, vfio, vhost
>     > targets, vhost-scsi device(vhost-block was also tested), etc, without
>     > errors or warnings reported.
>     > 
>     > But at the last step to run qemu, the process would somehow hang. And its
>     > cpu busy remained 100%.
>     > 
>     > I used `perf top -p` to monitor the process activity,
>     >   18.28%  [kernel]                      [k] vmx_vcpu_run
>     >    3.06%  [kernel]                      [k] vcpu_enter_guest
>     >    3.05%  [kernel]                      [k] system_call_after_swapgs
>     > 
>     > Do you have any ideas about what happened here?
>     > 
>     > Test environment: latest CentOS 7 kernel, nvme ssd, spdk v18.01.x,
>     > dpdk 17.11.1, qemu 2.12.0-rc2. Detail log is attached within this mail.
>     > 
> 
>     Have you tried any other versions? (2.11.1, or 2.12-rc4?)
> 
>     >
>     >
>     > Thanks,
>     > Bob
>     >
> 
> 

Re: [Qemu-devel] [SPDK] qemu process hung at boot-up, no explicit errors or warnings

[Bob Chen]

2018-04-21 1:34 GMT+08:00 John Snow <jsnow@redhat.com>:

>
>
> On 04/20/2018 07:13 AM, Bob Chen wrote:
> > 2.11.1 could work, qemu is no longer occupying 100% CPU. That's
> > interesting...
> >
>
> Does 2.12 use 100% even at the firmware menu? Maybe we're not giving
> this VM long enough to hit the spot that causes it to use 100%.
>
> > Now I can see the starting screen via vnc, says that not a bootable disk.
> >
>
> If it's stopping at the firmware I think the guest is not being taken
> into account, and the firmware is either trying to boot the wrong disk
> or your disk is corrupted. (Or it can't see your disk at all -- I'm
> personally not very familiar with using SPDK.)


> Can you share your command line with us?
>

I opened another thread to discuss this issue, looks like there's something
wrong with the spdk official doc.

However, 2.12 is still very likely to be buggy. I'm sure I waited long
enough but the CPU never dropped below 100%. The vnc screen stopped at
"Loading SeaBIOS ..."

Again, 2.10 and 2.11 are fine. Though "No bootable device" but CPU dropped
anyway.


>
> >
> > Is that because the guest OS's virtio vhost driver is not up to date
> enough?
> > ​
> >
> > Thanks,
> > Bob
> >
>
> On the QEMU-devel list and many other technical lists, can you please
> reply in-line instead of replying above? ("top posting")
>
> Thank you,
> --John
>
> > 2018-04-20 1:17 GMT+08:00 John Snow <jsnow@redhat.com
> > <mailto:jsnow@redhat.com>>:
> >
> >     Forwarding to qemu-block.
> >
> >     On 04/19/2018 06:13 AM, Bob Chen wrote:
> >     > Hi,
> >     >
> >     > I was trying to run qemu with spdk, referring to
> >     > http://www.spdk.io/doc/vhost.html#vhost_qemu_config
> >     <http://www.spdk.io/doc/vhost.html#vhost_qemu_config>
> >     >
> >     > Everything went well since I had already set up hugepages, vfio,
> vhost
> >     > targets, vhost-scsi device(vhost-block was also tested), etc,
> without
> >     > errors or warnings reported.
> >     >
> >     > But at the last step to run qemu, the process would somehow hang.
> And its
> >     > cpu busy remained 100%.
> >     >
> >     > I used `perf top -p` to monitor the process activity,
> >     >   18.28%  [kernel]                      [k] vmx_vcpu_run
> >     >    3.06%  [kernel]                      [k] vcpu_enter_guest
> >     >    3.05%  [kernel]                      [k]
> system_call_after_swapgs
> >     >
> >     > Do you have any ideas about what happened here?
> >     >
> >     > Test environment: latest CentOS 7 kernel, nvme ssd, spdk v18.01.x,
> >     > dpdk 17.11.1, qemu 2.12.0-rc2. Detail log is attached within this
> mail.
> >     >
> >
> >     Have you tried any other versions? (2.11.1, or 2.12-rc4?)
> >
> >     >
> >     >
> >     > Thanks,
> >     > Bob
> >     >
> >
> >
>