[xen-unstable test] 104131: regressions - FAIL

[xen-unstable test] 104131: regressions - FAIL

[osstest service owner]

flight 104131 xen-unstable real [real]
http://logs.test-lab.xenproject.org/osstest/logs/104131/

Regressions :-(

Tests which did not succeed and are blocking,
including tests which could not be run:
 test-amd64-i386-xl-qemuu-debianhvm-amd64 16 guest-stop   fail REGR. vs. 104119

Regressions which are regarded as allowable (not blocking):
 test-armhf-armhf-xl-rtds    15 guest-start/debian.repeat fail REGR. vs. 104119
 test-armhf-armhf-libvirt-xsm 13 saverestore-support-check    fail  like 104104
 test-armhf-armhf-libvirt     13 saverestore-support-check    fail  like 104119
 test-amd64-i386-xl-qemuu-win7-amd64 16 guest-stop             fail like 104119
 test-amd64-i386-xl-qemut-win7-amd64 16 guest-stop             fail like 104119
 test-amd64-amd64-xl-qemut-win7-amd64 16 guest-stop            fail like 104119
 test-amd64-amd64-xl-qemuu-win7-amd64 16 guest-stop            fail like 104119
 test-armhf-armhf-libvirt-qcow2 12 saverestore-support-check   fail like 104119
 test-armhf-armhf-libvirt-raw 12 saverestore-support-check    fail  like 104119
 test-amd64-amd64-xl-rtds      9 debian-install               fail  like 104119

Tests which did not succeed, but are not blocking:
 test-amd64-amd64-xl-pvh-amd  11 guest-start                  fail   never pass
 test-amd64-amd64-xl-pvh-intel 11 guest-start                  fail  never pass
 test-amd64-i386-libvirt      12 migrate-support-check        fail   never pass
 test-amd64-i386-libvirt-xsm  12 migrate-support-check        fail   never pass
 test-amd64-amd64-libvirt-xsm 12 migrate-support-check        fail   never pass
 test-amd64-amd64-libvirt     12 migrate-support-check        fail   never pass
 test-amd64-i386-libvirt-qemuu-debianhvm-amd64-xsm 10 migrate-support-check fail never pass
 test-amd64-amd64-libvirt-qemuu-debianhvm-amd64-xsm 10 migrate-support-check fail never pass
 test-armhf-armhf-xl-arndale  12 migrate-support-check        fail   never pass
 test-armhf-armhf-xl-arndale  13 saverestore-support-check    fail   never pass
 test-amd64-amd64-libvirt-vhd 11 migrate-support-check        fail   never pass
 test-amd64-amd64-qemuu-nested-amd 16 debian-hvm-install/l1/l2  fail never pass
 test-armhf-armhf-xl-xsm      12 migrate-support-check        fail   never pass
 test-armhf-armhf-xl-credit2  12 migrate-support-check        fail   never pass
 test-armhf-armhf-xl-xsm      13 saverestore-support-check    fail   never pass
 test-armhf-armhf-xl-cubietruck 12 migrate-support-check        fail never pass
 test-armhf-armhf-libvirt     12 migrate-support-check        fail   never pass
 test-armhf-armhf-xl-credit2  13 saverestore-support-check    fail   never pass
 test-armhf-armhf-xl-cubietruck 13 saverestore-support-check    fail never pass
 test-armhf-armhf-libvirt-xsm 12 migrate-support-check        fail   never pass
 test-armhf-armhf-libvirt-qcow2 11 migrate-support-check        fail never pass
 test-armhf-armhf-xl-rtds     12 migrate-support-check        fail   never pass
 test-armhf-armhf-xl-rtds     13 saverestore-support-check    fail   never pass
 test-armhf-armhf-libvirt-raw 11 migrate-support-check        fail   never pass
 test-armhf-armhf-xl-vhd      11 migrate-support-check        fail   never pass
 test-armhf-armhf-xl-vhd      12 saverestore-support-check    fail   never pass
 test-armhf-armhf-xl-multivcpu 12 migrate-support-check        fail  never pass
 test-armhf-armhf-xl-multivcpu 13 saverestore-support-check    fail  never pass
 test-armhf-armhf-xl          12 migrate-support-check        fail   never pass
 test-armhf-armhf-xl          13 saverestore-support-check    fail   never pass

version targeted for testing:
 xen                  0d045d65c19ac48b31344b566cbf82a0270e6e44
baseline version:
 xen                  ffc103c223a6d12e5221f66b7e96396a61ba1b20

Last test of basis   104119  2017-01-11 06:45:46 Z    0 days
Failing since        104126  2017-01-11 16:44:54 Z    0 days    2 attempts
Testing same since   104131  2017-01-11 22:43:41 Z    0 days    1 attempts

------------------------------------------------------------
People who touched revisions under test:
  Andrew Cooper <andrew.cooper3@citrix.com>
  Jan Beulich <jbeulich@suse.com>
  Kevin Tian <kevin.tian@intel.com>
  Stefano Stabellini <sstabellini@kernel.org>
  Wei Liu <wei.liu2@citrix.com>

jobs:
 build-amd64-xsm                                              pass    
 build-armhf-xsm                                              pass    
 build-i386-xsm                                               pass    
 build-amd64-xtf                                              pass    
 build-amd64                                                  pass    
 build-armhf                                                  pass    
 build-i386                                                   pass    
 build-amd64-libvirt                                          pass    
 build-armhf-libvirt                                          pass    
 build-i386-libvirt                                           pass    
 build-amd64-oldkern                                          pass    
 build-i386-oldkern                                           pass    
 build-amd64-prev                                             pass    
 build-i386-prev                                              pass    
 build-amd64-pvops                                            pass    
 build-armhf-pvops                                            pass    
 build-i386-pvops                                             pass    
 build-amd64-rumprun                                          pass    
 build-i386-rumprun                                           pass    
 test-xtf-amd64-amd64-1                                       pass    
 test-xtf-amd64-amd64-2                                       pass    
 test-xtf-amd64-amd64-3                                       pass    
 test-xtf-amd64-amd64-4                                       pass    
 test-xtf-amd64-amd64-5                                       pass    
 test-amd64-amd64-xl                                          pass    
 test-armhf-armhf-xl                                          pass    
 test-amd64-i386-xl                                           pass    
 test-amd64-amd64-xl-qemut-debianhvm-amd64-xsm                pass    
 test-amd64-i386-xl-qemut-debianhvm-amd64-xsm                 pass    
 test-amd64-amd64-libvirt-qemuu-debianhvm-amd64-xsm           pass    
 test-amd64-i386-libvirt-qemuu-debianhvm-amd64-xsm            pass    
 test-amd64-amd64-xl-qemuu-debianhvm-amd64-xsm                pass    
 test-amd64-i386-xl-qemuu-debianhvm-amd64-xsm                 pass    
 test-amd64-amd64-xl-qemut-stubdom-debianhvm-amd64-xsm        pass    
 test-amd64-i386-xl-qemut-stubdom-debianhvm-amd64-xsm         pass    
 test-amd64-amd64-libvirt-xsm                                 pass    
 test-armhf-armhf-libvirt-xsm                                 pass    
 test-amd64-i386-libvirt-xsm                                  pass    
 test-amd64-amd64-xl-xsm                                      pass    
 test-armhf-armhf-xl-xsm                                      pass    
 test-amd64-i386-xl-xsm                                       pass    
 test-amd64-amd64-qemuu-nested-amd                            fail    
 test-amd64-amd64-xl-pvh-amd                                  fail    
 test-amd64-i386-qemut-rhel6hvm-amd                           pass    
 test-amd64-i386-qemuu-rhel6hvm-amd                           pass    
 test-amd64-amd64-xl-qemut-debianhvm-amd64                    pass    
 test-amd64-i386-xl-qemut-debianhvm-amd64                     pass    
 test-amd64-amd64-xl-qemuu-debianhvm-amd64                    pass    
 test-amd64-i386-xl-qemuu-debianhvm-amd64                     fail    
 test-amd64-i386-freebsd10-amd64                              pass    
 test-amd64-amd64-xl-qemuu-ovmf-amd64                         pass    
 test-amd64-i386-xl-qemuu-ovmf-amd64                          pass    
 test-amd64-amd64-rumprun-amd64                               pass    
 test-amd64-amd64-xl-qemut-win7-amd64                         fail    
 test-amd64-i386-xl-qemut-win7-amd64                          fail    
 test-amd64-amd64-xl-qemuu-win7-amd64                         fail    
 test-amd64-i386-xl-qemuu-win7-amd64                          fail    
 test-armhf-armhf-xl-arndale                                  pass    
 test-amd64-amd64-xl-credit2                                  pass    
 test-armhf-armhf-xl-credit2                                  pass    
 test-armhf-armhf-xl-cubietruck                               pass    
 test-amd64-i386-freebsd10-i386                               pass    
 test-amd64-i386-rumprun-i386                                 pass    
 test-amd64-amd64-qemuu-nested-intel                          pass    
 test-amd64-amd64-xl-pvh-intel                                fail    
 test-amd64-i386-qemut-rhel6hvm-intel                         pass    
 test-amd64-i386-qemuu-rhel6hvm-intel                         pass    
 test-amd64-amd64-libvirt                                     pass    
 test-armhf-armhf-libvirt                                     pass    
 test-amd64-i386-libvirt                                      pass    
 test-amd64-amd64-migrupgrade                                 pass    
 test-amd64-i386-migrupgrade                                  pass    
 test-amd64-amd64-xl-multivcpu                                pass    
 test-armhf-armhf-xl-multivcpu                                pass    
 test-amd64-amd64-pair                                        pass    
 test-amd64-i386-pair                                         pass    
 test-amd64-amd64-libvirt-pair                                pass    
 test-amd64-i386-libvirt-pair                                 pass    
 test-amd64-amd64-amd64-pvgrub                                pass    
 test-amd64-amd64-i386-pvgrub                                 pass    
 test-amd64-amd64-pygrub                                      pass    
 test-armhf-armhf-libvirt-qcow2                               pass    
 test-amd64-amd64-xl-qcow2                                    pass    
 test-armhf-armhf-libvirt-raw                                 pass    
 test-amd64-i386-xl-raw                                       pass    
 test-amd64-amd64-xl-rtds                                     fail    
 test-armhf-armhf-xl-rtds                                     fail    
 test-amd64-i386-xl-qemut-winxpsp3-vcpus1                     pass    
 test-amd64-i386-xl-qemuu-winxpsp3-vcpus1                     pass    
 test-amd64-amd64-libvirt-vhd                                 pass    
 test-armhf-armhf-xl-vhd                                      pass    
 test-amd64-amd64-xl-qemut-winxpsp3                           pass    
 test-amd64-i386-xl-qemut-winxpsp3                            pass    
 test-amd64-amd64-xl-qemuu-winxpsp3                           pass    
 test-amd64-i386-xl-qemuu-winxpsp3                            pass    


------------------------------------------------------------
sg-report-flight on osstest.test-lab.xenproject.org
logs: /home/logs/logs
images: /home/logs/images

Logs, config files, etc. are available at
    http://logs.test-lab.xenproject.org/osstest/logs

Explanation of these reports, and of osstest in general, is at
    http://xenbits.xen.org/gitweb/?p=osstest.git;a=blob;f=README.email;hb=master
    http://xenbits.xen.org/gitweb/?p=osstest.git;a=blob;f=README;hb=master

Test harness code can be found at
    http://xenbits.xen.org/gitweb?p=osstest.git;a=summary


Not pushing.

(No revision log; it would be 528 lines long.)

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [xen-unstable test] 104131: regressions - FAIL

[Andrew Cooper]

On 12/01/2017 06:46, osstest service owner wrote:
> flight 104131 xen-unstable real [real]
> http://logs.test-lab.xenproject.org/osstest/logs/104131/
>
> Regressions :-(
>
> Tests which did not succeed and are blocking,
> including tests which could not be run:
>  test-amd64-i386-xl-qemuu-debianhvm-amd64 16 guest-stop   fail REGR. vs. 104119

Jan 12 01:25:17.397607 (XEN) Assertion 'intack.vector >= pt_vector' failed at intr.c:321
Jan 12 01:25:37.133596 (XEN) ----[ Xen-4.9-unstable  x86_64  debug=y   Not tainted ]----
Jan 12 01:25:37.141577 (XEN) CPU:    14
Jan 12 01:25:37.141607 (XEN) RIP:    e008:[<ffff82d0801ef7fc>] vmx_intr_assist+0x35e/0x51d
Jan 12 01:25:37.149617 (XEN) RFLAGS: 0000000000010202   CONTEXT: hypervisor (d15v0)
Jan 12 01:25:37.149655 (XEN) rax: 0000000000000038   rbx: ffff830079e1e000   rcx: 0000000000000030
Jan 12 01:25:37.157582 (XEN) rdx: 0000000000000000   rsi: 0000000000000030   rdi: ffff830079e1e000
Jan 12 01:25:37.165584 (XEN) rbp: ffff83047de2ff08   rsp: ffff83047de2fea8   r8:  ffff82c00022f000
Jan 12 01:25:37.173579 (XEN) r9:  ffff8301b63ede80   r10: ffff830176386560   r11: 000001955ee79bd0
Jan 12 01:25:37.181582 (XEN) r12: 0000000000003002   r13: 0000000000003002   r14: 0000000000000030
Jan 12 01:25:37.189584 (XEN) r15: ffff83023fec2000   cr0: 0000000080050033   cr4: 00000000003526e0
Jan 12 01:25:37.197572 (XEN) cr3: 0000000232edb000   cr2: 0000000002487034
Jan 12 01:25:37.205569 (XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: e008
Jan 12 01:25:37.205606 (XEN) Xen code around <ffff82d0801ef7fc> (vmx_intr_assist+0x35e/0x51d):
Jan 12 01:25:37.213575 (XEN)  41 0f b6 f6 39 f0 7e 02 <0f> 0b 48 89 df e8 51 20 00 00 b8 10 08 00 00 0f
Jan 12 01:25:37.221561 (XEN) Xen stack trace from rsp=ffff83047de2fea8:
Jan 12 01:25:37.229600 (XEN)    ffff82d08031aa80 00000038ffffffff ffff83047de2ffff ffff83023fec2000
Jan 12 01:25:37.237594 (XEN)    ffff83047de2fef8 ffff82d080130cb6 ffff830079e1e000 ffff830079e1e000
Jan 12 01:25:37.245588 (XEN)    ffff83007bae2000 000000000000000e ffff830233117000 ffff83023fec2000
Jan 12 01:25:37.253594 (XEN)    ffff83047de2fdc0 ffff82d0801fdeb1 0000000000000004 00000000000000c2
Jan 12 01:25:37.261584 (XEN)    0000000000000020 0000000000000007 ffff8800e8d28000 ffffffff81add0a0
Jan 12 01:25:37.269607 (XEN)    0000000000000246 0000000000000000 ffff880142400008 0000000000000004
Jan 12 01:25:37.277580 (XEN)    0000000000000036 0000000000000000 00000000000003f8 00000000000003f8
Jan 12 01:25:37.285584 (XEN)    ffffffff81add0a0 0000beef0000beef ffffffff813899a4 000000bf0000beef
Jan 12 01:25:37.293567 (XEN)    0000000000000002 ffff880147c03e08 000000000000beef 1cec835356e5beef
Jan 12 01:25:37.293606 (XEN)    085d8b002674beef 01dcb38b000cbeef 8914458d3174beef 2444c7100000000e
Jan 12 01:25:37.301586 (XEN)    ffff830079e1e000 00000031bfc37600 00000000003526e0
Jan 12 01:25:37.309607 (XEN) Xen call trace:
Jan 12 01:25:37.309639 (XEN)    [<ffff82d0801ef7fc>] vmx_intr_assist+0x35e/0x51d
Jan 12 01:25:37.317591 (XEN)    [<ffff82d0801fdeb1>] vmx_asm_vmexit_handler+0x41/0x120
Jan 12 01:25:37.325598 (XEN) 
Jan 12 01:25:37.325624 (XEN) 
Jan 12 01:25:37.325647 (XEN) ****************************************
Jan 12 01:25:37.333653 (XEN) Panic on CPU 14:
Jan 12 01:25:37.333684 (XEN) Assertion 'intack.vector >= pt_vector' failed at intr.c:321
Jan 12 01:25:37.341571 (XEN) ****************************************
Jan 12 01:25:37.341603 (XEN) 
Jan 12 01:25:37.341626 (XEN) Reboot in five seconds...
Jan 12 01:25:37.349566 (XEN) Resetting with ACPI MEMORY or I/O RESET_REG.

This is caused by "x86/apicv: fix RTC periodic timer and apicv issue".  It is not a deterministic issue, as it appears to have survived a week of testing already, but there is clearly something still problematic with the code.

~Andrew


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [xen-unstable test] 104131: regressions - FAIL

[Xuquan (Quan Xu)]

On January 12, 2017 5:14 PM, Andrew Cooper wrote:
>On 12/01/2017 06:46, osstest service owner wrote:
>> flight 104131 xen-unstable real [real]
>> http://logs.test-lab.xenproject.org/osstest/logs/104131/
>>
>> Regressions :-(
>>
>> Tests which did not succeed and are blocking, including tests which
>> could not be run:
>>  test-amd64-i386-xl-qemuu-debianhvm-amd64 16 guest-stop   fail
>REGR. vs. 104119
>
>Jan 12 01:25:17.397607 (XEN) Assertion 'intack.vector >= pt_vector' failed at
>intr.c:321
>Jan 12 01:25:37.133596 (XEN) ----[ Xen-4.9-unstable  x86_64  debug=y
>Not tainted ]----
>Jan 12 01:25:37.141577 (XEN) CPU:    14
>Jan 12 01:25:37.141607 (XEN) RIP:    e008:[<ffff82d0801ef7fc>]
>vmx_intr_assist+0x35e/0x51d
>Jan 12 01:25:37.149617 (XEN) RFLAGS: 0000000000010202   CONTEXT:
>hypervisor (d15v0)
>Jan 12 01:25:37.149655 (XEN) rax: 0000000000000038   rbx:
>ffff830079e1e000   rcx: 0000000000000030
>Jan 12 01:25:37.157582 (XEN) rdx: 0000000000000000   rsi:
>0000000000000030   rdi: ffff830079e1e000
>Jan 12 01:25:37.165584 (XEN) rbp: ffff83047de2ff08   rsp: ffff83047de2fea8
>r8:  ffff82c00022f000
>Jan 12 01:25:37.173579 (XEN) r9:  ffff8301b63ede80   r10:
>ffff830176386560   r11: 000001955ee79bd0
>Jan 12 01:25:37.181582 (XEN) r12: 0000000000003002   r13:
>0000000000003002   r14: 0000000000000030
>Jan 12 01:25:37.189584 (XEN) r15: ffff83023fec2000   cr0:
>0000000080050033   cr4: 00000000003526e0
>Jan 12 01:25:37.197572 (XEN) cr3: 0000000232edb000   cr2:
>0000000002487034
>Jan 12 01:25:37.205569 (XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000
>ss: 0000   cs: e008
>Jan 12 01:25:37.205606 (XEN) Xen code around <ffff82d0801ef7fc>
>(vmx_intr_assist+0x35e/0x51d):
>Jan 12 01:25:37.213575 (XEN)  41 0f b6 f6 39 f0 7e 02 <0f> 0b 48 89 df e8 51
>20 00 00 b8 10 08 00 00 0f Jan 12 01:25:37.221561 (XEN) Xen stack trace
>from rsp=ffff83047de2fea8:
>Jan 12 01:25:37.229600 (XEN)    ffff82d08031aa80 00000038ffffffff
>ffff83047de2ffff ffff83023fec2000
>Jan 12 01:25:37.237594 (XEN)    ffff83047de2fef8 ffff82d080130cb6
>ffff830079e1e000 ffff830079e1e000
>Jan 12 01:25:37.245588 (XEN)    ffff83007bae2000 000000000000000e
>ffff830233117000 ffff83023fec2000
>Jan 12 01:25:37.253594 (XEN)    ffff83047de2fdc0 ffff82d0801fdeb1
>0000000000000004 00000000000000c2
>Jan 12 01:25:37.261584 (XEN)    0000000000000020 0000000000000007
>ffff8800e8d28000 ffffffff81add0a0
>Jan 12 01:25:37.269607 (XEN)    0000000000000246 0000000000000000
>ffff880142400008 0000000000000004
>Jan 12 01:25:37.277580 (XEN)    0000000000000036 0000000000000000
>00000000000003f8 00000000000003f8
>Jan 12 01:25:37.285584 (XEN)    ffffffff81add0a0 0000beef0000beef
>ffffffff813899a4 000000bf0000beef
>Jan 12 01:25:37.293567 (XEN)    0000000000000002 ffff880147c03e08
>000000000000beef 1cec835356e5beef
>Jan 12 01:25:37.293606 (XEN)    085d8b002674beef 01dcb38b000cbeef
>8914458d3174beef 2444c7100000000e
>Jan 12 01:25:37.301586 (XEN)    ffff830079e1e000 00000031bfc37600
>00000000003526e0
>Jan 12 01:25:37.309607 (XEN) Xen call trace:
>Jan 12 01:25:37.309639 (XEN)    [<ffff82d0801ef7fc>]
>vmx_intr_assist+0x35e/0x51d
>Jan 12 01:25:37.317591 (XEN)    [<ffff82d0801fdeb1>]
>vmx_asm_vmexit_handler+0x41/0x120
>Jan 12 01:25:37.325598 (XEN)
>Jan 12 01:25:37.325624 (XEN)
>Jan 12 01:25:37.325647 (XEN)
>****************************************
>Jan 12 01:25:37.333653 (XEN) Panic on CPU 14:
>Jan 12 01:25:37.333684 (XEN) Assertion 'intack.vector >= pt_vector' failed at
>intr.c:321 Jan 12 01:25:37.341571 (XEN)
>****************************************
>Jan 12 01:25:37.341603 (XEN)
>Jan 12 01:25:37.341626 (XEN) Reboot in five seconds...
>Jan 12 01:25:37.349566 (XEN) Resetting with ACPI MEMORY or I/O
>RESET_REG.
>
>This is caused by "x86/apicv: fix RTC periodic timer and apicv issue".  It is
>not a deterministic issue, as it appears to have survived a week of testing
>already, but there is clearly something still problematic with the code.
>


Andrew,
If you have, could you give more information? Such as the value of intack.vector / pt_vector..
I guess, the reason may be that the intack.vector is ' uint8_t ' and the pt_vector is 'int'..

Or there is a corner case that intack.vector is __not__ the highest priority vector..

Kevin / Jan,  any thoughts?

Quan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [xen-unstable test] 104131: regressions - FAIL

[Chao Gao]

According the code around the assert: 
movzbl %r14b, %esi  41 0f b6 f6 
cmp %esi, %eax      39 f0
jle ...             7e 02
ud2                 <0f> 0b 
mov %rbx, %rdi      48 89 df
callq ...           e8 51 20 00 00 
mov $0x810, %eax    b8 10 08 00 00 

so I think one is 0x38 %eax, the other is 0x30 %esi

On Thu, Jan 12, 2017 at 12:07:53PM +0000, Xuquan (Quan Xu) wrote:
>On January 12, 2017 5:14 PM, Andrew Cooper wrote:
>>On 12/01/2017 06:46, osstest service owner wrote:
>>> flight 104131 xen-unstable real [real]
>>> http://logs.test-lab.xenproject.org/osstest/logs/104131/
>>>
>>> Regressions :-(
>>>
>>> Tests which did not succeed and are blocking, including tests which
>>> could not be run:
>>>  test-amd64-i386-xl-qemuu-debianhvm-amd64 16 guest-stop   fail
>>REGR. vs. 104119
>>
>>Jan 12 01:25:17.397607 (XEN) Assertion 'intack.vector >= pt_vector' failed at
>>intr.c:321
>>Jan 12 01:25:37.133596 (XEN) ----[ Xen-4.9-unstable  x86_64  debug=y
>>Not tainted ]----
>>Jan 12 01:25:37.141577 (XEN) CPU:    14
>>Jan 12 01:25:37.141607 (XEN) RIP:    e008:[<ffff82d0801ef7fc>]
>>vmx_intr_assist+0x35e/0x51d
>>Jan 12 01:25:37.149617 (XEN) RFLAGS: 0000000000010202   CONTEXT:
>>hypervisor (d15v0)
>>Jan 12 01:25:37.149655 (XEN) rax: 0000000000000038   rbx:
>>ffff830079e1e000   rcx: 0000000000000030
>>Jan 12 01:25:37.157582 (XEN) rdx: 0000000000000000   rsi:
>>0000000000000030   rdi: ffff830079e1e000
>>Jan 12 01:25:37.165584 (XEN) rbp: ffff83047de2ff08   rsp: ffff83047de2fea8
>>r8:  ffff82c00022f000
>>Jan 12 01:25:37.173579 (XEN) r9:  ffff8301b63ede80   r10:
>>ffff830176386560   r11: 000001955ee79bd0
>>Jan 12 01:25:37.181582 (XEN) r12: 0000000000003002   r13:
>>0000000000003002   r14: 0000000000000030
>>Jan 12 01:25:37.189584 (XEN) r15: ffff83023fec2000   cr0:
>>0000000080050033   cr4: 00000000003526e0
>>Jan 12 01:25:37.197572 (XEN) cr3: 0000000232edb000   cr2:
>>0000000002487034
>>Jan 12 01:25:37.205569 (XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000
>>ss: 0000   cs: e008
>>Jan 12 01:25:37.205606 (XEN) Xen code around <ffff82d0801ef7fc>
>>(vmx_intr_assist+0x35e/0x51d):
>>Jan 12 01:25:37.213575 (XEN)  41 0f b6 f6 39 f0 7e 02 <0f> 0b 48 89 df e8 51
>>20 00 00 b8 10 08 00 00 0f Jan 12 01:25:37.221561 (XEN) Xen stack trace
>>from rsp=ffff83047de2fea8:
>>Jan 12 01:25:37.229600 (XEN)    ffff82d08031aa80 00000038ffffffff
>>ffff83047de2ffff ffff83023fec2000
>>Jan 12 01:25:37.237594 (XEN)    ffff83047de2fef8 ffff82d080130cb6
>>ffff830079e1e000 ffff830079e1e000
>>Jan 12 01:25:37.245588 (XEN)    ffff83007bae2000 000000000000000e
>>ffff830233117000 ffff83023fec2000
>>Jan 12 01:25:37.253594 (XEN)    ffff83047de2fdc0 ffff82d0801fdeb1
>>0000000000000004 00000000000000c2
>>Jan 12 01:25:37.261584 (XEN)    0000000000000020 0000000000000007
>>ffff8800e8d28000 ffffffff81add0a0
>>Jan 12 01:25:37.269607 (XEN)    0000000000000246 0000000000000000
>>ffff880142400008 0000000000000004
>>Jan 12 01:25:37.277580 (XEN)    0000000000000036 0000000000000000
>>00000000000003f8 00000000000003f8
>>Jan 12 01:25:37.285584 (XEN)    ffffffff81add0a0 0000beef0000beef
>>ffffffff813899a4 000000bf0000beef
>>Jan 12 01:25:37.293567 (XEN)    0000000000000002 ffff880147c03e08
>>000000000000beef 1cec835356e5beef
>>Jan 12 01:25:37.293606 (XEN)    085d8b002674beef 01dcb38b000cbeef
>>8914458d3174beef 2444c7100000000e
>>Jan 12 01:25:37.301586 (XEN)    ffff830079e1e000 00000031bfc37600
>>00000000003526e0
>>Jan 12 01:25:37.309607 (XEN) Xen call trace:
>>Jan 12 01:25:37.309639 (XEN)    [<ffff82d0801ef7fc>]
>>vmx_intr_assist+0x35e/0x51d
>>Jan 12 01:25:37.317591 (XEN)    [<ffff82d0801fdeb1>]
>>vmx_asm_vmexit_handler+0x41/0x120
>>Jan 12 01:25:37.325598 (XEN)
>>Jan 12 01:25:37.325624 (XEN)
>>Jan 12 01:25:37.325647 (XEN)
>>****************************************
>>Jan 12 01:25:37.333653 (XEN) Panic on CPU 14:
>>Jan 12 01:25:37.333684 (XEN) Assertion 'intack.vector >= pt_vector' failed at
>>intr.c:321 Jan 12 01:25:37.341571 (XEN)
>>****************************************
>>Jan 12 01:25:37.341603 (XEN)
>>Jan 12 01:25:37.341626 (XEN) Reboot in five seconds...
>>Jan 12 01:25:37.349566 (XEN) Resetting with ACPI MEMORY or I/O
>>RESET_REG.
>>
>>This is caused by "x86/apicv: fix RTC periodic timer and apicv issue".  It is
>>not a deterministic issue, as it appears to have survived a week of testing
>>already, but there is clearly something still problematic with the code.
>>
>
>
>Andrew,
>If you have, could you give more information? Such as the value of intack.vector / pt_vector..
>I guess, the reason may be that the intack.vector is ' uint8_t ' and the pt_vector is 'int'..
>
>Or there is a corner case that intack.vector is __not__ the highest priority vector..
>
>Kevin / Jan,  any thoughts?
>
>Quan
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [xen-unstable test] 104131: regressions - FAIL

[Andrew Cooper]

On 12/01/17 12:07, Xuquan (Quan Xu) wrote:
> On January 12, 2017 5:14 PM, Andrew Cooper wrote:
>> On 12/01/2017 06:46, osstest service owner wrote:
>>> flight 104131 xen-unstable real [real]
>>> http://logs.test-lab.xenproject.org/osstest/logs/104131/
>>>
>>> Regressions :-(
>>>
>>> Tests which did not succeed and are blocking, including tests which
>>> could not be run:
>>>  test-amd64-i386-xl-qemuu-debianhvm-amd64 16 guest-stop   fail
>> REGR. vs. 104119
>>
>> Jan 12 01:25:17.397607 (XEN) Assertion 'intack.vector >= pt_vector' failed at
>> intr.c:321
>> Jan 12 01:25:37.133596 (XEN) ----[ Xen-4.9-unstable  x86_64  debug=y
>> Not tainted ]----
>> Jan 12 01:25:37.141577 (XEN) CPU:    14
>> Jan 12 01:25:37.141607 (XEN) RIP:    e008:[<ffff82d0801ef7fc>]
>> vmx_intr_assist+0x35e/0x51d
>> Jan 12 01:25:37.149617 (XEN) RFLAGS: 0000000000010202   CONTEXT:
>> hypervisor (d15v0)
>> Jan 12 01:25:37.149655 (XEN) rax: 0000000000000038   rbx:
>> ffff830079e1e000   rcx: 0000000000000030
>> Jan 12 01:25:37.157582 (XEN) rdx: 0000000000000000   rsi:
>> 0000000000000030   rdi: ffff830079e1e000
>> Jan 12 01:25:37.165584 (XEN) rbp: ffff83047de2ff08   rsp: ffff83047de2fea8
>> r8:  ffff82c00022f000
>> Jan 12 01:25:37.173579 (XEN) r9:  ffff8301b63ede80   r10:
>> ffff830176386560   r11: 000001955ee79bd0
>> Jan 12 01:25:37.181582 (XEN) r12: 0000000000003002   r13:
>> 0000000000003002   r14: 0000000000000030
>> Jan 12 01:25:37.189584 (XEN) r15: ffff83023fec2000   cr0:
>> 0000000080050033   cr4: 00000000003526e0
>> Jan 12 01:25:37.197572 (XEN) cr3: 0000000232edb000   cr2:
>> 0000000002487034
>> Jan 12 01:25:37.205569 (XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000
>> ss: 0000   cs: e008
>> Jan 12 01:25:37.205606 (XEN) Xen code around <ffff82d0801ef7fc>
>> (vmx_intr_assist+0x35e/0x51d):
>> Jan 12 01:25:37.213575 (XEN)  41 0f b6 f6 39 f0 7e 02 <0f> 0b 48 89 df e8 51
>> 20 00 00 b8 10 08 00 00 0f Jan 12 01:25:37.221561 (XEN) Xen stack trace
> >from rsp=ffff83047de2fea8:
>> Jan 12 01:25:37.229600 (XEN)    ffff82d08031aa80 00000038ffffffff
>> ffff83047de2ffff ffff83023fec2000
>> Jan 12 01:25:37.237594 (XEN)    ffff83047de2fef8 ffff82d080130cb6
>> ffff830079e1e000 ffff830079e1e000
>> Jan 12 01:25:37.245588 (XEN)    ffff83007bae2000 000000000000000e
>> ffff830233117000 ffff83023fec2000
>> Jan 12 01:25:37.253594 (XEN)    ffff83047de2fdc0 ffff82d0801fdeb1
>> 0000000000000004 00000000000000c2
>> Jan 12 01:25:37.261584 (XEN)    0000000000000020 0000000000000007
>> ffff8800e8d28000 ffffffff81add0a0
>> Jan 12 01:25:37.269607 (XEN)    0000000000000246 0000000000000000
>> ffff880142400008 0000000000000004
>> Jan 12 01:25:37.277580 (XEN)    0000000000000036 0000000000000000
>> 00000000000003f8 00000000000003f8
>> Jan 12 01:25:37.285584 (XEN)    ffffffff81add0a0 0000beef0000beef
>> ffffffff813899a4 000000bf0000beef
>> Jan 12 01:25:37.293567 (XEN)    0000000000000002 ffff880147c03e08
>> 000000000000beef 1cec835356e5beef
>> Jan 12 01:25:37.293606 (XEN)    085d8b002674beef 01dcb38b000cbeef
>> 8914458d3174beef 2444c7100000000e
>> Jan 12 01:25:37.301586 (XEN)    ffff830079e1e000 00000031bfc37600
>> 00000000003526e0
>> Jan 12 01:25:37.309607 (XEN) Xen call trace:
>> Jan 12 01:25:37.309639 (XEN)    [<ffff82d0801ef7fc>]
>> vmx_intr_assist+0x35e/0x51d
>> Jan 12 01:25:37.317591 (XEN)    [<ffff82d0801fdeb1>]
>> vmx_asm_vmexit_handler+0x41/0x120
>> Jan 12 01:25:37.325598 (XEN)
>> Jan 12 01:25:37.325624 (XEN)
>> Jan 12 01:25:37.325647 (XEN)
>> ****************************************
>> Jan 12 01:25:37.333653 (XEN) Panic on CPU 14:
>> Jan 12 01:25:37.333684 (XEN) Assertion 'intack.vector >= pt_vector' failed at
>> intr.c:321 Jan 12 01:25:37.341571 (XEN)
>> ****************************************
>> Jan 12 01:25:37.341603 (XEN)
>> Jan 12 01:25:37.341626 (XEN) Reboot in five seconds...
>> Jan 12 01:25:37.349566 (XEN) Resetting with ACPI MEMORY or I/O
>> RESET_REG.
>>
>> This is caused by "x86/apicv: fix RTC periodic timer and apicv issue".  It is
>> not a deterministic issue, as it appears to have survived a week of testing
>> already, but there is clearly something still problematic with the code.
>>
>
> Andrew,
> If you have, could you give more information?

No further information sorry.  This was found by the automated test system.

Full logs are available from
http://logs.test-lab.xenproject.org/osstest/logs/104131/test-amd64-i386-xl-qemuu-debianhvm-amd64/
but I doubt any of them will help in diagnosing the issue any further.

> Such as the value of intack.vector / pt_vector..
> I guess, the reason may be that the intack.vector is ' uint8_t ' and the pt_vector is 'int'..
>
> Or there is a corner case that intack.vector is __not__ the highest priority vector..
>
> Kevin / Jan,  any thoughts?

It happened during domain shutdown.  It might be an edge case
interaction with qemu-raised interrupts via hypercall?

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [xen-unstable test] 104131: regressions - FAIL

[Jan Beulich]

>>> On 12.01.17 at 13:15, <andrew.cooper3@citrix.com> wrote:
> On 12/01/17 12:07, Xuquan (Quan Xu) wrote:
>> On January 12, 2017 5:14 PM, Andrew Cooper wrote:
>>> On 12/01/2017 06:46, osstest service owner wrote:
>>>> flight 104131 xen-unstable real [real]
>>>> http://logs.test-lab.xenproject.org/osstest/logs/104131/ 
>>>>
>>>> Regressions :-(
>>>>
>>>> Tests which did not succeed and are blocking, including tests which
>>>> could not be run:
>>>>  test-amd64-i386-xl-qemuu-debianhvm-amd64 16 guest-stop   fail
>>> REGR. vs. 104119
>>>
>>> Jan 12 01:25:17.397607 (XEN) Assertion 'intack.vector >= pt_vector' failed at
>>> intr.c:321
>>> Jan 12 01:25:37.133596 (XEN) ----[ Xen-4.9-unstable  x86_64  debug=y
>>> Not tainted ]----
>>> Jan 12 01:25:37.141577 (XEN) CPU:    14
>>> Jan 12 01:25:37.141607 (XEN) RIP:    e008:[<ffff82d0801ef7fc>]
>>> vmx_intr_assist+0x35e/0x51d
>>> Jan 12 01:25:37.149617 (XEN) RFLAGS: 0000000000010202   CONTEXT:
>>> hypervisor (d15v0)
>>> Jan 12 01:25:37.149655 (XEN) rax: 0000000000000038   rbx:
>>> ffff830079e1e000   rcx: 0000000000000030
>>> Jan 12 01:25:37.157582 (XEN) rdx: 0000000000000000   rsi:
>>> 0000000000000030   rdi: ffff830079e1e000
>>> Jan 12 01:25:37.165584 (XEN) rbp: ffff83047de2ff08   rsp: ffff83047de2fea8
>>> r8:  ffff82c00022f000
>>> Jan 12 01:25:37.173579 (XEN) r9:  ffff8301b63ede80   r10:
>>> ffff830176386560   r11: 000001955ee79bd0
>>> Jan 12 01:25:37.181582 (XEN) r12: 0000000000003002   r13:
>>> 0000000000003002   r14: 0000000000000030
>>> Jan 12 01:25:37.189584 (XEN) r15: ffff83023fec2000   cr0:
>>> 0000000080050033   cr4: 00000000003526e0
>>> Jan 12 01:25:37.197572 (XEN) cr3: 0000000232edb000   cr2:
>>> 0000000002487034
>>> Jan 12 01:25:37.205569 (XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000
>>> ss: 0000   cs: e008
>>> Jan 12 01:25:37.205606 (XEN) Xen code around <ffff82d0801ef7fc>
>>> (vmx_intr_assist+0x35e/0x51d):
>>> Jan 12 01:25:37.213575 (XEN)  41 0f b6 f6 39 f0 7e 02 <0f> 0b 48 89 df e8 51
>>> 20 00 00 b8 10 08 00 00 0f Jan 12 01:25:37.221561 (XEN) Xen stack trace
>> >from rsp=ffff83047de2fea8:
>>> Jan 12 01:25:37.229600 (XEN)    ffff82d08031aa80 00000038ffffffff
>>> ffff83047de2ffff ffff83023fec2000
>>> Jan 12 01:25:37.237594 (XEN)    ffff83047de2fef8 ffff82d080130cb6
>>> ffff830079e1e000 ffff830079e1e000
>>> Jan 12 01:25:37.245588 (XEN)    ffff83007bae2000 000000000000000e
>>> ffff830233117000 ffff83023fec2000
>>> Jan 12 01:25:37.253594 (XEN)    ffff83047de2fdc0 ffff82d0801fdeb1
>>> 0000000000000004 00000000000000c2
>>> Jan 12 01:25:37.261584 (XEN)    0000000000000020 0000000000000007
>>> ffff8800e8d28000 ffffffff81add0a0
>>> Jan 12 01:25:37.269607 (XEN)    0000000000000246 0000000000000000
>>> ffff880142400008 0000000000000004
>>> Jan 12 01:25:37.277580 (XEN)    0000000000000036 0000000000000000
>>> 00000000000003f8 00000000000003f8
>>> Jan 12 01:25:37.285584 (XEN)    ffffffff81add0a0 0000beef0000beef
>>> ffffffff813899a4 000000bf0000beef
>>> Jan 12 01:25:37.293567 (XEN)    0000000000000002 ffff880147c03e08
>>> 000000000000beef 1cec835356e5beef
>>> Jan 12 01:25:37.293606 (XEN)    085d8b002674beef 01dcb38b000cbeef
>>> 8914458d3174beef 2444c7100000000e
>>> Jan 12 01:25:37.301586 (XEN)    ffff830079e1e000 00000031bfc37600
>>> 00000000003526e0
>>> Jan 12 01:25:37.309607 (XEN) Xen call trace:
>>> Jan 12 01:25:37.309639 (XEN)    [<ffff82d0801ef7fc>]
>>> vmx_intr_assist+0x35e/0x51d
>>> Jan 12 01:25:37.317591 (XEN)    [<ffff82d0801fdeb1>]
>>> vmx_asm_vmexit_handler+0x41/0x120
>>> Jan 12 01:25:37.325598 (XEN)
>>> Jan 12 01:25:37.325624 (XEN)
>>> Jan 12 01:25:37.325647 (XEN)
>>> ****************************************
>>> Jan 12 01:25:37.333653 (XEN) Panic on CPU 14:
>>> Jan 12 01:25:37.333684 (XEN) Assertion 'intack.vector >= pt_vector' failed at
>>> intr.c:321 Jan 12 01:25:37.341571 (XEN)
>>> ****************************************
>>> Jan 12 01:25:37.341603 (XEN)
>>> Jan 12 01:25:37.341626 (XEN) Reboot in five seconds...
>>> Jan 12 01:25:37.349566 (XEN) Resetting with ACPI MEMORY or I/O
>>> RESET_REG.
>>>
>>> This is caused by "x86/apicv: fix RTC periodic timer and apicv issue".  It is
>>> not a deterministic issue, as it appears to have survived a week of testing
>>> already, but there is clearly something still problematic with the code.
>>>
>>
>> Andrew,
>> If you have, could you give more information?
> 
> No further information sorry.  This was found by the automated test system.

But some can be gathered:

> Full logs are available from
> http://logs.test-lab.xenproject.org/osstest/logs/104131/test-amd64-i386-xl-q 
> emuu-debianhvm-amd64/
> but I doubt any of them will help in diagnosing the issue any further.
> 
>> Such as the value of intack.vector / pt_vector..

At leastb one of the two values is likely to live in a register, and
hence its value would be available in the dump. Just takes looking
at the disassembly.

>> I guess, the reason may be that the intack.vector is ' uint8_t ' and the pt_vector is 'int'..

That would be odd.

>> Or there is a corner case that intack.vector is __not__ the highest priority vector..

That's what I'm afraid of, and why I had asked to add the ASSERT().

Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [xen-unstable test] 104131: regressions - FAIL

[Tian, Kevin]

> From: Jan Beulich [mailto:JBeulich@suse.com]
> Sent: Thursday, January 12, 2017 8:26 PM
> 
> >>> On 12.01.17 at 13:15, <andrew.cooper3@citrix.com> wrote:
> > On 12/01/17 12:07, Xuquan (Quan Xu) wrote:
> >> On January 12, 2017 5:14 PM, Andrew Cooper wrote:
> >>> On 12/01/2017 06:46, osstest service owner wrote:
> >>>> flight 104131 xen-unstable real [real]
> >>>> http://logs.test-lab.xenproject.org/osstest/logs/104131/
> >>>>
> >>>> Regressions :-(
> >>>>
> >>>> Tests which did not succeed and are blocking, including tests which
> >>>> could not be run:
> >>>>  test-amd64-i386-xl-qemuu-debianhvm-amd64 16 guest-stop   fail
> >>> REGR. vs. 104119
> >>>
> >>> Jan 12 01:25:17.397607 (XEN) Assertion 'intack.vector >= pt_vector' failed at
> >>> intr.c:321
> >>> Jan 12 01:25:37.133596 (XEN) ----[ Xen-4.9-unstable  x86_64  debug=y
> >>> Not tainted ]----
> >>> Jan 12 01:25:37.141577 (XEN) CPU:    14
> >>> Jan 12 01:25:37.141607 (XEN) RIP:    e008:[<ffff82d0801ef7fc>]
> >>> vmx_intr_assist+0x35e/0x51d
> >>> Jan 12 01:25:37.149617 (XEN) RFLAGS: 0000000000010202   CONTEXT:
> >>> hypervisor (d15v0)
> >>> Jan 12 01:25:37.149655 (XEN) rax: 0000000000000038   rbx:
> >>> ffff830079e1e000   rcx: 0000000000000030
> >>> Jan 12 01:25:37.157582 (XEN) rdx: 0000000000000000   rsi:
> >>> 0000000000000030   rdi: ffff830079e1e000
> >>> Jan 12 01:25:37.165584 (XEN) rbp: ffff83047de2ff08   rsp: ffff83047de2fea8
> >>> r8:  ffff82c00022f000
> >>> Jan 12 01:25:37.173579 (XEN) r9:  ffff8301b63ede80   r10:
> >>> ffff830176386560   r11: 000001955ee79bd0
> >>> Jan 12 01:25:37.181582 (XEN) r12: 0000000000003002   r13:
> >>> 0000000000003002   r14: 0000000000000030
> >>> Jan 12 01:25:37.189584 (XEN) r15: ffff83023fec2000   cr0:
> >>> 0000000080050033   cr4: 00000000003526e0
> >>> Jan 12 01:25:37.197572 (XEN) cr3: 0000000232edb000   cr2:
> >>> 0000000002487034
> >>> Jan 12 01:25:37.205569 (XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000
> >>> ss: 0000   cs: e008
> >>> Jan 12 01:25:37.205606 (XEN) Xen code around <ffff82d0801ef7fc>
> >>> (vmx_intr_assist+0x35e/0x51d):
> >>> Jan 12 01:25:37.213575 (XEN)  41 0f b6 f6 39 f0 7e 02 <0f> 0b 48 89 df e8 51
> >>> 20 00 00 b8 10 08 00 00 0f Jan 12 01:25:37.221561 (XEN) Xen stack trace
> >> >from rsp=ffff83047de2fea8:
> >>> Jan 12 01:25:37.229600 (XEN)    ffff82d08031aa80 00000038ffffffff
> >>> ffff83047de2ffff ffff83023fec2000
> >>> Jan 12 01:25:37.237594 (XEN)    ffff83047de2fef8 ffff82d080130cb6
> >>> ffff830079e1e000 ffff830079e1e000
> >>> Jan 12 01:25:37.245588 (XEN)    ffff83007bae2000 000000000000000e
> >>> ffff830233117000 ffff83023fec2000
> >>> Jan 12 01:25:37.253594 (XEN)    ffff83047de2fdc0 ffff82d0801fdeb1
> >>> 0000000000000004 00000000000000c2
> >>> Jan 12 01:25:37.261584 (XEN)    0000000000000020 0000000000000007
> >>> ffff8800e8d28000 ffffffff81add0a0
> >>> Jan 12 01:25:37.269607 (XEN)    0000000000000246 0000000000000000
> >>> ffff880142400008 0000000000000004
> >>> Jan 12 01:25:37.277580 (XEN)    0000000000000036 0000000000000000
> >>> 00000000000003f8 00000000000003f8
> >>> Jan 12 01:25:37.285584 (XEN)    ffffffff81add0a0 0000beef0000beef
> >>> ffffffff813899a4 000000bf0000beef
> >>> Jan 12 01:25:37.293567 (XEN)    0000000000000002 ffff880147c03e08
> >>> 000000000000beef 1cec835356e5beef
> >>> Jan 12 01:25:37.293606 (XEN)    085d8b002674beef 01dcb38b000cbeef
> >>> 8914458d3174beef 2444c7100000000e
> >>> Jan 12 01:25:37.301586 (XEN)    ffff830079e1e000 00000031bfc37600
> >>> 00000000003526e0
> >>> Jan 12 01:25:37.309607 (XEN) Xen call trace:
> >>> Jan 12 01:25:37.309639 (XEN)    [<ffff82d0801ef7fc>]
> >>> vmx_intr_assist+0x35e/0x51d
> >>> Jan 12 01:25:37.317591 (XEN)    [<ffff82d0801fdeb1>]
> >>> vmx_asm_vmexit_handler+0x41/0x120
> >>> Jan 12 01:25:37.325598 (XEN)
> >>> Jan 12 01:25:37.325624 (XEN)
> >>> Jan 12 01:25:37.325647 (XEN)
> >>> ****************************************
> >>> Jan 12 01:25:37.333653 (XEN) Panic on CPU 14:
> >>> Jan 12 01:25:37.333684 (XEN) Assertion 'intack.vector >= pt_vector' failed at
> >>> intr.c:321 Jan 12 01:25:37.341571 (XEN)
> >>> ****************************************
> >>> Jan 12 01:25:37.341603 (XEN)
> >>> Jan 12 01:25:37.341626 (XEN) Reboot in five seconds...
> >>> Jan 12 01:25:37.349566 (XEN) Resetting with ACPI MEMORY or I/O
> >>> RESET_REG.
> >>>
> >>> This is caused by "x86/apicv: fix RTC periodic timer and apicv issue".  It is
> >>> not a deterministic issue, as it appears to have survived a week of testing
> >>> already, but there is clearly something still problematic with the code.
> >>>
> >>
> >> Andrew,
> >> If you have, could you give more information?
> >
> > No further information sorry.  This was found by the automated test system.
> 
> But some can be gathered:
> 
> > Full logs are available from
> > http://logs.test-lab.xenproject.org/osstest/logs/104131/test-amd64-i386-xl-q
> > emuu-debianhvm-amd64/
> > but I doubt any of them will help in diagnosing the issue any further.
> >
> >> Such as the value of intack.vector / pt_vector..
> 
> At leastb one of the two values is likely to live in a register, and
> hence its value would be available in the dump. Just takes looking
> at the disassembly.
> 
> >> I guess, the reason may be that the intack.vector is ' uint8_t ' and the pt_vector is 'int'..
> 
> That would be odd.
> 
> >> Or there is a corner case that intack.vector is __not__ the highest priority vector..
> 
> That's what I'm afraid of, and why I had asked to add the ASSERT().
> 

I cannot come up a valid reason for such situation (intack.vector is 0x30 
while pt_vector is 0x38 from Chao's data). pt_update_irq is invoked before
checking highest pending IRRs so pt_vector should be honored anyway. 
One possible reason is that being some reason pt_vector is not in vIRR at 
that point (due to some bug in the path from PIR to vIRR). However I 
didn't catch such bug simply by looking at code. We need reproduce this 
problem in developer side to find out actual reason. Andrew it'd be helpful
if you can help Quan/Chao to find out more test environment info.

One thing noted though. The original patch from Quan is actually orthogonal
to this ASSERT. Regardless of whether intack.vector is larger or smaller
than pt_vector, we always require the trick as long as pt_vector is not the
one being currently programmed to RVI. Then do we want to revert the whole
commit until the problem is finally fixed, or OK to just remove ASSERT 
(or replace with WARN_ON with more debug info) to unblock test system
before the fix is ready?

Thanks
Kevin

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [xen-unstable test] 104131: regressions - FAIL

[Xuquan (Quan Xu)]

On January 16, 2017 1:26 PM, Tian, Kevin wrote:
>> From: Jan Beulich [mailto:JBeulich@suse.com]
>> Sent: Thursday, January 12, 2017 8:26 PM
>>
>> >>> On 12.01.17 at 13:15, <andrew.cooper3@citrix.com> wrote:
>> > On 12/01/17 12:07, Xuquan (Quan Xu) wrote:
>> >> On January 12, 2017 5:14 PM, Andrew Cooper wrote:
>> >>> On 12/01/2017 06:46, osstest service owner wrote:
>> >>>> flight 104131 xen-unstable real [real]
>> >>>> http://logs.test-lab.xenproject.org/osstest/logs/104131/
>> >>>>
>> >>>> Regressions :-(
>> >>>>
>> >>>> Tests which did not succeed and are blocking, including tests
>> >>>> which could not be run:
>> >>>>  test-amd64-i386-xl-qemuu-debianhvm-amd64 16 guest-stop   fail
>> >>> REGR. vs. 104119
>> >>>
>> >>> Jan 12 01:25:17.397607 (XEN) Assertion 'intack.vector >=
>> >>> pt_vector' failed at
>> >>> intr.c:321
>> >>> Jan 12 01:25:37.133596 (XEN) ----[ Xen-4.9-unstable  x86_64
>> >>> debug=y Not tainted ]----
>> >>> Jan 12 01:25:37.141577 (XEN) CPU:    14
>> >>> Jan 12 01:25:37.141607 (XEN) RIP:    e008:[<ffff82d0801ef7fc>]
>> >>> vmx_intr_assist+0x35e/0x51d
>> >>> Jan 12 01:25:37.149617 (XEN) RFLAGS: 0000000000010202
>CONTEXT:
>> >>> hypervisor (d15v0)
>> >>> Jan 12 01:25:37.149655 (XEN) rax: 0000000000000038   rbx:
>> >>> ffff830079e1e000   rcx: 0000000000000030
>> >>> Jan 12 01:25:37.157582 (XEN) rdx: 0000000000000000   rsi:
>> >>> 0000000000000030   rdi: ffff830079e1e000
>> >>> Jan 12 01:25:37.165584 (XEN) rbp: ffff83047de2ff08   rsp:
>ffff83047de2fea8
>> >>> r8:  ffff82c00022f000
>> >>> Jan 12 01:25:37.173579 (XEN) r9:  ffff8301b63ede80   r10:
>> >>> ffff830176386560   r11: 000001955ee79bd0
>> >>> Jan 12 01:25:37.181582 (XEN) r12: 0000000000003002   r13:
>> >>> 0000000000003002   r14: 0000000000000030
>> >>> Jan 12 01:25:37.189584 (XEN) r15: ffff83023fec2000   cr0:
>> >>> 0000000080050033   cr4: 00000000003526e0
>> >>> Jan 12 01:25:37.197572 (XEN) cr3: 0000000232edb000   cr2:
>> >>> 0000000002487034
>> >>> Jan 12 01:25:37.205569 (XEN) ds: 0000   es: 0000   fs: 0000   gs:
>0000
>> >>> ss: 0000   cs: e008
>> >>> Jan 12 01:25:37.205606 (XEN) Xen code around <ffff82d0801ef7fc>
>> >>> (vmx_intr_assist+0x35e/0x51d):
>> >>> Jan 12 01:25:37.213575 (XEN)  41 0f b6 f6 39 f0 7e 02 <0f> 0b 48
>> >>> 89 df e8 51
>> >>> 20 00 00 b8 10 08 00 00 0f Jan 12 01:25:37.221561 (XEN) Xen stack
>> >>> trace
>> >> >from rsp=ffff83047de2fea8:
>> >>> Jan 12 01:25:37.229600 (XEN)    ffff82d08031aa80 00000038ffffffff
>> >>> ffff83047de2ffff ffff83023fec2000
>> >>> Jan 12 01:25:37.237594 (XEN)    ffff83047de2fef8 ffff82d080130cb6
>> >>> ffff830079e1e000 ffff830079e1e000
>> >>> Jan 12 01:25:37.245588 (XEN)    ffff83007bae2000
>000000000000000e
>> >>> ffff830233117000 ffff83023fec2000
>> >>> Jan 12 01:25:37.253594 (XEN)    ffff83047de2fdc0 ffff82d0801fdeb1
>> >>> 0000000000000004 00000000000000c2
>> >>> Jan 12 01:25:37.261584 (XEN)    0000000000000020
>0000000000000007
>> >>> ffff8800e8d28000 ffffffff81add0a0
>> >>> Jan 12 01:25:37.269607 (XEN)    0000000000000246
>0000000000000000
>> >>> ffff880142400008 0000000000000004
>> >>> Jan 12 01:25:37.277580 (XEN)    0000000000000036
>0000000000000000
>> >>> 00000000000003f8 00000000000003f8
>> >>> Jan 12 01:25:37.285584 (XEN)    ffffffff81add0a0 0000beef0000beef
>> >>> ffffffff813899a4 000000bf0000beef
>> >>> Jan 12 01:25:37.293567 (XEN)    0000000000000002
>ffff880147c03e08
>> >>> 000000000000beef 1cec835356e5beef
>> >>> Jan 12 01:25:37.293606 (XEN)    085d8b002674beef
>01dcb38b000cbeef
>> >>> 8914458d3174beef 2444c7100000000e
>> >>> Jan 12 01:25:37.301586 (XEN)    ffff830079e1e000
>00000031bfc37600
>> >>> 00000000003526e0
>> >>> Jan 12 01:25:37.309607 (XEN) Xen call trace:
>> >>> Jan 12 01:25:37.309639 (XEN)    [<ffff82d0801ef7fc>]
>> >>> vmx_intr_assist+0x35e/0x51d
>> >>> Jan 12 01:25:37.317591 (XEN)    [<ffff82d0801fdeb1>]
>> >>> vmx_asm_vmexit_handler+0x41/0x120
>> >>> Jan 12 01:25:37.325598 (XEN)
>> >>> Jan 12 01:25:37.325624 (XEN)
>> >>> Jan 12 01:25:37.325647 (XEN)
>> >>> ****************************************
>> >>> Jan 12 01:25:37.333653 (XEN) Panic on CPU 14:
>> >>> Jan 12 01:25:37.333684 (XEN) Assertion 'intack.vector >=
>> >>> pt_vector' failed at
>> >>> intr.c:321 Jan 12 01:25:37.341571 (XEN)
>> >>> ****************************************
>> >>> Jan 12 01:25:37.341603 (XEN)
>> >>> Jan 12 01:25:37.341626 (XEN) Reboot in five seconds...
>> >>> Jan 12 01:25:37.349566 (XEN) Resetting with ACPI MEMORY or I/O
>> >>> RESET_REG.
>> >>>
>> >>> This is caused by "x86/apicv: fix RTC periodic timer and apicv
>> >>> issue".  It is not a deterministic issue, as it appears to have
>> >>> survived a week of testing already, but there is clearly something still
>problematic with the code.
>> >>>
>> >>
>> >> Andrew,
>> >> If you have, could you give more information?
>> >
>> > No further information sorry.  This was found by the automated test
>system.
>>
>> But some can be gathered:
>>
>> > Full logs are available from
>> > http://logs.test-lab.xenproject.org/osstest/logs/104131/test-amd64-i
>> > 386-xl-q
>> > emuu-debianhvm-amd64/
>> > but I doubt any of them will help in diagnosing the issue any further.
>> >
>> >> Such as the value of intack.vector / pt_vector..
>>
>> At leastb one of the two values is likely to live in a register, and
>> hence its value would be available in the dump. Just takes looking at
>> the disassembly.
>>
>> >> I guess, the reason may be that the intack.vector is ' uint8_t ' and the
>pt_vector is 'int'..
>>
>> That would be odd.
>>
>> >> Or there is a corner case that intack.vector is __not__ the highest
>priority vector..
>>
>> That's what I'm afraid of, and why I had asked to add the ASSERT().
>>
>
>I cannot come up a valid reason for such situation (intack.vector is 0x30
>while pt_vector is 0x38 from Chao's data). pt_update_irq is invoked before
>checking highest pending IRRs so pt_vector should be honored anyway.
>One possible reason is that being some reason pt_vector is not in vIRR at
>that point (due to some bug in the path from PIR to vIRR). However I didn't
>catch such bug simply by looking at code. We need reproduce this problem
>in developer side to find out actual reason. Andrew it'd be helpful if you
>can help Quan/Chao to find out more test environment info.
>

I'll continue to follow up this issue..
However I don't have enough CPU-v3 machine for test it(occupied by another project).. I hope Chao could build some test environment.. 


Quan 




>One thing noted though. The original patch from Quan is actually
>orthogonal to this ASSERT. Regardless of whether intack.vector is larger or
>smaller than pt_vector, we always require the trick as long as pt_vector is
>not the one being currently programmed to RVI. Then do we want to revert
>the whole commit until the problem is finally fixed, or OK to just remove
>ASSERT (or replace with WARN_ON with more debug info) to unblock test
>system before the fix is ready?
>
>Thanks
>Kevin

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [xen-unstable test] 104131: regressions - FAIL

[Chao Gao]

On Mon, Jan 16, 2017 at 06:27:23AM +0000, Xuquan (Quan Xu) wrote:
>On January 16, 2017 1:26 PM, Tian, Kevin wrote:
>>> From: Jan Beulich [mailto:JBeulich@suse.com]
>>> Sent: Thursday, January 12, 2017 8:26 PM
>>>
>>> >>> On 12.01.17 at 13:15, <andrew.cooper3@citrix.com> wrote:
>>> > On 12/01/17 12:07, Xuquan (Quan Xu) wrote:
>>> >> On January 12, 2017 5:14 PM, Andrew Cooper wrote:
>>> >>> On 12/01/2017 06:46, osstest service owner wrote:
>>> >>>> flight 104131 xen-unstable real [real]
>>> >>>> http://logs.test-lab.xenproject.org/osstest/logs/104131/
>>> >>>>
>>> >>>> Regressions :-(
>>> >>>>
>>> >>>> Tests which did not succeed and are blocking, including tests
>>> >>>> which could not be run:
>>> >>>>  test-amd64-i386-xl-qemuu-debianhvm-amd64 16 guest-stop   fail
>>> >>> REGR. vs. 104119
>>> >>>
>>> >>> Jan 12 01:25:17.397607 (XEN) Assertion 'intack.vector >=
>>> >>> pt_vector' failed at
>>> >>> intr.c:321
>>> >>> Jan 12 01:25:37.133596 (XEN) ----[ Xen-4.9-unstable  x86_64
>>> >>> debug=y Not tainted ]----
>>> >>> Jan 12 01:25:37.141577 (XEN) CPU:    14
>>> >>> Jan 12 01:25:37.141607 (XEN) RIP:    e008:[<ffff82d0801ef7fc>]
>>> >>> vmx_intr_assist+0x35e/0x51d
>>> >>> Jan 12 01:25:37.149617 (XEN) RFLAGS: 0000000000010202
>>CONTEXT:
>>> >>> hypervisor (d15v0)
>>> >>> Jan 12 01:25:37.149655 (XEN) rax: 0000000000000038   rbx:
>>> >>> ffff830079e1e000   rcx: 0000000000000030
>>> >>> Jan 12 01:25:37.157582 (XEN) rdx: 0000000000000000   rsi:
>>> >>> 0000000000000030   rdi: ffff830079e1e000
>>> >>> Jan 12 01:25:37.165584 (XEN) rbp: ffff83047de2ff08   rsp:
>>ffff83047de2fea8
>>> >>> r8:  ffff82c00022f000
>>> >>> Jan 12 01:25:37.173579 (XEN) r9:  ffff8301b63ede80   r10:
>>> >>> ffff830176386560   r11: 000001955ee79bd0
>>> >>> Jan 12 01:25:37.181582 (XEN) r12: 0000000000003002   r13:
>>> >>> 0000000000003002   r14: 0000000000000030
>>> >>> Jan 12 01:25:37.189584 (XEN) r15: ffff83023fec2000   cr0:
>>> >>> 0000000080050033   cr4: 00000000003526e0
>>> >>> Jan 12 01:25:37.197572 (XEN) cr3: 0000000232edb000   cr2:
>>> >>> 0000000002487034
>>> >>> Jan 12 01:25:37.205569 (XEN) ds: 0000   es: 0000   fs: 0000   gs:
>>0000
>>> >>> ss: 0000   cs: e008
>>> >>> Jan 12 01:25:37.205606 (XEN) Xen code around <ffff82d0801ef7fc>
>>> >>> (vmx_intr_assist+0x35e/0x51d):
>>> >>> Jan 12 01:25:37.213575 (XEN)  41 0f b6 f6 39 f0 7e 02 <0f> 0b 48
>>> >>> 89 df e8 51
>>> >>> 20 00 00 b8 10 08 00 00 0f Jan 12 01:25:37.221561 (XEN) Xen stack
>>> >>> trace
>>> >> >from rsp=ffff83047de2fea8:
>>> >>> Jan 12 01:25:37.229600 (XEN)    ffff82d08031aa80 00000038ffffffff
>>> >>> ffff83047de2ffff ffff83023fec2000
>>> >>> Jan 12 01:25:37.237594 (XEN)    ffff83047de2fef8 ffff82d080130cb6
>>> >>> ffff830079e1e000 ffff830079e1e000
>>> >>> Jan 12 01:25:37.245588 (XEN)    ffff83007bae2000
>>000000000000000e
>>> >>> ffff830233117000 ffff83023fec2000
>>> >>> Jan 12 01:25:37.253594 (XEN)    ffff83047de2fdc0 ffff82d0801fdeb1
>>> >>> 0000000000000004 00000000000000c2
>>> >>> Jan 12 01:25:37.261584 (XEN)    0000000000000020
>>0000000000000007
>>> >>> ffff8800e8d28000 ffffffff81add0a0
>>> >>> Jan 12 01:25:37.269607 (XEN)    0000000000000246
>>0000000000000000
>>> >>> ffff880142400008 0000000000000004
>>> >>> Jan 12 01:25:37.277580 (XEN)    0000000000000036
>>0000000000000000
>>> >>> 00000000000003f8 00000000000003f8
>>> >>> Jan 12 01:25:37.285584 (XEN)    ffffffff81add0a0 0000beef0000beef
>>> >>> ffffffff813899a4 000000bf0000beef
>>> >>> Jan 12 01:25:37.293567 (XEN)    0000000000000002
>>ffff880147c03e08
>>> >>> 000000000000beef 1cec835356e5beef
>>> >>> Jan 12 01:25:37.293606 (XEN)    085d8b002674beef
>>01dcb38b000cbeef
>>> >>> 8914458d3174beef 2444c7100000000e
>>> >>> Jan 12 01:25:37.301586 (XEN)    ffff830079e1e000
>>00000031bfc37600
>>> >>> 00000000003526e0
>>> >>> Jan 12 01:25:37.309607 (XEN) Xen call trace:
>>> >>> Jan 12 01:25:37.309639 (XEN)    [<ffff82d0801ef7fc>]
>>> >>> vmx_intr_assist+0x35e/0x51d
>>> >>> Jan 12 01:25:37.317591 (XEN)    [<ffff82d0801fdeb1>]
>>> >>> vmx_asm_vmexit_handler+0x41/0x120
>>> >>> Jan 12 01:25:37.325598 (XEN)
>>> >>> Jan 12 01:25:37.325624 (XEN)
>>> >>> Jan 12 01:25:37.325647 (XEN)
>>> >>> ****************************************
>>> >>> Jan 12 01:25:37.333653 (XEN) Panic on CPU 14:
>>> >>> Jan 12 01:25:37.333684 (XEN) Assertion 'intack.vector >=
>>> >>> pt_vector' failed at
>>> >>> intr.c:321 Jan 12 01:25:37.341571 (XEN)
>>> >>> ****************************************
>>> >>> Jan 12 01:25:37.341603 (XEN)
>>> >>> Jan 12 01:25:37.341626 (XEN) Reboot in five seconds...
>>> >>> Jan 12 01:25:37.349566 (XEN) Resetting with ACPI MEMORY or I/O
>>> >>> RESET_REG.
>>> >>>
>>> >>> This is caused by "x86/apicv: fix RTC periodic timer and apicv
>>> >>> issue".  It is not a deterministic issue, as it appears to have
>>> >>> survived a week of testing already, but there is clearly something still
>>problematic with the code.
>>> >>>
>>> >>
>>> >> Andrew,
>>> >> If you have, could you give more information?
>>> >
>>> > No further information sorry.  This was found by the automated test
>>system.
>>>
>>> But some can be gathered:
>>>
>>> > Full logs are available from
>>> > http://logs.test-lab.xenproject.org/osstest/logs/104131/test-amd64-i
>>> > 386-xl-q
>>> > emuu-debianhvm-amd64/
>>> > but I doubt any of them will help in diagnosing the issue any further.
>>> >
>>> >> Such as the value of intack.vector / pt_vector..
>>>
>>> At leastb one of the two values is likely to live in a register, and
>>> hence its value would be available in the dump. Just takes looking at
>>> the disassembly.
>>>
>>> >> I guess, the reason may be that the intack.vector is ' uint8_t ' and the
>>pt_vector is 'int'..
>>>
>>> That would be odd.
>>>
>>> >> Or there is a corner case that intack.vector is __not__ the highest
>>priority vector..
>>>
>>> That's what I'm afraid of, and why I had asked to add the ASSERT().
>>>
>>
>>I cannot come up a valid reason for such situation (intack.vector is 0x30
>>while pt_vector is 0x38 from Chao's data). pt_update_irq is invoked before
>>checking highest pending IRRs so pt_vector should be honored anyway.
>>One possible reason is that being some reason pt_vector is not in vIRR at
>>that point (due to some bug in the path from PIR to vIRR). However I didn't
>>catch such bug simply by looking at code. We need reproduce this problem
>>in developer side to find out actual reason. Andrew it'd be helpful if you
>>can help Quan/Chao to find out more test environment info.
>>
>
>I'll continue to follow up this issue..
>However I don't have enough CPU-v3 machine for test it(occupied by another project).. I hope Chao could build some test environment.. 
>
No problem. When you come up with ideas or find some clues, I can verify them for you.

thanks
Chao
>
>Quan 
>
>
>
>
>>One thing noted though. The original patch from Quan is actually
>>orthogonal to this ASSERT. Regardless of whether intack.vector is larger or
>>smaller than pt_vector, we always require the trick as long as pt_vector is
>>not the one being currently programmed to RVI. Then do we want to revert
>>the whole commit until the problem is finally fixed, or OK to just remove
>>ASSERT (or replace with WARN_ON with more debug info) to unblock test
>>system before the fix is ready?
>>
>>Thanks
>>Kevin

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [xen-unstable test] 104131: regressions - FAIL

[Jan Beulich]

>>> On 16.01.17 at 06:25, <kevin.tian@intel.com> wrote:
> One thing noted though. The original patch from Quan is actually orthogonal
> to this ASSERT. Regardless of whether intack.vector is larger or smaller
> than pt_vector, we always require the trick as long as pt_vector is not the
> one being currently programmed to RVI.

I don't think the ASSERT() addition is orthogonal: It exchanges
intack.vector for pt_vector in the invocation of
vmx_set_eoi_exit_bitmap(), and during discussion of the patch
there at least intermediately was max() of the two used instead.
It was - iirc - one of you who suggested that the use of max()
there is unnecessary, which the ASSERT() triggering has now
shown is wrong.

> Then do we want to revert the whole
> commit until the problem is finally fixed, or OK to just remove ASSERT 
> (or replace with WARN_ON with more debug info) to unblock test system
> before the fix is ready?

Well, as the VMX maintainer I think the proposal of whether to
revert or wait should really come from you.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [xen-unstable test] 104131: regressions - FAIL

[Tian, Kevin]

> From: Jan Beulich [mailto:JBeulich@suse.com]
> Sent: Monday, January 16, 2017 7:00 PM
> 
> >>> On 16.01.17 at 06:25, <kevin.tian@intel.com> wrote:
> > One thing noted though. The original patch from Quan is actually orthogonal
> > to this ASSERT. Regardless of whether intack.vector is larger or smaller
> > than pt_vector, we always require the trick as long as pt_vector is not the
> > one being currently programmed to RVI.
> 
> I don't think the ASSERT() addition is orthogonal: It exchanges
> intack.vector for pt_vector in the invocation of
> vmx_set_eoi_exit_bitmap(), and during discussion of the patch
> there at least intermediately was max() of the two used instead.
> It was - iirc - one of you who suggested that the use of max()
> there is unnecessary, which the ASSERT() triggering has now
> shown is wrong.

Attached was my earlier comment:

--
> >>> On 20.12.16 at 06:37, <kevin.tian@intel.com> wrote:
> >>  From: Xuquan (Quan Xu) [mailto:xuquan8@huawei.com]
> >> Sent: Friday, December 16, 2016 5:40 PM
> >> -        if (pt_vector != -1)
> >> -            vmx_set_eoi_exit_bitmap(v, pt_vector);
> >> +        if ( pt_vector != -1 ) {
> >> +            if ( intack.vector > pt_vector )
> >> +                vmx_set_eoi_exit_bitmap(v, intack.vector);
> >> +            else
> >> +                vmx_set_eoi_exit_bitmap(v, pt_vector);
> >> +        }
> >
> > Above can be simplified as one line change:
> > 	if ( pt_vector != -1 )
> > 		vmx_set_eoi_exit_bitmap(v, intack.vector);
> 
> Hmm, I don't understand. Did you mean to use max() here? Or
> else how is this an equivalent of the originally proposed code?
> 

Original code is not 100% correct. The purpose is to set EOI exit
bitmap for any vector which may block injection of pt_vector - 
give chance to recognize pt_vector in future intack and then do pt 
intr post. The simplified code achieves this effect same as original
code if intack.vector >= vector. I cannot come up a case why
intack.vector might be smaller than vector. If this case happens,
we still need enable exit bitmap for intack.vector instead of
pt_vector for said purpose while original code did it wrong.

Thanks
Kevin
--

Using intack.vector is always expected here regardless of the 
comparison result between intack.vector and pt_vector. The 
reason why I was OK adding an ASSERT was simply to test 
whether intack.vecor<pt_vector does happen which is
orthogonal to the fix itself.

> 
> > Then do we want to revert the whole
> > commit until the problem is finally fixed, or OK to just remove ASSERT
> > (or replace with WARN_ON with more debug info) to unblock test system
> > before the fix is ready?
> 
> Well, as the VMX maintainer I think the proposal of whether to
> revert or wait should really come from you.
> 
> Jan

Andrew, how long do you usually tolerate a failure case in osstest?
I'm not sure how long it may take for developer to reproduce this
situation. If it has blocking impact in your side, I'd suggest go 
replacing ASSERT with more informative warn info before final 
root-cause, if Quan&Chao cannot reproduce in a short time (say
1 or 2wk or so).

Thanks
Kevin

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [xen-unstable test] 104131: regressions - FAIL

[Jan Beulich]

>>> On 18.01.17 at 05:57, <kevin.tian@intel.com> wrote:
> Attached was my earlier comment:
> 
> --
>> >>> On 20.12.16 at 06:37, <kevin.tian@intel.com> wrote:
>> >>  From: Xuquan (Quan Xu) [mailto:xuquan8@huawei.com]
>> >> Sent: Friday, December 16, 2016 5:40 PM
>> >> -        if (pt_vector != -1)
>> >> -            vmx_set_eoi_exit_bitmap(v, pt_vector);
>> >> +        if ( pt_vector != -1 ) {
>> >> +            if ( intack.vector > pt_vector )
>> >> +                vmx_set_eoi_exit_bitmap(v, intack.vector);
>> >> +            else
>> >> +                vmx_set_eoi_exit_bitmap(v, pt_vector);
>> >> +        }
>> >
>> > Above can be simplified as one line change:
>> > 	if ( pt_vector != -1 )
>> > 		vmx_set_eoi_exit_bitmap(v, intack.vector);
>> 
>> Hmm, I don't understand. Did you mean to use max() here? Or
>> else how is this an equivalent of the originally proposed code?
>> 
> 
> Original code is not 100% correct. The purpose is to set EOI exit
> bitmap for any vector which may block injection of pt_vector - 
> give chance to recognize pt_vector in future intack and then do pt 
> intr post. The simplified code achieves this effect same as original
> code if intack.vector >= vector. I cannot come up a case why
> intack.vector might be smaller than vector. If this case happens,
> we still need enable exit bitmap for intack.vector instead of
> pt_vector for said purpose while original code did it wrong.
> 
> Thanks
> Kevin
> --
> 
> Using intack.vector is always expected here regardless of the 
> comparison result between intack.vector and pt_vector. The 
> reason why I was OK adding an ASSERT was simply to test 
> whether intack.vecor<pt_vector does happen which is
> orthogonal to the fix itself.

Well, a vector lower than pt_vector can't block delivery. Or wait:
Don't we need to consider vector classes here, i.e.

            ASSERT((intack.vector >> 4) >= (pt_vector >> 4));

?

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [xen-unstable test] 104131: regressions - FAIL

[Tian, Kevin]

> From: Jan Beulich [mailto:JBeulich@suse.com]
> Sent: Wednesday, January 18, 2017 5:38 PM
> 
> >>> On 18.01.17 at 05:57, <kevin.tian@intel.com> wrote:
> > Attached was my earlier comment:
> >
> > --
> >> >>> On 20.12.16 at 06:37, <kevin.tian@intel.com> wrote:
> >> >>  From: Xuquan (Quan Xu) [mailto:xuquan8@huawei.com]
> >> >> Sent: Friday, December 16, 2016 5:40 PM
> >> >> -        if (pt_vector != -1)
> >> >> -            vmx_set_eoi_exit_bitmap(v, pt_vector);
> >> >> +        if ( pt_vector != -1 ) {
> >> >> +            if ( intack.vector > pt_vector )
> >> >> +                vmx_set_eoi_exit_bitmap(v, intack.vector);
> >> >> +            else
> >> >> +                vmx_set_eoi_exit_bitmap(v, pt_vector);
> >> >> +        }
> >> >
> >> > Above can be simplified as one line change:
> >> > 	if ( pt_vector != -1 )
> >> > 		vmx_set_eoi_exit_bitmap(v, intack.vector);
> >>
> >> Hmm, I don't understand. Did you mean to use max() here? Or
> >> else how is this an equivalent of the originally proposed code?
> >>
> >
> > Original code is not 100% correct. The purpose is to set EOI exit
> > bitmap for any vector which may block injection of pt_vector -
> > give chance to recognize pt_vector in future intack and then do pt
> > intr post. The simplified code achieves this effect same as original
> > code if intack.vector >= vector. I cannot come up a case why
> > intack.vector might be smaller than vector. If this case happens,
> > we still need enable exit bitmap for intack.vector instead of
> > pt_vector for said purpose while original code did it wrong.
> >
> > Thanks
> > Kevin
> > --
> >
> > Using intack.vector is always expected here regardless of the
> > comparison result between intack.vector and pt_vector. The
> > reason why I was OK adding an ASSERT was simply to test
> > whether intack.vecor<pt_vector does happen which is
> > orthogonal to the fix itself.
> 
> Well, a vector lower than pt_vector can't block delivery. Or wait:

There are two points here:

a) We need enable EOI exit bitmap when pt_vector is blocked.

b) As you said, ideally a vector lower than pt_vecotr cannot block

The patch fixed a) and then added an ASSERT to verify b). Strictly
speaking they are separate issues.

> Don't we need to consider vector classes here, i.e.
> 
>             ASSERT((intack.vector >> 4) >= (pt_vector >> 4));
> 
> ?
> 

However it still doesn't explain why original ASSERT is triggered.
vlapic_find_highest_vector actually finds the highest vector, instead
of highest class...

static int vlapic_find_highest_vector(const void *bitmap)
{
    const uint32_t *word = bitmap;
    unsigned int word_offset = NR_VECTORS / 32;

    /* Work backwards through the bitmap (first 32-bit word in every four). */
    while ( (word_offset != 0) && (word[(--word_offset)*4] == 0) )
        continue;

    return (fls(word[word_offset*4]) - 1) + (word_offset * 32);
}

Thanks
Kevin

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [xen-unstable test] 104131: regressions - FAIL

[Jan Beulich]

>>> On 18.01.17 at 11:23, <kevin.tian@intel.com> wrote:
>>  From: Jan Beulich [mailto:JBeulich@suse.com]
>> Sent: Wednesday, January 18, 2017 5:38 PM
>> 
>> >>> On 18.01.17 at 05:57, <kevin.tian@intel.com> wrote:
>> > Attached was my earlier comment:
>> >
>> > --
>> >> >>> On 20.12.16 at 06:37, <kevin.tian@intel.com> wrote:
>> >> >>  From: Xuquan (Quan Xu) [mailto:xuquan8@huawei.com]
>> >> >> Sent: Friday, December 16, 2016 5:40 PM
>> >> >> -        if (pt_vector != -1)
>> >> >> -            vmx_set_eoi_exit_bitmap(v, pt_vector);
>> >> >> +        if ( pt_vector != -1 ) {
>> >> >> +            if ( intack.vector > pt_vector )
>> >> >> +                vmx_set_eoi_exit_bitmap(v, intack.vector);
>> >> >> +            else
>> >> >> +                vmx_set_eoi_exit_bitmap(v, pt_vector);
>> >> >> +        }
>> >> >
>> >> > Above can be simplified as one line change:
>> >> > 	if ( pt_vector != -1 )
>> >> > 		vmx_set_eoi_exit_bitmap(v, intack.vector);
>> >>
>> >> Hmm, I don't understand. Did you mean to use max() here? Or
>> >> else how is this an equivalent of the originally proposed code?
>> >>
>> >
>> > Original code is not 100% correct. The purpose is to set EOI exit
>> > bitmap for any vector which may block injection of pt_vector -
>> > give chance to recognize pt_vector in future intack and then do pt
>> > intr post. The simplified code achieves this effect same as original
>> > code if intack.vector >= vector. I cannot come up a case why
>> > intack.vector might be smaller than vector. If this case happens,
>> > we still need enable exit bitmap for intack.vector instead of
>> > pt_vector for said purpose while original code did it wrong.
>> >
>> > Thanks
>> > Kevin
>> > --
>> >
>> > Using intack.vector is always expected here regardless of the
>> > comparison result between intack.vector and pt_vector. The
>> > reason why I was OK adding an ASSERT was simply to test
>> > whether intack.vecor<pt_vector does happen which is
>> > orthogonal to the fix itself.
>> 
>> Well, a vector lower than pt_vector can't block delivery. Or wait:
> 
> There are two points here:
> 
> a) We need enable EOI exit bitmap when pt_vector is blocked.
> 
> b) As you said, ideally a vector lower than pt_vecotr cannot block
> 
> The patch fixed a) and then added an ASSERT to verify b). Strictly
> speaking they are separate issues.

Okay, I think I finally understand your argumentation here.

>> Don't we need to consider vector classes here, i.e.
>> 
>>             ASSERT((intack.vector >> 4) >= (pt_vector >> 4));
>> 
>> ?
>> 
> 
> However it still doesn't explain why original ASSERT is triggered.
> vlapic_find_highest_vector actually finds the highest vector, instead
> of highest class...
> 
> static int vlapic_find_highest_vector(const void *bitmap)
> {
>     const uint32_t *word = bitmap;
>     unsigned int word_offset = NR_VECTORS / 32;
> 
>     /* Work backwards through the bitmap (first 32-bit word in every four). */
>     while ( (word_offset != 0) && (word[(--word_offset)*4] == 0) )
>         continue;
> 
>     return (fls(word[word_offset*4]) - 1) + (word_offset * 32);
> }

Well, perhaps a PIR -> IRR syncing issue then (I in particular note
the early bailing from vmx_sync_pir_to_irr())? I guess we'd want
to see the entire IRR array (and perhaps also PI state) if the check
in the assertion fails.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [xen-unstable test] 104131: regressions - FAIL

[Tian, Kevin]

> From: Jan Beulich [mailto:JBeulich@suse.com]
> Sent: Friday, January 20, 2017 7:49 PM
> 
> >>> On 18.01.17 at 11:23, <kevin.tian@intel.com> wrote:
> >>  From: Jan Beulich [mailto:JBeulich@suse.com]
> >> Sent: Wednesday, January 18, 2017 5:38 PM
> >>
> >> >>> On 18.01.17 at 05:57, <kevin.tian@intel.com> wrote:
> >> > Attached was my earlier comment:
> >> >
> >> > --
> >> >> >>> On 20.12.16 at 06:37, <kevin.tian@intel.com> wrote:
> >> >> >>  From: Xuquan (Quan Xu) [mailto:xuquan8@huawei.com]
> >> >> >> Sent: Friday, December 16, 2016 5:40 PM
> >> >> >> -        if (pt_vector != -1)
> >> >> >> -            vmx_set_eoi_exit_bitmap(v, pt_vector);
> >> >> >> +        if ( pt_vector != -1 ) {
> >> >> >> +            if ( intack.vector > pt_vector )
> >> >> >> +                vmx_set_eoi_exit_bitmap(v, intack.vector);
> >> >> >> +            else
> >> >> >> +                vmx_set_eoi_exit_bitmap(v, pt_vector);
> >> >> >> +        }
> >> >> >
> >> >> > Above can be simplified as one line change:
> >> >> > 	if ( pt_vector != -1 )
> >> >> > 		vmx_set_eoi_exit_bitmap(v, intack.vector);
> >> >>
> >> >> Hmm, I don't understand. Did you mean to use max() here? Or
> >> >> else how is this an equivalent of the originally proposed code?
> >> >>
> >> >
> >> > Original code is not 100% correct. The purpose is to set EOI exit
> >> > bitmap for any vector which may block injection of pt_vector -
> >> > give chance to recognize pt_vector in future intack and then do pt
> >> > intr post. The simplified code achieves this effect same as original
> >> > code if intack.vector >= vector. I cannot come up a case why
> >> > intack.vector might be smaller than vector. If this case happens,
> >> > we still need enable exit bitmap for intack.vector instead of
> >> > pt_vector for said purpose while original code did it wrong.
> >> >
> >> > Thanks
> >> > Kevin
> >> > --
> >> >
> >> > Using intack.vector is always expected here regardless of the
> >> > comparison result between intack.vector and pt_vector. The
> >> > reason why I was OK adding an ASSERT was simply to test
> >> > whether intack.vecor<pt_vector does happen which is
> >> > orthogonal to the fix itself.
> >>
> >> Well, a vector lower than pt_vector can't block delivery. Or wait:
> >
> > There are two points here:
> >
> > a) We need enable EOI exit bitmap when pt_vector is blocked.
> >
> > b) As you said, ideally a vector lower than pt_vecotr cannot block
> >
> > The patch fixed a) and then added an ASSERT to verify b). Strictly
> > speaking they are separate issues.
> 
> Okay, I think I finally understand your argumentation here.
> 
> >> Don't we need to consider vector classes here, i.e.
> >>
> >>             ASSERT((intack.vector >> 4) >= (pt_vector >> 4));
> >>
> >> ?
> >>
> >
> > However it still doesn't explain why original ASSERT is triggered.
> > vlapic_find_highest_vector actually finds the highest vector, instead
> > of highest class...
> >
> > static int vlapic_find_highest_vector(const void *bitmap)
> > {
> >     const uint32_t *word = bitmap;
> >     unsigned int word_offset = NR_VECTORS / 32;
> >
> >     /* Work backwards through the bitmap (first 32-bit word in every four). */
> >     while ( (word_offset != 0) && (word[(--word_offset)*4] == 0) )
> >         continue;
> >
> >     return (fls(word[word_offset*4]) - 1) + (word_offset * 32);
> > }
> 
> Well, perhaps a PIR -> IRR syncing issue then (I in particular note
> the early bailing from vmx_sync_pir_to_irr())? I guess we'd want
> to see the entire IRR array (and perhaps also PI state) if the check
> in the assertion fails.
> 

Yes, I asked Chao to add some debug info in that case. The problem now
is when we will reproduce the bug to see meaningful hint...

Thanks
Kevin

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [xen-unstable test] 104131: regressions - FAIL

[Jan Beulich]

>>> On 22.01.17 at 05:35, <kevin.tian@intel.com> wrote:
> Yes, I asked Chao to add some debug info in that case. The problem now
> is when we will reproduce the bug to see meaningful hint...

If written reasonably, feel free to submit the patch for considering to
put it into staging for a while. The bigger problem is that we'd then
need to constantly scan logs for incidents.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [xen-unstable test] 104131: regressions - FAIL

[Xuquan (Quan Xu)]

On January 18, 2017 5:38 PM, Jan Beulich wrote:
>>>> On 18.01.17 at 05:57, <kevin.tian@intel.com> wrote:
>> Attached was my earlier comment:
>>
>> --
>>> >>> On 20.12.16 at 06:37, <kevin.tian@intel.com> wrote:
>>> >>  From: Xuquan (Quan Xu) [mailto:xuquan8@huawei.com]
>>> >> Sent: Friday, December 16, 2016 5:40 PM
>>> >> -        if (pt_vector != -1)
>>> >> -            vmx_set_eoi_exit_bitmap(v, pt_vector);
>>> >> +        if ( pt_vector != -1 ) {
>>> >> +            if ( intack.vector > pt_vector )
>>> >> +                vmx_set_eoi_exit_bitmap(v, intack.vector);
>>> >> +            else
>>> >> +                vmx_set_eoi_exit_bitmap(v, pt_vector);
>>> >> +        }
>>> >
>>> > Above can be simplified as one line change:
>>> > 	if ( pt_vector != -1 )
>>> > 		vmx_set_eoi_exit_bitmap(v, intack.vector);
>>>
>>> Hmm, I don't understand. Did you mean to use max() here? Or else how
>>> is this an equivalent of the originally proposed code?
>>>
>>
>> Original code is not 100% correct. The purpose is to set EOI exit
>> bitmap for any vector which may block injection of pt_vector - give
>> chance to recognize pt_vector in future intack and then do pt intr
>> post. The simplified code achieves this effect same as original code
>> if intack.vector >= vector. I cannot come up a case why intack.vector
>> might be smaller than vector. If this case happens, we still need
>> enable exit bitmap for intack.vector instead of pt_vector for said
>> purpose while original code did it wrong.
>>
>> Thanks
>> Kevin
>> --
>>
>> Using intack.vector is always expected here regardless of the
>> comparison result between intack.vector and pt_vector. The reason why
>> I was OK adding an ASSERT was simply to test whether
>> intack.vecor<pt_vector does happen which is orthogonal to the fix
>> itself.
>
>Well, a vector lower than pt_vector can't block delivery. Or wait:
>Don't we need to consider vector classes here, i.e.
>
>            ASSERT((intack.vector >> 4) >= (pt_vector >> 4));
>

Jan, I can't follow vector classes.. could you explain more? Thanks..


Quan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [xen-unstable test] 104131: regressions - FAIL

[Jan Beulich]

>>> On 20.01.17 at 09:47, <xuquan8@huawei.com> wrote:
> Jan, I can't follow vector classes.. could you explain more? Thanks..

For determining vector priority, the LAPIC uses only the high 4 bits.
Iirc this is well documented in the SDM.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [xen-unstable test] 104131: regressions - FAIL

[Xuquan (Quan Xu)]

On January 16, 2017 1:26 PM, Tian, Kevin wrote:
>I cannot come up a valid reason for such situation (intack.vector is 0x30 while
>pt_vector is 0x38 from Chao's data). pt_update_irq is invoked before checking
>highest pending IRRs so pt_vector should be honored anyway.
>One possible reason is that being some reason pt_vector is not in vIRR at that
>point (due to some bug in the path from PIR to vIRR). 

btw, for PIR.. I find that there might be a bug in __vmx_deliver_posted_interrupt()...
why test_and_set_bit(VCPU_KICK_SOFTIRQ, &softirq_pending(cpu)) ??

static void __vmx_deliver_posted_interrupt(struct vcpu *v)
{
...
        if ( !test_and_set_bit(VCPU_KICK_SOFTIRQ, &softirq_pending(cpu))
...
}

Suppose that vCPUx is in guest mode, there are two (even more) interrupts to vCPUx..
As the bit is set when delivers the first interrupt... the second interrupt is pending until next VM entry -- by PIR to vIRR..


Quan




However I didn't catch
>such bug simply by looking at code. We need reproduce this problem in
>developer side to find out actual reason. Andrew it'd be helpful if you can help
>Quan/Chao to find out more test environment info.
>
>One thing noted though. The original patch from Quan is actually orthogonal to
>this ASSERT. Regardless of whether intack.vector is larger or smaller than
>pt_vector, we always require the trick as long as pt_vector is not the one being
>currently programmed to RVI. Then do we want to revert the whole commit
>until the problem is finally fixed, or OK to just remove ASSERT (or replace with
>WARN_ON with more debug info) to unblock test system before the fix is
>ready?
>
>Thanks
>Kevin

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [xen-unstable test] 104131: regressions - FAIL

[Xuquan (Quan Xu)]

On January 20, 2017 5:09 PM, Quan Xu wrote:
>btw, for PIR.. I find that there might be a bug in
>__vmx_deliver_posted_interrupt()...
>why test_and_set_bit(VCPU_KICK_SOFTIRQ, &softirq_pending(cpu)) ??
>
>static void __vmx_deliver_posted_interrupt(struct vcpu *v) { ...
>        if ( !test_and_set_bit(VCPU_KICK_SOFTIRQ,
>&softirq_pending(cpu)) ...
>}
>
>Suppose that vCPUx is in guest mode, there are two (even more) interrupts
>to vCPUx..
>As the bit is set when delivers the first interrupt... the second interrupt is
>pending until next VM entry -- by PIR to vIRR..
>

Jan , Kevin
Correct me if I am wrong...

Quan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [xen-unstable test] 104131: regressions - FAIL

[Tian, Kevin]

> From: Xuquan (Quan Xu) [mailto:xuquan8@huawei.com]
> Sent: Monday, January 23, 2017 6:57 PM
> 
> On January 20, 2017 5:09 PM, Quan Xu wrote:
> >btw, for PIR.. I find that there might be a bug in
> >__vmx_deliver_posted_interrupt()...
> >why test_and_set_bit(VCPU_KICK_SOFTIRQ, &softirq_pending(cpu)) ??
> >
> >static void __vmx_deliver_posted_interrupt(struct vcpu *v) { ...
> >        if ( !test_and_set_bit(VCPU_KICK_SOFTIRQ,
> >&softirq_pending(cpu)) ...
> >}
> >
> >Suppose that vCPUx is in guest mode, there are two (even more) interrupts
> >to vCPUx..
> >As the bit is set when delivers the first interrupt... the second interrupt is
> >pending until next VM entry -- by PIR to vIRR..
> >
> 
> Jan , Kevin
> Correct me if I am wrong...
> 
> Quan

I don't quite understand the point here. Can you elaborate?

Thanks
Kevin

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [xen-unstable test] 104131: regressions - FAIL

[Xuquan (Quan Xu)]

On February 08, 2017 2:51 PM, Tian, Kevin wrote:
>> From: Xuquan (Quan Xu) [mailto:xuquan8@huawei.com]
>> Sent: Monday, January 23, 2017 6:57 PM
>>
>> On January 20, 2017 5:09 PM, Quan Xu wrote:
>> >btw, for PIR.. I find that there might be a bug in
>> >__vmx_deliver_posted_interrupt()...
>> >why test_and_set_bit(VCPU_KICK_SOFTIRQ, &softirq_pending(cpu)) ??
>> >
>> >static void __vmx_deliver_posted_interrupt(struct vcpu *v) { ...
>> >        if ( !test_and_set_bit(VCPU_KICK_SOFTIRQ,
>> >&softirq_pending(cpu)) ...
>> >}
>> >
>> >Suppose that vCPUx is in guest mode, there are two (even more)
>> >interrupts to vCPUx..
>> >As the bit is set when delivers the first interrupt... the second
>> >interrupt is pending until next VM entry -- by PIR to vIRR..
>> >
>>
>> Jan , Kevin
>> Correct me if I am wrong...
>>
>> Quan
>
>I don't quite understand the point here. Can you elaborate?
>

Assumed vCPU is in guest_mode..
When apicv is enabled, hypervisor calls vmx_deliver_posted_intr(), then __vmx_deliver_posted_interrupt() to deliver interrupt, but no vmexit (also no vcpu_kick() )..
In __vmx_deliver_posted_interrupt(), it is __conditional__ to deliver posted interrupt. if posted interrupt is not delivered, the posted interrupt is 
pending until next VM entry -- by PIR to vIRR.. 

one condition is :
In __vmx_deliver_posted_interrupt(),  ' if ( !test_and_set_bit(VCPU_KICK_SOFTIRQ, &softirq_pending(cpu))' ..

Specifically, we did verify it by RES interrupt, which is used for smp_reschedule_interrupt..
We even cost more time to deliver RES interrupt than no-apicv in average..

If RES interrupt (no. 1) is delivered by posted way (the vcpu is still guest_mode).. when tries to deliver next-coming RES interrupt (no. 2) by posted way,
The next-coming RES interrupt (no. 2) is not delivered, as we set the VCPU_KICK_SOFTIRQ bit when we deliver RES interrupt (no. 1)..

Then the next-coming RES interrupt (no. 2) is pending until next VM entry -- by PIR to vIRR..


We can fix it as below(I don't think this is a best one, it is better to set the VCPU_KICK_SOFTIRQ bit, but not test it):

--- a/xen/arch/x86/hvm/vmx/vmx.c
+++ b/xen/arch/x86/hvm/vmx/vmx.c
@@ -1846,7 +1846,7 @@ static void __vmx_deliver_posted_interrupt(struct vcpu *v)
     {
         unsigned int cpu = v->processor;

-        if ( !test_and_set_bit(VCPU_KICK_SOFTIRQ, &softirq_pending(cpu))
+        if ( !test_bit(VCPU_KICK_SOFTIRQ, &softirq_pending(cpu))
              && (cpu != smp_processor_id()) )
             send_IPI_mask(cpumask_of(cpu), posted_intr_vector);
     }



To be honest, I really spent several days for the original awkward-description:).. 

Happy Spring Festival!!
Quan






_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [xen-unstable test] 104131: regressions - FAIL

[Jan Beulich]

>>> On 08.02.17 at 09:27, <xuquan8@huawei.com> wrote:
> Assumed vCPU is in guest_mode..
> When apicv is enabled, hypervisor calls vmx_deliver_posted_intr(), then 
> __vmx_deliver_posted_interrupt() to deliver interrupt, but no vmexit (also no 
> vcpu_kick() )..
> In __vmx_deliver_posted_interrupt(), it is __conditional__ to deliver posted 
> interrupt. if posted interrupt is not delivered, the posted interrupt is 
> pending until next VM entry -- by PIR to vIRR.. 
> 
> one condition is :
> In __vmx_deliver_posted_interrupt(),  ' if ( 
> !test_and_set_bit(VCPU_KICK_SOFTIRQ, &softirq_pending(cpu))' ..
> 
> Specifically, we did verify it by RES interrupt, which is used for 
> smp_reschedule_interrupt..
> We even cost more time to deliver RES interrupt than no-apicv in average..
> 
> If RES interrupt (no. 1) is delivered by posted way (the vcpu is still 
> guest_mode).. when tries to deliver next-coming RES interrupt (no. 2) by 
> posted way,
> The next-coming RES interrupt (no. 2) is not delivered, as we set the 
> VCPU_KICK_SOFTIRQ bit when we deliver RES interrupt (no. 1)..
> 
> Then the next-coming RES interrupt (no. 2) is pending until next VM entry -- by 
> PIR to vIRR..
> 
> 
> We can fix it as below(I don't think this is a best one, it is better to set 
> the VCPU_KICK_SOFTIRQ bit, but not test it):
> 
> --- a/xen/arch/x86/hvm/vmx/vmx.c
> +++ b/xen/arch/x86/hvm/vmx/vmx.c
> @@ -1846,7 +1846,7 @@ static void __vmx_deliver_posted_interrupt(struct vcpu *v)
>      {
>          unsigned int cpu = v->processor;
> 
> -        if ( !test_and_set_bit(VCPU_KICK_SOFTIRQ, &softirq_pending(cpu))
> +        if ( !test_bit(VCPU_KICK_SOFTIRQ, &softirq_pending(cpu))
>               && (cpu != smp_processor_id()) )
>              send_IPI_mask(cpumask_of(cpu), posted_intr_vector);
>      }

While I don't think I fully understand your description, the line you
change here has always been puzzling me: If we were to raise a
softirq here, we ought to call cpu_raise_softirq() instead of partly
open coding what it does. So I think not marking that softirq
pending (but doing this incompletely) is a valid change in any case.
But I'll have to defer to Kevin in the hopes that he fully
understands what you explain above as well as him knowing why
this was a test-and-set here in the first place.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [xen-unstable test] 104131: regressions - FAIL

[Xuquan (Quan Xu)]

On February 08, 2017 4:52 PM, Jan Beulich wrote:
>>>> On 08.02.17 at 09:27, <xuquan8@huawei.com> wrote:
>> Assumed vCPU is in guest_mode..
>> When apicv is enabled, hypervisor calls vmx_deliver_posted_intr(),
>> then
>> __vmx_deliver_posted_interrupt() to deliver interrupt, but no vmexit
>> (also no
>> vcpu_kick() )..
>> In __vmx_deliver_posted_interrupt(), it is __conditional__ to deliver
>> posted interrupt. if posted interrupt is not delivered, the posted
>> interrupt is pending until next VM entry -- by PIR to vIRR..
>>
>> one condition is :
>> In __vmx_deliver_posted_interrupt(),  ' if (
>> !test_and_set_bit(VCPU_KICK_SOFTIRQ, &softirq_pending(cpu))' ..
>>
>> Specifically, we did verify it by RES interrupt, which is used for
>> smp_reschedule_interrupt..
>> We even cost more time to deliver RES interrupt than no-apicv in
>average..
>>
>> If RES interrupt (no. 1) is delivered by posted way (the vcpu is still
>> guest_mode).. when tries to deliver next-coming RES interrupt (no. 2)
>> by posted way, The next-coming RES interrupt (no. 2) is not delivered,
>> as we set the VCPU_KICK_SOFTIRQ bit when we deliver RES interrupt (no.
>> 1)..
>>
>> Then the next-coming RES interrupt (no. 2) is pending until next VM
>> entry -- by PIR to vIRR..
>>
>>
>> We can fix it as below(I don't think this is a best one, it is better
>> to set the VCPU_KICK_SOFTIRQ bit, but not test it):
>>
>> --- a/xen/arch/x86/hvm/vmx/vmx.c
>> +++ b/xen/arch/x86/hvm/vmx/vmx.c
>> @@ -1846,7 +1846,7 @@ static void
>__vmx_deliver_posted_interrupt(struct vcpu *v)
>>      {
>>          unsigned int cpu = v->processor;
>>
>> -        if ( !test_and_set_bit(VCPU_KICK_SOFTIRQ,
>&softirq_pending(cpu))
>> +        if ( !test_bit(VCPU_KICK_SOFTIRQ, &softirq_pending(cpu))
>>               && (cpu != smp_processor_id()) )
>>              send_IPI_mask(cpumask_of(cpu), posted_intr_vector);
>>      }
>
>While I don't think I fully understand your description, 

Sorry!!

>the line you change
>here has always been puzzling me: If we were to raise a softirq here, we
>ought to call cpu_raise_softirq() instead of partly open coding what it does.
>So I think not marking that softirq pending (but doing this incompletely) is
>a valid change in any case.

As comments in pi_notification_interrupt()  -- xen/arch/x86/hvm/vmx/vmx.c
((((
     *
     * we need to set VCPU_KICK_SOFTIRQ for the current cpu, just like
     * __vmx_deliver_posted_interrupt(). So the pending interrupt in PIRR will
     * be synced to vIRR before VM-Exit in time.
     *
))))

I think setting VCPU_KICK_SOFTIRQ bit -- the pending interrupt in PIRR will be synced to vIRR before VM-Exit in time.
That's also why i said it is better to set the VCPU_KICK_SOFTIRQ bit, but not test it..


>But I'll have to defer to Kevin in the hopes that he fully understands what
>you explain above as well as him knowing why this was a test-and-set here
>in the first place.
>

To me, this test-and-set is a bug.

Quan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [xen-unstable test] 104131: regressions - FAIL

[Chao Gao]

On Wed, Feb 08, 2017 at 10:15:28AM +0000, Xuquan (Quan Xu) wrote:
>On February 08, 2017 4:52 PM, Jan Beulich wrote:
>>>>> On 08.02.17 at 09:27, <xuquan8@huawei.com> wrote:
>>> Assumed vCPU is in guest_mode..
>>> When apicv is enabled, hypervisor calls vmx_deliver_posted_intr(),
>>> then
>>> __vmx_deliver_posted_interrupt() to deliver interrupt, but no vmexit
>>> (also no
>>> vcpu_kick() )..
>>> In __vmx_deliver_posted_interrupt(), it is __conditional__ to deliver
>>> posted interrupt. if posted interrupt is not delivered, the posted
>>> interrupt is pending until next VM entry -- by PIR to vIRR..
>>>
>>> one condition is :
>>> In __vmx_deliver_posted_interrupt(),  ' if (
>>> !test_and_set_bit(VCPU_KICK_SOFTIRQ, &softirq_pending(cpu))' ..
>>>
>>> Specifically, we did verify it by RES interrupt, which is used for
>>> smp_reschedule_interrupt..
>>> We even cost more time to deliver RES interrupt than no-apicv in
>>average..
>>>
>>> If RES interrupt (no. 1) is delivered by posted way (the vcpu is still
>>> guest_mode).. when tries to deliver next-coming RES interrupt (no. 2)
>>> by posted way, The next-coming RES interrupt (no. 2) is not delivered,
>>> as we set the VCPU_KICK_SOFTIRQ bit when we deliver RES interrupt (no.
>>> 1)..
>>>
>>> Then the next-coming RES interrupt (no. 2) is pending until next VM
>>> entry -- by PIR to vIRR..
>>>
>>>
>>> We can fix it as below(I don't think this is a best one, it is better
>>> to set the VCPU_KICK_SOFTIRQ bit, but not test it):
>>>
>>> --- a/xen/arch/x86/hvm/vmx/vmx.c
>>> +++ b/xen/arch/x86/hvm/vmx/vmx.c
>>> @@ -1846,7 +1846,7 @@ static void
>>__vmx_deliver_posted_interrupt(struct vcpu *v)
>>>      {
>>>          unsigned int cpu = v->processor;
>>>
>>> -        if ( !test_and_set_bit(VCPU_KICK_SOFTIRQ,
>>&softirq_pending(cpu))
>>> +        if ( !test_bit(VCPU_KICK_SOFTIRQ, &softirq_pending(cpu))
>>>               && (cpu != smp_processor_id()) )
>>>              send_IPI_mask(cpumask_of(cpu), posted_intr_vector);
>>>      }
>>
>>While I don't think I fully understand your description, 
>
>Sorry!!
>
>>the line you change
>>here has always been puzzling me: If we were to raise a softirq here, we
>>ought to call cpu_raise_softirq() instead of partly open coding what it does.
>>So I think not marking that softirq pending (but doing this incompletely) is
>>a valid change in any case.
>
>As comments in pi_notification_interrupt()  -- xen/arch/x86/hvm/vmx/vmx.c
>((((
>     *
>     * we need to set VCPU_KICK_SOFTIRQ for the current cpu, just like
>     * __vmx_deliver_posted_interrupt(). So the pending interrupt in PIRR will
>     * be synced to vIRR before VM-Exit in time.
>     *
>))))
>
>I think setting VCPU_KICK_SOFTIRQ bit -- the pending interrupt in PIRR will be synced to vIRR before VM-Exit in time.
>That's also why i said it is better to set the VCPU_KICK_SOFTIRQ bit, but not test it..
>

I think there is a typo. It should be "before VM-Entry in time". It set 
VCPU_KICK_SOFTIRQ bit only to jump to vmx_do_vmentry again instead of
entering guest directly. Jumping to vmx_do_vmentry again can re-sync the PIR to
vIRR in vmx_intr_assist(). In root-mode, cpu treat the pi notification interrupt
as normal interrupt, so cpu will run the interrupt handler pi_notification_interrupt()
instead of syncing PIR to vIRR automatically. Receiving a pi notificatio interrupt
means some interrupts have been posted in PIR. Setting that bit is to deliver 
these new arrival interrupts before this VM-entry. 

Thanks
chao

>
>>But I'll have to defer to Kevin in the hopes that he fully understands what
>>you explain above as well as him knowing why this was a test-and-set here
>>in the first place.
>>
>
>To me, this test-and-set is a bug.
>
>Quan
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [xen-unstable test] 104131: regressions - FAIL

[Xuquan (Quan Xu)]

On February 08, 2017 4:22 PM, Chao Gao wrote:
>On Wed, Feb 08, 2017 at 10:15:28AM +0000, Xuquan (Quan Xu) wrote:
>>On February 08, 2017 4:52 PM, Jan Beulich wrote:
>>>>>> On 08.02.17 at 09:27, <xuquan8@huawei.com> wrote:
>>>> Assumed vCPU is in guest_mode..
>>>> When apicv is enabled, hypervisor calls vmx_deliver_posted_intr(),
>>>> then
>>>> __vmx_deliver_posted_interrupt() to deliver interrupt, but no vmexit
>>>> (also no
>>>> vcpu_kick() )..
>>>> In __vmx_deliver_posted_interrupt(), it is __conditional__ to
>>>> deliver posted interrupt. if posted interrupt is not delivered, the
>>>> posted interrupt is pending until next VM entry -- by PIR to vIRR..
>>>>
>>>> one condition is :
>>>> In __vmx_deliver_posted_interrupt(),  ' if (
>>>> !test_and_set_bit(VCPU_KICK_SOFTIRQ, &softirq_pending(cpu))' ..
>>>>
>>>> Specifically, we did verify it by RES interrupt, which is used for
>>>> smp_reschedule_interrupt..
>>>> We even cost more time to deliver RES interrupt than no-apicv in
>>>average..
>>>>
>>>> If RES interrupt (no. 1) is delivered by posted way (the vcpu is
>>>> still guest_mode).. when tries to deliver next-coming RES interrupt
>>>> (no. 2) by posted way, The next-coming RES interrupt (no. 2) is not
>>>> delivered, as we set the VCPU_KICK_SOFTIRQ bit when we deliver RES
>interrupt (no.
>>>> 1)..
>>>>
>>>> Then the next-coming RES interrupt (no. 2) is pending until next VM
>>>> entry -- by PIR to vIRR..
>>>>
>>>>
>>>> We can fix it as below(I don't think this is a best one, it is
>>>> better to set the VCPU_KICK_SOFTIRQ bit, but not test it):
>>>>
>>>> --- a/xen/arch/x86/hvm/vmx/vmx.c
>>>> +++ b/xen/arch/x86/hvm/vmx/vmx.c
>>>> @@ -1846,7 +1846,7 @@ static void
>>>__vmx_deliver_posted_interrupt(struct vcpu *v)
>>>>      {
>>>>          unsigned int cpu = v->processor;
>>>>
>>>> -        if ( !test_and_set_bit(VCPU_KICK_SOFTIRQ,
>>>&softirq_pending(cpu))
>>>> +        if ( !test_bit(VCPU_KICK_SOFTIRQ, &softirq_pending(cpu))
>>>>               && (cpu != smp_processor_id()) )
>>>>              send_IPI_mask(cpumask_of(cpu), posted_intr_vector);
>>>>      }
>>>
>>>While I don't think I fully understand your description,
>>
>>Sorry!!
>>
>>>the line you change
>>>here has always been puzzling me: If we were to raise a softirq here,
>>>we ought to call cpu_raise_softirq() instead of partly open coding what it
>does.
>>>So I think not marking that softirq pending (but doing this
>>>incompletely) is a valid change in any case.
>>
>>As comments in pi_notification_interrupt()  --
>>xen/arch/x86/hvm/vmx/vmx.c ((((
>>     *
>>     * we need to set VCPU_KICK_SOFTIRQ for the current cpu, just like
>>     * __vmx_deliver_posted_interrupt(). So the pending interrupt in
>PIRR will
>>     * be synced to vIRR before VM-Exit in time.
>>     *
>>))))
>>
>>I think setting VCPU_KICK_SOFTIRQ bit -- the pending interrupt in PIRR will
>be synced to vIRR before VM-Exit in time.
>>That's also why i said it is better to set the VCPU_KICK_SOFTIRQ bit, but
>not test it..
>>
>
>I think there is a typo. It should be "before VM-Entry in time". It set
>VCPU_KICK_SOFTIRQ bit only to jump to vmx_do_vmentry again instead of
>entering guest directly. Jumping to vmx_do_vmentry again can re-sync the
>PIR to vIRR in vmx_intr_assist(). 

impressive analysis..
chao, could you show the related code?

Quan


>In root-mode, cpu treat the pi notification
>interrupt as normal interrupt, so cpu will run the interrupt handler
>pi_notification_interrupt() instead of syncing PIR to vIRR automatically.
>Receiving a pi notificatio interrupt means some interrupts have been
>posted in PIR. Setting that bit is to deliver these new arrival interrupts
>before this VM-entry.
>


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [xen-unstable test] 104131: regressions - FAIL

[Chao Gao]

On Thu, Feb 09, 2017 at 08:51:46AM +0000, Xuquan (Quan Xu) wrote:
>On February 08, 2017 4:22 PM, Chao Gao wrote:
>>On Wed, Feb 08, 2017 at 10:15:28AM +0000, Xuquan (Quan Xu) wrote:
>>>On February 08, 2017 4:52 PM, Jan Beulich wrote:
>>>>>>> On 08.02.17 at 09:27, <xuquan8@huawei.com> wrote:
>>>>> Assumed vCPU is in guest_mode..
>>>>> When apicv is enabled, hypervisor calls vmx_deliver_posted_intr(),
>>>>> then
>>>>> __vmx_deliver_posted_interrupt() to deliver interrupt, but no vmexit
>>>>> (also no
>>>>> vcpu_kick() )..
>>>>> In __vmx_deliver_posted_interrupt(), it is __conditional__ to
>>>>> deliver posted interrupt. if posted interrupt is not delivered, the
>>>>> posted interrupt is pending until next VM entry -- by PIR to vIRR..
>>>>>
>>>>> one condition is :
>>>>> In __vmx_deliver_posted_interrupt(),  ' if (
>>>>> !test_and_set_bit(VCPU_KICK_SOFTIRQ, &softirq_pending(cpu))' ..
>>>>>
>>>>> Specifically, we did verify it by RES interrupt, which is used for
>>>>> smp_reschedule_interrupt..
>>>>> We even cost more time to deliver RES interrupt than no-apicv in
>>>>average..
>>>>>
>>>>> If RES interrupt (no. 1) is delivered by posted way (the vcpu is
>>>>> still guest_mode).. when tries to deliver next-coming RES interrupt
>>>>> (no. 2) by posted way, The next-coming RES interrupt (no. 2) is not
>>>>> delivered, as we set the VCPU_KICK_SOFTIRQ bit when we deliver RES
>>interrupt (no.
>>>>> 1)..
>>>>>
>>>>> Then the next-coming RES interrupt (no. 2) is pending until next VM
>>>>> entry -- by PIR to vIRR..
>>>>>
>>>>>
>>>>> We can fix it as below(I don't think this is a best one, it is
>>>>> better to set the VCPU_KICK_SOFTIRQ bit, but not test it):
>>>>>
>>>>> --- a/xen/arch/x86/hvm/vmx/vmx.c
>>>>> +++ b/xen/arch/x86/hvm/vmx/vmx.c
>>>>> @@ -1846,7 +1846,7 @@ static void
>>>>__vmx_deliver_posted_interrupt(struct vcpu *v)
>>>>>      {
>>>>>          unsigned int cpu = v->processor;
>>>>>
>>>>> -        if ( !test_and_set_bit(VCPU_KICK_SOFTIRQ,
>>>>&softirq_pending(cpu))
>>>>> +        if ( !test_bit(VCPU_KICK_SOFTIRQ, &softirq_pending(cpu))
>>>>>               && (cpu != smp_processor_id()) )
>>>>>              send_IPI_mask(cpumask_of(cpu), posted_intr_vector);
>>>>>      }
>>>>
>>>>While I don't think I fully understand your description,
>>>
>>>Sorry!!
>>>
>>>>the line you change
>>>>here has always been puzzling me: If we were to raise a softirq here,
>>>>we ought to call cpu_raise_softirq() instead of partly open coding what it
>>does.
>>>>So I think not marking that softirq pending (but doing this
>>>>incompletely) is a valid change in any case.
>>>
>>>As comments in pi_notification_interrupt()  --
>>>xen/arch/x86/hvm/vmx/vmx.c ((((
>>>     *
>>>     * we need to set VCPU_KICK_SOFTIRQ for the current cpu, just like
>>>     * __vmx_deliver_posted_interrupt(). So the pending interrupt in
>>PIRR will
>>>     * be synced to vIRR before VM-Exit in time.
>>>     *
>>>))))
>>>
>>>I think setting VCPU_KICK_SOFTIRQ bit -- the pending interrupt in PIRR will
>>be synced to vIRR before VM-Exit in time.
>>>That's also why i said it is better to set the VCPU_KICK_SOFTIRQ bit, but
>>not test it..
>>>
>>
>>I think there is a typo. It should be "before VM-Entry in time". It set
>>VCPU_KICK_SOFTIRQ bit only to jump to vmx_do_vmentry again instead of
>>entering guest directly. Jumping to vmx_do_vmentry again can re-sync the
>>PIR to vIRR in vmx_intr_assist(). 
>
>impressive analysis..
>chao, could you show the related code?
>

In xen/arch/x86/vmx/entry.S, 
.Lvmx_do_vmentry:
	call vmx_intr_assist
	... 
	cli
	cmp %ecx,(%rdx,%rax,1)
	jnz .Lvmx_process_softirqs
and 
.Lvmx_process_softirqs:
	sti
	call do_softirq
	jmp .Lvmx_do_vmentry	

In vmx_intr_assist(), PIR is synced to vIRR. After vmx_intr_assist(), 
if some interrupts are posted in PIR, VCPU_KICK_SOFTIRQ is set 
in pi_nofitication_interrupt() and it will jump to vmx_process_softirqs
and jump to vmx_do_vmentry again.

Thanks
Chao

>Quan
>
>
>>In root-mode, cpu treat the pi notification
>>interrupt as normal interrupt, so cpu will run the interrupt handler
>>pi_notification_interrupt() instead of syncing PIR to vIRR automatically.
>>Receiving a pi notificatio interrupt means some interrupts have been
>>posted in PIR. Setting that bit is to deliver these new arrival interrupts
>>before this VM-entry.
>>
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [xen-unstable test] 104131: regressions - FAIL

[Tian, Kevin]

> From: Jan Beulich [mailto:JBeulich@suse.com]
> Sent: Wednesday, February 08, 2017 4:52 PM
> 
> >>> On 08.02.17 at 09:27, <xuquan8@huawei.com> wrote:
> > Assumed vCPU is in guest_mode..
> > When apicv is enabled, hypervisor calls vmx_deliver_posted_intr(), then
> > __vmx_deliver_posted_interrupt() to deliver interrupt, but no vmexit (also no
> > vcpu_kick() )..
> > In __vmx_deliver_posted_interrupt(), it is __conditional__ to deliver posted
> > interrupt. if posted interrupt is not delivered, the posted interrupt is
> > pending until next VM entry -- by PIR to vIRR..
> >
> > one condition is :
> > In __vmx_deliver_posted_interrupt(),  ' if (
> > !test_and_set_bit(VCPU_KICK_SOFTIRQ, &softirq_pending(cpu))' ..
> >
> > Specifically, we did verify it by RES interrupt, which is used for
> > smp_reschedule_interrupt..
> > We even cost more time to deliver RES interrupt than no-apicv in average..
> >
> > If RES interrupt (no. 1) is delivered by posted way (the vcpu is still
> > guest_mode).. when tries to deliver next-coming RES interrupt (no. 2) by
> > posted way,
> > The next-coming RES interrupt (no. 2) is not delivered, as we set the
> > VCPU_KICK_SOFTIRQ bit when we deliver RES interrupt (no. 1)..
> >
> > Then the next-coming RES interrupt (no. 2) is pending until next VM entry -- by
> > PIR to vIRR..
> >
> >
> > We can fix it as below(I don't think this is a best one, it is better to set
> > the VCPU_KICK_SOFTIRQ bit, but not test it):
> >
> > --- a/xen/arch/x86/hvm/vmx/vmx.c
> > +++ b/xen/arch/x86/hvm/vmx/vmx.c
> > @@ -1846,7 +1846,7 @@ static void __vmx_deliver_posted_interrupt(struct vcpu *v)
> >      {
> >          unsigned int cpu = v->processor;
> >
> > -        if ( !test_and_set_bit(VCPU_KICK_SOFTIRQ, &softirq_pending(cpu))
> > +        if ( !test_bit(VCPU_KICK_SOFTIRQ, &softirq_pending(cpu))
> >               && (cpu != smp_processor_id()) )
> >              send_IPI_mask(cpumask_of(cpu), posted_intr_vector);
> >      }
> 
> While I don't think I fully understand your description, the line you
> change here has always been puzzling me: If we were to raise a
> softirq here, we ought to call cpu_raise_softirq() instead of partly
> open coding what it does. 

We require posted_intr_vector for target CPU to ack/deliver virtual 
interrupt in non-root mode. cpu_raise_softirq uses a different vector,
which cannot trigger such effect. 

> So I think not marking that softirq
> pending (but doing this incompletely) is a valid change in any case.
> But I'll have to defer to Kevin in the hopes that he fully
> understands what you explain above as well as him knowing why
> this was a test-and-set here in the first place.
> 

I agree we have a misuse of softirq mechanism here. If guest 
is already in non-root mode, the 1st posted interrupt will be directly 
delivered to guest (leaving softirq being set w/o actually incurring a 
VM-exit - breaking desired softirq behavior). Then further posted 
interrupts will skip the IPI, stay in PIR and not noted until another 
VM-exit happens. Looks Quan observes such delay of delivery in
his experiments.

I'm OK to remove the set here. Actually since it's an optimization
for less IPIs, we'd better check softirq_pending(cpu) directly 
instead of sticking to one bit only.

Thanks
Kevin





_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [xen-unstable test] 104131: regressions - FAIL

[Xuquan (Quan Xu)]

On February 13, 2017 4:21 PM, Tian, Kevin wrote:
>> From: Jan Beulich [mailto:JBeulich@suse.com]
>> Sent: Wednesday, February 08, 2017 4:52 PM
>>
>> >>> On 08.02.17 at 09:27, <xuquan8@huawei.com> wrote:
>> > Assumed vCPU is in guest_mode..
>> > When apicv is enabled, hypervisor calls vmx_deliver_posted_intr(),
>> > then
>> > __vmx_deliver_posted_interrupt() to deliver interrupt, but no vmexit
>> > (also no
>> > vcpu_kick() )..
>> > In __vmx_deliver_posted_interrupt(), it is __conditional__ to
>> > deliver posted interrupt. if posted interrupt is not delivered, the
>> > posted interrupt is pending until next VM entry -- by PIR to vIRR..
>> >
>> > one condition is :
>> > In __vmx_deliver_posted_interrupt(),  ' if (
>> > !test_and_set_bit(VCPU_KICK_SOFTIRQ, &softirq_pending(cpu))' ..
>> >
>> > Specifically, we did verify it by RES interrupt, which is used for
>> > smp_reschedule_interrupt..
>> > We even cost more time to deliver RES interrupt than no-apicv in
>average..
>> >
>> > If RES interrupt (no. 1) is delivered by posted way (the vcpu is
>> > still guest_mode).. when tries to deliver next-coming RES interrupt
>> > (no. 2) by posted way, The next-coming RES interrupt (no. 2) is not
>> > delivered, as we set the VCPU_KICK_SOFTIRQ bit when we deliver RES
>> > interrupt (no. 1)..
>> >
>> > Then the next-coming RES interrupt (no. 2) is pending until next VM
>> > entry -- by PIR to vIRR..
>> >
>> >
>> > We can fix it as below(I don't think this is a best one, it is
>> > better to set the VCPU_KICK_SOFTIRQ bit, but not test it):
>> >
>> > --- a/xen/arch/x86/hvm/vmx/vmx.c
>> > +++ b/xen/arch/x86/hvm/vmx/vmx.c
>> > @@ -1846,7 +1846,7 @@ static void
>__vmx_deliver_posted_interrupt(struct vcpu *v)
>> >      {
>> >          unsigned int cpu = v->processor;
>> >
>> > -        if ( !test_and_set_bit(VCPU_KICK_SOFTIRQ,
>&softirq_pending(cpu))
>> > +        if ( !test_bit(VCPU_KICK_SOFTIRQ, &softirq_pending(cpu))
>> >               && (cpu != smp_processor_id()) )
>> >              send_IPI_mask(cpumask_of(cpu), posted_intr_vector);
>> >      }
>>
>> While I don't think I fully understand your description, the line you
>> change here has always been puzzling me: If we were to raise a softirq
>> here, we ought to call cpu_raise_softirq() instead of partly open
>> coding what it does.
>
>We require posted_intr_vector for target CPU to ack/deliver virtual
>interrupt in non-root mode. cpu_raise_softirq uses a different vector, which
>cannot trigger such effect.
>


Kevin,

I can't follow this 'to ack'..
As I understand, the posted_intr_vector is to call event_check_interrupt() [ or pi_notification_interrupt() ] to writes zero to the EOI register in the local APIC --
this dismisses the interrupt with the posted interrupt notification vector from the local APIC.

What does this ack refer to?






>> So I think not marking that softirq
>> pending (but doing this incompletely) is a valid change in any case.
>> But I'll have to defer to Kevin in the hopes that he fully understands
>> what you explain above as well as him knowing why this was a
>> test-and-set here in the first place.
>>
>
>I agree we have a misuse of softirq mechanism here. If guest is already in
>non-root mode, the 1st posted interrupt will be directly delivered to guest
>(leaving softirq being set w/o actually incurring a VM-exit - breaking desired
>softirq behavior). Then further posted interrupts will skip the IPI, stay in PIR
>and not noted until another VM-exit happens. Looks Quan observes such
>delay of delivery in his experiments.
>
>I'm OK to remove the set here. Actually since it's an optimization for less
>IPIs, we'd better check softirq_pending(cpu) directly instead of sticking to
>one bit only.
>
>Thanks
>Kevin
>
>
>


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [xen-unstable test] 104131: regressions - FAIL

[Tian, Kevin]

> From: Xuquan (Quan Xu) [mailto:xuquan8@huawei.com]
> Sent: Monday, February 20, 2017 8:04 PM
> 
> On February 13, 2017 4:21 PM, Tian, Kevin wrote:
> >> From: Jan Beulich [mailto:JBeulich@suse.com]
> >> Sent: Wednesday, February 08, 2017 4:52 PM
> >>
> >> >>> On 08.02.17 at 09:27, <xuquan8@huawei.com> wrote:
> >> > Assumed vCPU is in guest_mode..
> >> > When apicv is enabled, hypervisor calls vmx_deliver_posted_intr(),
> >> > then
> >> > __vmx_deliver_posted_interrupt() to deliver interrupt, but no vmexit
> >> > (also no
> >> > vcpu_kick() )..
> >> > In __vmx_deliver_posted_interrupt(), it is __conditional__ to
> >> > deliver posted interrupt. if posted interrupt is not delivered, the
> >> > posted interrupt is pending until next VM entry -- by PIR to vIRR..
> >> >
> >> > one condition is :
> >> > In __vmx_deliver_posted_interrupt(),  ' if (
> >> > !test_and_set_bit(VCPU_KICK_SOFTIRQ, &softirq_pending(cpu))' ..
> >> >
> >> > Specifically, we did verify it by RES interrupt, which is used for
> >> > smp_reschedule_interrupt..
> >> > We even cost more time to deliver RES interrupt than no-apicv in
> >average..
> >> >
> >> > If RES interrupt (no. 1) is delivered by posted way (the vcpu is
> >> > still guest_mode).. when tries to deliver next-coming RES interrupt
> >> > (no. 2) by posted way, The next-coming RES interrupt (no. 2) is not
> >> > delivered, as we set the VCPU_KICK_SOFTIRQ bit when we deliver RES
> >> > interrupt (no. 1)..
> >> >
> >> > Then the next-coming RES interrupt (no. 2) is pending until next VM
> >> > entry -- by PIR to vIRR..
> >> >
> >> >
> >> > We can fix it as below(I don't think this is a best one, it is
> >> > better to set the VCPU_KICK_SOFTIRQ bit, but not test it):
> >> >
> >> > --- a/xen/arch/x86/hvm/vmx/vmx.c
> >> > +++ b/xen/arch/x86/hvm/vmx/vmx.c
> >> > @@ -1846,7 +1846,7 @@ static void
> >__vmx_deliver_posted_interrupt(struct vcpu *v)
> >> >      {
> >> >          unsigned int cpu = v->processor;
> >> >
> >> > -        if ( !test_and_set_bit(VCPU_KICK_SOFTIRQ,
> >&softirq_pending(cpu))
> >> > +        if ( !test_bit(VCPU_KICK_SOFTIRQ, &softirq_pending(cpu))
> >> >               && (cpu != smp_processor_id()) )
> >> >              send_IPI_mask(cpumask_of(cpu), posted_intr_vector);
> >> >      }
> >>
> >> While I don't think I fully understand your description, the line you
> >> change here has always been puzzling me: If we were to raise a softirq
> >> here, we ought to call cpu_raise_softirq() instead of partly open
> >> coding what it does.
> >
> >We require posted_intr_vector for target CPU to ack/deliver virtual
> >interrupt in non-root mode. cpu_raise_softirq uses a different vector, which
> >cannot trigger such effect.
> >
> 
> 
> Kevin,
> 
> I can't follow this 'to ack'..
> As I understand, the posted_intr_vector is to call event_check_interrupt() [ or
> pi_notification_interrupt() ] to writes zero to the EOI register in the local APIC --
> this dismisses the interrupt with the posted interrupt notification vector from the local
> APIC.
> 
> What does this ack refer to?
> 

Please look at SDM. 'ack' means evaluation of pending vIRRs when CPU is
in non-root mode which results in direct virtual interrupt delivery w/o incurring
VM-exit.

Thanks
Kevin

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [xen-unstable test] 104131: regressions - FAIL

[Tian, Kevin]

> From: Tian, Kevin
> Sent: Monday, February 13, 2017 4:21 PM
> 
> > From: Jan Beulich [mailto:JBeulich@suse.com]
> > Sent: Wednesday, February 08, 2017 4:52 PM
> >
> > >>> On 08.02.17 at 09:27, <xuquan8@huawei.com> wrote:
> > > Assumed vCPU is in guest_mode..
> > > When apicv is enabled, hypervisor calls vmx_deliver_posted_intr(), then
> > > __vmx_deliver_posted_interrupt() to deliver interrupt, but no vmexit (also no
> > > vcpu_kick() )..
> > > In __vmx_deliver_posted_interrupt(), it is __conditional__ to deliver posted
> > > interrupt. if posted interrupt is not delivered, the posted interrupt is
> > > pending until next VM entry -- by PIR to vIRR..
> > >
> > > one condition is :
> > > In __vmx_deliver_posted_interrupt(),  ' if (
> > > !test_and_set_bit(VCPU_KICK_SOFTIRQ, &softirq_pending(cpu))' ..
> > >
> > > Specifically, we did verify it by RES interrupt, which is used for
> > > smp_reschedule_interrupt..
> > > We even cost more time to deliver RES interrupt than no-apicv in average..
> > >
> > > If RES interrupt (no. 1) is delivered by posted way (the vcpu is still
> > > guest_mode).. when tries to deliver next-coming RES interrupt (no. 2) by
> > > posted way,
> > > The next-coming RES interrupt (no. 2) is not delivered, as we set the
> > > VCPU_KICK_SOFTIRQ bit when we deliver RES interrupt (no. 1)..
> > >
> > > Then the next-coming RES interrupt (no. 2) is pending until next VM entry -- by
> > > PIR to vIRR..
> > >
> > >
> > > We can fix it as below(I don't think this is a best one, it is better to set
> > > the VCPU_KICK_SOFTIRQ bit, but not test it):
> > >
> > > --- a/xen/arch/x86/hvm/vmx/vmx.c
> > > +++ b/xen/arch/x86/hvm/vmx/vmx.c
> > > @@ -1846,7 +1846,7 @@ static void __vmx_deliver_posted_interrupt(struct vcpu *v)
> > >      {
> > >          unsigned int cpu = v->processor;
> > >
> > > -        if ( !test_and_set_bit(VCPU_KICK_SOFTIRQ, &softirq_pending(cpu))
> > > +        if ( !test_bit(VCPU_KICK_SOFTIRQ, &softirq_pending(cpu))
> > >               && (cpu != smp_processor_id()) )
> > >              send_IPI_mask(cpumask_of(cpu), posted_intr_vector);
> > >      }
> >
> > While I don't think I fully understand your description, the line you
> > change here has always been puzzling me: If we were to raise a
> > softirq here, we ought to call cpu_raise_softirq() instead of partly
> > open coding what it does.
> 
> We require posted_intr_vector for target CPU to ack/deliver virtual
> interrupt in non-root mode. cpu_raise_softirq uses a different vector,
> which cannot trigger such effect.
> 
> > So I think not marking that softirq
> > pending (but doing this incompletely) is a valid change in any case.
> > But I'll have to defer to Kevin in the hopes that he fully
> > understands what you explain above as well as him knowing why
> > this was a test-and-set here in the first place.
> >
> 
> I agree we have a misuse of softirq mechanism here. If guest
> is already in non-root mode, the 1st posted interrupt will be directly
> delivered to guest (leaving softirq being set w/o actually incurring a
> VM-exit - breaking desired softirq behavior). Then further posted
> interrupts will skip the IPI, stay in PIR and not noted until another
> VM-exit happens. Looks Quan observes such delay of delivery in
> his experiments.
> 
> I'm OK to remove the set here. Actually since it's an optimization
> for less IPIs, we'd better check softirq_pending(cpu) directly
> instead of sticking to one bit only.
>

sent too fast... Quan, can you work out a patch following this
suggestion and see whether your slow-delivery issue is solved?
(hope I understand your issue correctly here).

Thanks
Kevin

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [xen-unstable test] 104131: regressions - FAIL

[Xuquan (Quan Xu)]

On February 13, 2017 4:24 PM, Tian, Kevin wrote:
>> From: Tian, Kevin
>> Sent: Monday, February 13, 2017 4:21 PM
>>
>> > From: Jan Beulich [mailto:JBeulich@suse.com]
>> > Sent: Wednesday, February 08, 2017 4:52 PM
>> >
>> > >>> On 08.02.17 at 09:27, <xuquan8@huawei.com> wrote:
>> > > Assumed vCPU is in guest_mode..
>> > > When apicv is enabled, hypervisor calls vmx_deliver_posted_intr(),
>> > > then
>> > > __vmx_deliver_posted_interrupt() to deliver interrupt, but no
>> > > vmexit (also no
>> > > vcpu_kick() )..
>> > > In __vmx_deliver_posted_interrupt(), it is __conditional__ to
>> > > deliver posted interrupt. if posted interrupt is not delivered,
>> > > the posted interrupt is pending until next VM entry -- by PIR to vIRR..
>> > >
>> > > one condition is :
>> > > In __vmx_deliver_posted_interrupt(),  ' if (
>> > > !test_and_set_bit(VCPU_KICK_SOFTIRQ, &softirq_pending(cpu))' ..
>> > >
>> > > Specifically, we did verify it by RES interrupt, which is used for
>> > > smp_reschedule_interrupt..
>> > > We even cost more time to deliver RES interrupt than no-apicv in
>average..
>> > >
>> > > If RES interrupt (no. 1) is delivered by posted way (the vcpu is
>> > > still guest_mode).. when tries to deliver next-coming RES
>> > > interrupt (no. 2) by posted way, The next-coming RES interrupt
>> > > (no. 2) is not delivered, as we set the VCPU_KICK_SOFTIRQ bit when
>> > > we deliver RES interrupt (no. 1)..
>> > >
>> > > Then the next-coming RES interrupt (no. 2) is pending until next
>> > > VM entry -- by PIR to vIRR..
>> > >
>> > >
>> > > We can fix it as below(I don't think this is a best one, it is
>> > > better to set the VCPU_KICK_SOFTIRQ bit, but not test it):
>> > >
>> > > --- a/xen/arch/x86/hvm/vmx/vmx.c
>> > > +++ b/xen/arch/x86/hvm/vmx/vmx.c
>> > > @@ -1846,7 +1846,7 @@ static void
>__vmx_deliver_posted_interrupt(struct vcpu *v)
>> > >      {
>> > >          unsigned int cpu = v->processor;
>> > >
>> > > -        if ( !test_and_set_bit(VCPU_KICK_SOFTIRQ,
>&softirq_pending(cpu))
>> > > +        if ( !test_bit(VCPU_KICK_SOFTIRQ, &softirq_pending(cpu))
>> > >               && (cpu != smp_processor_id()) )
>> > >              send_IPI_mask(cpumask_of(cpu), posted_intr_vector);
>> > >      }
>> >
>> > While I don't think I fully understand your description, the line
>> > you change here has always been puzzling me: If we were to raise a
>> > softirq here, we ought to call cpu_raise_softirq() instead of partly
>> > open coding what it does.
>>
>> We require posted_intr_vector for target CPU to ack/deliver virtual
>> interrupt in non-root mode. cpu_raise_softirq uses a different vector,
>> which cannot trigger such effect.
>>
>> > So I think not marking that softirq
>> > pending (but doing this incompletely) is a valid change in any case.
>> > But I'll have to defer to Kevin in the hopes that he fully
>> > understands what you explain above as well as him knowing why this
>> > was a test-and-set here in the first place.
>> >
>>
>> I agree we have a misuse of softirq mechanism here. If guest is
>> already in non-root mode, the 1st posted interrupt will be directly
>> delivered to guest (leaving softirq being set w/o actually incurring a
>> VM-exit - breaking desired softirq behavior). Then further posted
>> interrupts will skip the IPI, stay in PIR and not noted until another
>> VM-exit happens. Looks Quan observes such delay of delivery in his
>> experiments.
>>
>> I'm OK to remove the set here. Actually since it's an optimization for
>> less IPIs, we'd better check softirq_pending(cpu) directly instead of
>> sticking to one bit only.
>>
>
>sent too fast... Quan, can you work out a patch following this suggestion
>and see whether your slow-delivery issue is solved?
>(hope I understand your issue correctly here).
>

Cool, Very Correct!! 
Sure, I will send out a patch in coming days..

Quan










_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel