From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Hu, Robert" Subject: Re: [xen-unstable test] 65141: regressions - FAIL Date: Tue, 8 Dec 2015 08:06:05 +0000 Message-ID: <9E79D1C9A97CFD4097BCE431828FDD31023DB12A@SHSMSX103.ccr.corp.intel.com> References: <22104.31917.532615.949661@mariner.uk.xensource.com> <9E79D1C9A97CFD4097BCE431828FDD31023BAE97@SHSMSX103.ccr.corp.intel.com> <1449052492.4424.30.camel@citrix.com> <1449064269.4424.73.camel@citrix.com> <1449302981.3451.3.camel@citrix.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <1449302981.3451.3.camel@citrix.com> Content-Language: en-US List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Ian Campbell , Ian Jackson , "Nakajima, Jun" , "Tian, Kevin" Cc: "xen-devel@lists.xensource.com" , Andrew Cooper , osstest service owner , "Wang, Yong Y" , Jan Beulich , "Jin, Gordon" List-Id: xen-devel@lists.xenproject.org > -----Original Message----- > From: Ian Campbell [mailto:ian.campbell@citrix.com] > Sent: Saturday, December 5, 2015 4:10 PM > To: Hu, Robert ; Ian Jackson > ; Nakajima, Jun ; > Tian, Kevin > Cc: xen-devel@lists.xensource.com; osstest service owner > ; Jan Beulich ; > Andrew Cooper > Subject: Re: [xen-unstable test] 65141: regressions - FAIL > > On Wed, 2015-12-02 at 13:51 +0000, Ian Campbell wrote: > > > http://osstest.test-lab.xenproject.org/~osstest/pub/logs/65301/ > > > > I think that ought to give a baseline for the bisector to work with. I'll > > prod it to do so. > > Results are below. TL;DR: d02e84b9d9d "vVMX: use latched VMCS machine > address" is somehow at fault. > > It appears to be somewhat machine specific, the one this has been > failing on is godello* which says "CPU0: Intel(R) Xeon(R) CPU E3-1220 > v3 @ 3.10GHz stepping 03" in its serial log. > > Andy suggested this might be related to cpu_has_vmx_vmcs_shadowing > so Haswell and newer vs IvyBridge and older. > > Ian. > > branch xen-unstable > xenbranch xen-unstable > job test-amd64-amd64-qemuu-nested-intel > testid debian-hvm-install/l1/l2 > > Tree: linux git://xenbits.xen.org/linux-pvops.git > Tree: linuxfirmware git://xenbits.xen.org/osstest/linux-firmware.git > Tree: qemu git://xenbits.xen.org/qemu-xen-traditional.git > Tree: qemuu git://xenbits.xen.org/qemu-xen.git > Tree: xen git://xenbits.xen.org/xen.git > > *** Found and reproduced problem changeset *** > > Bug is in tree: xen git://xenbits.xen.org/xen.git > Bug introduced: d02e84b9d9d16b6b56186f0dfdcb3c90b83c82a3 > Bug not present: 3b47431691409004c7218f6a6ba5c9c0bcf483ea > Last fail repro: http://logs.test-lab.xenproject.org/osstest/logs/65388/ > > > commit d02e84b9d9d16b6b56186f0dfdcb3c90b83c82a3 > Author: Jan Beulich > Date: Tue Nov 24 12:07:27 2015 +0100 > > vVMX: use latched VMCS machine address > > Instead of calling domain_page_map_to_mfn() over and over, latch > the > guest VMCS machine address unconditionally (i.e. independent of > whether > VMCS shadowing is supported by the hardware). > > Since this requires altering the parameters of __[gs]et_vmcs{,_real}() > (and hence all their callers) anyway, take the opportunity to also drop > the bogus double underscores from their names (and from > __[gs]et_vmcs_virtual() as well). > > Signed-off-by: Jan Beulich > Acked-by: Kevin Tian > > > For bisection revision-tuple graph see: > > http://logs.test-lab.xenproject.org/osstest/results-adhoc/bisect/xen-unstabl > e/test-amd64-amd64-qemuu-nested-intel.debian-hvm-install--l1--l2.html > Revision IDs in each graph node refer, respectively, to the Trees above. > > ---------------------------------------- > Running cs-bisection-step > --graph-out=/home/logs/results-adhoc/bisect/xen-unstable/test-amd64-am > d64-qemuu-nested-intel.debian-hvm-install--l1--l2 > --summary-out=tmp/65388.bisection-summary > --blessings=real,real-bisect,adhoc-bisect --basis-template=65301 > --basis-flight=65301 xen-unstable test-amd64-amd64-qemuu-nested-intel > debian-hvm-install/l1/l2 > Searching for failure / basis pass: > 65314 fail [host=godello1] / template as basis? using template as basis. > Failure / basis pass flights: 65314 / 65301 > (tree with no url: ovmf) > (tree with no url: seabios) > Tree: linux git://xenbits.xen.org/linux-pvops.git > Tree: linuxfirmware git://xenbits.xen.org/osstest/linux-firmware.git > Tree: qemu git://xenbits.xen.org/qemu-xen-traditional.git > Tree: qemuu git://xenbits.xen.org/qemu-xen.git > Tree: xen git://xenbits.xen.org/xen.git > Latest 769b79eb206ad5b0249a08665fefb913c3d1998e > c530a75c1e6a472b0eb9558310b518f0dfcd8860 > bc00cad75d8bcc3ba696992bec219c21db8406aa > 3fb401edbd8e9741c611bfddf6a2032ca91f55ed > 2c4f313a7e62c7e559a469d4af4c3d03c49afa43 > Basis pass 1230ae0e99e05ced8a945a1a2c5762ce5c6c97c9 > c530a75c1e6a472b0eb9558310b518f0dfcd8860 > bc00cad75d8bcc3ba696992bec219c21db8406aa > 816609b2841297925a223ec377c336360e044ee5 > d07f63fa6e70350b23e7acbde06129247c4e655d > Generating revisions with ./adhoc-revtuple-generator > git://xenbits.xen.org/linux-pvops.git#1230ae0e99e05ced8a945a1a2c5762ce > 5c6c97c9-769b79eb206ad5b0249a08665fefb913c3d1998e > git://xenbits.xen.org/osstest/linux-firmware.git#c530a75c1e6a472b0eb955 > 8310b518f0dfcd8860-c530a75c1e6a472b0eb9558310b518f0dfcd8860 > git://xenbits.xen.org/qemu-xen-traditional.git#bc00cad75d8bcc3ba696992b > ec219c21db8406aa-bc00cad75d8bcc3ba696992bec219c21db8406aa > git://xenbits.xen.org/qemu-xen.git#816609b2841297925a223ec377c336360 > e044ee5-3fb401edbd8e9741c611bfddf6a2032ca91f55ed > git://xenbits.xen.org/xen.git#d07f63fa6e70350b23e7acbde06129247c4e655 > d-2c4f313a7e62c7e559a469d4af4c3d03c49afa43 > Loaded 17133 nodes in revision graph > Searching for test results: > 65114 [host=italia0] > 65141 fail 769b79eb206ad5b0249a08665fefb913c3d1998e > c530a75c1e6a472b0eb9558310b518f0dfcd8860 > bc00cad75d8bcc3ba696992bec219c21db8406aa > 3fb401edbd8e9741c611bfddf6a2032ca91f55ed > b1d398b67781140d1c6efd05778d0ad4103b2a32 > 65162 fail 769b79eb206ad5b0249a08665fefb913c3d1998e > c530a75c1e6a472b0eb9558310b518f0dfcd8860 > bc00cad75d8bcc3ba696992bec219c21db8406aa > 3fb401edbd8e9741c611bfddf6a2032ca91f55ed > 713b7e4ef2aa4ec3ae697cde9c81d5a57548f9b1 > 65164 fail 769b79eb206ad5b0249a08665fefb913c3d1998e > c530a75c1e6a472b0eb9558310b518f0dfcd8860 > bc00cad75d8bcc3ba696992bec219c21db8406aa > 3fb401edbd8e9741c611bfddf6a2032ca91f55ed > b1d398b67781140d1c6efd05778d0ad4103b2a32 > 65186 fail 769b79eb206ad5b0249a08665fefb913c3d1998e > c530a75c1e6a472b0eb9558310b518f0dfcd8860 > bc00cad75d8bcc3ba696992bec219c21db8406aa > 3fb401edbd8e9741c611bfddf6a2032ca91f55ed > b1d398b67781140d1c6efd05778d0ad4103b2a32 > 65217 fail 769b79eb206ad5b0249a08665fefb913c3d1998e > c530a75c1e6a472b0eb9558310b518f0dfcd8860 > bc00cad75d8bcc3ba696992bec219c21db8406aa > 3fb401edbd8e9741c611bfddf6a2032ca91f55ed > b1d398b67781140d1c6efd05778d0ad4103b2a32 > 65233 fail 769b79eb206ad5b0249a08665fefb913c3d1998e > c530a75c1e6a472b0eb9558310b518f0dfcd8860 > bc00cad75d8bcc3ba696992bec219c21db8406aa > 3fb401edbd8e9741c611bfddf6a2032ca91f55ed > b1d398b67781140d1c6efd05778d0ad4103b2a32 > 65287 fail 769b79eb206ad5b0249a08665fefb913c3d1998e > c530a75c1e6a472b0eb9558310b518f0dfcd8860 > bc00cad75d8bcc3ba696992bec219c21db8406aa > 3fb401edbd8e9741c611bfddf6a2032ca91f55ed > c8eb0ec277ae387e78d685523e0fee633e46f046 > 65333 pass 1230ae0e99e05ced8a945a1a2c5762ce5c6c97c9 > c530a75c1e6a472b0eb9558310b518f0dfcd8860 > bc00cad75d8bcc3ba696992bec219c21db8406aa > 816609b2841297925a223ec377c336360e044ee5 > d07f63fa6e70350b23e7acbde06129247c4e655d > 65314 fail 769b79eb206ad5b0249a08665fefb913c3d1998e > c530a75c1e6a472b0eb9558310b518f0dfcd8860 > bc00cad75d8bcc3ba696992bec219c21db8406aa > 3fb401edbd8e9741c611bfddf6a2032ca91f55ed > 2c4f313a7e62c7e559a469d4af4c3d03c49afa43 > 65354 pass 769b79eb206ad5b0249a08665fefb913c3d1998e > c530a75c1e6a472b0eb9558310b518f0dfcd8860 > bc00cad75d8bcc3ba696992bec219c21db8406aa > 816609b2841297925a223ec377c336360e044ee5 > c35eefded2992fc9b979f99190422527650872fd > 65267 fail 769b79eb206ad5b0249a08665fefb913c3d1998e > c530a75c1e6a472b0eb9558310b518f0dfcd8860 > bc00cad75d8bcc3ba696992bec219c21db8406aa > 3fb401edbd8e9741c611bfddf6a2032ca91f55ed > 4c6cd64519f9bc270a7278128c94e4b66e3d2077 > 65301 pass 1230ae0e99e05ced8a945a1a2c5762ce5c6c97c9 > c530a75c1e6a472b0eb9558310b518f0dfcd8860 > bc00cad75d8bcc3ba696992bec219c21db8406aa > 816609b2841297925a223ec377c336360e044ee5 > d07f63fa6e70350b23e7acbde06129247c4e655d > 65351 fail 769b79eb206ad5b0249a08665fefb913c3d1998e > c530a75c1e6a472b0eb9558310b518f0dfcd8860 > bc00cad75d8bcc3ba696992bec219c21db8406aa > 3fb401edbd8e9741c611bfddf6a2032ca91f55ed > 0ba0faa7f02fb310a330579adeef3534e6eca5a8 > 65343 pass 997badf1e03dfd39854094f7767e1a4cf5ed310b > c530a75c1e6a472b0eb9558310b518f0dfcd8860 > bc00cad75d8bcc3ba696992bec219c21db8406aa > 816609b2841297925a223ec377c336360e044ee5 > d07f63fa6e70350b23e7acbde06129247c4e655d > 65325 fail 769b79eb206ad5b0249a08665fefb913c3d1998e > c530a75c1e6a472b0eb9558310b518f0dfcd8860 > bc00cad75d8bcc3ba696992bec219c21db8406aa > 3fb401edbd8e9741c611bfddf6a2032ca91f55ed > c8eb0ec277ae387e78d685523e0fee633e46f046 > 65345 pass e7a10d9297c1abfd27138d86a13f7b6435634a46 > c530a75c1e6a472b0eb9558310b518f0dfcd8860 > bc00cad75d8bcc3ba696992bec219c21db8406aa > 816609b2841297925a223ec377c336360e044ee5 > d07f63fa6e70350b23e7acbde06129247c4e655d > 65341 fail 769b79eb206ad5b0249a08665fefb913c3d1998e > c530a75c1e6a472b0eb9558310b518f0dfcd8860 > bc00cad75d8bcc3ba696992bec219c21db8406aa > 3fb401edbd8e9741c611bfddf6a2032ca91f55ed > 2c4f313a7e62c7e559a469d4af4c3d03c49afa43 > 65342 fail 769b79eb206ad5b0249a08665fefb913c3d1998e > c530a75c1e6a472b0eb9558310b518f0dfcd8860 > bc00cad75d8bcc3ba696992bec219c21db8406aa > 3fb401edbd8e9741c611bfddf6a2032ca91f55ed > c87303c04738b0e837da6e891eb561de0bf1b64e > 65362 fail 769b79eb206ad5b0249a08665fefb913c3d1998e > c530a75c1e6a472b0eb9558310b518f0dfcd8860 > bc00cad75d8bcc3ba696992bec219c21db8406aa > 3fb401edbd8e9741c611bfddf6a2032ca91f55ed > d02e84b9d9d16b6b56186f0dfdcb3c90b83c82a3 > 65367 pass 769b79eb206ad5b0249a08665fefb913c3d1998e > c530a75c1e6a472b0eb9558310b518f0dfcd8860 > bc00cad75d8bcc3ba696992bec219c21db8406aa > 816609b2841297925a223ec377c336360e044ee5 > 827db7b26384ce083df7154d77f13379b2cf4121 > 65370 pass 769b79eb206ad5b0249a08665fefb913c3d1998e > c530a75c1e6a472b0eb9558310b518f0dfcd8860 > bc00cad75d8bcc3ba696992bec219c21db8406aa > 3fb401edbd8e9741c611bfddf6a2032ca91f55ed > 827db7b26384ce083df7154d77f13379b2cf4121 > 65374 pass 769b79eb206ad5b0249a08665fefb913c3d1998e > c530a75c1e6a472b0eb9558310b518f0dfcd8860 > bc00cad75d8bcc3ba696992bec219c21db8406aa > 3fb401edbd8e9741c611bfddf6a2032ca91f55ed > 3b47431691409004c7218f6a6ba5c9c0bcf483ea > 65379 fail 769b79eb206ad5b0249a08665fefb913c3d1998e > c530a75c1e6a472b0eb9558310b518f0dfcd8860 > bc00cad75d8bcc3ba696992bec219c21db8406aa > 3fb401edbd8e9741c611bfddf6a2032ca91f55ed > d02e84b9d9d16b6b56186f0dfdcb3c90b83c82a3 > 65380 pass 769b79eb206ad5b0249a08665fefb913c3d1998e > c530a75c1e6a472b0eb9558310b518f0dfcd8860 > bc00cad75d8bcc3ba696992bec219c21db8406aa > 3fb401edbd8e9741c611bfddf6a2032ca91f55ed > 3b47431691409004c7218f6a6ba5c9c0bcf483ea > 65383 fail 769b79eb206ad5b0249a08665fefb913c3d1998e > c530a75c1e6a472b0eb9558310b518f0dfcd8860 > bc00cad75d8bcc3ba696992bec219c21db8406aa > 3fb401edbd8e9741c611bfddf6a2032ca91f55ed > d02e84b9d9d16b6b56186f0dfdcb3c90b83c82a3 > 65384 pass 769b79eb206ad5b0249a08665fefb913c3d1998e > c530a75c1e6a472b0eb9558310b518f0dfcd8860 > bc00cad75d8bcc3ba696992bec219c21db8406aa > 3fb401edbd8e9741c611bfddf6a2032ca91f55ed > 3b47431691409004c7218f6a6ba5c9c0bcf483ea > 65388 fail 769b79eb206ad5b0249a08665fefb913c3d1998e > c530a75c1e6a472b0eb9558310b518f0dfcd8860 > bc00cad75d8bcc3ba696992bec219c21db8406aa > 3fb401edbd8e9741c611bfddf6a2032ca91f55ed > d02e84b9d9d16b6b56186f0dfdcb3c90b83c82a3 > Searching for interesting versions > Result found: flight 65301 (pass), for basis pass > Result found: flight 65314 (fail), for basis failure > Repro found: flight 65333 (pass), for basis pass > Repro found: flight 65341 (fail), for basis failure > 0 revisions at 769b79eb206ad5b0249a08665fefb913c3d1998e > c530a75c1e6a472b0eb9558310b518f0dfcd8860 > bc00cad75d8bcc3ba696992bec219c21db8406aa > 3fb401edbd8e9741c611bfddf6a2032ca91f55ed > 3b47431691409004c7218f6a6ba5c9c0bcf483ea > No revisions left to test, checking graph state. > Result found: flight 65374 (pass), for last pass > Result found: flight 65379 (fail), for first failure > Repro found: flight 65380 (pass), for last pass > Repro found: flight 65383 (fail), for first failure > Repro found: flight 65384 (pass), for last pass > Repro found: flight 65388 (fail), for first failure > > *** Found and reproduced problem changeset *** > > Bug is in tree: xen git://xenbits.xen.org/xen.git > Bug introduced: d02e84b9d9d16b6b56186f0dfdcb3c90b83c82a3 > Bug not present: 3b47431691409004c7218f6a6ba5c9c0bcf483ea > Last fail repro: http://logs.test-lab.xenproject.org/osstest/logs/65388/ > > > commit d02e84b9d9d16b6b56186f0dfdcb3c90b83c82a3 > Author: Jan Beulich > Date: Tue Nov 24 12:07:27 2015 +0100 > > vVMX: use latched VMCS machine address > > Instead of calling domain_page_map_to_mfn() over and over, latch > the > guest VMCS machine address unconditionally (i.e. independent of > whether > VMCS shadowing is supported by the hardware). > > Since this requires altering the parameters of __[gs]et_vmcs{,_real}() > (and hence all their callers) anyway, take the opportunity to also drop > the bogus double underscores from their names (and from > __[gs]et_vmcs_virtual() as well). > > Signed-off-by: Jan Beulich > Acked-by: Kevin Tian > > Revision graph left in > /home/logs/results-adhoc/bisect/xen-unstable/test-amd64-amd64-qemuu-n > ested-intel.debian-hvm-install--l1--l2.{dot,ps,png,html,svg}. > ---------------------------------------- > 65388: tolerable ALL FAIL > > flight 65388 xen-unstable adhoc-bisect [real] > http://logs.test-lab.xenproject.org/osstest/logs/65388/ > > Failures :-/ but no regressions. > > Tests which did not succeed, > including tests which could not be run: > test-amd64-amd64-qemuu-nested-intel 16 debian-hvm-install/l1/l2 fail > baseline untested > > > jobs: > test-amd64-amd64-qemuu-nested-intel fail [Hu, Robert] I tried to reproduce the failure in my environment, with your Qemu/Xen/Dom0 /Seabios/OVMF designation, but failed at even earlier stage -- normal L1 guest installation. xen be: vkbd-0: initialise() failed xen be: vkbd-0: initialise() failed xen be: vkbd-0: initialise() failed Not sure if this related to linux-pvops tree, for I previously using linux-stable tree. For your failure, as Kevin mentioned in other mail, we will find someone to look into. Would you find out the detailed log of ' debian-hvm-install--l1--l2' step? so that he can start analyze, as I cannot reproduce it right now. (I didn't managed to find out the log in your web links) > > > ------------------------------------------------------------ > sg-report-flight on osstest.test-lab.xenproject.org > logs: /home/logs/logs > images: /home/logs/images > > Logs, config files, etc. are available at > http://logs.test-lab.xenproject.org/osstest/logs > > Explanation of these reports, and of osstest in general, is at > > http://xenbits.xen.org/gitweb/?p=osstest.git;a=blob;f=README.email;hb=m > aster > > http://xenbits.xen.org/gitweb/?p=osstest.git;a=blob;f=README;hb=master > > Test harness code can be found at > http://xenbits.xen.org/gitweb?p=osstest.git;a=summary