* [linux-3.18 bisection] complete test-amd64-i386-xl-qemut-debianhvm-amd64
@ 2015-07-27 21:53 osstest service owner
0 siblings, 0 replies; 2+ messages in thread
From: osstest service owner @ 2015-07-27 21:53 UTC (permalink / raw)
To: xen-devel, osstest-admin
branch xen-unstable
xen branch xen-unstable
job test-amd64-i386-xl-qemut-debianhvm-amd64
test guest-saverestore
Tree: linux git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git
Tree: linuxfirmware git://xenbits.xen.org/osstest/linux-firmware.git
Tree: qemu git://xenbits.xen.org/staging/qemu-xen-unstable.git
Tree: qemuu git://xenbits.xen.org/staging/qemu-upstream-unstable.git
Tree: xen git://xenbits.xen.org/xen.git
*** Found and reproduced problem changeset ***
Bug is in tree: xen git://xenbits.xen.org/xen.git
Bug introduced: 3a9ace0147d48af49ffd34628f9510f248f2f588
Bug not present: d9c879039393bb14760966bf7076a2d40d45b124
commit 3a9ace0147d48af49ffd34628f9510f248f2f588
Author: Andrew Cooper <andrew.cooper3@citrix.com>
Date: Fri Jun 12 17:21:41 2015 +0100
tools/libxc+libxl+xl: Restore v2 streams
This is a complicated set of changes which must be done together for
bisectability.
* libxl-save-helper is updated to unconditionally use libxc migration
v2.
* libxl compatibility workarounds in libxc are disabled for restore
operations.
* libxl__stream_read_start() is logically spliced into the event
location where libxl__xc_domain_restore() used to reside.
* Ownership of the save_helper_state moves to stream_read_state.
The parameters 'hvm', 'pae', and 'superpages' were previously
superfluous, and are completely unused in migration
v2. callbacks->toolstack_restore is handled via a migration v2 record
now, rather than via a callback from libxc.
NB: this change breaks Remus. Further untangling needs to happen
before Remus will function.
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Acked-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
CC: Ian Campbell <Ian.Campbell@citrix.com>
CC: Wei Liu <wei.liu2@citrix.com>
---
v4:
* Don't use _init() needlessly
v3:
* Simplify from v2.
* Alter the ownership of save_helper_state
v2:
* Drop "legacy_width" from the IDL
* Gain a LIBXL_HAVE_ to signify support of migration v2 streams
For bisection revision-tuple graph see:
http://logs.test-lab.xenproject.org/osstest/results/bisect/linux-3.18/test-amd64-i386-xl-qemut-debianhvm-amd64.guest-saverestore.html
Revision IDs in each graph node refer, respectively, to the Trees above.
----------------------------------------
Searching for failure / basis pass:
59825 fail [host=merlot0] / 59807 [host=huxelrebe0] 59785 [host=huxelrebe1] 59766 [host=elbling1] 59697 [host=fiano1] 59665 [host=elbling0] 59640 [host=italia0] 59623 [host=chardonnay0] 59604 [host=fiano0] 59587 [host=elbling0] 59564 [host=italia1] 59520 [host=elbling1] 59474 [host=merlot1] 59452 [host=chardonnay1] 59412 [host=rimava1] 59319 [host=fiano1] 59222 [host=italia0] 59177 [host=huxelrebe0] 59117 [host=huxelrebe1] 59075 [host=pinot1] 59050 [host=chardonnay0] 59041 [host=elbling0] 59015 [host=fiano0] 59001 [host=italia1] 58987 [host=italia0] 58976 [host=elbling1] 58558 [host=pinot1] 58524 [host=italia1] 58402 [host=fiano1] 58355 [host=fiano0] 58306 [host=chardonnay1] 58263 [host=chardonnay0] 58222 [host=huxelrebe1] 58185 [host=italia0] 58146 [host=merlot1] 58111 [host=huxelrebe0]
58064 [host=rimava0] 57968 [host=rimava1] 57904 [host=elbling0] 57853 [host=elbling1] 57788 ok.
Failure / basis pass flights: 59825 / 57788
(tree with no url: ovmf)
(tree with no url: seabios)
Tree: linux git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git
Tree: linuxfirmware git://xenbits.xen.org/osstest/linux-firmware.git
Tree: qemu git://xenbits.xen.org/staging/qemu-xen-unstable.git
Tree: qemuu git://xenbits.xen.org/staging/qemu-upstream-unstable.git
Tree: xen git://xenbits.xen.org/xen.git
Latest 22a6cbf9f36ee3ae2878efcbdde33e6ca00b9c4b c530a75c1e6a472b0eb9558310b518f0dfcd8860 3e2e51ecc1120bd59537ed19b6bc7066511c7e2e c4a962ec0c61aa9b860a3635c8424472e6c2cc2c 21d9b079e53805b68047d60d28cde224d09bbb40
Basis pass 51af817611f2c0987030d024f24fc7ea95dd33e6 c530a75c1e6a472b0eb9558310b518f0dfcd8860 4de1422ea306832b6ef2cba34e9febb73dd139a7 b2da824bc5ad35fb9f1e74a203d7be96a7b0345e d6b6bd8374ac30597495d457829ce7ad6e8b7016
Generating revisions with ./adhoc-revtuple-generator git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git#51af817611f2c0987030d024f24fc7ea95dd33e6-22a6cbf9f36ee3ae2878efcbdde33e6ca00b9c4b git://xenbits.xen.org/osstest/linux-firmware.git#c530a75c1e6a472b0eb9558310b518f0dfcd8860-c530a75c1e6a472b0eb9558310b518f0dfcd8860 git://xenbits.xen.org/staging/qemu-xen-unstable.git#4de1422ea306832b6ef2cba34e9febb73dd139a7-3e2e51ecc1120bd59537ed19b6bc7066511c7e2e git://xenbits.xen.org/staging/qemu-upstream-unstable.git#b2da824bc5ad35fb9f1e74a203d7be96a7b0345e-c4a962ec0c61aa9b860a3635c8424472e6c2cc2c git://xenbits.xen.org/xen.git#d6b6bd8374ac30597495d457829ce7ad6e8b7016-21d9b079e53805b68047d60d28cde224d09bbb40
+ exec
+ sh -xe
+ cd /home/osstest/repos/linux-stable
+ git remote set-url origin git://cache:9419/git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git
+ git fetch -p origin +refs/heads/*:refs/remotes/origin/*
+ exec
+ sh -xe
+ cd /home/osstest/repos/qemu-xen-unstable
+ git remote set-url origin git://cache:9419/git://xenbits.xen.org/staging/qemu-xen-unstable.git
+ git fetch -p origin +refs/heads/*:refs/remotes/origin/*
+ exec
+ sh -xe
+ cd /home/osstest/repos/qemu-upstream-unstable
+ git remote set-url origin git://cache:9419/git://xenbits.xen.org/staging/qemu-upstream-unstable.git
+ git fetch -p origin +refs/heads/*:refs/remotes/origin/*
+ exec
+ sh -xe
+ cd /home/osstest/repos/xen
+ git remote set-url origin git://cache:9419/git://xenbits.xen.org/xen.git
+ git fetch -p origin +refs/heads/*:refs/remotes/origin/*
+ exec
+ sh -xe
+ cd /home/osstest/repos/linux-stable
+ git remote set-url origin git://cache:9419/git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git
+ git fetch -p origin +refs/heads/*:refs/remotes/origin/*
+ exec
+ sh -xe
+ cd /home/osstest/repos/qemu-xen-unstable
+ git remote set-url origin git://cache:9419/git://xenbits.xen.org/staging/qemu-xen-unstable.git
+ git fetch -p origin +refs/heads/*:refs/remotes/origin/*
+ exec
+ sh -xe
+ cd /home/osstest/repos/qemu-upstream-unstable
+ git remote set-url origin git://cache:9419/git://xenbits.xen.org/staging/qemu-upstream-unstable.git
+ git fetch -p origin +refs/heads/*:refs/remotes/origin/*
+ exec
+ sh -xe
+ cd /home/osstest/repos/xen
+ git remote set-url origin git://cache:9419/git://xenbits.xen.org/xen.git
+ git fetch -p origin +refs/heads/*:refs/remotes/origin/*
Loaded 12457 nodes in revision graph
Searching for test results:
57713 [host=pinot0]
57788 pass 51af817611f2c0987030d024f24fc7ea95dd33e6 c530a75c1e6a472b0eb9558310b518f0dfcd8860 4de1422ea306832b6ef2cba34e9febb73dd139a7 b2da824bc5ad35fb9f1e74a203d7be96a7b0345e d6b6bd8374ac30597495d457829ce7ad6e8b7016
57853 [host=elbling1]
57968 [host=rimava1]
57904 [host=elbling0]
58064 [host=rimava0]
58111 [host=huxelrebe0]
58222 [host=huxelrebe1]
58146 [host=merlot1]
58185 [host=italia0]
58263 [host=chardonnay0]
58306 [host=chardonnay1]
58355 [host=fiano0]
58402 [host=fiano1]
58558 [host=pinot1]
58524 [host=italia1]
58987 [host=italia0]
58976 [host=elbling1]
59001 [host=italia1]
59027 [host=elbling0]
59015 [host=fiano0]
59041 [host=elbling0]
59075 [host=pinot1]
59050 [host=chardonnay0]
59117 [host=huxelrebe1]
59177 [host=huxelrebe0]
59222 [host=italia0]
59319 [host=fiano1]
59412 [host=rimava1]
59452 [host=chardonnay1]
59474 [host=merlot1]
59520 [host=elbling1]
59564 [host=italia1]
59604 [host=fiano0]
59587 [host=elbling0]
59640 [host=italia0]
59623 [host=chardonnay0]
59665 [host=elbling0]
59734 []
59697 [host=fiano1]
59757 []
59785 [host=huxelrebe1]
59766 [host=elbling1]
59746 []
59825 fail 22a6cbf9f36ee3ae2878efcbdde33e6ca00b9c4b c530a75c1e6a472b0eb9558310b518f0dfcd8860 3e2e51ecc1120bd59537ed19b6bc7066511c7e2e c4a962ec0c61aa9b860a3635c8424472e6c2cc2c 21d9b079e53805b68047d60d28cde224d09bbb40
59807 [host=huxelrebe0]
59902 blocked c46ed6527b0fbebabb494a648e1d8ec0dee8e0d8 c530a75c1e6a472b0eb9558310b518f0dfcd8860 3e2e51ecc1120bd59537ed19b6bc7066511c7e2e c4a962ec0c61aa9b860a3635c8424472e6c2cc2c d26cf404befff4f39ad095d6b03759c807b2b1fe
59866 pass 51af817611f2c0987030d024f24fc7ea95dd33e6 c530a75c1e6a472b0eb9558310b518f0dfcd8860 4de1422ea306832b6ef2cba34e9febb73dd139a7 b2da824bc5ad35fb9f1e74a203d7be96a7b0345e d6b6bd8374ac30597495d457829ce7ad6e8b7016
59881 fail 22a6cbf9f36ee3ae2878efcbdde33e6ca00b9c4b c530a75c1e6a472b0eb9558310b518f0dfcd8860 3e2e51ecc1120bd59537ed19b6bc7066511c7e2e c4a962ec0c61aa9b860a3635c8424472e6c2cc2c 21d9b079e53805b68047d60d28cde224d09bbb40
59885 blocked a3759241250e4ef7872ac0727a3c2b8d6f379f8f c530a75c1e6a472b0eb9558310b518f0dfcd8860 3e2e51ecc1120bd59537ed19b6bc7066511c7e2e c4a962ec0c61aa9b860a3635c8424472e6c2cc2c 27c5541986667a728a1bf54c63ede8796aab79d8
59929 pass 866cebe251f4fb2b435f4ecfe6d3bb4025938533 c530a75c1e6a472b0eb9558310b518f0dfcd8860 3e2e51ecc1120bd59537ed19b6bc7066511c7e2e c4a962ec0c61aa9b860a3635c8424472e6c2cc2c 4012b9a4660e2db686d0592fc91318b7fd89b3de
60002 pass 71e331989c7b5bdf5b910d718ce206f431323039 c530a75c1e6a472b0eb9558310b518f0dfcd8860 3e2e51ecc1120bd59537ed19b6bc7066511c7e2e c4a962ec0c61aa9b860a3635c8424472e6c2cc2c d9c879039393bb14760966bf7076a2d40d45b124
59906 pass 90b934b19c15a1f1a8140e93af719cb741e4cf40 c530a75c1e6a472b0eb9558310b518f0dfcd8860 3e2e51ecc1120bd59537ed19b6bc7066511c7e2e c4a962ec0c61aa9b860a3635c8424472e6c2cc2c 5f33fa2bca6354fad1decfeda723c046311e85cc
59994 fail 71e331989c7b5bdf5b910d718ce206f431323039 c530a75c1e6a472b0eb9558310b518f0dfcd8860 3e2e51ecc1120bd59537ed19b6bc7066511c7e2e c4a962ec0c61aa9b860a3635c8424472e6c2cc2c 7c4027c6a0bddcde74df2bf5c16421f2cbb19971
59915 pass 3ca9f5f9f498a7db78949c9573d95de24fcfde73 c530a75c1e6a472b0eb9558310b518f0dfcd8860 38609ae72b0a9e09b42be94f469fef928a1049fa 579e90432e995d6cb6f8520aca557fa6646f94ec a622b5ade2bdf79ad95e6088a4041e75253c43f3
59919 pass e479bdcf6cd952ee6b54d428d31a027d1c66d7ac c530a75c1e6a472b0eb9558310b518f0dfcd8860 3e2e51ecc1120bd59537ed19b6bc7066511c7e2e c4a962ec0c61aa9b860a3635c8424472e6c2cc2c c40317f11b3f05e7c06a2213560c8471081f2662
59953 fail 71e331989c7b5bdf5b910d718ce206f431323039 c530a75c1e6a472b0eb9558310b518f0dfcd8860 3e2e51ecc1120bd59537ed19b6bc7066511c7e2e c4a962ec0c61aa9b860a3635c8424472e6c2cc2c a8bc99b981c5ad773bd646f5986e616d26fb94d7
59971 fail 71e331989c7b5bdf5b910d718ce206f431323039 c530a75c1e6a472b0eb9558310b518f0dfcd8860 3e2e51ecc1120bd59537ed19b6bc7066511c7e2e c4a962ec0c61aa9b860a3635c8424472e6c2cc2c 681ce1681622a46d111cfdc4fc07e4cb565ae131
59924 pass d5ced3d143593bf844f5d23fed9fd9d3fdb5b083 c530a75c1e6a472b0eb9558310b518f0dfcd8860 3e2e51ecc1120bd59537ed19b6bc7066511c7e2e c4a962ec0c61aa9b860a3635c8424472e6c2cc2c b9dbe33d15a038500bcc3226a3ca31ee215122cd
59938 pass 71e331989c7b5bdf5b910d718ce206f431323039 c530a75c1e6a472b0eb9558310b518f0dfcd8860 3e2e51ecc1120bd59537ed19b6bc7066511c7e2e c4a962ec0c61aa9b860a3635c8424472e6c2cc2c d3ea9652585877a1431486786281a178b977c6b3
59998 fail 71e331989c7b5bdf5b910d718ce206f431323039 c530a75c1e6a472b0eb9558310b518f0dfcd8860 3e2e51ecc1120bd59537ed19b6bc7066511c7e2e c4a962ec0c61aa9b860a3635c8424472e6c2cc2c 3a9ace0147d48af49ffd34628f9510f248f2f588
59982 fail 71e331989c7b5bdf5b910d718ce206f431323039 c530a75c1e6a472b0eb9558310b518f0dfcd8860 3e2e51ecc1120bd59537ed19b6bc7066511c7e2e c4a962ec0c61aa9b860a3635c8424472e6c2cc2c 7eaec00dd938087e4ce21a9fb06f2be44fc41945
59985 pass 71e331989c7b5bdf5b910d718ce206f431323039 c530a75c1e6a472b0eb9558310b518f0dfcd8860 3e2e51ecc1120bd59537ed19b6bc7066511c7e2e c4a962ec0c61aa9b860a3635c8424472e6c2cc2c d9c879039393bb14760966bf7076a2d40d45b124
60016 fail 71e331989c7b5bdf5b910d718ce206f431323039 c530a75c1e6a472b0eb9558310b518f0dfcd8860 3e2e51ecc1120bd59537ed19b6bc7066511c7e2e c4a962ec0c61aa9b860a3635c8424472e6c2cc2c 3a9ace0147d48af49ffd34628f9510f248f2f588
60031 pass 71e331989c7b5bdf5b910d718ce206f431323039 c530a75c1e6a472b0eb9558310b518f0dfcd8860 3e2e51ecc1120bd59537ed19b6bc7066511c7e2e c4a962ec0c61aa9b860a3635c8424472e6c2cc2c d9c879039393bb14760966bf7076a2d40d45b124
60037 fail 71e331989c7b5bdf5b910d718ce206f431323039 c530a75c1e6a472b0eb9558310b518f0dfcd8860 3e2e51ecc1120bd59537ed19b6bc7066511c7e2e c4a962ec0c61aa9b860a3635c8424472e6c2cc2c 3a9ace0147d48af49ffd34628f9510f248f2f588
Searching for interesting versions
Result found: flight 57788 (pass), for basis pass
Result found: flight 59825 (fail), for basis failure
Repro found: flight 59866 (pass), for basis pass
Repro found: flight 59881 (fail), for basis failure
0 revisions at 71e331989c7b5bdf5b910d718ce206f431323039 c530a75c1e6a472b0eb9558310b518f0dfcd8860 3e2e51ecc1120bd59537ed19b6bc7066511c7e2e c4a962ec0c61aa9b860a3635c8424472e6c2cc2c d9c879039393bb14760966bf7076a2d40d45b124
No revisions left to test, checking graph state.
Result found: flight 59985 (pass), for last pass
Result found: flight 59998 (fail), for first failure
Repro found: flight 60002 (pass), for last pass
Repro found: flight 60016 (fail), for first failure
Repro found: flight 60031 (pass), for last pass
Repro found: flight 60037 (fail), for first failure
*** Found and reproduced problem changeset ***
Bug is in tree: xen git://xenbits.xen.org/xen.git
Bug introduced: 3a9ace0147d48af49ffd34628f9510f248f2f588
Bug not present: d9c879039393bb14760966bf7076a2d40d45b124
+ exec
+ sh -xe
+ cd /home/osstest/repos/xen
+ git remote set-url origin git://cache:9419/git://xenbits.xen.org/xen.git
+ git fetch -p origin +refs/heads/*:refs/remotes/origin/*
commit 3a9ace0147d48af49ffd34628f9510f248f2f588
Author: Andrew Cooper <andrew.cooper3@citrix.com>
Date: Fri Jun 12 17:21:41 2015 +0100
tools/libxc+libxl+xl: Restore v2 streams
This is a complicated set of changes which must be done together for
bisectability.
* libxl-save-helper is updated to unconditionally use libxc migration
v2.
* libxl compatibility workarounds in libxc are disabled for restore
operations.
* libxl__stream_read_start() is logically spliced into the event
location where libxl__xc_domain_restore() used to reside.
* Ownership of the save_helper_state moves to stream_read_state.
The parameters 'hvm', 'pae', and 'superpages' were previously
superfluous, and are completely unused in migration
v2. callbacks->toolstack_restore is handled via a migration v2 record
now, rather than via a callback from libxc.
NB: this change breaks Remus. Further untangling needs to happen
before Remus will function.
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Acked-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
CC: Ian Campbell <Ian.Campbell@citrix.com>
CC: Wei Liu <wei.liu2@citrix.com>
---
v4:
* Don't use _init() needlessly
v3:
* Simplify from v2.
* Alter the ownership of save_helper_state
v2:
* Drop "legacy_width" from the IDL
* Gain a LIBXL_HAVE_ to signify support of migration v2 streams
dot: graph is too large for cairo-renderer bitmaps. Scaling by 0.588065 to fit
pnmtopng: 105 colors found
Revision graph left in /home/logs/results/bisect/linux-3.18/test-amd64-i386-xl-qemut-debianhvm-amd64.guest-saverestore.{dot,ps,png,html}.
----------------------------------------
60037: tolerable ALL FAIL
flight 60037 linux-3.18 real-bisect [real]
http://logs.test-lab.xenproject.org/osstest/logs/60037/
Failures :-/ but no regressions.
Tests which did not succeed,
including tests which could not be run:
test-amd64-i386-xl-qemut-debianhvm-amd64 11 guest-saverestore fail baseline untested
jobs:
test-amd64-i386-xl-qemut-debianhvm-amd64 fail
------------------------------------------------------------
sg-report-flight on osstest.test-lab.xenproject.org
logs: /home/logs/logs
images: /home/logs/images
Logs, config files, etc. are available at
http://logs.test-lab.xenproject.org/osstest/logs
Explanation of these reports, and of osstest in general, is at
http://xenbits.xen.org/gitweb/?p=osstest.git;a=blob;f=README.email;hb=master
http://xenbits.xen.org/gitweb/?p=osstest.git;a=blob;f=README;hb=master
Test harness code can be found at
http://xenbits.xen.org/gitweb?p=osstest.git;a=summary
^ permalink raw reply [flat|nested] 2+ messages in thread
* [linux-3.18 bisection] complete test-amd64-i386-xl-qemut-debianhvm-amd64
@ 2016-07-19 20:29 osstest service owner
0 siblings, 0 replies; 2+ messages in thread
From: osstest service owner @ 2016-07-19 20:29 UTC (permalink / raw)
To: xen-devel, osstest-admin
branch xen-unstable
xenbranch xen-unstable
job test-amd64-i386-xl-qemut-debianhvm-amd64
testid debian-hvm-install
Tree: linux git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git
Tree: linuxfirmware git://xenbits.xen.org/osstest/linux-firmware.git
Tree: qemu git://xenbits.xen.org/qemu-xen-traditional.git
Tree: qemuu git://xenbits.xen.org/qemu-xen.git
Tree: xen git://xenbits.xen.org/xen.git
*** Found and reproduced problem changeset ***
Bug is in tree: linux git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git
Bug introduced: a2d8c514753276394d68414f563591f174ef86cb
Bug not present: 8f620446135b64ca6f96cf32066a76d64e79a388
Last fail repro: http://logs.test-lab.xenproject.org/osstest/logs/97669/
commit a2d8c514753276394d68414f563591f174ef86cb
Author: Lukasz Odzioba <lukasz.odzioba@intel.com>
Date: Fri Jun 24 14:50:01 2016 -0700
mm/swap.c: flush lru pvecs on compound page arrival
[ Upstream commit 8f182270dfec432e93fae14f9208a6b9af01009f ]
Currently we can have compound pages held on per cpu pagevecs, which
leads to a lot of memory unavailable for reclaim when needed. In the
systems with hundreads of processors it can be GBs of memory.
On of the way of reproducing the problem is to not call munmap
explicitly on all mapped regions (i.e. after receiving SIGTERM). After
that some pages (with THP enabled also huge pages) may end up on
lru_add_pvec, example below.
void main() {
#pragma omp parallel
{
size_t size = 55 * 1000 * 1000; // smaller than MEM/CPUS
void *p = mmap(NULL, size, PROT_READ | PROT_WRITE,
MAP_PRIVATE | MAP_ANONYMOUS , -1, 0);
if (p != MAP_FAILED)
memset(p, 0, size);
//munmap(p, size); // uncomment to make the problem go away
}
}
When we run it with THP enabled it will leave significant amount of
memory on lru_add_pvec. This memory will be not reclaimed if we hit
OOM, so when we run above program in a loop:
for i in `seq 100`; do ./a.out; done
many processes (95% in my case) will be killed by OOM.
The primary point of the LRU add cache is to save the zone lru_lock
contention with a hope that more pages will belong to the same zone and
so their addition can be batched. The huge page is already a form of
batched addition (it will add 512 worth of memory in one go) so skipping
the batching seems like a safer option when compared to a potential
excess in the caching which can be quite large and much harder to fix
because lru_add_drain_all is way to expensive and it is not really clear
what would be a good moment to call it.
Similarly we can reproduce the problem on lru_deactivate_pvec by adding:
madvise(p, size, MADV_FREE); after memset.
This patch flushes lru pvecs on compound page arrival making the problem
less severe - after applying it kill rate of above example drops to 0%,
due to reducing maximum amount of memory held on pvec from 28MB (with
THP) to 56kB per CPU.
Suggested-by: Michal Hocko <mhocko@suse.com>
Link: http://lkml.kernel.org/r/1466180198-18854-1-git-send-email-lukasz.odzioba@intel.com
Signed-off-by: Lukasz Odzioba <lukasz.odzioba@intel.com>
Acked-by: Michal Hocko <mhocko@suse.com>
Cc: Kirill Shutemov <kirill.shutemov@linux.intel.com>
Cc: Andrea Arcangeli <aarcange@redhat.com>
Cc: Vladimir Davydov <vdavydov@parallels.com>
Cc: Ming Li <mingli199x@qq.com>
Cc: Minchan Kim <minchan@kernel.org>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Sasha Levin <sasha.levin@oracle.com>
For bisection revision-tuple graph see:
http://logs.test-lab.xenproject.org/osstest/results/bisect/linux-3.18/test-amd64-i386-xl-qemut-debianhvm-amd64.debian-hvm-install.html
Revision IDs in each graph node refer, respectively, to the Trees above.
----------------------------------------
Running cs-bisection-step --graph-out=/home/logs/results/bisect/linux-3.18/test-amd64-i386-xl-qemut-debianhvm-amd64.debian-hvm-install --summary-out=tmp/97669.bisection-summary --basis-template=96188 --blessings=real,real-bisect linux-3.18 test-amd64-i386-xl-qemut-debianhvm-amd64 debian-hvm-install
Searching for failure / basis pass:
97592 fail [host=huxelrebe0] / 96188 [host=huxelrebe1] 96161 [host=chardonnay1] 95844 [host=baroque1] 95809 [host=pinot1] 95597 ok.
Failure / basis pass flights: 97592 / 95597
(tree with no url: minios)
(tree with no url: ovmf)
(tree with no url: seabios)
Tree: linux git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git
Tree: linuxfirmware git://xenbits.xen.org/osstest/linux-firmware.git
Tree: qemu git://xenbits.xen.org/qemu-xen-traditional.git
Tree: qemuu git://xenbits.xen.org/qemu-xen.git
Tree: xen git://xenbits.xen.org/xen.git
Latest 0ac0a856d986c1ab240753479f5e50fdfab82b14 c530a75c1e6a472b0eb9558310b518f0dfcd8860 6e20809727261599e8527c456eb078c0e89139a1 44a072f0de0d57c95c2212bbce02888832b7b74f b48be35ac86cd6369124cf06ca3006d086095297
Basis pass b5076139991c6b12c62346d9880eec1d4227d99f c530a75c1e6a472b0eb9558310b518f0dfcd8860 df553c056104e3dd8a2bd2e72539a57c4c085bae 44a072f0de0d57c95c2212bbce02888832b7b74f c2a17869d5dcd845d646bf4db122cad73596a2be
Generating revisions with ./adhoc-revtuple-generator git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git#b5076139991c6b12c62346d9880eec1d4227d99f-0ac0a856d986c1ab240753479f5e50fdfab82b14 git://xenbits.xen.org/osstest/linux-firmware.git#c530a75c1e6a472b0eb9558310b518f0dfcd8860-c530a75c1e6a472b0eb9558310b518f0dfcd8860 git://xenbits.xen.org/qemu-xen-traditional.git#df553c056104e3dd8a2bd2e72539a57c4c085bae-6e20809727261599e8527c456eb078c0e89139a1 git://xenbits.xen.org/qemu-xen.git#44a072f0de0d57c95c2212bbce02888832b7b74f-44a072f0de0d57c95c2212bbce02888832b7b74f git://xenbits.xen.org/xen.git#c2a17869d5dcd845d646bf4db122cad73596a2be-b48be35ac86cd6369124cf06ca3006d086095297
Loaded 3004 nodes in revision graph
Searching for test results:
95406 [host=elbling1]
95458 [host=rimava1]
95521 [host=italia0]
95597 pass b5076139991c6b12c62346d9880eec1d4227d99f c530a75c1e6a472b0eb9558310b518f0dfcd8860 df553c056104e3dd8a2bd2e72539a57c4c085bae 44a072f0de0d57c95c2212bbce02888832b7b74f c2a17869d5dcd845d646bf4db122cad73596a2be
95809 [host=pinot1]
95844 [host=baroque1]
96161 [host=chardonnay1]
96188 [host=huxelrebe1]
97278 fail irrelevant
97289 fail irrelevant
97319 fail irrelevant
97377 fail irrelevant
97426 fail irrelevant
97533 fail 0ac0a856d986c1ab240753479f5e50fdfab82b14 c530a75c1e6a472b0eb9558310b518f0dfcd8860 6e20809727261599e8527c456eb078c0e89139a1 44a072f0de0d57c95c2212bbce02888832b7b74f b48be35ac86cd6369124cf06ca3006d086095297
97617 pass d259ae2b8a5635dd30148bf76d34c0b421791b5b c530a75c1e6a472b0eb9558310b518f0dfcd8860 6e20809727261599e8527c456eb078c0e89139a1 44a072f0de0d57c95c2212bbce02888832b7b74f 7da483b0236d8974cc97f81780dcf8e559a63175
97640 fail 6ded7184675a9f27e801dc7749a8ccd5d898b4e1 c530a75c1e6a472b0eb9558310b518f0dfcd8860 6e20809727261599e8527c456eb078c0e89139a1 44a072f0de0d57c95c2212bbce02888832b7b74f 22ea8ad02e465e32cd40887c750b55c3a997a288
97656 fail a2d8c514753276394d68414f563591f174ef86cb c530a75c1e6a472b0eb9558310b518f0dfcd8860 6e20809727261599e8527c456eb078c0e89139a1 44a072f0de0d57c95c2212bbce02888832b7b74f 22ea8ad02e465e32cd40887c750b55c3a997a288
97624 fail 848110b885d7003887eb599f247323e4a9ce832e c530a75c1e6a472b0eb9558310b518f0dfcd8860 6e20809727261599e8527c456eb078c0e89139a1 44a072f0de0d57c95c2212bbce02888832b7b74f 22ea8ad02e465e32cd40887c750b55c3a997a288
97668 pass 8f620446135b64ca6f96cf32066a76d64e79a388 c530a75c1e6a472b0eb9558310b518f0dfcd8860 6e20809727261599e8527c456eb078c0e89139a1 44a072f0de0d57c95c2212bbce02888832b7b74f 22ea8ad02e465e32cd40887c750b55c3a997a288
97599 pass b5076139991c6b12c62346d9880eec1d4227d99f c530a75c1e6a472b0eb9558310b518f0dfcd8860 df553c056104e3dd8a2bd2e72539a57c4c085bae 44a072f0de0d57c95c2212bbce02888832b7b74f c2a17869d5dcd845d646bf4db122cad73596a2be
97628 fail 1e7429d49b1cef08bef8c4bd0ef42c3e14164488 c530a75c1e6a472b0eb9558310b518f0dfcd8860 6e20809727261599e8527c456eb078c0e89139a1 44a072f0de0d57c95c2212bbce02888832b7b74f 22ea8ad02e465e32cd40887c750b55c3a997a288
97592 fail 0ac0a856d986c1ab240753479f5e50fdfab82b14 c530a75c1e6a472b0eb9558310b518f0dfcd8860 6e20809727261599e8527c456eb078c0e89139a1 44a072f0de0d57c95c2212bbce02888832b7b74f b48be35ac86cd6369124cf06ca3006d086095297
97612 fail 0ac0a856d986c1ab240753479f5e50fdfab82b14 c530a75c1e6a472b0eb9558310b518f0dfcd8860 6e20809727261599e8527c456eb078c0e89139a1 44a072f0de0d57c95c2212bbce02888832b7b74f b48be35ac86cd6369124cf06ca3006d086095297
97632 pass 5634b6de989d03714ef8c894022c3910095bfc2b c530a75c1e6a472b0eb9558310b518f0dfcd8860 6e20809727261599e8527c456eb078c0e89139a1 44a072f0de0d57c95c2212bbce02888832b7b74f 22ea8ad02e465e32cd40887c750b55c3a997a288
97659 pass 8f620446135b64ca6f96cf32066a76d64e79a388 c530a75c1e6a472b0eb9558310b518f0dfcd8860 6e20809727261599e8527c456eb078c0e89139a1 44a072f0de0d57c95c2212bbce02888832b7b74f 22ea8ad02e465e32cd40887c750b55c3a997a288
97645 fail f1f702e8044c1fb8791111b71b9cb2ff8b9c6e92 c530a75c1e6a472b0eb9558310b518f0dfcd8860 6e20809727261599e8527c456eb078c0e89139a1 44a072f0de0d57c95c2212bbce02888832b7b74f 22ea8ad02e465e32cd40887c750b55c3a997a288
97635 pass 4c2b0216cdf54e81f7c0e841b5bb1116701ae25b c530a75c1e6a472b0eb9558310b518f0dfcd8860 6e20809727261599e8527c456eb078c0e89139a1 44a072f0de0d57c95c2212bbce02888832b7b74f 22ea8ad02e465e32cd40887c750b55c3a997a288
97669 fail a2d8c514753276394d68414f563591f174ef86cb c530a75c1e6a472b0eb9558310b518f0dfcd8860 6e20809727261599e8527c456eb078c0e89139a1 44a072f0de0d57c95c2212bbce02888832b7b74f 22ea8ad02e465e32cd40887c750b55c3a997a288
97652 pass 8f620446135b64ca6f96cf32066a76d64e79a388 c530a75c1e6a472b0eb9558310b518f0dfcd8860 6e20809727261599e8527c456eb078c0e89139a1 44a072f0de0d57c95c2212bbce02888832b7b74f 22ea8ad02e465e32cd40887c750b55c3a997a288
97662 fail a2d8c514753276394d68414f563591f174ef86cb c530a75c1e6a472b0eb9558310b518f0dfcd8860 6e20809727261599e8527c456eb078c0e89139a1 44a072f0de0d57c95c2212bbce02888832b7b74f 22ea8ad02e465e32cd40887c750b55c3a997a288
Searching for interesting versions
Result found: flight 95597 (pass), for basis pass
Result found: flight 97533 (fail), for basis failure
Repro found: flight 97599 (pass), for basis pass
Repro found: flight 97612 (fail), for basis failure
0 revisions at 8f620446135b64ca6f96cf32066a76d64e79a388 c530a75c1e6a472b0eb9558310b518f0dfcd8860 6e20809727261599e8527c456eb078c0e89139a1 44a072f0de0d57c95c2212bbce02888832b7b74f 22ea8ad02e465e32cd40887c750b55c3a997a288
No revisions left to test, checking graph state.
Result found: flight 97652 (pass), for last pass
Result found: flight 97656 (fail), for first failure
Repro found: flight 97659 (pass), for last pass
Repro found: flight 97662 (fail), for first failure
Repro found: flight 97668 (pass), for last pass
Repro found: flight 97669 (fail), for first failure
*** Found and reproduced problem changeset ***
Bug is in tree: linux git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git
Bug introduced: a2d8c514753276394d68414f563591f174ef86cb
Bug not present: 8f620446135b64ca6f96cf32066a76d64e79a388
Last fail repro: http://logs.test-lab.xenproject.org/osstest/logs/97669/
commit a2d8c514753276394d68414f563591f174ef86cb
Author: Lukasz Odzioba <lukasz.odzioba@intel.com>
Date: Fri Jun 24 14:50:01 2016 -0700
mm/swap.c: flush lru pvecs on compound page arrival
[ Upstream commit 8f182270dfec432e93fae14f9208a6b9af01009f ]
Currently we can have compound pages held on per cpu pagevecs, which
leads to a lot of memory unavailable for reclaim when needed. In the
systems with hundreads of processors it can be GBs of memory.
On of the way of reproducing the problem is to not call munmap
explicitly on all mapped regions (i.e. after receiving SIGTERM). After
that some pages (with THP enabled also huge pages) may end up on
lru_add_pvec, example below.
void main() {
#pragma omp parallel
{
size_t size = 55 * 1000 * 1000; // smaller than MEM/CPUS
void *p = mmap(NULL, size, PROT_READ | PROT_WRITE,
MAP_PRIVATE | MAP_ANONYMOUS , -1, 0);
if (p != MAP_FAILED)
memset(p, 0, size);
//munmap(p, size); // uncomment to make the problem go away
}
}
When we run it with THP enabled it will leave significant amount of
memory on lru_add_pvec. This memory will be not reclaimed if we hit
OOM, so when we run above program in a loop:
for i in `seq 100`; do ./a.out; done
many processes (95% in my case) will be killed by OOM.
The primary point of the LRU add cache is to save the zone lru_lock
contention with a hope that more pages will belong to the same zone and
so their addition can be batched. The huge page is already a form of
batched addition (it will add 512 worth of memory in one go) so skipping
the batching seems like a safer option when compared to a potential
excess in the caching which can be quite large and much harder to fix
because lru_add_drain_all is way to expensive and it is not really clear
what would be a good moment to call it.
Similarly we can reproduce the problem on lru_deactivate_pvec by adding:
madvise(p, size, MADV_FREE); after memset.
This patch flushes lru pvecs on compound page arrival making the problem
less severe - after applying it kill rate of above example drops to 0%,
due to reducing maximum amount of memory held on pvec from 28MB (with
THP) to 56kB per CPU.
Suggested-by: Michal Hocko <mhocko@suse.com>
Link: http://lkml.kernel.org/r/1466180198-18854-1-git-send-email-lukasz.odzioba@intel.com
Signed-off-by: Lukasz Odzioba <lukasz.odzioba@intel.com>
Acked-by: Michal Hocko <mhocko@suse.com>
Cc: Kirill Shutemov <kirill.shutemov@linux.intel.com>
Cc: Andrea Arcangeli <aarcange@redhat.com>
Cc: Vladimir Davydov <vdavydov@parallels.com>
Cc: Ming Li <mingli199x@qq.com>
Cc: Minchan Kim <minchan@kernel.org>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Sasha Levin <sasha.levin@oracle.com>
pnmtopng: 63 colors found
Revision graph left in /home/logs/results/bisect/linux-3.18/test-amd64-i386-xl-qemut-debianhvm-amd64.debian-hvm-install.{dot,ps,png,html,svg}.
----------------------------------------
97669: tolerable ALL FAIL
flight 97669 linux-3.18 real-bisect [real]
http://logs.test-lab.xenproject.org/osstest/logs/97669/
Failures :-/ but no regressions.
Tests which did not succeed,
including tests which could not be run:
test-amd64-i386-xl-qemut-debianhvm-amd64 9 debian-hvm-install fail baseline untested
jobs:
test-amd64-i386-xl-qemut-debianhvm-amd64 fail
------------------------------------------------------------
sg-report-flight on osstest.test-lab.xenproject.org
logs: /home/logs/logs
images: /home/logs/images
Logs, config files, etc. are available at
http://logs.test-lab.xenproject.org/osstest/logs
Explanation of these reports, and of osstest in general, is at
http://xenbits.xen.org/gitweb/?p=osstest.git;a=blob;f=README.email;hb=master
http://xenbits.xen.org/gitweb/?p=osstest.git;a=blob;f=README;hb=master
Test harness code can be found at
http://xenbits.xen.org/gitweb?p=osstest.git;a=summary
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2016-07-19 20:29 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2015-07-27 21:53 [linux-3.18 bisection] complete test-amd64-i386-xl-qemut-debianhvm-amd64 osstest service owner
2016-07-19 20:29 osstest service owner
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).