From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751715AbdHJEO5 (ORCPT ); Thu, 10 Aug 2017 00:14:57 -0400 Received: from mail-pg0-f67.google.com ([74.125.83.67]:37899 "EHLO mail-pg0-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750868AbdHJEOz (ORCPT ); Thu, 10 Aug 2017 00:14:55 -0400 Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 10.3 \(3273\)) Subject: Re: [lkp-robot] [mm] 7674270022: will-it-scale.per_process_ops -19.3% regression From: Nadav Amit In-Reply-To: <20170810041353.GB2042@bbox> Date: Wed, 9 Aug 2017 21:14:50 -0700 Cc: Ye Xiaolong , "open list:MEMORY MANAGEMENT" , LKML , Andrew Morton , Ingo Molnar , Russell King , Tony Luck , Martin Schwidefsky , "David S. Miller" , Heiko Carstens , Yoshinori Sato , Jeff Dike , linux-arch@vger.kernel.org, lkp@01.org Message-Id: <80589593-6F0E-4421-9279-681D5B388100@gmail.com> References: <20170802000818.4760-7-namit@vmware.com> <20170808011923.GE25554@yexl-desktop> <20170808022830.GA28570@bbox> <93CA4B47-95C2-43A2-8E92-B142CAB1DAF7@gmail.com> <970B5DC5-BFC2-461E-AC46-F71B3691D301@gmail.com> <20170808080821.GA31730@bbox> <20170809025902.GA17616@yexl-desktop> <20170810041353.GB2042@bbox> To: Minchan Kim X-Mailer: Apple Mail (2.3273) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by nfs id v7A4F3LI016235 Minchan Kim wrote: > On Wed, Aug 09, 2017 at 10:59:02AM +0800, Ye Xiaolong wrote: >> On 08/08, Minchan Kim wrote: >>> On Mon, Aug 07, 2017 at 10:51:00PM -0700, Nadav Amit wrote: >>>> Nadav Amit wrote: >>>> >>>>> Minchan Kim wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> On Tue, Aug 08, 2017 at 09:19:23AM +0800, kernel test robot wrote: >>>>>>> Greeting, >>>>>>> >>>>>>> FYI, we noticed a -19.3% regression of will-it-scale.per_process_ops due to commit: >>>>>>> >>>>>>> >>>>>>> commit: 76742700225cad9df49f05399381ac3f1ec3dc60 ("mm: fix MADV_[FREE|DONTNEED] TLB flush miss problem") >>>>>>> url: https://github.com/0day-ci/linux/commits/Nadav-Amit/mm-migrate-prevent-racy-access-to-tlb_flush_pending/20170802-205715 >>>>>>> >>>>>>> >>>>>>> in testcase: will-it-scale >>>>>>> on test machine: 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 64G memory >>>>>>> with following parameters: >>>>>>> >>>>>>> nr_task: 16 >>>>>>> mode: process >>>>>>> test: brk1 >>>>>>> cpufreq_governor: performance >>>>>>> >>>>>>> test-description: Will It Scale takes a testcase and runs it from 1 through to n parallel copies to see if the testcase will scale. It builds both a process and threads based test in order to see any differences between the two. >>>>>>> test-url: https://github.com/antonblanchard/will-it-scale >>>>>> >>>>>> Thanks for the report. >>>>>> Could you explain what kinds of workload you are testing? >>>>>> >>>>>> Does it calls frequently madvise(MADV_DONTNEED) in parallel on multiple >>>>>> threads? >>>>> >>>>> According to the description it is "testcase:brk increase/decrease of one >>>>> page”. According to the mode it spawns multiple processes, not threads. >>>>> >>>>> Since a single page is unmapped each time, and the iTLB-loads increase >>>>> dramatically, I would suspect that for some reason a full TLB flush is >>>>> caused during do_munmap(). >>>>> >>>>> If I find some free time, I’ll try to profile the workload - but feel free >>>>> to beat me to it. >>>> >>>> The root-cause appears to be that tlb_finish_mmu() does not call >>>> dec_tlb_flush_pending() - as it should. Any chance you can take care of it? >>> >>> Oops, but with second looking, it seems it's not my fault. ;-) >>> https://marc.info/?l=linux-mm&m=150156699114088&w=2 >>> >>> Anyway, thanks for the pointing out. >>> xiaolong.ye, could you retest with this fix? >> >> I've queued tests for 5 times and results show this patch (e8f682574e4 "mm: >> decrease tlb flush pending count in tlb_finish_mmu") does help recover the >> performance back. >> >> 378005bdbac0a2ec 76742700225cad9df49f053993 e8f682574e45b6406dadfffeb4 >> ---------------- -------------------------- -------------------------- >> %stddev change %stddev change %stddev >> \ | \ | \ >> 3405093 -19% 2747088 -2% 3348752 will-it-scale.per_process_ops >> 1280 ± 3% -2% 1257 ± 3% -6% 1207 vmstat.system.cs >> 2702 ± 18% 11% 3002 ± 19% 17% 3156 ± 18% numa-vmstat.node0.nr_mapped >> 10765 ± 18% 11% 11964 ± 19% 17% 12588 ± 18% numa-meminfo.node0.Mapped >> 0.00 ± 47% -40% 0.00 ± 45% -84% 0.00 ± 42% mpstat.cpu.soft% >> >> Thanks, >> Xiaolong > > Thanks for the testing! Sorry again for screwing your patch, Minchan. From mboxrd@z Thu Jan 1 00:00:00 1970 From: Nadav Amit Subject: Re: [lkp-robot] [mm] 7674270022: will-it-scale.per_process_ops -19.3% regression Date: Wed, 9 Aug 2017 21:14:50 -0700 Message-ID: <80589593-6F0E-4421-9279-681D5B388100@gmail.com> References: <20170802000818.4760-7-namit@vmware.com> <20170808011923.GE25554@yexl-desktop> <20170808022830.GA28570@bbox> <93CA4B47-95C2-43A2-8E92-B142CAB1DAF7@gmail.com> <970B5DC5-BFC2-461E-AC46-F71B3691D301@gmail.com> <20170808080821.GA31730@bbox> <20170809025902.GA17616@yexl-desktop> <20170810041353.GB2042@bbox> Mime-Version: 1.0 (Mac OS X Mail 10.3 \(3273\)) Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Return-path: In-Reply-To: <20170810041353.GB2042@bbox> Sender: owner-linux-mm@kvack.org To: Minchan Kim Cc: Ye Xiaolong , "open list:MEMORY MANAGEMENT" , LKML , Andrew Morton , Ingo Molnar , Russell King , Tony Luck , Martin Schwidefsky , "David S. Miller" , Heiko Carstens , Yoshinori Sato , Jeff Dike , linux-arch@vger.kernel.org, lkp@01.org List-Id: linux-arch.vger.kernel.org Minchan Kim wrote: > On Wed, Aug 09, 2017 at 10:59:02AM +0800, Ye Xiaolong wrote: >> On 08/08, Minchan Kim wrote: >>> On Mon, Aug 07, 2017 at 10:51:00PM -0700, Nadav Amit wrote: >>>> Nadav Amit wrote: >>>>=20 >>>>> Minchan Kim wrote: >>>>>=20 >>>>>> Hi, >>>>>>=20 >>>>>> On Tue, Aug 08, 2017 at 09:19:23AM +0800, kernel test robot = wrote: >>>>>>> Greeting, >>>>>>>=20 >>>>>>> FYI, we noticed a -19.3% regression of = will-it-scale.per_process_ops due to commit: >>>>>>>=20 >>>>>>>=20 >>>>>>> commit: 76742700225cad9df49f05399381ac3f1ec3dc60 ("mm: fix = MADV_[FREE|DONTNEED] TLB flush miss problem") >>>>>>> url: = https://github.com/0day-ci/linux/commits/Nadav-Amit/mm-migrate-prevent-rac= y-access-to-tlb_flush_pending/20170802-205715 >>>>>>>=20 >>>>>>>=20 >>>>>>> in testcase: will-it-scale >>>>>>> on test machine: 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ = 2.20GHz with 64G memory >>>>>>> with following parameters: >>>>>>>=20 >>>>>>> nr_task: 16 >>>>>>> mode: process >>>>>>> test: brk1 >>>>>>> cpufreq_governor: performance >>>>>>>=20 >>>>>>> test-description: Will It Scale takes a testcase and runs it = from 1 through to n parallel copies to see if the testcase will scale. = It builds both a process and threads based test in order to see any = differences between the two. >>>>>>> test-url: https://github.com/antonblanchard/will-it-scale >>>>>>=20 >>>>>> Thanks for the report. >>>>>> Could you explain what kinds of workload you are testing? >>>>>>=20 >>>>>> Does it calls frequently madvise(MADV_DONTNEED) in parallel on = multiple >>>>>> threads? >>>>>=20 >>>>> According to the description it is "testcase:brk increase/decrease = of one >>>>> page=E2=80=9D. According to the mode it spawns multiple processes, = not threads. >>>>>=20 >>>>> Since a single page is unmapped each time, and the iTLB-loads = increase >>>>> dramatically, I would suspect that for some reason a full TLB = flush is >>>>> caused during do_munmap(). >>>>>=20 >>>>> If I find some free time, I=E2=80=99ll try to profile the workload = - but feel free >>>>> to beat me to it. >>>>=20 >>>> The root-cause appears to be that tlb_finish_mmu() does not call >>>> dec_tlb_flush_pending() - as it should. Any chance you can take = care of it? >>>=20 >>> Oops, but with second looking, it seems it's not my fault. ;-) >>> https://marc.info/?l=3Dlinux-mm&m=3D150156699114088&w=3D2 >>>=20 >>> Anyway, thanks for the pointing out. >>> xiaolong.ye, could you retest with this fix? >>=20 >> I've queued tests for 5 times and results show this patch = (e8f682574e4 "mm: >> decrease tlb flush pending count in tlb_finish_mmu") does help = recover the >> performance back. >>=20 >> 378005bdbac0a2ec 76742700225cad9df49f053993 = e8f682574e45b6406dadfffeb4 =20 >> ---------------- -------------------------- = -------------------------- =20 >> %stddev change %stddev change = %stddev >> \ | \ | \ = =20 >> 3405093 -19% 2747088 -2% 3348752 = will-it-scale.per_process_ops >> 1280 =C2=B1 3% -2% 1257 =C2=B1 3% -6% = 1207 vmstat.system.cs >> 2702 =C2=B1 18% 11% 3002 =C2=B1 19% 17% = 3156 =C2=B1 18% numa-vmstat.node0.nr_mapped >> 10765 =C2=B1 18% 11% 11964 =C2=B1 19% 17% = 12588 =C2=B1 18% numa-meminfo.node0.Mapped >> 0.00 =C2=B1 47% -40% 0.00 =C2=B1 45% -84% = 0.00 =C2=B1 42% mpstat.cpu.soft% >>=20 >> Thanks, >> Xiaolong >=20 > Thanks for the testing! Sorry again for screwing your patch, Minchan. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Content-Type: multipart/mixed; boundary="===============3077107280308552643==" MIME-Version: 1.0 From: Nadav Amit To: lkp@lists.01.org Subject: Re: [lkp-robot] [mm] 7674270022: will-it-scale.per_process_ops -19.3% regression Date: Wed, 09 Aug 2017 21:14:50 -0700 Message-ID: <80589593-6F0E-4421-9279-681D5B388100@gmail.com> In-Reply-To: <20170810041353.GB2042@bbox> List-Id: --===============3077107280308552643== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Minchan Kim wrote: > On Wed, Aug 09, 2017 at 10:59:02AM +0800, Ye Xiaolong wrote: >> On 08/08, Minchan Kim wrote: >>> On Mon, Aug 07, 2017 at 10:51:00PM -0700, Nadav Amit wrote: >>>> Nadav Amit wrote: >>>> = >>>>> Minchan Kim wrote: >>>>> = >>>>>> Hi, >>>>>> = >>>>>> On Tue, Aug 08, 2017 at 09:19:23AM +0800, kernel test robot wrote: >>>>>>> Greeting, >>>>>>> = >>>>>>> FYI, we noticed a -19.3% regression of will-it-scale.per_process_op= s due to commit: >>>>>>> = >>>>>>> = >>>>>>> commit: 76742700225cad9df49f05399381ac3f1ec3dc60 ("mm: fix MADV_[FR= EE|DONTNEED] TLB flush miss problem") >>>>>>> url: https://github.com/0day-ci/linux/commits/Nadav-Amit/mm-migrate= -prevent-racy-access-to-tlb_flush_pending/20170802-205715 >>>>>>> = >>>>>>> = >>>>>>> in testcase: will-it-scale >>>>>>> on test machine: 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20G= Hz with 64G memory >>>>>>> with following parameters: >>>>>>> = >>>>>>> nr_task: 16 >>>>>>> mode: process >>>>>>> test: brk1 >>>>>>> cpufreq_governor: performance >>>>>>> = >>>>>>> test-description: Will It Scale takes a testcase and runs it from 1= through to n parallel copies to see if the testcase will scale. It builds = both a process and threads based test in order to see any differences betwe= en the two. >>>>>>> test-url: https://github.com/antonblanchard/will-it-scale >>>>>> = >>>>>> Thanks for the report. >>>>>> Could you explain what kinds of workload you are testing? >>>>>> = >>>>>> Does it calls frequently madvise(MADV_DONTNEED) in parallel on multi= ple >>>>>> threads? >>>>> = >>>>> According to the description it is "testcase:brk increase/decrease of= one >>>>> page=E2=80=9D. According to the mode it spawns multiple processes, no= t threads. >>>>> = >>>>> Since a single page is unmapped each time, and the iTLB-loads increase >>>>> dramatically, I would suspect that for some reason a full TLB flush is >>>>> caused during do_munmap(). >>>>> = >>>>> If I find some free time, I=E2=80=99ll try to profile the workload - = but feel free >>>>> to beat me to it. >>>> = >>>> The root-cause appears to be that tlb_finish_mmu() does not call >>>> dec_tlb_flush_pending() - as it should. Any chance you can take care o= f it? >>> = >>> Oops, but with second looking, it seems it's not my fault. ;-) >>> https://marc.info/?l=3Dlinux-mm&m=3D150156699114088&w=3D2 >>> = >>> Anyway, thanks for the pointing out. >>> xiaolong.ye, could you retest with this fix? >> = >> I've queued tests for 5 times and results show this patch (e8f682574e4 "= mm: >> decrease tlb flush pending count in tlb_finish_mmu") does help recover t= he >> performance back. >> = >> 378005bdbac0a2ec 76742700225cad9df49f053993 e8f682574e45b6406dadfffeb4= = >> ---------------- -------------------------- --------------------------= = >> %stddev change %stddev change %stddev >> \ | \ | \ = >> 3405093 -19% 2747088 -2% 3348752 = will-it-scale.per_process_ops >> 1280 =C2=B1 3% -2% 1257 =C2=B1 3% -6% 1= 207 vmstat.system.cs >> 2702 =C2=B1 18% 11% 3002 =C2=B1 19% 17% 3= 156 =C2=B1 18% numa-vmstat.node0.nr_mapped >> 10765 =C2=B1 18% 11% 11964 =C2=B1 19% 17% 12= 588 =C2=B1 18% numa-meminfo.node0.Mapped >> 0.00 =C2=B1 47% -40% 0.00 =C2=B1 45% -84% 0= .00 =C2=B1 42% mpstat.cpu.soft% >> = >> Thanks, >> Xiaolong > = > Thanks for the testing! Sorry again for screwing your patch, Minchan. --===============3077107280308552643==--