From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751187AbdEaMjh convert rfc822-to-8bit (ORCPT ); Wed, 31 May 2017 08:39:37 -0400 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:37166 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751125AbdEaMjf (ORCPT ); Wed, 31 May 2017 08:39:35 -0400 Date: Wed, 31 May 2017 15:39:22 +0300 User-Agent: K-9 Mail for Android In-Reply-To: <20170531120822.GL27783@dhcp22.suse.cz> References: <20170524103947.GC3063@rapoport-lnx> <20170524111800.GD14733@dhcp22.suse.cz> <20170524142735.GF3063@rapoport-lnx> <20170530074408.GA7969@dhcp22.suse.cz> <20170530101921.GA25738@rapoport-lnx> <20170530103930.GB7969@dhcp22.suse.cz> <20170530140456.GA8412@redhat.com> <20170530143941.GK7969@dhcp22.suse.cz> <20170530154326.GB8412@redhat.com> <20170531120822.GL27783@dhcp22.suse.cz> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8BIT Subject: Re: [PATCH] mm: introduce MADV_CLR_HUGEPAGE To: Michal Hocko , Andrea Arcangeli CC: Vlastimil Babka , "Kirill A. Shutemov" , Andrew Morton , Arnd Bergmann , "Kirill A. Shutemov" , Pavel Emelyanov , linux-mm , lkml , Linux API From: Mike Rapoprt X-TM-AS-GCONF: 00 x-cbid: 17053112-0040-0000-0000-0000039DCEBD X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 17053112-0041-0000-0000-000025916B23 Message-Id: <8FA5E4C2-D289-4AF5-AA09-6C199E58F9A5@linux.vnet.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2017-05-31_05:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 spamscore=0 suspectscore=0 malwarescore=0 phishscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1703280000 definitions=main-1705310232 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On May 31, 2017 3:08:22 PM GMT+03:00, Michal Hocko wrote: >On Tue 30-05-17 17:43:26, Andrea Arcangeli wrote: >> On Tue, May 30, 2017 at 04:39:41PM +0200, Michal Hocko wrote: >> > I sysctl for the mapcount can be increased, right? I also assume >that >> > those vmas will get merged after the post copy is done. >> >> Assuming you enlarge the sysctl to the worst possible case, with >64bit >> address space you can have billions of VMAs if you're migrating 4T of >> RAM and you're unlucky and the address space gets fragmented. The >> unswappable kernel memory overhead would be relatively large >> (i.e. dozen gigabytes of RAM in vm_area_struct slab), and each >> find_vma operation would need to walk ~40 steps across that large vma >> rbtree. There's a reason the sysctl exist. Not to tell all those >> unnecessary vma mangling operations would be protected by the >mmap_sem >> for writing. >> >> Not creating a ton of vmas and enabling vma-less pte mangling with a >> single large vma and only using mmap_sem for reading during all the >> pte mangling, is one of the primary design motivations for >> userfaultfd. > >Yes, I am aware of fallouts of too many vmas. I was asking merely to >learn whether this will really happen under the the specific usecase >Mike is after. That depends on the application access pattern in the period between the pre-dump is finished and the application is frozen. If the accesses are random enough, the dirty pages that would be post copied could get spread all over the address space. >> > I understand that part but it sounds awfully one purpose thing to >me. >> > Are we going to add other MADVISE_RESET_$FOO to clear other flags >just >> > because we can race in this specific use case? >> >> Those already exists, see for example MADV_NORMAL, clearing >> ~VM_RAND_READ & ~VM_SEQ_READ after calling MADV_SEQUENTIAL or >> MADV_RANDOM. > >I would argue that MADV_NORMAL is everything but a clear madvise >command. Why doesn't it clear all the sticky MADV* flags? That would be helpful :) Still, the problem here is more with the naming that with the action. If it was called MADV_DEFAULT_READ or something, it would be fine, wouldn't it? >> Or MADV_DOFORK after MADV_DONTFORK. MADV_DONTDUMP after MADV_DODUMP. >Etc.. >> >> > But we already have MADV_HUGEPAGE, MADV_NOHUGEPAGE and prctl to >> > enable/disable thp. Doesn't that sound little bit too much for a >single >> > feature to you? >> >> MADV_NOHUGEPAGE doesn't mean clearing the flag set with >> MADV_HUGEPAGE. MADV_NOHUGEPAGE disables THP on the region if the >> global sysfs "enabled" tune is set to "always". MADV_HUGEPAGE enables >> THP if the global "enabled" sysfs tune is set to "madvise". The two >> MADV_NOHUGEPAGE and MADV_HUGEPAGE are needed to leverage the >three-way >> setting of "never" "madvise" "always" of the global tune. >> >> The "madvise" global tune exists if you want to save RAM and you >don't >> care much about performance but still allowing apps like QEMU where >no >> memory is lost by enabling THP, to use THP. >> >> There's no way to clear either of those two flags and bring back the >> default behavior of the global sysfs tune, so it's not redundant at >> the very least. > >Yes I am not a huge fan of the current MADV*HUGEPAGE semantic but I >would really like to see a strong usecase for adding another command on >top. Well, another command makes the semantic a bit better, IMHO... > From what Mike said a global disable THP for the whole process >while the post-copy is in progress is a better solution anyway. For the CRIU usecase, disabling THP for a while and re-enabling it back will do the trick, provided VMAs flags are not affected, like in the patch you've sent. Moreover, we may even get away with ioctl(UFFDIO_COPY) if it's overhead shows to be negligible​. Still, I believe that MADV_RESET_HUGEPAGE (or some better named) command has the value on its own. -- Sincerely yours, Mike. From mboxrd@z Thu Jan 1 00:00:00 1970 From: Mike Rapoprt Subject: Re: [PATCH] mm: introduce MADV_CLR_HUGEPAGE Date: Wed, 31 May 2017 15:39:22 +0300 Message-ID: <8FA5E4C2-D289-4AF5-AA09-6C199E58F9A5@linux.vnet.ibm.com> References: <20170524103947.GC3063@rapoport-lnx> <20170524111800.GD14733@dhcp22.suse.cz> <20170524142735.GF3063@rapoport-lnx> <20170530074408.GA7969@dhcp22.suse.cz> <20170530101921.GA25738@rapoport-lnx> <20170530103930.GB7969@dhcp22.suse.cz> <20170530140456.GA8412@redhat.com> <20170530143941.GK7969@dhcp22.suse.cz> <20170530154326.GB8412@redhat.com> <20170531120822.GL27783@dhcp22.suse.cz> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8BIT Return-path: In-Reply-To: <20170531120822.GL27783-2MMpYkNvuYDjFM9bn6wA6Q@public.gmane.org> Sender: linux-api-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org To: Michal Hocko , Andrea Arcangeli Cc: Vlastimil Babka , "Kirill A. Shutemov" , Andrew Morton , Arnd Bergmann , "Kirill A. Shutemov" , Pavel Emelyanov , linux-mm , lkml , Linux API List-Id: linux-api@vger.kernel.org On May 31, 2017 3:08:22 PM GMT+03:00, Michal Hocko wrote: >On Tue 30-05-17 17:43:26, Andrea Arcangeli wrote: >> On Tue, May 30, 2017 at 04:39:41PM +0200, Michal Hocko wrote: >> > I sysctl for the mapcount can be increased, right? I also assume >that >> > those vmas will get merged after the post copy is done. >> >> Assuming you enlarge the sysctl to the worst possible case, with >64bit >> address space you can have billions of VMAs if you're migrating 4T of >> RAM and you're unlucky and the address space gets fragmented. The >> unswappable kernel memory overhead would be relatively large >> (i.e. dozen gigabytes of RAM in vm_area_struct slab), and each >> find_vma operation would need to walk ~40 steps across that large vma >> rbtree. There's a reason the sysctl exist. Not to tell all those >> unnecessary vma mangling operations would be protected by the >mmap_sem >> for writing. >> >> Not creating a ton of vmas and enabling vma-less pte mangling with a >> single large vma and only using mmap_sem for reading during all the >> pte mangling, is one of the primary design motivations for >> userfaultfd. > >Yes, I am aware of fallouts of too many vmas. I was asking merely to >learn whether this will really happen under the the specific usecase >Mike is after. That depends on the application access pattern in the period between the pre-dump is finished and the application is frozen. If the accesses are random enough, the dirty pages that would be post copied could get spread all over the address space. >> > I understand that part but it sounds awfully one purpose thing to >me. >> > Are we going to add other MADVISE_RESET_$FOO to clear other flags >just >> > because we can race in this specific use case? >> >> Those already exists, see for example MADV_NORMAL, clearing >> ~VM_RAND_READ & ~VM_SEQ_READ after calling MADV_SEQUENTIAL or >> MADV_RANDOM. > >I would argue that MADV_NORMAL is everything but a clear madvise >command. Why doesn't it clear all the sticky MADV* flags? That would be helpful :) Still, the problem here is more with the naming that with the action. If it was called MADV_DEFAULT_READ or something, it would be fine, wouldn't it? >> Or MADV_DOFORK after MADV_DONTFORK. MADV_DONTDUMP after MADV_DODUMP. >Etc.. >> >> > But we already have MADV_HUGEPAGE, MADV_NOHUGEPAGE and prctl to >> > enable/disable thp. Doesn't that sound little bit too much for a >single >> > feature to you? >> >> MADV_NOHUGEPAGE doesn't mean clearing the flag set with >> MADV_HUGEPAGE. MADV_NOHUGEPAGE disables THP on the region if the >> global sysfs "enabled" tune is set to "always". MADV_HUGEPAGE enables >> THP if the global "enabled" sysfs tune is set to "madvise". The two >> MADV_NOHUGEPAGE and MADV_HUGEPAGE are needed to leverage the >three-way >> setting of "never" "madvise" "always" of the global tune. >> >> The "madvise" global tune exists if you want to save RAM and you >don't >> care much about performance but still allowing apps like QEMU where >no >> memory is lost by enabling THP, to use THP. >> >> There's no way to clear either of those two flags and bring back the >> default behavior of the global sysfs tune, so it's not redundant at >> the very least. > >Yes I am not a huge fan of the current MADV*HUGEPAGE semantic but I >would really like to see a strong usecase for adding another command on >top. Well, another command makes the semantic a bit better, IMHO... > From what Mike said a global disable THP for the whole process >while the post-copy is in progress is a better solution anyway. For the CRIU usecase, disabling THP for a while and re-enabling it back will do the trick, provided VMAs flags are not affected, like in the patch you've sent. Moreover, we may even get away with ioctl(UFFDIO_COPY) if it's overhead shows to be negligible​. Still, I believe that MADV_RESET_HUGEPAGE (or some better named) command has the value on its own. -- Sincerely yours, Mike. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pf0-f197.google.com (mail-pf0-f197.google.com [209.85.192.197]) by kanga.kvack.org (Postfix) with ESMTP id C1F9D6B02B4 for ; Wed, 31 May 2017 08:39:37 -0400 (EDT) Received: by mail-pf0-f197.google.com with SMTP id m5so14041216pfc.1 for ; Wed, 31 May 2017 05:39:37 -0700 (PDT) Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com. [148.163.156.1]) by mx.google.com with ESMTPS id p7si16281986pgn.150.2017.05.31.05.39.36 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 31 May 2017 05:39:36 -0700 (PDT) Received: from pps.filterd (m0098409.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.20/8.16.0.20) with SMTP id v4VCdQCn041656 for ; Wed, 31 May 2017 08:39:35 -0400 Received: from e06smtp10.uk.ibm.com (e06smtp10.uk.ibm.com [195.75.94.106]) by mx0a-001b2d01.pphosted.com with ESMTP id 2asj42wba9-1 (version=TLSv1.2 cipher=AES256-SHA bits=256 verify=NOT) for ; Wed, 31 May 2017 08:39:34 -0400 Received: from localhost by e06smtp10.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Wed, 31 May 2017 13:39:29 +0100 Date: Wed, 31 May 2017 15:39:22 +0300 In-Reply-To: <20170531120822.GL27783@dhcp22.suse.cz> References: <20170524103947.GC3063@rapoport-lnx> <20170524111800.GD14733@dhcp22.suse.cz> <20170524142735.GF3063@rapoport-lnx> <20170530074408.GA7969@dhcp22.suse.cz> <20170530101921.GA25738@rapoport-lnx> <20170530103930.GB7969@dhcp22.suse.cz> <20170530140456.GA8412@redhat.com> <20170530143941.GK7969@dhcp22.suse.cz> <20170530154326.GB8412@redhat.com> <20170531120822.GL27783@dhcp22.suse.cz> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Subject: Re: [PATCH] mm: introduce MADV_CLR_HUGEPAGE From: Mike Rapoprt Message-Id: <8FA5E4C2-D289-4AF5-AA09-6C199E58F9A5@linux.vnet.ibm.com> Sender: owner-linux-mm@kvack.org List-ID: To: Michal Hocko , Andrea Arcangeli Cc: Vlastimil Babka , "Kirill A. Shutemov" , Andrew Morton , Arnd Bergmann , "Kirill A. Shutemov" , Pavel Emelyanov , linux-mm , lkml , Linux API On May 31, 2017 3:08:22 PM GMT+03:00, Michal Hocko w= rote: >On Tue 30-05-17 17:43:26, Andrea Arcangeli wrote: >> On Tue, May 30, 2017 at 04:39:41PM +0200, Michal Hocko wrote: >> > I sysctl for the mapcount can be increased, right? I also assume >that >> > those vmas will get merged after the post copy is done=2E >>=20 >> Assuming you enlarge the sysctl to the worst possible case, with >64bit >> address space you can have billions of VMAs if you're migrating 4T of >> RAM and you're unlucky and the address space gets fragmented=2E The >> unswappable kernel memory overhead would be relatively large >> (i=2Ee=2E dozen gigabytes of RAM in vm_area_struct slab), and each >> find_vma operation would need to walk ~40 steps across that large vma >> rbtree=2E There's a reason the sysctl exist=2E Not to tell all those >> unnecessary vma mangling operations would be protected by the >mmap_sem >> for writing=2E >>=20 >> Not creating a ton of vmas and enabling vma-less pte mangling with a >> single large vma and only using mmap_sem for reading during all the >> pte mangling, is one of the primary design motivations for >> userfaultfd=2E > >Yes, I am aware of fallouts of too many vmas=2E I was asking merely to >learn whether this will really happen under the the specific usecase >Mike is after=2E That depends on the application access pattern in the period between the p= re-dump is finished and the application is frozen=2E If the accesses are ra= ndom enough, the dirty pages that would be post copied could get spread all= over the address space=2E >> > I understand that part but it sounds awfully one purpose thing to >me=2E >> > Are we going to add other MADVISE_RESET_$FOO to clear other flags >just >> > because we can race in this specific use case? >>=20 >> Those already exists, see for example MADV_NORMAL, clearing >> ~VM_RAND_READ & ~VM_SEQ_READ after calling MADV_SEQUENTIAL or >> MADV_RANDOM=2E > >I would argue that MADV_NORMAL is everything but a clear madvise >command=2E Why doesn't it clear all the sticky MADV* flags? That would be helpful :) Still, the problem here is more with the naming that with the action=2E If= it was called MADV_DEFAULT_READ or something, it would be fine, wouldn't i= t? >> Or MADV_DOFORK after MADV_DONTFORK=2E MADV_DONTDUMP after MADV_DODUMP= =2E >Etc=2E=2E >> >> > But we already have MADV_HUGEPAGE, MADV_NOHUGEPAGE and prctl to >> > enable/disable thp=2E Doesn't that sound little bit too much for a >single >> > feature to you? >>=20 >> MADV_NOHUGEPAGE doesn't mean clearing the flag set with >> MADV_HUGEPAGE=2E MADV_NOHUGEPAGE disables THP on the region if the >> global sysfs "enabled" tune is set to "always"=2E MADV_HUGEPAGE enables >> THP if the global "enabled" sysfs tune is set to "madvise"=2E The two >> MADV_NOHUGEPAGE and MADV_HUGEPAGE are needed to leverage the >three-way >> setting of "never" "madvise" "always" of the global tune=2E >>=20 >> The "madvise" global tune exists if you want to save RAM and you >don't >> care much about performance but still allowing apps like QEMU where >no >> memory is lost by enabling THP, to use THP=2E >>=20 >> There's no way to clear either of those two flags and bring back the >> default behavior of the global sysfs tune, so it's not redundant at >> the very least=2E > >Yes I am not a huge fan of the current MADV*HUGEPAGE semantic but I >would really like to see a strong usecase for adding another command on >top=2E=20 Well, another command makes the semantic a bit better, IMHO=2E=2E=2E > From what Mike said a global disable THP for the whole process >while the post-copy is in progress is a better solution anyway=2E For the CRIU usecase, disabling THP for a while and re-enabling it back wi= ll do the trick, provided VMAs flags are not affected, like in the patch yo= u've sent=2E Moreover, we may even get away with ioctl(UFFDIO_COPY) if it's overhead sh= ows to be negligible=E2=80=8B=2E Still, I believe that MADV_RESET_HUGEPAGE (or some better named) command h= as the value on its own=2E -- Sincerely yours, Mike=2E -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org