From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.8 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id AC076C3F2D9 for ; Mon, 2 Mar 2020 17:40:51 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 8DB2122B48 for ; Mon, 2 Mar 2020 17:40:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727459AbgCBRku (ORCPT ); Mon, 2 Mar 2020 12:40:50 -0500 Received: from mga01.intel.com ([192.55.52.88]:7421 "EHLO mga01.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727159AbgCBRks (ORCPT ); Mon, 2 Mar 2020 12:40:48 -0500 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga001.jf.intel.com ([10.7.209.18]) by fmsmga101.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 02 Mar 2020 09:40:48 -0800 X-IronPort-AV: E=Sophos;i="5.70,507,1574150400"; d="scan'208";a="319136975" Received: from ahduyck-desk1.jf.intel.com ([10.7.198.76]) by orsmga001-auth.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 02 Mar 2020 09:40:48 -0800 Message-ID: Subject: Re: [PATCH v1 06/11] mm: Allow to offline unmovable PageOffline() pages via MEM_GOING_OFFLINE From: Alexander Duyck To: David Hildenbrand , linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, virtio-dev@lists.oasis-open.org, virtualization@lists.linux-foundation.org, kvm@vger.kernel.org, Michal Hocko , Andrew Morton , "Michael S . Tsirkin" , Juergen Gross , Konrad Rzeszutek Wilk , Pavel Tatashin , Vlastimil Babka , Johannes Weiner , Anthony Yznaga , Michal Hocko , Oscar Salvador , Mel Gorman , Mike Rapoport , Dan Williams , Anshuman Khandual , Qian Cai , Pingfan Liu Date: Mon, 02 Mar 2020 09:40:47 -0800 In-Reply-To: <20200302134941.315212-7-david@redhat.com> References: <20200302134941.315212-1-david@redhat.com> <20200302134941.315212-7-david@redhat.com> Content-Type: text/plain; charset="UTF-8" User-Agent: Evolution 3.32.5 (3.32.5-1.fc30) MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Mon, 2020-03-02 at 14:49 +0100, David Hildenbrand wrote: > virtio-mem wants to allow to offline memory blocks of which some parts > were unplugged (allocated via alloc_contig_range()), especially, to later > offline and remove completely unplugged memory blocks. The important part > is that PageOffline() has to remain set until the section is offline, so > these pages will never get accessed (e.g., when dumping). The pages should > not be handed back to the buddy (which would require clearing PageOffline() > and result in issues if offlining fails and the pages are suddenly in the > buddy). > > Let's allow to do that by allowing to isolate any PageOffline() page > when offlining. This way, we can reach the memory hotplug notifier > MEM_GOING_OFFLINE, where the driver can signal that he is fine with > offlining this page by dropping its reference count. PageOffline() pages > with a reference count of 0 can then be skipped when offlining the > pages (like if they were free, however they are not in the buddy). > > Anybody who uses PageOffline() pages and does not agree to offline them > (e.g., Hyper-V balloon, XEN balloon, VMWare balloon for 2MB pages) will not > decrement the reference count and make offlining fail when trying to > migrate such an unmovable page. So there should be no observable change. > Same applies to balloon compaction users (movable PageOffline() pages), the > pages will simply be migrated. > > Note 1: If offlining fails, a driver has to increment the reference > count again in MEM_CANCEL_OFFLINE. > > Note 2: A driver that makes use of this has to be aware that re-onlining > the memory block has to be handled by hooking into onlining code > (online_page_callback_t), resetting the page PageOffline() and > not giving them to the buddy. > > Cc: Andrew Morton > Cc: Juergen Gross > Cc: Konrad Rzeszutek Wilk > Cc: Pavel Tatashin > Cc: Alexander Duyck > Cc: Vlastimil Babka > Cc: Johannes Weiner > Cc: Anthony Yznaga > Cc: Michal Hocko > Cc: Oscar Salvador > Cc: Mel Gorman > Cc: Mike Rapoport > Cc: Dan Williams > Cc: Anshuman Khandual > Cc: Qian Cai > Cc: Pingfan Liu > Signed-off-by: David Hildenbrand This addresses the core concerns I had with the patch. Reviewed-by: Alexander Duyck From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.8 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 92D18C3F2D1 for ; Mon, 2 Mar 2020 17:40:51 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 60A3120857 for ; Mon, 2 Mar 2020 17:40:51 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 60A3120857 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.intel.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id E8A4C6B0003; Mon, 2 Mar 2020 12:40:50 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E39A36B0005; Mon, 2 Mar 2020 12:40:50 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D77276B0006; Mon, 2 Mar 2020 12:40:50 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0240.hostedemail.com [216.40.44.240]) by kanga.kvack.org (Postfix) with ESMTP id C12326B0003 for ; Mon, 2 Mar 2020 12:40:50 -0500 (EST) Received: from smtpin30.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 6D2D92C78 for ; Mon, 2 Mar 2020 17:40:50 +0000 (UTC) X-FDA: 76551137460.30.field34_7ba881af4330d X-HE-Tag: field34_7ba881af4330d X-Filterd-Recvd-Size: 4855 Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by imf03.hostedemail.com (Postfix) with ESMTP for ; Mon, 2 Mar 2020 17:40:49 +0000 (UTC) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga001.jf.intel.com ([10.7.209.18]) by orsmga103.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 02 Mar 2020 09:40:48 -0800 X-IronPort-AV: E=Sophos;i="5.70,507,1574150400"; d="scan'208";a="319136975" Received: from ahduyck-desk1.jf.intel.com ([10.7.198.76]) by orsmga001-auth.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 02 Mar 2020 09:40:48 -0800 Message-ID: Subject: Re: [PATCH v1 06/11] mm: Allow to offline unmovable PageOffline() pages via MEM_GOING_OFFLINE From: Alexander Duyck To: David Hildenbrand , linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, virtio-dev@lists.oasis-open.org, virtualization@lists.linux-foundation.org, kvm@vger.kernel.org, Michal Hocko , Andrew Morton , "Michael S . Tsirkin" , Juergen Gross , Konrad Rzeszutek Wilk , Pavel Tatashin , Vlastimil Babka , Johannes Weiner , Anthony Yznaga , Michal Hocko , Oscar Salvador , Mel Gorman , Mike Rapoport , Dan Williams , Anshuman Khandual , Qian Cai , Pingfan Liu Date: Mon, 02 Mar 2020 09:40:47 -0800 In-Reply-To: <20200302134941.315212-7-david@redhat.com> References: <20200302134941.315212-1-david@redhat.com> <20200302134941.315212-7-david@redhat.com> Content-Type: text/plain; charset="UTF-8" User-Agent: Evolution 3.32.5 (3.32.5-1.fc30) MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Mon, 2020-03-02 at 14:49 +0100, David Hildenbrand wrote: > virtio-mem wants to allow to offline memory blocks of which some parts > were unplugged (allocated via alloc_contig_range()), especially, to later > offline and remove completely unplugged memory blocks. The important part > is that PageOffline() has to remain set until the section is offline, so > these pages will never get accessed (e.g., when dumping). The pages should > not be handed back to the buddy (which would require clearing PageOffline() > and result in issues if offlining fails and the pages are suddenly in the > buddy). > > Let's allow to do that by allowing to isolate any PageOffline() page > when offlining. This way, we can reach the memory hotplug notifier > MEM_GOING_OFFLINE, where the driver can signal that he is fine with > offlining this page by dropping its reference count. PageOffline() pages > with a reference count of 0 can then be skipped when offlining the > pages (like if they were free, however they are not in the buddy). > > Anybody who uses PageOffline() pages and does not agree to offline them > (e.g., Hyper-V balloon, XEN balloon, VMWare balloon for 2MB pages) will not > decrement the reference count and make offlining fail when trying to > migrate such an unmovable page. So there should be no observable change. > Same applies to balloon compaction users (movable PageOffline() pages), the > pages will simply be migrated. > > Note 1: If offlining fails, a driver has to increment the reference > count again in MEM_CANCEL_OFFLINE. > > Note 2: A driver that makes use of this has to be aware that re-onlining > the memory block has to be handled by hooking into onlining code > (online_page_callback_t), resetting the page PageOffline() and > not giving them to the buddy. > > Cc: Andrew Morton > Cc: Juergen Gross > Cc: Konrad Rzeszutek Wilk > Cc: Pavel Tatashin > Cc: Alexander Duyck > Cc: Vlastimil Babka > Cc: Johannes Weiner > Cc: Anthony Yznaga > Cc: Michal Hocko > Cc: Oscar Salvador > Cc: Mel Gorman > Cc: Mike Rapoport > Cc: Dan Williams > Cc: Anshuman Khandual > Cc: Qian Cai > Cc: Pingfan Liu > Signed-off-by: David Hildenbrand This addresses the core concerns I had with the patch. Reviewed-by: Alexander Duyck From mboxrd@z Thu Jan 1 00:00:00 1970 From: Alexander Duyck Subject: Re: [PATCH v1 06/11] mm: Allow to offline unmovable PageOffline() pages via MEM_GOING_OFFLINE Date: Mon, 02 Mar 2020 09:40:47 -0800 Message-ID: References: <20200302134941.315212-1-david@redhat.com> <20200302134941.315212-7-david@redhat.com> Mime-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <20200302134941.315212-7-david@redhat.com> Sender: linux-kernel-owner@vger.kernel.org To: David Hildenbrand , linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, virtio-dev@lists.oasis-open.org, virtualization@lists.linux-foundation.org, kvm@vger.kernel.org, Michal Hocko , Andrew Morton , "Michael S . Tsirkin" , Juergen Gross , Konrad Rzeszutek Wilk , Pavel Tatashin , Vlastimil Babka , Johannes Weiner , Anthony Yznaga , Michal Hocko , Oscar Salvador , Mel Gorman , Mike Rapoport , Dan Williams , Anshuman Khandual , Qian Cai , Pingfan Liu List-Id: virtualization@lists.linuxfoundation.org On Mon, 2020-03-02 at 14:49 +0100, David Hildenbrand wrote: > virtio-mem wants to allow to offline memory blocks of which some parts > were unplugged (allocated via alloc_contig_range()), especially, to later > offline and remove completely unplugged memory blocks. The important part > is that PageOffline() has to remain set until the section is offline, so > these pages will never get accessed (e.g., when dumping). The pages should > not be handed back to the buddy (which would require clearing PageOffline() > and result in issues if offlining fails and the pages are suddenly in the > buddy). > > Let's allow to do that by allowing to isolate any PageOffline() page > when offlining. This way, we can reach the memory hotplug notifier > MEM_GOING_OFFLINE, where the driver can signal that he is fine with > offlining this page by dropping its reference count. PageOffline() pages > with a reference count of 0 can then be skipped when offlining the > pages (like if they were free, however they are not in the buddy). > > Anybody who uses PageOffline() pages and does not agree to offline them > (e.g., Hyper-V balloon, XEN balloon, VMWare balloon for 2MB pages) will not > decrement the reference count and make offlining fail when trying to > migrate such an unmovable page. So there should be no observable change. > Same applies to balloon compaction users (movable PageOffline() pages), the > pages will simply be migrated. > > Note 1: If offlining fails, a driver has to increment the reference > count again in MEM_CANCEL_OFFLINE. > > Note 2: A driver that makes use of this has to be aware that re-onlining > the memory block has to be handled by hooking into onlining code > (online_page_callback_t), resetting the page PageOffline() and > not giving them to the buddy. > > Cc: Andrew Morton > Cc: Juergen Gross > Cc: Konrad Rzeszutek Wilk > Cc: Pavel Tatashin > Cc: Alexander Duyck > Cc: Vlastimil Babka > Cc: Johannes Weiner > Cc: Anthony Yznaga > Cc: Michal Hocko > Cc: Oscar Salvador > Cc: Mel Gorman > Cc: Mike Rapoport > Cc: Dan Williams > Cc: Anshuman Khandual > Cc: Qian Cai > Cc: Pingfan Liu > Signed-off-by: David Hildenbrand This addresses the core concerns I had with the patch. Reviewed-by: Alexander Duyck From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: virtio-dev-return-6869-cohuck=redhat.com@lists.oasis-open.org Sender: List-Post: List-Help: List-Unsubscribe: List-Subscribe: Received: from lists.oasis-open.org (oasis-open.org [10.110.1.242]) by lists.oasis-open.org (Postfix) with ESMTP id 210DA985CF2 for ; Mon, 2 Mar 2020 17:40:52 +0000 (UTC) Message-ID: From: Alexander Duyck Date: Mon, 02 Mar 2020 09:40:47 -0800 In-Reply-To: <20200302134941.315212-7-david@redhat.com> References: <20200302134941.315212-1-david@redhat.com> <20200302134941.315212-7-david@redhat.com> MIME-Version: 1.0 Subject: [virtio-dev] Re: [PATCH v1 06/11] mm: Allow to offline unmovable PageOffline() pages via MEM_GOING_OFFLINE Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable To: David Hildenbrand , linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, virtio-dev@lists.oasis-open.org, virtualization@lists.linux-foundation.org, kvm@vger.kernel.org, Michal Hocko , Andrew Morton , "Michael S . Tsirkin" , Juergen Gross , Konrad Rzeszutek Wilk , Pavel Tatashin , Vlastimil Babka , Johannes Weiner , Anthony Yznaga , Michal Hocko , Oscar Salvador , Mel Gorman , Mike Rapoport , Dan Williams , Anshuman Khandual , Qian Cai , Pingfan Liu List-ID: On Mon, 2020-03-02 at 14:49 +0100, David Hildenbrand wrote: > virtio-mem wants to allow to offline memory blocks of which some parts > were unplugged (allocated via alloc_contig_range()), especially, to later > offline and remove completely unplugged memory blocks. The important part > is that PageOffline() has to remain set until the section is offline, so > these pages will never get accessed (e.g., when dumping). The pages shoul= d > not be handed back to the buddy (which would require clearing PageOffline= () > and result in issues if offlining fails and the pages are suddenly in the > buddy). >=20 > Let's allow to do that by allowing to isolate any PageOffline() page > when offlining. This way, we can reach the memory hotplug notifier > MEM_GOING_OFFLINE, where the driver can signal that he is fine with > offlining this page by dropping its reference count. PageOffline() pages > with a reference count of 0 can then be skipped when offlining the > pages (like if they were free, however they are not in the buddy). >=20 > Anybody who uses PageOffline() pages and does not agree to offline them > (e.g., Hyper-V balloon, XEN balloon, VMWare balloon for 2MB pages) will n= ot > decrement the reference count and make offlining fail when trying to > migrate such an unmovable page. So there should be no observable change. > Same applies to balloon compaction users (movable PageOffline() pages), t= he > pages will simply be migrated. >=20 > Note 1: If offlining fails, a driver has to increment the reference > =09count again in MEM_CANCEL_OFFLINE. >=20 > Note 2: A driver that makes use of this has to be aware that re-onlining > =09the memory block has to be handled by hooking into onlining code > =09(online_page_callback_t), resetting the page PageOffline() and > =09not giving them to the buddy. >=20 > Cc: Andrew Morton > Cc: Juergen Gross > Cc: Konrad Rzeszutek Wilk > Cc: Pavel Tatashin > Cc: Alexander Duyck > Cc: Vlastimil Babka > Cc: Johannes Weiner > Cc: Anthony Yznaga > Cc: Michal Hocko > Cc: Oscar Salvador > Cc: Mel Gorman > Cc: Mike Rapoport > Cc: Dan Williams > Cc: Anshuman Khandual > Cc: Qian Cai > Cc: Pingfan Liu > Signed-off-by: David Hildenbrand This addresses the core concerns I had with the patch. Reviewed-by: Alexander Duyck --------------------------------------------------------------------- To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org