[PATCH 0/2] Drivers: hv: balloon

[PATCH 0/2] Drivers: hv: balloon

[K. Y. Srinivasan]

Support 2M page allocations when memory is ballooned out of the
guest. Hyper-V Dynamic Memory protocol is optimized around the ability
to move memory in 2M chunks.

K. Y. Srinivasan (2):
  mm: Export split_page()
  Drivers: hv: balloon: Support 2M page allocations for ballooning

 drivers/hv/hv_balloon.c |   18 ++++++++++++++++--
 mm/page_alloc.c         |    1 +
 2 files changed, 17 insertions(+), 2 deletions(-)

-- 
1.7.4.1

[PATCH 1/2] mm: Export split_page()

[K. Y. Srinivasan]

The split_page() function will be very useful for balloon drivers. On Hyper-V,
it will be very efficient to use 2M allocations in the guest as this (a) makes
the ballooning protocol with the host that much more efficient and (b) moving
memory in 2M chunks minimizes fragmentation in the host. Export the split_page()
function to let the guest allocations be in 2M chunks while the host is free to
return this memory at arbitrary granularity.

Signed-off-by: K. Y. Srinivasan <kys@microsoft.com>
---
 mm/page_alloc.c |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 6cacfee..7e0ead6 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -1404,6 +1404,7 @@ void split_page(struct page *page, unsigned int order)
 	for (i = 1; i < (1 << order); i++)
 		set_page_refcounted(page + i);
 }
+EXPORT_SYMBOL_GPL(split_page);
 
 static int __isolate_free_page(struct page *page, unsigned int order)
 {
-- 
1.7.4.1

[PATCH 2/2] Drivers: hv: balloon: Support 2M page allocations for ballooning

[K. Y. Srinivasan]

While ballooning memory out of the guest, attempt 2M allocations first.
If 2M allocations fail, then go for 4K allocations. In cases where we
have performed 2M allocations, split this 2M page so that we can free this
page at 4K granularity (when the host returns the memory).

Signed-off-by: K. Y. Srinivasan <kys@microsoft.com>
---
 drivers/hv/hv_balloon.c |   18 ++++++++++++++++--
 1 files changed, 16 insertions(+), 2 deletions(-)

diff --git a/drivers/hv/hv_balloon.c b/drivers/hv/hv_balloon.c
index 2cf7d4e..71655b4 100644
--- a/drivers/hv/hv_balloon.c
+++ b/drivers/hv/hv_balloon.c
@@ -997,6 +997,14 @@ static int  alloc_balloon_pages(struct hv_dynmem_device *dm, int num_pages,
 
 		dm->num_pages_ballooned += alloc_unit;
 
+		/*
+		 * If we allocatted 2M pages; split them so we
+		 * can free them in any order we get.
+		 */
+
+		if (alloc_unit != 1)
+			split_page(pg, get_order(alloc_unit << PAGE_SHIFT));
+
 		bl_resp->range_count++;
 		bl_resp->range_array[i].finfo.start_page =
 			page_to_pfn(pg);
@@ -1023,9 +1031,10 @@ static void balloon_up(struct work_struct *dummy)
 
 
 	/*
-	 * Currently, we only support 4k allocations.
+	 * We will attempt 2M allocations. However, if we fail to
+	 * allocate 2M chunks, we will go back to 4k allocations.
 	 */
-	alloc_unit = 1;
+	alloc_unit = 512;
 
 	while (!done) {
 		bl_resp = (struct dm_balloon_response *)send_buffer;
@@ -1041,6 +1050,11 @@ static void balloon_up(struct work_struct *dummy)
 						bl_resp, alloc_unit,
 						 &alloc_error);
 
+		if ((alloc_error) && (alloc_unit != 1)) {
+			alloc_unit = 1;
+			continue;
+		}
+
 		if ((alloc_error) || (num_ballooned == num_pages)) {
 			bl_resp->more_pages = 0;
 			done = true;
-- 
1.7.4.1

Re: [PATCH 2/2] Drivers: hv: balloon: Support 2M page allocations for ballooning

[Michal Hocko]

On Sat 16-03-13 14:42:05, K. Y. Srinivasan wrote:
> While ballooning memory out of the guest, attempt 2M allocations first.
> If 2M allocations fail, then go for 4K allocations. In cases where we
> have performed 2M allocations, split this 2M page so that we can free this
> page at 4K granularity (when the host returns the memory).

Maybe I am missing something but what is the advantage of 2M allocation
when you split it up immediately so you are not using it as a huge page?

[...]
-- 
Michal Hocko
SUSE Labs

Re: [PATCH 2/2] Drivers: hv: balloon: Support 2M page allocations for ballooning

[Michal Hocko]

On Mon 18-03-13 11:52:57, Michal Hocko wrote:
> On Sat 16-03-13 14:42:05, K. Y. Srinivasan wrote:
> > While ballooning memory out of the guest, attempt 2M allocations first.
> > If 2M allocations fail, then go for 4K allocations. In cases where we
> > have performed 2M allocations, split this 2M page so that we can free this
> > page at 4K granularity (when the host returns the memory).
> 
> Maybe I am missing something but what is the advantage of 2M allocation
> when you split it up immediately so you are not using it as a huge page?

OK, it seems that Patch 1/2 says some advantages. That description
should be part of this patch IMO.
You are saying that a) it reduces fragmentation on the host b) it makes
ballooning more effective.

Have you measured any effects on the ability to allocate huge pages on
the host? What prevents from depleting high order pages on the host with
many guests? Also have you measured any effect on THP allocation success
rate on the host?

Could you clarify how this helps ballooning when you split the page
right after you allocate it?
-- 
Michal Hocko
SUSE Labs

Re: [PATCH 1/2] mm: Export split_page()

[Michal Hocko]

On Sat 16-03-13 14:42:04, K. Y. Srinivasan wrote:
> The split_page() function will be very useful for balloon drivers. On Hyper-V,
> it will be very efficient to use 2M allocations in the guest as this (a) makes
> the ballooning protocol with the host that much more efficient and (b) moving
> memory in 2M chunks minimizes fragmentation in the host. Export the split_page()
> function to let the guest allocations be in 2M chunks while the host is free to
> return this memory at arbitrary granularity.
> 
> Signed-off-by: K. Y. Srinivasan <kys@microsoft.com>

I do not have any objections to exporting the symbol (at least we
prevent drivers code from inventing their own split_page) but the
Hyper-V specific description should go into Hyper-V patch IMO.

So for the export with a short note that the symbol will be used by
Hyper-V
Acked-by: Michal Hocko <mhocko@suse.cz>

> ---
>  mm/page_alloc.c |    1 +
>  1 files changed, 1 insertions(+), 0 deletions(-)
> 
> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> index 6cacfee..7e0ead6 100644
> --- a/mm/page_alloc.c
> +++ b/mm/page_alloc.c
> @@ -1404,6 +1404,7 @@ void split_page(struct page *page, unsigned int order)
>  	for (i = 1; i < (1 << order); i++)
>  		set_page_refcounted(page + i);
>  }
> +EXPORT_SYMBOL_GPL(split_page);
>  
>  static int __isolate_free_page(struct page *page, unsigned int order)
>  {
> -- 
> 1.7.4.1
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

-- 
Michal Hocko
SUSE Labs

RE: [PATCH 1/2] mm: Export split_page()

[KY Srinivasan]

> -----Original Message-----
> From: Michal Hocko [mailto:mhocko@suse.cz]
> Sent: Monday, March 18, 2013 7:04 AM
> To: KY Srinivasan
> Cc: gregkh@linuxfoundation.org; linux-kernel@vger.kernel.org;
> devel@linuxdriverproject.org; olaf@aepfle.de; apw@canonical.com;
> andi@firstfloor.org; akpm@linux-foundation.org; linux-mm@kvack.org;
> kamezawa.hiroyuki@gmail.com; hannes@cmpxchg.org; yinghan@google.com
> Subject: Re: [PATCH 1/2] mm: Export split_page()
> 
> On Sat 16-03-13 14:42:04, K. Y. Srinivasan wrote:
> > The split_page() function will be very useful for balloon drivers. On Hyper-V,
> > it will be very efficient to use 2M allocations in the guest as this (a) makes
> > the ballooning protocol with the host that much more efficient and (b) moving
> > memory in 2M chunks minimizes fragmentation in the host. Export the
> split_page()
> > function to let the guest allocations be in 2M chunks while the host is free to
> > return this memory at arbitrary granularity.
> >
> > Signed-off-by: K. Y. Srinivasan <kys@microsoft.com>
> 
> I do not have any objections to exporting the symbol (at least we
> prevent drivers code from inventing their own split_page) but the
> Hyper-V specific description should go into Hyper-V patch IMO.
> 
> So for the export with a short note that the symbol will be used by
> Hyper-V

Will do.

K. Y
> Acked-by: Michal Hocko <mhocko@suse.cz>
> 
> > ---
> >  mm/page_alloc.c |    1 +
> >  1 files changed, 1 insertions(+), 0 deletions(-)
> >
> > diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> > index 6cacfee..7e0ead6 100644
> > --- a/mm/page_alloc.c
> > +++ b/mm/page_alloc.c
> > @@ -1404,6 +1404,7 @@ void split_page(struct page *page, unsigned int order)
> >  	for (i = 1; i < (1 << order); i++)
> >  		set_page_refcounted(page + i);
> >  }
> > +EXPORT_SYMBOL_GPL(split_page);
> >
> >  static int __isolate_free_page(struct page *page, unsigned int order)
> >  {
> > --
> > 1.7.4.1
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > Please read the FAQ at  http://www.tux.org/lkml/
> 
> --
> Michal Hocko
> SUSE Labs
> 

RE: [PATCH 2/2] Drivers: hv: balloon: Support 2M page allocations for ballooning

[KY Srinivasan]

> -----Original Message-----
> From: Michal Hocko [mailto:mhocko@suse.cz]
> Sent: Monday, March 18, 2013 6:53 AM
> To: KY Srinivasan
> Cc: gregkh@linuxfoundation.org; linux-kernel@vger.kernel.org;
> devel@linuxdriverproject.org; olaf@aepfle.de; apw@canonical.com;
> andi@firstfloor.org; akpm@linux-foundation.org; linux-mm@kvack.org;
> kamezawa.hiroyuki@gmail.com; hannes@cmpxchg.org; yinghan@google.com
> Subject: Re: [PATCH 2/2] Drivers: hv: balloon: Support 2M page allocations for
> ballooning
> 
> On Sat 16-03-13 14:42:05, K. Y. Srinivasan wrote:
> > While ballooning memory out of the guest, attempt 2M allocations first.
> > If 2M allocations fail, then go for 4K allocations. In cases where we
> > have performed 2M allocations, split this 2M page so that we can free this
> > page at 4K granularity (when the host returns the memory).
> 
> Maybe I am missing something but what is the advantage of 2M allocation
> when you split it up immediately so you are not using it as a huge page?

The Hyper-V ballooning protocol specifies the pages being ballooned as page ranges -
start_pfn: number_of_pfns. So, when the guest balloon is inflating and I am able to
allocate 2M pages, I will be able to represent 512 contiguous pages in one 64 bit entry and this makes
the ballooning operation that much more efficient. The reason I split the page is that the host does not
guarantee that when it returns the memory to the guest, it will return in any particular granularity and so
I have to be able to free this memory in 4K granularity. This is the corner case that I will have to handle.

Regards,

K. Y
> 
> [...]
> --
> Michal Hocko
> SUSE Labs
> 

Re: [PATCH 2/2] Drivers: hv: balloon: Support 2M page allocations for ballooning

[Michal Hocko]

On Mon 18-03-13 13:44:05, KY Srinivasan wrote:
> 
> 
> > -----Original Message-----
> > From: Michal Hocko [mailto:mhocko@suse.cz]
> > Sent: Monday, March 18, 2013 6:53 AM
> > To: KY Srinivasan
> > Cc: gregkh@linuxfoundation.org; linux-kernel@vger.kernel.org;
> > devel@linuxdriverproject.org; olaf@aepfle.de; apw@canonical.com;
> > andi@firstfloor.org; akpm@linux-foundation.org; linux-mm@kvack.org;
> > kamezawa.hiroyuki@gmail.com; hannes@cmpxchg.org; yinghan@google.com
> > Subject: Re: [PATCH 2/2] Drivers: hv: balloon: Support 2M page allocations for
> > ballooning
> > 
> > On Sat 16-03-13 14:42:05, K. Y. Srinivasan wrote:
> > > While ballooning memory out of the guest, attempt 2M allocations first.
> > > If 2M allocations fail, then go for 4K allocations. In cases where we
> > > have performed 2M allocations, split this 2M page so that we can free this
> > > page at 4K granularity (when the host returns the memory).
> > 
> > Maybe I am missing something but what is the advantage of 2M allocation
> > when you split it up immediately so you are not using it as a huge page?
> 
> The Hyper-V ballooning protocol specifies the pages being ballooned as
> page ranges - start_pfn: number_of_pfns. So, when the guest balloon
> is inflating and I am able to allocate 2M pages, I will be able to
> represent 512 contiguous pages in one 64 bit entry and this makes the
> ballooning operation that much more efficient. The reason I split the
> page is that the host does not guarantee that when it returns the
> memory to the guest, it will return in any particular granularity and
> so I have to be able to free this memory in 4K granularity. This is
> the corner case that I will have to handle.

Thanks for the clarification. I think this information would be valuable
in the changelog.
-- 
Michal Hocko
SUSE Labs

RE: [PATCH 2/2] Drivers: hv: balloon: Support 2M page allocations for ballooning

[KY Srinivasan]

> -----Original Message-----
> From: Michal Hocko [mailto:mhocko@suse.cz]
> Sent: Monday, March 18, 2013 10:13 AM
> To: KY Srinivasan
> Cc: gregkh@linuxfoundation.org; linux-kernel@vger.kernel.org;
> devel@linuxdriverproject.org; olaf@aepfle.de; apw@canonical.com;
> andi@firstfloor.org; akpm@linux-foundation.org; linux-mm@kvack.org;
> kamezawa.hiroyuki@gmail.com; hannes@cmpxchg.org; yinghan@google.com
> Subject: Re: [PATCH 2/2] Drivers: hv: balloon: Support 2M page allocations for
> ballooning
> 
> On Mon 18-03-13 13:44:05, KY Srinivasan wrote:
> >
> >
> > > -----Original Message-----
> > > From: Michal Hocko [mailto:mhocko@suse.cz]
> > > Sent: Monday, March 18, 2013 6:53 AM
> > > To: KY Srinivasan
> > > Cc: gregkh@linuxfoundation.org; linux-kernel@vger.kernel.org;
> > > devel@linuxdriverproject.org; olaf@aepfle.de; apw@canonical.com;
> > > andi@firstfloor.org; akpm@linux-foundation.org; linux-mm@kvack.org;
> > > kamezawa.hiroyuki@gmail.com; hannes@cmpxchg.org;
> yinghan@google.com
> > > Subject: Re: [PATCH 2/2] Drivers: hv: balloon: Support 2M page allocations for
> > > ballooning
> > >
> > > On Sat 16-03-13 14:42:05, K. Y. Srinivasan wrote:
> > > > While ballooning memory out of the guest, attempt 2M allocations first.
> > > > If 2M allocations fail, then go for 4K allocations. In cases where we
> > > > have performed 2M allocations, split this 2M page so that we can free this
> > > > page at 4K granularity (when the host returns the memory).
> > >
> > > Maybe I am missing something but what is the advantage of 2M allocation
> > > when you split it up immediately so you are not using it as a huge page?
> >
> > The Hyper-V ballooning protocol specifies the pages being ballooned as
> > page ranges - start_pfn: number_of_pfns. So, when the guest balloon
> > is inflating and I am able to allocate 2M pages, I will be able to
> > represent 512 contiguous pages in one 64 bit entry and this makes the
> > ballooning operation that much more efficient. The reason I split the
> > page is that the host does not guarantee that when it returns the
> > memory to the guest, it will return in any particular granularity and
> > so I have to be able to free this memory in 4K granularity. This is
> > the corner case that I will have to handle.
> 
> Thanks for the clarification. I think this information would be valuable
> in the changelog.

Thanks Michal. I will resend the patches with the changes you have suggested. What is your
recommendation with regards which tree the mm patch needs to go through; the Hyper-V balloon
driver patch will go through Greg's tree.

Regards,

K. Y 
> --
> Michal Hocko
> SUSE Labs
> 

Re: [PATCH 2/2] Drivers: hv: balloon: Support 2M page allocations for ballooning

[Michal Hocko]

On Mon 18-03-13 15:08:36, KY Srinivasan wrote:
> What is your recommendation with regards which tree the mm patch needs
> to go through; the Hyper-V balloon driver patch will go through Greg's
> tree.

I would say via Andrew but there are dependencies between those two so a
single tree would be less confusing /me thinks.
-- 
Michal Hocko
SUSE Labs