Re: xen: Can't insert balloon page into VM userspace (WAS Re: [Xen-devel] [linux-linus bisection] complete test-arm64-arm64-xl-xsm)

Re: xen: Can't insert balloon page into VM userspace (WAS Re: [Xen-devel] [linux-linus bisection] complete test-arm64-arm64-xl-xsm)

[Matthew Wilcox]

On Tue, Mar 12, 2019 at 05:05:39PM +0000, Julien Grall wrote:
> On 3/12/19 3:59 PM, Julien Grall wrote:
> > It looks like all the arm test for linus [1] and next [2] tree
> > are now failing. x86 seems to be mostly ok.
> > 
> > The bisector fingered the following commit:
> > 
> > commit 0ee930e6cafa048c1925893d0ca89918b2814f2c
> > Author: Matthew Wilcox <willy@infradead.org>
> > Date:   Tue Mar 5 15:46:06 2019 -0800
> > 
> >      mm/memory.c: prevent mapping typed pages to userspace
> >      Pages which use page_type must never be mapped to userspace as it would
> >      destroy their page type.  Add an explicit check for this instead of
> >      assuming that kernel drivers always get this right.

Oh good, it found a real problem.

> It turns out the problem is because the balloon driver will call
> __SetPageOffline() on allocated page. Therefore the page has a type and
> vm_insert_pages will deny the insertion.
> 
> My knowledge is quite limited in this area. So I am not sure how we can
> solve the problem.
> 
> I would appreciate if someone could provide input of to fix the mapping.

I don't know the balloon driver, so I don't know why it was doing this,
but what it was doing was Wrong and has been since 2014 with:

commit d6d86c0a7f8ddc5b38cf089222cb1d9540762dc2
Author: Konstantin Khlebnikov <k.khlebnikov@samsung.com>
Date:   Thu Oct 9 15:29:27 2014 -0700

    mm/balloon_compaction: redesign ballooned pages management

If ballooned pages are supposed to be mapped into userspace, you can't mark
them as ballooned pages using the mapcount field.

Re: xen: Can't insert balloon page into VM userspace (WAS Re: [Xen-devel] [linux-linus bisection] complete test-arm64-arm64-xl-xsm)

[David Hildenbrand]

On 12.03.19 18:14, Matthew Wilcox wrote:
> On Tue, Mar 12, 2019 at 05:05:39PM +0000, Julien Grall wrote:
>> On 3/12/19 3:59 PM, Julien Grall wrote:
>>> It looks like all the arm test for linus [1] and next [2] tree
>>> are now failing. x86 seems to be mostly ok.
>>>
>>> The bisector fingered the following commit:
>>>
>>> commit 0ee930e6cafa048c1925893d0ca89918b2814f2c
>>> Author: Matthew Wilcox <willy@infradead.org>
>>> Date:   Tue Mar 5 15:46:06 2019 -0800
>>>
>>>      mm/memory.c: prevent mapping typed pages to userspace
>>>      Pages which use page_type must never be mapped to userspace as it would
>>>      destroy their page type.  Add an explicit check for this instead of
>>>      assuming that kernel drivers always get this right.
> 
> Oh good, it found a real problem.
> 
>> It turns out the problem is because the balloon driver will call
>> __SetPageOffline() on allocated page. Therefore the page has a type and
>> vm_insert_pages will deny the insertion.
>>
>> My knowledge is quite limited in this area. So I am not sure how we can
>> solve the problem.
>>
>> I would appreciate if someone could provide input of to fix the mapping.
> 
> I don't know the balloon driver, so I don't know why it was doing this,
> but what it was doing was Wrong and has been since 2014 with:
> 
> commit d6d86c0a7f8ddc5b38cf089222cb1d9540762dc2
> Author: Konstantin Khlebnikov <k.khlebnikov@samsung.com>
> Date:   Thu Oct 9 15:29:27 2014 -0700
> 
>     mm/balloon_compaction: redesign ballooned pages management
> 
> If ballooned pages are supposed to be mapped into userspace, you can't mark
> them as ballooned pages using the mapcount field.
> 

Asking myself why anybody would want to map balloon inflated pages into
user space (this just sounds plain wrong but my understanding to what
XEN balloon driver does might be limited), but I assume the easy fix
would be to revert


commit 2f085ff37d08ecbc7849d5abb9424bd7927dda1d
Author: David Hildenbrand <david@redhat.com>
Date:   Wed Mar 6 11:42:24 2019 +1100

    xen/balloon: mark inflated pages PG_offline

    Mark inflated and never onlined pages PG_offline, to tell the world that
    the content is stale and should not be dumped.


-- 

Thanks,

David / dhildenb

Re: [Xen-devel] xen: Can't insert balloon page into VM userspace (WAS Re: [linux-linus bisection] complete test-arm64-arm64-xl-xsm)

[Andrew Cooper]

On 12/03/2019 17:18, David Hildenbrand wrote:
> On 12.03.19 18:14, Matthew Wilcox wrote:
>> On Tue, Mar 12, 2019 at 05:05:39PM +0000, Julien Grall wrote:
>>> On 3/12/19 3:59 PM, Julien Grall wrote:
>>>> It looks like all the arm test for linus [1] and next [2] tree
>>>> are now failing. x86 seems to be mostly ok.
>>>>
>>>> The bisector fingered the following commit:
>>>>
>>>> commit 0ee930e6cafa048c1925893d0ca89918b2814f2c
>>>> Author: Matthew Wilcox <willy@infradead.org>
>>>> Date:   Tue Mar 5 15:46:06 2019 -0800
>>>>
>>>>      mm/memory.c: prevent mapping typed pages to userspace
>>>>      Pages which use page_type must never be mapped to userspace as it would
>>>>      destroy their page type.  Add an explicit check for this instead of
>>>>      assuming that kernel drivers always get this right.
>> Oh good, it found a real problem.
>>
>>> It turns out the problem is because the balloon driver will call
>>> __SetPageOffline() on allocated page. Therefore the page has a type and
>>> vm_insert_pages will deny the insertion.
>>>
>>> My knowledge is quite limited in this area. So I am not sure how we can
>>> solve the problem.
>>>
>>> I would appreciate if someone could provide input of to fix the mapping.
>> I don't know the balloon driver, so I don't know why it was doing this,
>> but what it was doing was Wrong and has been since 2014 with:
>>
>> commit d6d86c0a7f8ddc5b38cf089222cb1d9540762dc2
>> Author: Konstantin Khlebnikov <k.khlebnikov@samsung.com>
>> Date:   Thu Oct 9 15:29:27 2014 -0700
>>
>>     mm/balloon_compaction: redesign ballooned pages management
>>
>> If ballooned pages are supposed to be mapped into userspace, you can't mark
>> them as ballooned pages using the mapcount field.
>>
> Asking myself why anybody would want to map balloon inflated pages into
> user space (this just sounds plain wrong but my understanding to what
> XEN balloon driver does might be limited), but I assume the easy fix
> would be to revert

I suspect the bug here is that the balloon driver is (ab)used for a
second purpose - to create a hole in pfn space to map some other bits of
shared memory into.

I think at the end of the day, what is needed is a struct page_info
which looks like normal RAM, but the backing for which can be altered by
hypercall to map other things.

~Andrew

Re: [Xen-devel] xen: Can't insert balloon page into VM userspace (WAS Re: [linux-linus bisection] complete test-arm64-arm64-xl-xsm)

[Boris Ostrovsky]

On 3/12/19 1:24 PM, Andrew Cooper wrote:
> On 12/03/2019 17:18, David Hildenbrand wrote:
>> On 12.03.19 18:14, Matthew Wilcox wrote:
>>> On Tue, Mar 12, 2019 at 05:05:39PM +0000, Julien Grall wrote:
>>>> On 3/12/19 3:59 PM, Julien Grall wrote:
>>>>> It looks like all the arm test for linus [1] and next [2] tree
>>>>> are now failing. x86 seems to be mostly ok.
>>>>>
>>>>> The bisector fingered the following commit:
>>>>>
>>>>> commit 0ee930e6cafa048c1925893d0ca89918b2814f2c
>>>>> Author: Matthew Wilcox <willy@infradead.org>
>>>>> Date:   Tue Mar 5 15:46:06 2019 -0800
>>>>>
>>>>>      mm/memory.c: prevent mapping typed pages to userspace
>>>>>      Pages which use page_type must never be mapped to userspace as it would
>>>>>      destroy their page type.  Add an explicit check for this instead of
>>>>>      assuming that kernel drivers always get this right.
>>> Oh good, it found a real problem.
>>>
>>>> It turns out the problem is because the balloon driver will call
>>>> __SetPageOffline() on allocated page. Therefore the page has a type and
>>>> vm_insert_pages will deny the insertion.
>>>>
>>>> My knowledge is quite limited in this area. So I am not sure how we can
>>>> solve the problem.
>>>>
>>>> I would appreciate if someone could provide input of to fix the mapping.
>>> I don't know the balloon driver, so I don't know why it was doing this,
>>> but what it was doing was Wrong and has been since 2014 with:
>>>
>>> commit d6d86c0a7f8ddc5b38cf089222cb1d9540762dc2
>>> Author: Konstantin Khlebnikov <k.khlebnikov@samsung.com>
>>> Date:   Thu Oct 9 15:29:27 2014 -0700
>>>
>>>     mm/balloon_compaction: redesign ballooned pages management
>>>
>>> If ballooned pages are supposed to be mapped into userspace, you can't mark
>>> them as ballooned pages using the mapcount field.
>>>
>> Asking myself why anybody would want to map balloon inflated pages into
>> user space (this just sounds plain wrong but my understanding to what
>> XEN balloon driver does might be limited), but I assume the easy fix
>> would be to revert
> I suspect the bug here is that the balloon driver is (ab)used for a
> second purpose

Yes. And its name is alloc_xenballooned_pages().

-boris

>  - to create a hole in pfn space to map some other bits of
> shared memory into.
>
> I think at the end of the day, what is needed is a struct page_info
> which looks like normal RAM, but the backing for which can be altered by
> hypercall to map other things.
>
> ~Andrew

Re: [Xen-devel] xen: Can't insert balloon page into VM userspace (WAS Re: [linux-linus bisection] complete test-arm64-arm64-xl-xsm)

[Andrew Cooper]

On 12/03/2019 18:02, Boris Ostrovsky wrote:
> On 3/12/19 1:24 PM, Andrew Cooper wrote:
>> On 12/03/2019 17:18, David Hildenbrand wrote:
>>> On 12.03.19 18:14, Matthew Wilcox wrote:
>>>> On Tue, Mar 12, 2019 at 05:05:39PM +0000, Julien Grall wrote:
>>>>> On 3/12/19 3:59 PM, Julien Grall wrote:
>>>>>> It looks like all the arm test for linus [1] and next [2] tree
>>>>>> are now failing. x86 seems to be mostly ok.
>>>>>>
>>>>>> The bisector fingered the following commit:
>>>>>>
>>>>>> commit 0ee930e6cafa048c1925893d0ca89918b2814f2c
>>>>>> Author: Matthew Wilcox <willy@infradead.org>
>>>>>> Date:   Tue Mar 5 15:46:06 2019 -0800
>>>>>>
>>>>>>      mm/memory.c: prevent mapping typed pages to userspace
>>>>>>      Pages which use page_type must never be mapped to userspace as it would
>>>>>>      destroy their page type.  Add an explicit check for this instead of
>>>>>>      assuming that kernel drivers always get this right.
>>>> Oh good, it found a real problem.
>>>>
>>>>> It turns out the problem is because the balloon driver will call
>>>>> __SetPageOffline() on allocated page. Therefore the page has a type and
>>>>> vm_insert_pages will deny the insertion.
>>>>>
>>>>> My knowledge is quite limited in this area. So I am not sure how we can
>>>>> solve the problem.
>>>>>
>>>>> I would appreciate if someone could provide input of to fix the mapping.
>>>> I don't know the balloon driver, so I don't know why it was doing this,
>>>> but what it was doing was Wrong and has been since 2014 with:
>>>>
>>>> commit d6d86c0a7f8ddc5b38cf089222cb1d9540762dc2
>>>> Author: Konstantin Khlebnikov <k.khlebnikov@samsung.com>
>>>> Date:   Thu Oct 9 15:29:27 2014 -0700
>>>>
>>>>     mm/balloon_compaction: redesign ballooned pages management
>>>>
>>>> If ballooned pages are supposed to be mapped into userspace, you can't mark
>>>> them as ballooned pages using the mapcount field.
>>>>
>>> Asking myself why anybody would want to map balloon inflated pages into
>>> user space (this just sounds plain wrong but my understanding to what
>>> XEN balloon driver does might be limited), but I assume the easy fix
>>> would be to revert
>> I suspect the bug here is that the balloon driver is (ab)used for a
>> second purpose
> Yes. And its name is alloc_xenballooned_pages().

FWIW, I did express my views that this was a BadIdea(tm) when that logic
was first introduced.

But yes - now is clearly the time to fix this properly.

~Andrew

Re: [Xen-devel] xen: Can't insert balloon page into VM userspace (WAS Re: [linux-linus bisection] complete test-arm64-arm64-xl-xsm)

[David Hildenbrand]

On 12.03.19 19:02, Boris Ostrovsky wrote:
> On 3/12/19 1:24 PM, Andrew Cooper wrote:
>> On 12/03/2019 17:18, David Hildenbrand wrote:
>>> On 12.03.19 18:14, Matthew Wilcox wrote:
>>>> On Tue, Mar 12, 2019 at 05:05:39PM +0000, Julien Grall wrote:
>>>>> On 3/12/19 3:59 PM, Julien Grall wrote:
>>>>>> It looks like all the arm test for linus [1] and next [2] tree
>>>>>> are now failing. x86 seems to be mostly ok.
>>>>>>
>>>>>> The bisector fingered the following commit:
>>>>>>
>>>>>> commit 0ee930e6cafa048c1925893d0ca89918b2814f2c
>>>>>> Author: Matthew Wilcox <willy@infradead.org>
>>>>>> Date:   Tue Mar 5 15:46:06 2019 -0800
>>>>>>
>>>>>>      mm/memory.c: prevent mapping typed pages to userspace
>>>>>>      Pages which use page_type must never be mapped to userspace as it would
>>>>>>      destroy their page type.  Add an explicit check for this instead of
>>>>>>      assuming that kernel drivers always get this right.
>>>> Oh good, it found a real problem.
>>>>
>>>>> It turns out the problem is because the balloon driver will call
>>>>> __SetPageOffline() on allocated page. Therefore the page has a type and
>>>>> vm_insert_pages will deny the insertion.
>>>>>
>>>>> My knowledge is quite limited in this area. So I am not sure how we can
>>>>> solve the problem.
>>>>>
>>>>> I would appreciate if someone could provide input of to fix the mapping.
>>>> I don't know the balloon driver, so I don't know why it was doing this,
>>>> but what it was doing was Wrong and has been since 2014 with:
>>>>
>>>> commit d6d86c0a7f8ddc5b38cf089222cb1d9540762dc2
>>>> Author: Konstantin Khlebnikov <k.khlebnikov@samsung.com>
>>>> Date:   Thu Oct 9 15:29:27 2014 -0700
>>>>
>>>>     mm/balloon_compaction: redesign ballooned pages management
>>>>
>>>> If ballooned pages are supposed to be mapped into userspace, you can't mark
>>>> them as ballooned pages using the mapcount field.
>>>>
>>> Asking myself why anybody would want to map balloon inflated pages into
>>> user space (this just sounds plain wrong but my understanding to what
>>> XEN balloon driver does might be limited), but I assume the easy fix
>>> would be to revert
>> I suspect the bug here is that the balloon driver is (ab)used for a
>> second purpose
> 
> Yes. And its name is alloc_xenballooned_pages().
> 

Haven't had a look at the code yet, but would another temporary fix be
to clear/set PG_offline when allocating/freeing a ballooned page?
(assuming here that only such pages will be mapped to user space)

-- 

Thanks,

David / dhildenb

Re: [Xen-devel] xen: Can't insert balloon page into VM userspace (WAS Re: [linux-linus bisection] complete test-arm64-arm64-xl-xsm)

[David Hildenbrand]

On 12.03.19 19:23, David Hildenbrand wrote:
> On 12.03.19 19:02, Boris Ostrovsky wrote:
>> On 3/12/19 1:24 PM, Andrew Cooper wrote:
>>> On 12/03/2019 17:18, David Hildenbrand wrote:
>>>> On 12.03.19 18:14, Matthew Wilcox wrote:
>>>>> On Tue, Mar 12, 2019 at 05:05:39PM +0000, Julien Grall wrote:
>>>>>> On 3/12/19 3:59 PM, Julien Grall wrote:
>>>>>>> It looks like all the arm test for linus [1] and next [2] tree
>>>>>>> are now failing. x86 seems to be mostly ok.
>>>>>>>
>>>>>>> The bisector fingered the following commit:
>>>>>>>
>>>>>>> commit 0ee930e6cafa048c1925893d0ca89918b2814f2c
>>>>>>> Author: Matthew Wilcox <willy@infradead.org>
>>>>>>> Date:   Tue Mar 5 15:46:06 2019 -0800
>>>>>>>
>>>>>>>      mm/memory.c: prevent mapping typed pages to userspace
>>>>>>>      Pages which use page_type must never be mapped to userspace as it would
>>>>>>>      destroy their page type.  Add an explicit check for this instead of
>>>>>>>      assuming that kernel drivers always get this right.
>>>>> Oh good, it found a real problem.
>>>>>
>>>>>> It turns out the problem is because the balloon driver will call
>>>>>> __SetPageOffline() on allocated page. Therefore the page has a type and
>>>>>> vm_insert_pages will deny the insertion.
>>>>>>
>>>>>> My knowledge is quite limited in this area. So I am not sure how we can
>>>>>> solve the problem.
>>>>>>
>>>>>> I would appreciate if someone could provide input of to fix the mapping.
>>>>> I don't know the balloon driver, so I don't know why it was doing this,
>>>>> but what it was doing was Wrong and has been since 2014 with:
>>>>>
>>>>> commit d6d86c0a7f8ddc5b38cf089222cb1d9540762dc2
>>>>> Author: Konstantin Khlebnikov <k.khlebnikov@samsung.com>
>>>>> Date:   Thu Oct 9 15:29:27 2014 -0700
>>>>>
>>>>>     mm/balloon_compaction: redesign ballooned pages management
>>>>>
>>>>> If ballooned pages are supposed to be mapped into userspace, you can't mark
>>>>> them as ballooned pages using the mapcount field.
>>>>>
>>>> Asking myself why anybody would want to map balloon inflated pages into
>>>> user space (this just sounds plain wrong but my understanding to what
>>>> XEN balloon driver does might be limited), but I assume the easy fix
>>>> would be to revert
>>> I suspect the bug here is that the balloon driver is (ab)used for a
>>> second purpose
>>
>> Yes. And its name is alloc_xenballooned_pages().
>>
> 
> Haven't had a look at the code yet, but would another temporary fix be
> to clear/set PG_offline when allocating/freeing a ballooned page?
> (assuming here that only such pages will be mapped to user space)
> 

I guess something like this could do the trick if I understood it correctly:

diff --git a/drivers/xen/balloon.c b/drivers/xen/balloon.c
index 39b229f9e256..d37dd5bb7a8f 100644
--- a/drivers/xen/balloon.c
+++ b/drivers/xen/balloon.c
@@ -604,6 +604,7 @@ int alloc_xenballooned_pages(int nr_pages, struct
page **pages)
        while (pgno < nr_pages) {
                page = balloon_retrieve(true);
                if (page) {
+                       __ClearPageOffline(page);
                        pages[pgno++] = page;
 #ifdef CONFIG_XEN_HAVE_PVMMU
                        /*
@@ -645,8 +646,10 @@ void free_xenballooned_pages(int nr_pages, struct
page **pages)
        mutex_lock(&balloon_mutex);

        for (i = 0; i < nr_pages; i++) {
-               if (pages[i])
+               if (pages[i]) {
+                       __SetPageOffline(pages[i]);
                        balloon_append(pages[i]);
+               }
        }

        balloon_stats.target_unpopulated -= nr_pages;


At least this way, the pages allocated (and thus eventually mapped to
user space) would not be marked, but the other ones would remain marked
and could be excluded by makedumptool.

-- 

Thanks,

David / dhildenb

Re: [Xen-devel] xen: Can't insert balloon page into VM userspace (WAS Re: [linux-linus bisection] complete test-arm64-arm64-xl-xsm)

[Juergen Gross]

On 12/03/2019 20:46, David Hildenbrand wrote:
> On 12.03.19 19:23, David Hildenbrand wrote:
>> On 12.03.19 19:02, Boris Ostrovsky wrote:
>>> On 3/12/19 1:24 PM, Andrew Cooper wrote:
>>>> On 12/03/2019 17:18, David Hildenbrand wrote:
>>>>> On 12.03.19 18:14, Matthew Wilcox wrote:
>>>>>> On Tue, Mar 12, 2019 at 05:05:39PM +0000, Julien Grall wrote:
>>>>>>> On 3/12/19 3:59 PM, Julien Grall wrote:
>>>>>>>> It looks like all the arm test for linus [1] and next [2] tree
>>>>>>>> are now failing. x86 seems to be mostly ok.
>>>>>>>>
>>>>>>>> The bisector fingered the following commit:
>>>>>>>>
>>>>>>>> commit 0ee930e6cafa048c1925893d0ca89918b2814f2c
>>>>>>>> Author: Matthew Wilcox <willy@infradead.org>
>>>>>>>> Date:   Tue Mar 5 15:46:06 2019 -0800
>>>>>>>>
>>>>>>>>      mm/memory.c: prevent mapping typed pages to userspace
>>>>>>>>      Pages which use page_type must never be mapped to userspace as it would
>>>>>>>>      destroy their page type.  Add an explicit check for this instead of
>>>>>>>>      assuming that kernel drivers always get this right.
>>>>>> Oh good, it found a real problem.
>>>>>>
>>>>>>> It turns out the problem is because the balloon driver will call
>>>>>>> __SetPageOffline() on allocated page. Therefore the page has a type and
>>>>>>> vm_insert_pages will deny the insertion.
>>>>>>>
>>>>>>> My knowledge is quite limited in this area. So I am not sure how we can
>>>>>>> solve the problem.
>>>>>>>
>>>>>>> I would appreciate if someone could provide input of to fix the mapping.
>>>>>> I don't know the balloon driver, so I don't know why it was doing this,
>>>>>> but what it was doing was Wrong and has been since 2014 with:
>>>>>>
>>>>>> commit d6d86c0a7f8ddc5b38cf089222cb1d9540762dc2
>>>>>> Author: Konstantin Khlebnikov <k.khlebnikov@samsung.com>
>>>>>> Date:   Thu Oct 9 15:29:27 2014 -0700
>>>>>>
>>>>>>     mm/balloon_compaction: redesign ballooned pages management
>>>>>>
>>>>>> If ballooned pages are supposed to be mapped into userspace, you can't mark
>>>>>> them as ballooned pages using the mapcount field.
>>>>>>
>>>>> Asking myself why anybody would want to map balloon inflated pages into
>>>>> user space (this just sounds plain wrong but my understanding to what
>>>>> XEN balloon driver does might be limited), but I assume the easy fix
>>>>> would be to revert
>>>> I suspect the bug here is that the balloon driver is (ab)used for a
>>>> second purpose
>>>
>>> Yes. And its name is alloc_xenballooned_pages().
>>>
>>
>> Haven't had a look at the code yet, but would another temporary fix be
>> to clear/set PG_offline when allocating/freeing a ballooned page?
>> (assuming here that only such pages will be mapped to user space)
>>
> 
> I guess something like this could do the trick if I understood it correctly:
> 
> diff --git a/drivers/xen/balloon.c b/drivers/xen/balloon.c
> index 39b229f9e256..d37dd5bb7a8f 100644
> --- a/drivers/xen/balloon.c
> +++ b/drivers/xen/balloon.c
> @@ -604,6 +604,7 @@ int alloc_xenballooned_pages(int nr_pages, struct
> page **pages)
>         while (pgno < nr_pages) {
>                 page = balloon_retrieve(true);
>                 if (page) {
> +                       __ClearPageOffline(page);
>                         pages[pgno++] = page;
>  #ifdef CONFIG_XEN_HAVE_PVMMU
>                         /*
> @@ -645,8 +646,10 @@ void free_xenballooned_pages(int nr_pages, struct
> page **pages)
>         mutex_lock(&balloon_mutex);
> 
>         for (i = 0; i < nr_pages; i++) {
> -               if (pages[i])
> +               if (pages[i]) {
> +                       __SetPageOffline(pages[i]);
>                         balloon_append(pages[i]);
> +               }
>         }
> 
>         balloon_stats.target_unpopulated -= nr_pages;
> 
> 
> At least this way, the pages allocated (and thus eventually mapped to
> user space) would not be marked, but the other ones would remain marked
> and could be excluded by makedumptool.
> 

I think this patch should do the trick. Julien, could you give it a
try? On x86 I can't reproduce your problem easily as dom0 is PV with
plenty of unpopulated pages for grant memory not suffering from
missing "offline" bit.


Juergen

Re: [Xen-devel] xen: Can't insert balloon page into VM userspace (WAS Re: [linux-linus bisection] complete test-arm64-arm64-xl-xsm)

[Julien Grall]

Hi,

On 3/14/19 8:37 AM, Juergen Gross wrote:
> On 12/03/2019 20:46, David Hildenbrand wrote:
>> On 12.03.19 19:23, David Hildenbrand wrote:
>>
>> I guess something like this could do the trick if I understood it correctly:
>>
>> diff --git a/drivers/xen/balloon.c b/drivers/xen/balloon.c
>> index 39b229f9e256..d37dd5bb7a8f 100644
>> --- a/drivers/xen/balloon.c
>> +++ b/drivers/xen/balloon.c
>> @@ -604,6 +604,7 @@ int alloc_xenballooned_pages(int nr_pages, struct
>> page **pages)
>>          while (pgno < nr_pages) {
>>                  page = balloon_retrieve(true);
>>                  if (page) {
>> +                       __ClearPageOffline(page);
>>                          pages[pgno++] = page;
>>   #ifdef CONFIG_XEN_HAVE_PVMMU
>>                          /*
>> @@ -645,8 +646,10 @@ void free_xenballooned_pages(int nr_pages, struct
>> page **pages)
>>          mutex_lock(&balloon_mutex);
>>
>>          for (i = 0; i < nr_pages; i++) {
>> -               if (pages[i])
>> +               if (pages[i]) {
>> +                       __SetPageOffline(pages[i]);
>>                          balloon_append(pages[i]);
>> +               }
>>          }
>>
>>          balloon_stats.target_unpopulated -= nr_pages;
>>
>>
>> At least this way, the pages allocated (and thus eventually mapped to
>> user space) would not be marked, but the other ones would remain marked
>> and could be excluded by makedumptool.
>>
> 
> I think this patch should do the trick. Julien, could you give it a
> try? On x86 I can't reproduce your problem easily as dom0 is PV with
> plenty of unpopulated pages for grant memory not suffering from
> missing "offline" bit.

Sure. I managed to get the console working with the patch suggested by 
David. Feel free to add my tested-by if when you resend it as is.

Cheers,

-- 
Julien Grall

Re: [Xen-devel] xen: Can't insert balloon page into VM userspace (WAS Re: [linux-linus bisection] complete test-arm64-arm64-xl-xsm)

[David Hildenbrand]

On 14.03.19 15:12, Julien Grall wrote:
> Hi,
> 
> On 3/14/19 8:37 AM, Juergen Gross wrote:
>> On 12/03/2019 20:46, David Hildenbrand wrote:
>>> On 12.03.19 19:23, David Hildenbrand wrote:
>>>
>>> I guess something like this could do the trick if I understood it correctly:
>>>
>>> diff --git a/drivers/xen/balloon.c b/drivers/xen/balloon.c
>>> index 39b229f9e256..d37dd5bb7a8f 100644
>>> --- a/drivers/xen/balloon.c
>>> +++ b/drivers/xen/balloon.c
>>> @@ -604,6 +604,7 @@ int alloc_xenballooned_pages(int nr_pages, struct
>>> page **pages)
>>>          while (pgno < nr_pages) {
>>>                  page = balloon_retrieve(true);
>>>                  if (page) {
>>> +                       __ClearPageOffline(page);
>>>                          pages[pgno++] = page;
>>>   #ifdef CONFIG_XEN_HAVE_PVMMU
>>>                          /*
>>> @@ -645,8 +646,10 @@ void free_xenballooned_pages(int nr_pages, struct
>>> page **pages)
>>>          mutex_lock(&balloon_mutex);
>>>
>>>          for (i = 0; i < nr_pages; i++) {
>>> -               if (pages[i])
>>> +               if (pages[i]) {
>>> +                       __SetPageOffline(pages[i]);
>>>                          balloon_append(pages[i]);
>>> +               }
>>>          }
>>>
>>>          balloon_stats.target_unpopulated -= nr_pages;
>>>
>>>
>>> At least this way, the pages allocated (and thus eventually mapped to
>>> user space) would not be marked, but the other ones would remain marked
>>> and could be excluded by makedumptool.
>>>
>>
>> I think this patch should do the trick. Julien, could you give it a
>> try? On x86 I can't reproduce your problem easily as dom0 is PV with
>> plenty of unpopulated pages for grant memory not suffering from
>> missing "offline" bit.
> 
> Sure. I managed to get the console working with the patch suggested by 
> David. Feel free to add my tested-by if when you resend it as is.
> 

Thanks, I will send as proper patch later!

Cheers!

> Cheers,
> 


-- 

Thanks,

David / dhildenb

Re: [Xen-devel] xen: Can't insert balloon page into VM userspace (WAS Re: [linux-linus bisection] complete test-arm64-arm64-xl-xsm)

[Juergen Gross]

On 14/03/2019 15:12, Julien Grall wrote:
> Hi,
> 
> On 3/14/19 8:37 AM, Juergen Gross wrote:
>> On 12/03/2019 20:46, David Hildenbrand wrote:
>>> On 12.03.19 19:23, David Hildenbrand wrote:
>>>
>>> I guess something like this could do the trick if I understood it
>>> correctly:
>>>
>>> diff --git a/drivers/xen/balloon.c b/drivers/xen/balloon.c
>>> index 39b229f9e256..d37dd5bb7a8f 100644
>>> --- a/drivers/xen/balloon.c
>>> +++ b/drivers/xen/balloon.c
>>> @@ -604,6 +604,7 @@ int alloc_xenballooned_pages(int nr_pages, struct
>>> page **pages)
>>>          while (pgno < nr_pages) {
>>>                  page = balloon_retrieve(true);
>>>                  if (page) {
>>> +                       __ClearPageOffline(page);
>>>                          pages[pgno++] = page;
>>>   #ifdef CONFIG_XEN_HAVE_PVMMU
>>>                          /*
>>> @@ -645,8 +646,10 @@ void free_xenballooned_pages(int nr_pages, struct
>>> page **pages)
>>>          mutex_lock(&balloon_mutex);
>>>
>>>          for (i = 0; i < nr_pages; i++) {
>>> -               if (pages[i])
>>> +               if (pages[i]) {
>>> +                       __SetPageOffline(pages[i]);
>>>                          balloon_append(pages[i]);
>>> +               }
>>>          }
>>>
>>>          balloon_stats.target_unpopulated -= nr_pages;
>>>
>>>
>>> At least this way, the pages allocated (and thus eventually mapped to
>>> user space) would not be marked, but the other ones would remain marked
>>> and could be excluded by makedumptool.
>>>
>>
>> I think this patch should do the trick. Julien, could you give it a
>> try? On x86 I can't reproduce your problem easily as dom0 is PV with
>> plenty of unpopulated pages for grant memory not suffering from
>> missing "offline" bit.
> 
> Sure. I managed to get the console working with the patch suggested by
> David. Feel free to add my tested-by if when you resend it as is.

David, could you please send a proper patch with your Sob?


Juergen

Re: [Xen-devel] xen: Can't insert balloon page into VM userspace (WAS Re: [linux-linus bisection] complete test-arm64-arm64-xl-xsm)

[David Hildenbrand]

On 14.03.19 15:15, Juergen Gross wrote:
> On 14/03/2019 15:12, Julien Grall wrote:
>> Hi,
>>
>> On 3/14/19 8:37 AM, Juergen Gross wrote:
>>> On 12/03/2019 20:46, David Hildenbrand wrote:
>>>> On 12.03.19 19:23, David Hildenbrand wrote:
>>>>
>>>> I guess something like this could do the trick if I understood it
>>>> correctly:
>>>>
>>>> diff --git a/drivers/xen/balloon.c b/drivers/xen/balloon.c
>>>> index 39b229f9e256..d37dd5bb7a8f 100644
>>>> --- a/drivers/xen/balloon.c
>>>> +++ b/drivers/xen/balloon.c
>>>> @@ -604,6 +604,7 @@ int alloc_xenballooned_pages(int nr_pages, struct
>>>> page **pages)
>>>>          while (pgno < nr_pages) {
>>>>                  page = balloon_retrieve(true);
>>>>                  if (page) {
>>>> +                       __ClearPageOffline(page);
>>>>                          pages[pgno++] = page;
>>>>   #ifdef CONFIG_XEN_HAVE_PVMMU
>>>>                          /*
>>>> @@ -645,8 +646,10 @@ void free_xenballooned_pages(int nr_pages, struct
>>>> page **pages)
>>>>          mutex_lock(&balloon_mutex);
>>>>
>>>>          for (i = 0; i < nr_pages; i++) {
>>>> -               if (pages[i])
>>>> +               if (pages[i]) {
>>>> +                       __SetPageOffline(pages[i]);
>>>>                          balloon_append(pages[i]);
>>>> +               }
>>>>          }
>>>>
>>>>          balloon_stats.target_unpopulated -= nr_pages;
>>>>
>>>>
>>>> At least this way, the pages allocated (and thus eventually mapped to
>>>> user space) would not be marked, but the other ones would remain marked
>>>> and could be excluded by makedumptool.
>>>>
>>>
>>> I think this patch should do the trick. Julien, could you give it a
>>> try? On x86 I can't reproduce your problem easily as dom0 is PV with
>>> plenty of unpopulated pages for grant memory not suffering from
>>> missing "offline" bit.
>>
>> Sure. I managed to get the console working with the patch suggested by
>> David. Feel free to add my tested-by if when you resend it as is.
> 
> David, could you please send a proper patch with your Sob?
> 

Yes, on it :)

Cheers!

> 
> Juergen
> 


-- 

Thanks,

David / dhildenb

Re: xen: Can't insert balloon page into VM userspace (WAS Re: [Xen-devel] [linux-linus bisection] complete test-arm64-arm64-xl-xsm)

[Julien Grall]

Hi David,

On 3/12/19 5:18 PM, David Hildenbrand wrote:
> On 12.03.19 18:14, Matthew Wilcox wrote:
>> On Tue, Mar 12, 2019 at 05:05:39PM +0000, Julien Grall wrote:
>>> On 3/12/19 3:59 PM, Julien Grall wrote:
>>>> It looks like all the arm test for linus [1] and next [2] tree
>>>> are now failing. x86 seems to be mostly ok.
>>>>
>>>> The bisector fingered the following commit:
>>>>
>>>> commit 0ee930e6cafa048c1925893d0ca89918b2814f2c
>>>> Author: Matthew Wilcox <willy@infradead.org>
>>>> Date:   Tue Mar 5 15:46:06 2019 -0800
>>>>
>>>>       mm/memory.c: prevent mapping typed pages to userspace
>>>>       Pages which use page_type must never be mapped to userspace as it would
>>>>       destroy their page type.  Add an explicit check for this instead of
>>>>       assuming that kernel drivers always get this right.
>>
>> Oh good, it found a real problem.
>>
>>> It turns out the problem is because the balloon driver will call
>>> __SetPageOffline() on allocated page. Therefore the page has a type and
>>> vm_insert_pages will deny the insertion.
>>>
>>> My knowledge is quite limited in this area. So I am not sure how we can
>>> solve the problem.
>>>
>>> I would appreciate if someone could provide input of to fix the mapping.
>>
>> I don't know the balloon driver, so I don't know why it was doing this,
>> but what it was doing was Wrong and has been since 2014 with:
>>
>> commit d6d86c0a7f8ddc5b38cf089222cb1d9540762dc2
>> Author: Konstantin Khlebnikov <k.khlebnikov@samsung.com>
>> Date:   Thu Oct 9 15:29:27 2014 -0700
>>
>>      mm/balloon_compaction: redesign ballooned pages management
>>
>> If ballooned pages are supposed to be mapped into userspace, you can't mark
>> them as ballooned pages using the mapcount field.
>>
> 
> Asking myself why anybody would want to map balloon inflated pages into
> user space (this just sounds plain wrong but my understanding to what
> XEN balloon driver does might be limited), but I assume the easy fix
> would be to revert

Balloon pages are used to map foreign guest pages. As backend PV drivers 
may live in userspace (e.g QEMU, Xenconsoled...) we need to be able to
to insert balloon pages in the VM.

> 
> 
> commit 2f085ff37d08ecbc7849d5abb9424bd7927dda1d

I guess you meant 77c4adf6a6df6f8f39807eaed48eb73d0eb4261e?

I have reverted the patch and can now access the guest console. Is there 
a way to keep this patch and at the same time mapping the page in the 
userspace?


> Author: David Hildenbrand <david@redhat.com>
> Date:   Wed Mar 6 11:42:24 2019 +1100
> 
>      xen/balloon: mark inflated pages PG_offline
> 
>      Mark inflated and never onlined pages PG_offline, to tell the world that
>      the content is stale and should not be dumped.
> 
> 

Cheers,

-- 
Julien Grall

Re: xen: Can't insert balloon page into VM userspace (WAS Re: [Xen-devel] [linux-linus bisection] complete test-arm64-arm64-xl-xsm)

[David Hildenbrand]

On 12.03.19 18:39, Julien Grall wrote:
> Hi David,
> 
> On 3/12/19 5:18 PM, David Hildenbrand wrote:
>> On 12.03.19 18:14, Matthew Wilcox wrote:
>>> On Tue, Mar 12, 2019 at 05:05:39PM +0000, Julien Grall wrote:
>>>> On 3/12/19 3:59 PM, Julien Grall wrote:
>>>>> It looks like all the arm test for linus [1] and next [2] tree
>>>>> are now failing. x86 seems to be mostly ok.
>>>>>
>>>>> The bisector fingered the following commit:
>>>>>
>>>>> commit 0ee930e6cafa048c1925893d0ca89918b2814f2c
>>>>> Author: Matthew Wilcox <willy@infradead.org>
>>>>> Date:   Tue Mar 5 15:46:06 2019 -0800
>>>>>
>>>>>       mm/memory.c: prevent mapping typed pages to userspace
>>>>>       Pages which use page_type must never be mapped to userspace as it would
>>>>>       destroy their page type.  Add an explicit check for this instead of
>>>>>       assuming that kernel drivers always get this right.
>>>
>>> Oh good, it found a real problem.
>>>
>>>> It turns out the problem is because the balloon driver will call
>>>> __SetPageOffline() on allocated page. Therefore the page has a type and
>>>> vm_insert_pages will deny the insertion.
>>>>
>>>> My knowledge is quite limited in this area. So I am not sure how we can
>>>> solve the problem.
>>>>
>>>> I would appreciate if someone could provide input of to fix the mapping.
>>>
>>> I don't know the balloon driver, so I don't know why it was doing this,
>>> but what it was doing was Wrong and has been since 2014 with:
>>>
>>> commit d6d86c0a7f8ddc5b38cf089222cb1d9540762dc2
>>> Author: Konstantin Khlebnikov <k.khlebnikov@samsung.com>
>>> Date:   Thu Oct 9 15:29:27 2014 -0700
>>>
>>>      mm/balloon_compaction: redesign ballooned pages management
>>>
>>> If ballooned pages are supposed to be mapped into userspace, you can't mark
>>> them as ballooned pages using the mapcount field.
>>>
>>
>> Asking myself why anybody would want to map balloon inflated pages into
>> user space (this just sounds plain wrong but my understanding to what
>> XEN balloon driver does might be limited), but I assume the easy fix
>> would be to revert
> 
> Balloon pages are used to map foreign guest pages. As backend PV drivers 
> may live in userspace (e.g QEMU, Xenconsoled...) we need to be able to
> to insert balloon pages in the VM.

Okay, so this is really XEN specific (especially looking at Andrew's
reply). All other balloon drivers told the hypervisor that the inflated
page is dead and it is not going to be use before telling the hypervisor
otherwise. Mapping to user space would violate that contract (and even
be considered harmful in some hypervisor implementations).

> 
>>
>>
>> commit 2f085ff37d08ecbc7849d5abb9424bd7927dda1d
> 
> I guess you meant 77c4adf6a6df6f8f39807eaed48eb73d0eb4261e?

Yes indeed, no idea where that commit id came from :)

> 
> I have reverted the patch and can now access the guest console. Is there 
> a way to keep this patch and at the same time mapping the page in the 
> userspace?

Not without another page flag. And we all know that is unlikely to happen :)

Thanks!

> 
> 
>> Author: David Hildenbrand <david@redhat.com>
>> Date:   Wed Mar 6 11:42:24 2019 +1100
>>
>>      xen/balloon: mark inflated pages PG_offline
>>
>>      Mark inflated and never onlined pages PG_offline, to tell the world that
>>      the content is stale and should not be dumped.
>>
>>
> 
> Cheers,
> 


-- 

Thanks,

David / dhildenb

Re: xen: Can't insert balloon page into VM userspace (WAS Re: [Xen-devel] [linux-linus bisection] complete test-arm64-arm64-xl-xsm)

[David Hildenbrand]

On 12.03.19 18:14, Matthew Wilcox wrote:
> On Tue, Mar 12, 2019 at 05:05:39PM +0000, Julien Grall wrote:
>> On 3/12/19 3:59 PM, Julien Grall wrote:
>>> It looks like all the arm test for linus [1] and next [2] tree
>>> are now failing. x86 seems to be mostly ok.
>>>
>>> The bisector fingered the following commit:
>>>
>>> commit 0ee930e6cafa048c1925893d0ca89918b2814f2c
>>> Author: Matthew Wilcox <willy@infradead.org>
>>> Date:   Tue Mar 5 15:46:06 2019 -0800
>>>
>>>      mm/memory.c: prevent mapping typed pages to userspace
>>>      Pages which use page_type must never be mapped to userspace as it would
>>>      destroy their page type.  Add an explicit check for this instead of
>>>      assuming that kernel drivers always get this right.
> 
> Oh good, it found a real problem.
> 
>> It turns out the problem is because the balloon driver will call
>> __SetPageOffline() on allocated page. Therefore the page has a type and
>> vm_insert_pages will deny the insertion.
>>
>> My knowledge is quite limited in this area. So I am not sure how we can
>> solve the problem.
>>
>> I would appreciate if someone could provide input of to fix the mapping.
> 
> I don't know the balloon driver, so I don't know why it was doing this,
> but what it was doing was Wrong and has been since 2014 with:

Just to clarify on that point, XEN balloon does not use balloon
compaction as far as I know (only virtio-balloon and as far as I know
now also vmware balloon). Both of them don't map any such pages to user
space, so it never was and isn't a problem.

> 
> commit d6d86c0a7f8ddc5b38cf089222cb1d9540762dc2
> Author: Konstantin Khlebnikov <k.khlebnikov@samsung.com>
> Date:   Thu Oct 9 15:29:27 2014 -0700
> 
>     mm/balloon_compaction: redesign ballooned pages management
> 
> If ballooned pages are supposed to be mapped into userspace, you can't mark
> them as ballooned pages using the mapcount field.
> 


-- 

Thanks,

David / dhildenb

Re: xen: Can't insert balloon page into VM userspace (WAS Re: [Xen-devel] [linux-linus bisection] complete test-arm64-arm64-xl-xsm)

[Nadav Amit]

> On Mar 12, 2019, at 10:23 AM, David Hildenbrand <david@redhat.com> wrote:
> 
> On 12.03.19 18:14, Matthew Wilcox wrote:
>> On Tue, Mar 12, 2019 at 05:05:39PM +0000, Julien Grall wrote:
>>> On 3/12/19 3:59 PM, Julien Grall wrote:
>>>> It looks like all the arm test for linus [1] and next [2] tree
>>>> are now failing. x86 seems to be mostly ok.
>>>> 
>>>> The bisector fingered the following commit:
>>>> 
>>>> commit 0ee930e6cafa048c1925893d0ca89918b2814f2c
>>>> Author: Matthew Wilcox <willy@infradead.org>
>>>> Date:   Tue Mar 5 15:46:06 2019 -0800
>>>> 
>>>>     mm/memory.c: prevent mapping typed pages to userspace
>>>>     Pages which use page_type must never be mapped to userspace as it would
>>>>     destroy their page type.  Add an explicit check for this instead of
>>>>     assuming that kernel drivers always get this right.
>> 
>> Oh good, it found a real problem.
>> 
>>> It turns out the problem is because the balloon driver will call
>>> __SetPageOffline() on allocated page. Therefore the page has a type and
>>> vm_insert_pages will deny the insertion.
>>> 
>>> My knowledge is quite limited in this area. So I am not sure how we can
>>> solve the problem.
>>> 
>>> I would appreciate if someone could provide input of to fix the mapping.
>> 
>> I don't know the balloon driver, so I don't know why it was doing this,
>> but what it was doing was Wrong and has been since 2014 with:
> 
> Just to clarify on that point, XEN balloon does not use balloon
> compaction as far as I know (only virtio-balloon and as far as I know
> now also vmware balloon). Both of them don't map any such pages to user
> space, so it never was and isn't a problem.

I still need to submit the next version of the patches, but yes, we are not
about to map them to userspace (dah).