how to avoid allocating or freeze MOVABLE memory in userspace

how to avoid allocating or freeze MOVABLE memory in userspace

[Haojian Zhuang]

Hi all,

I have one question on memory migration. As we know, malloc() from
user app will allocate MIGRATE_MOVABLE pages. But if we want to use
this memory as DMA usage, we can't accept MIGRATE_MOVABLE type. Could
we change its behavior before DMA working?

Thanks
Haojian

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: how to avoid allocating or freeze MOVABLE memory in userspace

[Christoph Lameter]

On Fri, 13 Apr 2012, Haojian Zhuang wrote:

> I have one question on memory migration. As we know, malloc() from
> user app will allocate MIGRATE_MOVABLE pages. But if we want to use
> this memory as DMA usage, we can't accept MIGRATE_MOVABLE type. Could
> we change its behavior before DMA working?

MIGRATE_MOVABLE works fine for DMA. If you keep a reference from a device
driver to user pages then you will have to increase the page refcount
which will in turn pin the page and make it non movable for as long as you
keep the refcount.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: how to avoid allocating or freeze MOVABLE memory in userspace

[Haojian Zhuang]

On Sat, Apr 14, 2012 at 2:27 AM, Christoph Lameter <cl@linux.com> wrote:
> On Fri, 13 Apr 2012, Haojian Zhuang wrote:
>
>> I have one question on memory migration. As we know, malloc() from
>> user app will allocate MIGRATE_MOVABLE pages. But if we want to use
>> this memory as DMA usage, we can't accept MIGRATE_MOVABLE type. Could
>> we change its behavior before DMA working?
>
> MIGRATE_MOVABLE works fine for DMA. If you keep a reference from a device
> driver to user pages then you will have to increase the page refcount
> which will in turn pin the page and make it non movable for as long as you
> keep the refcount.

Hi Christoph,

Thanks for your illustration. But it's a little abstract. Could you
give me a simple example
or show me the code?

Best Regards
Haojian

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: how to avoid allocating or freeze MOVABLE memory in userspace

[Christoph Lameter]

On Sat, 14 Apr 2012, Haojian Zhuang wrote:

> On Sat, Apr 14, 2012 at 2:27 AM, Christoph Lameter <cl@linux.com> wrote:
> > On Fri, 13 Apr 2012, Haojian Zhuang wrote:
> >
> >> I have one question on memory migration. As we know, malloc() from
> >> user app will allocate MIGRATE_MOVABLE pages. But if we want to use
> >> this memory as DMA usage, we can't accept MIGRATE_MOVABLE type. Could
> >> we change its behavior before DMA working?
> >
> > MIGRATE_MOVABLE works fine for DMA. If you keep a reference from a device
> > driver to user pages then you will have to increase the page refcount
> > which will in turn pin the page and make it non movable for as long as you
> > keep the refcount.
>
> Hi Christoph,
>
> Thanks for your illustration. But it's a little abstract. Could you
> give me a simple example
> or show me the code?

Run get_user_pages() on the memory you are interest in pinning. See how
other drivers do that by looking up other use cases. F.e. ib_umem_get()
does a similar thing.



--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: how to avoid allocating or freeze MOVABLE memory in userspace

[Haojian Zhuang]

On Mon, Apr 16, 2012 at 9:55 PM, Christoph Lameter <cl@linux.com> wrote:
> On Sat, 14 Apr 2012, Haojian Zhuang wrote:
>
>> On Sat, Apr 14, 2012 at 2:27 AM, Christoph Lameter <cl@linux.com> wrote:
>> > On Fri, 13 Apr 2012, Haojian Zhuang wrote:
>> >
>> >> I have one question on memory migration. As we know, malloc() from
>> >> user app will allocate MIGRATE_MOVABLE pages. But if we want to use
>> >> this memory as DMA usage, we can't accept MIGRATE_MOVABLE type. Could
>> >> we change its behavior before DMA working?
>> >
>> > MIGRATE_MOVABLE works fine for DMA. If you keep a reference from a device
>> > driver to user pages then you will have to increase the page refcount
>> > which will in turn pin the page and make it non movable for as long as you
>> > keep the refcount.
>>
>> Hi Christoph,
>>
>> Thanks for your illustration. But it's a little abstract. Could you
>> give me a simple example
>> or show me the code?
>
> Run get_user_pages() on the memory you are interest in pinning. See how
> other drivers do that by looking up other use cases. F.e. ib_umem_get()
> does a similar thing.
>
>
Got it. And I think there's conflict in CMA.

For example, user process A malloc() memory, page->_count is 1. After
using get_user_pages()
in device driver for DMA usage, page->_count becomes 2.

If the page is in CMA region, it results migrate_pages() returns
-EAGAIN. But error handling in CMA is in below.

                ret = alloc_contig_range(pfn, pfn + count, MIGRATE_CMA);
                if (ret == 0) {
                        bitmap_set(cma->bitmap, pageno, count);
                        break;
                } else if (ret != -EBUSY) {
                        goto error;
                }

Since EAGAIN doesn't equal to EBUSY, dma_alloc_from_contiguous()
aborts. Should dma_alloc_from_contiguous() handle EAGAIN?

Best Regards
Haojian

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: how to avoid allocating or freeze MOVABLE memory in userspace

[Christoph Lameter]

On Wed, 18 Apr 2012, Haojian Zhuang wrote:

> > Run get_user_pages() on the memory you are interest in pinning. See how
> > other drivers do that by looking up other use cases. F.e. ib_umem_get()
> > does a similar thing.
> >
> >
> Got it. And I think there's conflict in CMA.
>
> For example, user process A malloc() memory, page->_count is 1. After
> using get_user_pages()
> in device driver for DMA usage, page->_count becomes 2.
>
> If the page is in CMA region, it results migrate_pages() returns
> -EAGAIN. But error handling in CMA is in below.

The increase of the page count should be temporary. That is why
migrate_pages() uses -EAGAIN to signify a temporary inability to migrate
the page.

Xen uses a page flag for pinned pages. IMHO that could be generalized and
used instead of increasing the page count. Or it could be checked in
addition and change the return value of migrate_pages().

>                 ret = alloc_contig_range(pfn, pfn + count, MIGRATE_CMA);
>                 if (ret == 0) {
>                         bitmap_set(cma->bitmap, pageno, count);
>                         break;
>                 } else if (ret != -EBUSY) {
>                         goto error;
>                 }
>
> Since EAGAIN doesn't equal to EBUSY, dma_alloc_from_contiguous()
> aborts. Should dma_alloc_from_contiguous() handle EAGAIN?

You need to talk to the CMA developers for this one. If there is a pinned
page in that range then definitely alloc_contig_range needs to fail. In
the case of EAGAIN (and correct marking of pinned pages elsewhere in the
kernel) we could handle the EGAIN return value by trying again.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

RE: how to avoid allocating or freeze MOVABLE memory in userspace

[Marek Szyprowski]

Hello,

On Wednesday, April 18, 2012 9:37 AM Haojian Zhuang wrote:

> On Mon, Apr 16, 2012 at 9:55 PM, Christoph Lameter <cl@linux.com> wrote:
> > On Sat, 14 Apr 2012, Haojian Zhuang wrote:
> >
> >> On Sat, Apr 14, 2012 at 2:27 AM, Christoph Lameter <cl@linux.com> wrote:
> >> > On Fri, 13 Apr 2012, Haojian Zhuang wrote:
> >> >
> >> >> I have one question on memory migration. As we know, malloc() from
> >> >> user app will allocate MIGRATE_MOVABLE pages. But if we want to use
> >> >> this memory as DMA usage, we can't accept MIGRATE_MOVABLE type. Could
> >> >> we change its behavior before DMA working?
> >> >
> >> > MIGRATE_MOVABLE works fine for DMA. If you keep a reference from a device
> >> > driver to user pages then you will have to increase the page refcount
> >> > which will in turn pin the page and make it non movable for as long as you
> >> > keep the refcount.
> >>
> >> Hi Christoph,
> >>
> >> Thanks for your illustration. But it's a little abstract. Could you
> >> give me a simple example
> >> or show me the code?
> >
> > Run get_user_pages() on the memory you are interest in pinning. See how
> > other drivers do that by looking up other use cases. F.e. ib_umem_get()
> > does a similar thing.
> >
> >
> Got it. And I think there's conflict in CMA.
> 
> For example, user process A malloc() memory, page->_count is 1. After
> using get_user_pages()
> in device driver for DMA usage, page->_count becomes 2.
> 
> If the page is in CMA region, it results migrate_pages() returns
> -EAGAIN. But error handling in CMA is in below.
> 
>                 ret = alloc_contig_range(pfn, pfn + count, MIGRATE_CMA);
>                 if (ret == 0) {
>                         bitmap_set(cma->bitmap, pageno, count);
>                         break;
>                 } else if (ret != -EBUSY) {
>                         goto error;
>                 }
> 
> Since EAGAIN doesn't equal to EBUSY, dma_alloc_from_contiguous()
> aborts. Should dma_alloc_from_contiguous() handle EAGAIN?

Yes, it definitely should threat EAGAIN the same way as EBUSY. I think
I've double checked that alloc_contig_range return only EBUSY in case of
migration failure, but it looks that I need to check it once again. Thanks
for spotting the possible bug.

Best regards
-- 
Marek Szyprowski
Samsung Poland R&D Center


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

RE: how to avoid allocating or freeze MOVABLE memory in userspace

[Marek Szyprowski]

Hello,

On Wednesday, April 18, 2012 9:37 AM Haojian Zhuang wrote:

> On Mon, Apr 16, 2012 at 9:55 PM, Christoph Lameter <cl@linux.com> wrote:
> > On Sat, 14 Apr 2012, Haojian Zhuang wrote:
> >
> >> On Sat, Apr 14, 2012 at 2:27 AM, Christoph Lameter <cl@linux.com> wrote:
> >> > On Fri, 13 Apr 2012, Haojian Zhuang wrote:
> >> >
> >> >> I have one question on memory migration. As we know, malloc() from
> >> >> user app will allocate MIGRATE_MOVABLE pages. But if we want to use
> >> >> this memory as DMA usage, we can't accept MIGRATE_MOVABLE type. Could
> >> >> we change its behavior before DMA working?
> >> >
> >> > MIGRATE_MOVABLE works fine for DMA. If you keep a reference from a device
> >> > driver to user pages then you will have to increase the page refcount
> >> > which will in turn pin the page and make it non movable for as long as you
> >> > keep the refcount.
> >>
> >> Hi Christoph,
> >>
> >> Thanks for your illustration. But it's a little abstract. Could you
> >> give me a simple example
> >> or show me the code?
> >
> > Run get_user_pages() on the memory you are interest in pinning. See how
> > other drivers do that by looking up other use cases. F.e. ib_umem_get()
> > does a similar thing.
> >
> >
> Got it. And I think there's conflict in CMA.
> 
> For example, user process A malloc() memory, page->_count is 1. After
> using get_user_pages()
> in device driver for DMA usage, page->_count becomes 2.
> 
> If the page is in CMA region, it results migrate_pages() returns
> -EAGAIN. But error handling in CMA is in below.
> 
>                 ret = alloc_contig_range(pfn, pfn + count, MIGRATE_CMA);
>                 if (ret == 0) {
>                         bitmap_set(cma->bitmap, pageno, count);
>                         break;
>                 } else if (ret != -EBUSY) {
>                         goto error;
>                 }
> 
> Since EAGAIN doesn't equal to EBUSY, dma_alloc_from_contiguous()
> aborts. Should dma_alloc_from_contiguous() handle EAGAIN?
> 

I've checked again and it is really not possible for migrate_pages() to return -EAGAIN, 
so the CMA code is correct. The only special case which needs retry is -EBUSY.

Best regards
-- 
Marek Szyprowski
Samsung Poland R&D Center


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>