Assigning contiguous memory to a driver domain

Assigning contiguous memory to a driver domain

[Rafal Wojtczuk]

Hello,

Could someone guide me in the right direction with the topic of assigning
contiguous memory to a domain.

I have an issue with a PV domain that is assigned a PCI device. Sometimes, the 
driver fails to load
Sep 13 10:36:43 localhost kernel: [  103.651858] iwlagn 0000:00:01.0:
firmware: requesting iwlwifi-4965-2.ucode
Sep 13 10:36:43 localhost kernel: [  103.669105] iwlagn 0000:00:01.0: loaded
firmware version 228.61.2.24
Sep 13 10:36:43 localhost kernel: [  103.669263] iwlagn 0000:00:01.0: failed
to allocate pci memory

The reason seems to be that the domain does not have enough contiguous
memory, in mfn terms.

Increasing the memory for the domain does not seem to help, unless one
assigns a significant fraction of all system RAM.

I am aware of xen-swiotlb
(http://groups.google.com/group/linux.kernel/browse_thread/thread/2840cb59bec23594),
however these patches do not apply to the kernel being used
(http://qubes-os.org/gitweb/?p=joanna/kernel-dom0.git;a=summary), so I am
looking for something else.

Is it correct that Xen 3.4.3 makes no effort to assign contiguous memory to
a domain ? The arch_setup_meminit function (is it the one responsible for
initial memory allocation ?) does

715     /* allocate guest memory */
716     for ( i = rc = allocsz = 0; (i < dom->total_pages) && !rc; i += allocsz )
717     {
718         allocsz = dom->total_pages - i;
719         if ( allocsz > 1024*1024 )
720             allocsz = 1024*1024;  
721         rc = xc_domain_memory_populate_physmap(
722             dom->guest_xc, dom->guest_domid, allocsz, 0, 0, &dom->p2m_host[i]);
723     }
so, it calls xc_domain_memory_populate_physmap with the fourth parameter
(extent order) being 0, so potentially, there can be literally no contiguous
mfns in the domain ?

In practice, is it safe to assume that the following scenario:
1) boot dom0 with 4G of RAM
2) balolon dom0 down to 1.5G
3) create a PV driver domain (netvm) with, say, 200MB of RAM
will result in netvm having sufficiently contiguous memory ? It seems to
work for me.

Otherwise, does it make sense to add contiguous memory to a domain (via
calling xc_domain_memory_populate_physmap with larger extent_order) just
after domain creation (create the domain paused, and call it before
unpausing it) ? Is there anything else that needs to be done after
calling this function so that the guest is able to use this additional
memory ?

Regards,
Rafal Wojtczuk

Re: Assigning contiguous memory to a driver domain

[Jan Beulich]

>>> On 14.09.10 at 11:24, Rafal Wojtczuk <rafal@invisiblethingslab.com> wrote:
> Hello,
> 
> Could someone guide me in the right direction with the topic of assigning
> contiguous memory to a domain.
> 
> I have an issue with a PV domain that is assigned a PCI device. Sometimes, 
> the 
> driver fails to load
> Sep 13 10:36:43 localhost kernel: [  103.651858] iwlagn 0000:00:01.0:
> firmware: requesting iwlwifi-4965-2.ucode
> Sep 13 10:36:43 localhost kernel: [  103.669105] iwlagn 0000:00:01.0: loaded
> firmware version 228.61.2.24
> Sep 13 10:36:43 localhost kernel: [  103.669263] iwlagn 0000:00:01.0: failed
> to allocate pci memory
> 
> The reason seems to be that the domain does not have enough contiguous
> memory, in mfn terms.

No, how (dis)contiguous the memory of a domain is doesn't matter
here. What matters is whether the domain can *make* the requested
memory contiguous, and that depends on how much contiguous
memory Xen has at the point of the allocation.

Jan

Re: Assigning contiguous memory to a driver domain

[Rafal Wojtczuk]

On Tue, Sep 14, 2010 at 10:36:56AM +0100, Jan Beulich wrote:
> >>> On 14.09.10 at 11:24, Rafal Wojtczuk <rafal@invisiblethingslab.com> wrote:
> > Hello,
> > 
> > Could someone guide me in the right direction with the topic of assigning
> > contiguous memory to a domain.
> > 
> > I have an issue with a PV domain that is assigned a PCI device. Sometimes, 
> > the 
> > driver fails to load
> > Sep 13 10:36:43 localhost kernel: [  103.651858] iwlagn 0000:00:01.0:
> > firmware: requesting iwlwifi-4965-2.ucode
> > Sep 13 10:36:43 localhost kernel: [  103.669105] iwlagn 0000:00:01.0: loaded
> > firmware version 228.61.2.24
> > Sep 13 10:36:43 localhost kernel: [  103.669263] iwlagn 0000:00:01.0: failed
> > to allocate pci memory
> > 
> > The reason seems to be that the domain does not have enough contiguous
> > memory, in mfn terms.
> 
> No, how (dis)contiguous the memory of a domain is doesn't matter
> here. What matters is whether the domain can *make* the requested
> memory contiguous, and that depends on how much contiguous
> memory Xen has at the point of the allocation.

Ah, so you are saying that regardless of whether a domain has some
contiguous memory, the driver will call xen_create_contiguous_region when allocating
memory (via dma_alloc_coherent ?).

Slightly out-of-the-list-scope: is there a convention when a driver should
allocate DMA-able memory ? Is it safe to assume that as soon as the driver
has loaded, it will no longer need to call xen_create_contiguous_region
anymore and we can use up all free Xen memory ? Particularly, would a
network card driver need to "*make* the requested memory contiguous"
whenever creating a skbuff [*] (and creating skbuffs is a frequent thing during
network card life)? 

Thank you for your answer,
regards,
Rafal Wojtczuk

[*] Admittedly, for a device with a sane MTU, skbuff should fit in a single
frame, so no issue with contiguousness here, but thinking generic.

Re: Assigning contiguous memory to a driver domain

[Rafal Wojtczuk]

On Wed, Sep 15, 2010 at 11:08:42AM +0200, Rafal Wojtczuk wrote:
> On Tue, Sep 14, 2010 at 10:36:56AM +0100, Jan Beulich wrote:
> > >>> On 14.09.10 at 11:24, Rafal Wojtczuk <rafal@invisiblethingslab.com> wrote:
> > > Hello,
> > > 
> > > Could someone guide me in the right direction with the topic of assigning
> > > contiguous memory to a domain.
> > > 
> > > I have an issue with a PV domain that is assigned a PCI device. Sometimes, 
> > > the 
> > > driver fails to load
> > > Sep 13 10:36:43 localhost kernel: [  103.651858] iwlagn 0000:00:01.0:
> > > firmware: requesting iwlwifi-4965-2.ucode
> > > Sep 13 10:36:43 localhost kernel: [  103.669105] iwlagn 0000:00:01.0: loaded
> > > firmware version 228.61.2.24
> > > Sep 13 10:36:43 localhost kernel: [  103.669263] iwlagn 0000:00:01.0: failed
> > > to allocate pci memory
> > > 
> > > The reason seems to be that the domain does not have enough contiguous
> > > memory, in mfn terms.
> > 
> > No, how (dis)contiguous the memory of a domain is doesn't matter
> > here. What matters is whether the domain can *make* the requested
> > memory contiguous, and that depends on how much contiguous
> > memory Xen has at the point of the allocation.
> 
> Ah, so you are saying that regardless of whether a domain has some
> contiguous memory, the driver will call xen_create_contiguous_region when allocating
> memory (via dma_alloc_coherent ?).
> 
> Slightly out-of-the-list-scope: is there a convention when a driver should
> allocate DMA-able memory ? Is it safe to assume that as soon as the driver
> has loaded, it will no longer need to call xen_create_contiguous_region
> anymore and we can use up all free Xen memory ? 

Hmm, at least in case of tg3 driver, if xen free memory = 0, after I have
done "ifconfig eth0 down" in the driver domain, the subsequent "ifconfig eth0 up" 
failed with "SIOCSIFFLAGS: Cannot allocate memory".

So, it looks like in order to make a PV driver domain work, there must be some Xen free memory 
all the time ? Moreover, this free Xen memory must be contiguous to some
extent; is there any way to assure this ?

Regards,
Rafal Wojtczuk

Re: Assigning contiguous memory to a driver domain

[Jan Beulich]

>>> On 15.09.10 at 11:39, Rafal Wojtczuk <rafal@invisiblethingslab.com> wrote:
> On Wed, Sep 15, 2010 at 11:08:42AM +0200, Rafal Wojtczuk wrote:
>> On Tue, Sep 14, 2010 at 10:36:56AM +0100, Jan Beulich wrote:
>> > >>> On 14.09.10 at 11:24, Rafal Wojtczuk <rafal@invisiblethingslab.com> wrote:
>> > > Hello,
>> > > 
>> > > Could someone guide me in the right direction with the topic of assigning
>> > > contiguous memory to a domain.
>> > > 
>> > > I have an issue with a PV domain that is assigned a PCI device. Sometimes, 
> 
>> > > the 
>> > > driver fails to load
>> > > Sep 13 10:36:43 localhost kernel: [  103.651858] iwlagn 0000:00:01.0:
>> > > firmware: requesting iwlwifi-4965-2.ucode
>> > > Sep 13 10:36:43 localhost kernel: [  103.669105] iwlagn 0000:00:01.0: 
> loaded
>> > > firmware version 228.61.2.24
>> > > Sep 13 10:36:43 localhost kernel: [  103.669263] iwlagn 0000:00:01.0: 
> failed
>> > > to allocate pci memory
>> > > 
>> > > The reason seems to be that the domain does not have enough contiguous
>> > > memory, in mfn terms.
>> > 
>> > No, how (dis)contiguous the memory of a domain is doesn't matter
>> > here. What matters is whether the domain can *make* the requested
>> > memory contiguous, and that depends on how much contiguous
>> > memory Xen has at the point of the allocation.
>> 
>> Ah, so you are saying that regardless of whether a domain has some
>> contiguous memory, the driver will call xen_create_contiguous_region when 
> allocating
>> memory (via dma_alloc_coherent ?).
>> 
>> Slightly out-of-the-list-scope: is there a convention when a driver should
>> allocate DMA-able memory ? Is it safe to assume that as soon as the driver
>> has loaded, it will no longer need to call xen_create_contiguous_region
>> anymore and we can use up all free Xen memory ? 
> 
> Hmm, at least in case of tg3 driver, if xen free memory = 0, after I have
> done "ifconfig eth0 down" in the driver domain, the subsequent "ifconfig 
> eth0 up" 
> failed with "SIOCSIFFLAGS: Cannot allocate memory".

Sure - why would the driver waste resources when the device may
not be used.

> So, it looks like in order to make a PV driver domain work, there must be 
> some Xen free memory 
> all the time ?

Potentially yes, but this really depends on how the respective
driver is written.

> Moreover, this free Xen memory must be contiguous to some
> extent; is there any way to assure this ?

No.

Jan

Re: Assigning contiguous memory to a driver domain

[Joanna Rutkowska]

[-- Attachment #1.1: Type: text/plain, Size: 3190 bytes --]

On 09/15/10 11:50, Jan Beulich wrote:
>>>> On 15.09.10 at 11:39, Rafal Wojtczuk <rafal@invisiblethingslab.com> wrote:
>> On Wed, Sep 15, 2010 at 11:08:42AM +0200, Rafal Wojtczuk wrote:
>>> On Tue, Sep 14, 2010 at 10:36:56AM +0100, Jan Beulich wrote:
>>>>>>> On 14.09.10 at 11:24, Rafal Wojtczuk <rafal@invisiblethingslab.com> wrote:
>>>>> Hello,
>>>>>
>>>>> Could someone guide me in the right direction with the topic of assigning
>>>>> contiguous memory to a domain.
>>>>>
>>>>> I have an issue with a PV domain that is assigned a PCI device. Sometimes, 
>>
>>>>> the 
>>>>> driver fails to load
>>>>> Sep 13 10:36:43 localhost kernel: [  103.651858] iwlagn 0000:00:01.0:
>>>>> firmware: requesting iwlwifi-4965-2.ucode
>>>>> Sep 13 10:36:43 localhost kernel: [  103.669105] iwlagn 0000:00:01.0: 
>> loaded
>>>>> firmware version 228.61.2.24
>>>>> Sep 13 10:36:43 localhost kernel: [  103.669263] iwlagn 0000:00:01.0: 
>> failed
>>>>> to allocate pci memory
>>>>>
>>>>> The reason seems to be that the domain does not have enough contiguous
>>>>> memory, in mfn terms.
>>>>
>>>> No, how (dis)contiguous the memory of a domain is doesn't matter
>>>> here. What matters is whether the domain can *make* the requested
>>>> memory contiguous, and that depends on how much contiguous
>>>> memory Xen has at the point of the allocation.
>>>
>>> Ah, so you are saying that regardless of whether a domain has some
>>> contiguous memory, the driver will call xen_create_contiguous_region when 
>> allocating
>>> memory (via dma_alloc_coherent ?).
>>>
>>> Slightly out-of-the-list-scope: is there a convention when a driver should
>>> allocate DMA-able memory ? Is it safe to assume that as soon as the driver
>>> has loaded, it will no longer need to call xen_create_contiguous_region
>>> anymore and we can use up all free Xen memory ? 
>>
>> Hmm, at least in case of tg3 driver, if xen free memory = 0, after I have
>> done "ifconfig eth0 down" in the driver domain, the subsequent "ifconfig 
>> eth0 up" 
>> failed with "SIOCSIFFLAGS: Cannot allocate memory".
> 
> Sure - why would the driver waste resources when the device may
> not be used.
> 
>> So, it looks like in order to make a PV driver domain work, there must be 
>> some Xen free memory 
>> all the time ?
> 
> Potentially yes, but this really depends on how the respective
> driver is written.
> 
>> Moreover, this free Xen memory must be contiguous to some
>> extent; is there any way to assure this ?
> 
> No.
> 

Wait! Are you saying there is no *way* to guarantee proper operation of
a driver domain in Xen?

Sure, we can tune our memory balancer to always keep some 100MB (or
200MB, or maybe 500MB?) of xen free memory, and *hope* that it will
contain enough continues pages, in case some driver in some driver
domain calls dma_alloc_coherent(), so the call will succeed.

But this is not a good solution: not only because it's a waste of memory
(I'd rather use this memory for Dom0/storage domain page cache instead)
but also, and most importantly, because I don't want to build a system
based on *hope*!

Can we do something about it?

joanna.


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 518 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Assigning contiguous memory to a driver domain

[Jan Beulich]

>>> On 15.09.10 at 12:42, Joanna Rutkowska <joanna@invisiblethingslab.com> wrote:
> On 09/15/10 11:50, Jan Beulich wrote:
>>>>> On 15.09.10 at 11:39, Rafal Wojtczuk <rafal@invisiblethingslab.com> wrote:
>>> On Wed, Sep 15, 2010 at 11:08:42AM +0200, Rafal Wojtczuk wrote:
>>>> On Tue, Sep 14, 2010 at 10:36:56AM +0100, Jan Beulich wrote:
>>>>>>>> On 14.09.10 at 11:24, Rafal Wojtczuk <rafal@invisiblethingslab.com> wrote:
>>>>>> Hello,
>>>>>>
>>>>>> Could someone guide me in the right direction with the topic of assigning
>>>>>> contiguous memory to a domain.
>>>>>>
>>>>>> I have an issue with a PV domain that is assigned a PCI device. Sometimes, 
>>>
>>>>>> the 
>>>>>> driver fails to load
>>>>>> Sep 13 10:36:43 localhost kernel: [  103.651858] iwlagn 0000:00:01.0:
>>>>>> firmware: requesting iwlwifi-4965-2.ucode
>>>>>> Sep 13 10:36:43 localhost kernel: [  103.669105] iwlagn 0000:00:01.0: 
>>> loaded
>>>>>> firmware version 228.61.2.24
>>>>>> Sep 13 10:36:43 localhost kernel: [  103.669263] iwlagn 0000:00:01.0: 
>>> failed
>>>>>> to allocate pci memory
>>>>>>
>>>>>> The reason seems to be that the domain does not have enough contiguous
>>>>>> memory, in mfn terms.
>>>>>
>>>>> No, how (dis)contiguous the memory of a domain is doesn't matter
>>>>> here. What matters is whether the domain can *make* the requested
>>>>> memory contiguous, and that depends on how much contiguous
>>>>> memory Xen has at the point of the allocation.
>>>>
>>>> Ah, so you are saying that regardless of whether a domain has some
>>>> contiguous memory, the driver will call xen_create_contiguous_region when 
>>> allocating
>>>> memory (via dma_alloc_coherent ?).
>>>>
>>>> Slightly out-of-the-list-scope: is there a convention when a driver should
>>>> allocate DMA-able memory ? Is it safe to assume that as soon as the driver
>>>> has loaded, it will no longer need to call xen_create_contiguous_region
>>>> anymore and we can use up all free Xen memory ? 
>>>
>>> Hmm, at least in case of tg3 driver, if xen free memory = 0, after I have
>>> done "ifconfig eth0 down" in the driver domain, the subsequent "ifconfig 
>>> eth0 up" 
>>> failed with "SIOCSIFFLAGS: Cannot allocate memory".
>> 
>> Sure - why would the driver waste resources when the device may
>> not be used.
>> 
>>> So, it looks like in order to make a PV driver domain work, there must be 
>>> some Xen free memory 
>>> all the time ?
>> 
>> Potentially yes, but this really depends on how the respective
>> driver is written.
>> 
>>> Moreover, this free Xen memory must be contiguous to some
>>> extent; is there any way to assure this ?
>> 
>> No.
>> 
> 
> Wait! Are you saying there is no *way* to guarantee proper operation of
> a driver domain in Xen?

No, that's to strict a statement. The admin can guarantee this. Xen
on its own can't.

> Sure, we can tune our memory balancer to always keep some 100MB (or
> 200MB, or maybe 500MB?) of xen free memory, and *hope* that it will
> contain enough continues pages, in case some driver in some driver
> domain calls dma_alloc_coherent(), so the call will succeed.

It likely doesn't need to be that much, because allocation happens
top-down. That is, Xen will try to avoid the (no longer explicitly
separate) DMA pool unless explicitly requested (or unless otherwise
not being able to fulfill a request). Therefore, if you always keep
just enough memory in Xen so that it'll never touch a reasonably
small set of pages you want to be available for DMA in guests,
these regions (the combined set of what Xen has and what was
given out as contiguous memory to the driver domain) will remain
contiguous. It'll get more problematic if you have more than one
driver domain.

Further, most drivers will prefer to allocate DMA memory once
(at startup or when enabling use of the device) rather than
repeatedly in the I/O path. In that case, you don't even need
to continuously make guarantees on available memory in Xen.

> But this is not a good solution: not only because it's a waste of memory
> (I'd rather use this memory for Dom0/storage domain page cache instead)
> but also, and most importantly, because I don't want to build a system
> based on *hope*!
> 
> Can we do something about it?

I can't easily imagine anything beyond how it's working today.

Jan

Re: Assigning contiguous memory to a driver domain

[Joanna Rutkowska]

[-- Attachment #1.1: Type: text/plain, Size: 5100 bytes --]

On 09/15/10 12:59, Jan Beulich wrote:
>>>> On 15.09.10 at 12:42, Joanna Rutkowska <joanna@invisiblethingslab.com> wrote:
>> On 09/15/10 11:50, Jan Beulich wrote:
>>>>>> On 15.09.10 at 11:39, Rafal Wojtczuk <rafal@invisiblethingslab.com> wrote:
>>>> On Wed, Sep 15, 2010 at 11:08:42AM +0200, Rafal Wojtczuk wrote:
>>>>> On Tue, Sep 14, 2010 at 10:36:56AM +0100, Jan Beulich wrote:
>>>>>>>>> On 14.09.10 at 11:24, Rafal Wojtczuk <rafal@invisiblethingslab.com> wrote:
>>>>>>> Hello,
>>>>>>>
>>>>>>> Could someone guide me in the right direction with the topic of assigning
>>>>>>> contiguous memory to a domain.
>>>>>>>
>>>>>>> I have an issue with a PV domain that is assigned a PCI device. Sometimes, 
>>>>
>>>>>>> the 
>>>>>>> driver fails to load
>>>>>>> Sep 13 10:36:43 localhost kernel: [  103.651858] iwlagn 0000:00:01.0:
>>>>>>> firmware: requesting iwlwifi-4965-2.ucode
>>>>>>> Sep 13 10:36:43 localhost kernel: [  103.669105] iwlagn 0000:00:01.0: 
>>>> loaded
>>>>>>> firmware version 228.61.2.24
>>>>>>> Sep 13 10:36:43 localhost kernel: [  103.669263] iwlagn 0000:00:01.0: 
>>>> failed
>>>>>>> to allocate pci memory
>>>>>>>
>>>>>>> The reason seems to be that the domain does not have enough contiguous
>>>>>>> memory, in mfn terms.
>>>>>>
>>>>>> No, how (dis)contiguous the memory of a domain is doesn't matter
>>>>>> here. What matters is whether the domain can *make* the requested
>>>>>> memory contiguous, and that depends on how much contiguous
>>>>>> memory Xen has at the point of the allocation.
>>>>>
>>>>> Ah, so you are saying that regardless of whether a domain has some
>>>>> contiguous memory, the driver will call xen_create_contiguous_region when 
>>>> allocating
>>>>> memory (via dma_alloc_coherent ?).
>>>>>
>>>>> Slightly out-of-the-list-scope: is there a convention when a driver should
>>>>> allocate DMA-able memory ? Is it safe to assume that as soon as the driver
>>>>> has loaded, it will no longer need to call xen_create_contiguous_region
>>>>> anymore and we can use up all free Xen memory ? 
>>>>
>>>> Hmm, at least in case of tg3 driver, if xen free memory = 0, after I have
>>>> done "ifconfig eth0 down" in the driver domain, the subsequent "ifconfig 
>>>> eth0 up" 
>>>> failed with "SIOCSIFFLAGS: Cannot allocate memory".
>>>
>>> Sure - why would the driver waste resources when the device may
>>> not be used.
>>>
>>>> So, it looks like in order to make a PV driver domain work, there must be 
>>>> some Xen free memory 
>>>> all the time ?
>>>
>>> Potentially yes, but this really depends on how the respective
>>> driver is written.
>>>
>>>> Moreover, this free Xen memory must be contiguous to some
>>>> extent; is there any way to assure this ?
>>>
>>> No.
>>>
>>
>> Wait! Are you saying there is no *way* to guarantee proper operation of
>> a driver domain in Xen?
> 
> No, that's to strict a statement. The admin can guarantee this. Xen
> on its own can't.
> 
Can they? How? Your answer below suggest it cannot be *guaranteed*...

>> Sure, we can tune our memory balancer to always keep some 100MB (or
>> 200MB, or maybe 500MB?) of xen free memory, and *hope* that it will
>> contain enough continues pages, in case some driver in some driver
>> domain calls dma_alloc_coherent(), so the call will succeed.
> 
> It likely doesn't need to be that much, because allocation happens
> top-down. That is, Xen will try to avoid the (no longer explicitly
> separate) DMA pool unless explicitly requested (or unless otherwise
> not being able to fulfill a request). Therefore, if you always keep
> just enough memory in Xen so that it'll never touch a reasonably
> small set of pages you want to be available for DMA in guests,
> these regions (the combined set of what Xen has and what was
> given out as contiguous memory to the driver domain) will remain
> contiguous. It'll get more problematic if you have more than one
> driver domain.
> 

Well, we do have 2 driver domains in Qubes: one is the netvm (NIC
drivers), and the other one is Dom0 (storage, video, etc). BTW, how is
it that the drivers in Dom0 never has this problem with allocating
continues memory (even if we keep xen free memory = 0)?

> Further, most drivers will prefer to allocate DMA memory once
> (at startup or when enabling use of the device) rather than
> repeatedly in the I/O path. In that case, you don't even need
> to continuously make guarantees on available memory in Xen.
> 

But it is not feasible to modify/ensure e.g. all the Linux networking
drivers behave like this. That would require review/patching of hundreds
of drivers, unless there is some generic way to ensure that?

joanna.

>> But this is not a good solution: not only because it's a waste of memory
>> (I'd rather use this memory for Dom0/storage domain page cache instead)
>> but also, and most importantly, because I don't want to build a system
>> based on *hope*!
>>
>> Can we do something about it?
> 
> I can't easily imagine anything beyond how it's working today.
> 
> Jan



[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 518 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Assigning contiguous memory to a driver domain

[Jan Beulich]

>>> On 15.09.10 at 13:07, Joanna Rutkowska <joanna@invisiblethingslab.com> wrote:
> On 09/15/10 12:59, Jan Beulich wrote:
>>>>> On 15.09.10 at 12:42, Joanna Rutkowska <joanna@invisiblethingslab.com> wrote:
>>> Wait! Are you saying there is no *way* to guarantee proper operation of
>>> a driver domain in Xen?
>> 
>> No, that's to strict a statement. The admin can guarantee this. Xen
>> on its own can't.
>> 
> Can they? How? Your answer below suggest it cannot be *guaranteed*...

Perhaps we have a different understanding of what "guaranteeing"
here means.

>>> Sure, we can tune our memory balancer to always keep some 100MB (or
>>> 200MB, or maybe 500MB?) of xen free memory, and *hope* that it will
>>> contain enough continues pages, in case some driver in some driver
>>> domain calls dma_alloc_coherent(), so the call will succeed.
>> 
>> It likely doesn't need to be that much, because allocation happens
>> top-down. That is, Xen will try to avoid the (no longer explicitly
>> separate) DMA pool unless explicitly requested (or unless otherwise
>> not being able to fulfill a request). Therefore, if you always keep
>> just enough memory in Xen so that it'll never touch a reasonably
>> small set of pages you want to be available for DMA in guests,
>> these regions (the combined set of what Xen has and what was
>> given out as contiguous memory to the driver domain) will remain
>> contiguous. It'll get more problematic if you have more than one
>> driver domain.
>> 
> 
> Well, we do have 2 driver domains in Qubes: one is the netvm (NIC
> drivers), and the other one is Dom0 (storage, video, etc). BTW, how is
> it that the drivers in Dom0 never has this problem with allocating
> continues memory (even if we keep xen free memory = 0)?

Merely because post-boot there would not usually be allocations.

>> Further, most drivers will prefer to allocate DMA memory once
>> (at startup or when enabling use of the device) rather than
>> repeatedly in the I/O path. In that case, you don't even need
>> to continuously make guarantees on available memory in Xen.
>> 
> 
> But it is not feasible to modify/ensure e.g. all the Linux networking
> drivers behave like this. That would require review/patching of hundreds
> of drivers, unless there is some generic way to ensure that?

The fact that in Dom0 things work flawlessly with most drivers
(and yes, I know of occasional exceptions, though usually it's a
shortage of swiotlb space that we get reported, not one of Xen
memory) speaks for itself I think. But yes, there is a level of
uncertainty which, afaict, can be overcome only by auditing
individual drivers (or groups of them if they share a common
framework).

And you certainly realize that there are other things some
drivers do that don't work under Xen without modification -
these cases also only get handled as people run into them.
When a pattern results from the necessary changes, one
can certainly go through the rest of the tree to find other
instances needing fixing. But that only helps until the next
such issue gets found (or introduced anew).

Jan

Re: Assigning contiguous memory to a driver domain

[Goswin von Brederlow]

"Jan Beulich" <JBeulich@novell.com> writes:

>> Well, we do have 2 driver domains in Qubes: one is the netvm (NIC
>> drivers), and the other one is Dom0 (storage, video, etc). BTW, how is
>> it that the drivers in Dom0 never has this problem with allocating
>> continues memory (even if we keep xen free memory = 0)?
>
> Merely because post-boot there would not usually be allocations.

It is also called fragmentation.

dom0 gets its memory at the start and probably all in one big
chunk. Other domains have to use the bits and pices that happen to be
free when the domain is started.

MfG
        Goswin

Re: Assigning contiguous memory to a driver domain

[Rafal Wojtczuk]

On Wed, Sep 15, 2010 at 01:58:30PM +0200, Goswin von Brederlow wrote:
> "Jan Beulich" <JBeulich@novell.com> writes:
> 
> >> Well, we do have 2 driver domains in Qubes: one is the netvm (NIC
> >> drivers), and the other one is Dom0 (storage, video, etc). BTW, how is
> >> it that the drivers in Dom0 never has this problem with allocating
> >> continues memory (even if we keep xen free memory = 0)?
> >
> > Merely because post-boot there would not usually be allocations.
> 
> It is also called fragmentation.
> 
> dom0 gets its memory at the start and probably all in one big
> chunk. Other domains have to use the bits and pices that happen to be
> free when the domain is started.

It does not look right. Jan has already written
* No, how (dis)contiguous the memory of a domain is doesn't matter
* here. What matters is whether the domain can *make* the requested
* memory contiguous, and that depends on how much contiguous
* memory Xen has at the point of the allocation.

> > Merely because post-boot there would not usually be allocations.
It is not too convincing, either. Firstly, assuming there is no dom0mem
parameter to Xen, dom0 gets all memory initially, so when dom0 boots the
first time, Xen free memory = 0 ? And drivers load fine.
Secondly, by default on Fedora, the pm-utils scripts tell NetworkManager to
turn off networking just before entering S3 sleep, and NetworkManager downs all
interfaces. It brings them up on resume. And apparently there are some
drivers, like tg3, that needs to alloc when bringing up the device. Still,
suspend/resume works when all devices are assigned to dom0.

Is it possible that swiotlb layer works differently in dom0 case ? Swiotlb
is switched on by default in dom0 case, is it correct ?

Regards,
Rafal Wojtczuk

Re: Assigning contiguous memory to a driver domain

[Jan Beulich]

>>> On 15.09.10 at 14:06, Rafal Wojtczuk <rafal@invisiblethingslab.com> wrote:
> It is not too convincing, either. Firstly, assuming there is no dom0mem
> parameter to Xen, dom0 gets all memory initially, so when dom0 boots the
> first time, Xen free memory = 0 ? And drivers load fine.

Xen reserves 1/16th of the memory (up to 128Mb) in this case for this
very purpose. Otherwise, Dom0 would neither be able to set up its
swiotlb nor would any coherent/consistent allocations work.

> Secondly, by default on Fedora, the pm-utils scripts tell NetworkManager to
> turn off networking just before entering S3 sleep, and NetworkManager downs 
> all
> interfaces. It brings them up on resume. And apparently there are some
> drivers, like tg3, that needs to alloc when bringing up the device. Still,
> suspend/resume works when all devices are assigned to dom0.

Because on suspend the driver frees the memory which on resume
it will allocate back?

> Is it possible that swiotlb layer works differently in dom0 case ? Swiotlb
> is switched on by default in dom0 case, is it correct ?

Of course.

Jan

Re: Assigning contiguous memory to a driver domain

[Rafal Wojtczuk]

On Wed, Sep 15, 2010 at 02:49:37PM +0100, Jan Beulich wrote:
> >>> On 15.09.10 at 14:06, Rafal Wojtczuk <rafal@invisiblethingslab.com> wrote:
> > It is not too convincing, either. Firstly, assuming there is no dom0mem
> > parameter to Xen, dom0 gets all memory initially, so when dom0 boots the
> > first time, Xen free memory = 0 ? And drivers load fine.
> 
> Xen reserves 1/16th of the memory (up to 128Mb) in this case for this
> very purpose. Otherwise, Dom0 would neither be able to set up its
> swiotlb nor would any coherent/consistent allocations work.
I see.

> > Secondly, by default on Fedora, the pm-utils scripts tell NetworkManager to
> > turn off networking just before entering S3 sleep, and NetworkManager downs 
> > all
> > interfaces. It brings them up on resume. And apparently there are some
> > drivers, like tg3, that needs to alloc when bringing up the device. Still,
> > suspend/resume works when all devices are assigned to dom0.
> 
> Because on suspend the driver frees the memory which on resume
> it will allocate back?
I am a bit lost.
By "frees the memory" you mean "return contiguous memory to Xen free memory" ?
Does it really work this way ?
If so, it requires nonzero Xen free memory ? And that is why when I do
"ifconfig eth0 down; ifconfig eth0 up" in the driver domain the second one
fails ?

Regards,
Rafal Wojtczuk

Re: Assigning contiguous memory to a driver domain

[Jan Beulich]

>>> On 15.09.10 at 16:44, Rafal Wojtczuk <rafal@invisiblethingslab.com> wrote:
> On Wed, Sep 15, 2010 at 02:49:37PM +0100, Jan Beulich wrote:
>> Because on suspend the driver frees the memory which on resume
>> it will allocate back?
> I am a bit lost.
> By "frees the memory" you mean "return contiguous memory to Xen free memory" 
> ?
> Does it really work this way ?

Yes - the "special" memory gets exchanged back to "normal" memory
upon freeing of it by the driver. The exception is if Xen has no "normal"
memory left to give back out in exchange - in that case the domain will
retain the "special" memory indefinitely. Yes, you can call this a leak,
but no, I don't think there's much you can do about it (without adding
likely rather complex extra code).

> If so, it requires nonzero Xen free memory ? And that is why when I do
> "ifconfig eth0 down; ifconfig eth0 up" in the driver domain the second one
> fails ?

Generally the second "up" shouldn't fail as long as the prior "down"
properly returned all resources. See the restrictions above.

Jan

Re: Assigning contiguous memory to a driver domain

[Goswin von Brederlow]

Rafal Wojtczuk <rafal@invisiblethingslab.com> writes:

> On Wed, Sep 15, 2010 at 01:58:30PM +0200, Goswin von Brederlow wrote:
>> "Jan Beulich" <JBeulich@novell.com> writes:
>> 
>> >> Well, we do have 2 driver domains in Qubes: one is the netvm (NIC
>> >> drivers), and the other one is Dom0 (storage, video, etc). BTW, how is
>> >> it that the drivers in Dom0 never has this problem with allocating
>> >> continues memory (even if we keep xen free memory = 0)?
>> >
>> > Merely because post-boot there would not usually be allocations.
>> 
>> It is also called fragmentation.
>> 
>> dom0 gets its memory at the start and probably all in one big
>> chunk. Other domains have to use the bits and pices that happen to be
>> free when the domain is started.
>
> It does not look right. Jan has already written
> * No, how (dis)contiguous the memory of a domain is doesn't matter
> * here. What matters is whether the domain can *make* the requested
> * memory contiguous, and that depends on how much contiguous
> * memory Xen has at the point of the allocation.

But that is exactly the point. By the time the later domains start the
remaining memory is fragmented and there just might not be a physical
continious chunk of memory there that XEN can give the domain.

The more memory is free the greater the chance that a sufficient chunk
exists. But that is just a probability game. In the worst case you can
have 50% free and no chunk larger than a page.

MfG
        Goswin

Re: Assigning contiguous memory to a driver domain

[Konrad Rzeszutek Wilk]

On Wed, Sep 15, 2010 at 04:29:28PM +0100, Jan Beulich wrote:
> >>> On 15.09.10 at 16:44, Rafal Wojtczuk <rafal@invisiblethingslab.com> wrote:
> > On Wed, Sep 15, 2010 at 02:49:37PM +0100, Jan Beulich wrote:
> >> Because on suspend the driver frees the memory which on resume
> >> it will allocate back?
> > I am a bit lost.
> > By "frees the memory" you mean "return contiguous memory to Xen free memory" 
> > ?
> > Does it really work this way ?
> 
> Yes - the "special" memory gets exchanged back to "normal" memory
> upon freeing of it by the driver. The exception is if Xen has no "normal"
> memory left to give back out in exchange - in that case the domain will
> retain the "special" memory indefinitely. Yes, you can call this a leak,
> but no, I don't think there's much you can do about it (without adding
> likely rather complex extra code).

Let me expand this. During bootup Xen-SWIOTLB  (which for DomU you have
to enable via the 'iommu=soft'), allocated 32 2MB chunks of contingous
memory under the 4GB limit. Those chunks stay in DomU and are used
during the the runtime of the DomU. They don't go back to Xen unless the
domain has been terminated. Any of the DMA operations that any driver
does go through the SWIOTLB bufer if the physical (mfn) for the DMA is
outside the driver capabilities (say, your ping buffer is allocated above
the 4GB, and your r8169 can only do 32-bit, then SWIOTLB would be utilized
to "bounce" the memory).


> 
> > If so, it requires nonzero Xen free memory ? And that is why when I do
> > "ifconfig eth0 down; ifconfig eth0 up" in the driver domain the second one
> > fails ?

There are couple of things happening when you do ifconfig eth0 up. The 
PTE used for virtual address for the BARs are updated with the _PAGE_IOMAP
which means that the GMFN->PFN->MFN is shortcircuited to be GMFN->MFN.
Obviously that doesn' use any Xen heap memory. The next thing is
that the the driver might allocate coherent DMA mappings. Those are the
ones I think Jan is referring to. For coherent DMA mappings we just
do page_alloc and then we swap the memory behind those pages with Xen
to be under the 32-bit limit (xen_create_contiguous_region).
Naturally when the driver is unloaded the de-allocation will call
xen_destroy_contiguous_region. Loking at the code I think it swaps with
the highest bit order (so with memory close to the end of physical space).


> 
> Generally the second "up" shouldn't fail as long as the prior "down"
> properly returned all resources. See the restrictions above.

Yeah, it might be worth looking at what it is doing to cause this. The e1000/igb
are pretty good at cleaning everying so you can do ifup;ifdown indefinitly.

In reference to the Xen-SWIOTLB for other versions that upstream, there are a couple
of implementations at:
git://git.kernel.org/pub/scm/linux/kernel/git/konrad/swiotlb-2.6.git

for different Linux versions. Which version of kernel do you guys use?
> 
> Jan
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: Assigning contiguous memory to a driver domain

[Jeremy Fitzhardinge]

 On 09/20/2010 12:48 PM, Konrad Rzeszutek Wilk wrote:
> Let me expand this. During bootup Xen-SWIOTLB  (which for DomU you have
> to enable via the 'iommu=soft'), allocated 32 2MB chunks of contingous
> memory under the 4GB limit. Those chunks stay in DomU and are used
> during the the runtime of the DomU. They don't go back to Xen unless the
> domain has been terminated. Any of the DMA operations that any driver
> does go through the SWIOTLB bufer if the physical (mfn) for the DMA is
> outside the driver capabilities (say, your ping buffer is allocated above
> the 4GB, and your r8169 can only do 32-bit, then SWIOTLB would be utilized
> to "bounce" the memory).

BTW, are there any hooks to make sure these pages are still contiguous
after migration/restore?

    J

Re: Assigning contiguous memory to a driver domain

[Konrad Rzeszutek Wilk]

On Mon, Sep 20, 2010 at 01:27:24PM -0700, Jeremy Fitzhardinge wrote:
>  On 09/20/2010 12:48 PM, Konrad Rzeszutek Wilk wrote:
> > Let me expand this. During bootup Xen-SWIOTLB  (which for DomU you have
> > to enable via the 'iommu=soft'), allocated 32 2MB chunks of contingous
> > memory under the 4GB limit. Those chunks stay in DomU and are used
> > during the the runtime of the DomU. They don't go back to Xen unless the
> > domain has been terminated. Any of the DMA operations that any driver
> > does go through the SWIOTLB bufer if the physical (mfn) for the DMA is
> > outside the driver capabilities (say, your ping buffer is allocated above
> > the 4GB, and your r8169 can only do 32-bit, then SWIOTLB would be utilized
> > to "bounce" the memory).
> 
> BTW, are there any hooks to make sure these pages are still contiguous
> after migration/restore?

<shudders>I don't really know. We do save the P2M map, so those mappings
are saved in the guest. But for the MFNs that have been exchanged - I don't
believe the hypervior is notified about which MFNs have been transfered
to another domain? Or these MFNs extracted during save/resume?

Re: Assigning contiguous memory to a driver domain

[Jeremy Fitzhardinge]

 On 09/20/2010 02:41 PM, Konrad Rzeszutek Wilk wrote:
> On Mon, Sep 20, 2010 at 01:27:24PM -0700, Jeremy Fitzhardinge wrote:
>>  On 09/20/2010 12:48 PM, Konrad Rzeszutek Wilk wrote:
>>> Let me expand this. During bootup Xen-SWIOTLB  (which for DomU you have
>>> to enable via the 'iommu=soft'), allocated 32 2MB chunks of contingous
>>> memory under the 4GB limit. Those chunks stay in DomU and are used
>>> during the the runtime of the DomU. They don't go back to Xen unless the
>>> domain has been terminated. Any of the DMA operations that any driver
>>> does go through the SWIOTLB bufer if the physical (mfn) for the DMA is
>>> outside the driver capabilities (say, your ping buffer is allocated above
>>> the 4GB, and your r8169 can only do 32-bit, then SWIOTLB would be utilized
>>> to "bounce" the memory).
>> BTW, are there any hooks to make sure these pages are still contiguous
>> after migration/restore?
> <shudders>I don't really know. We do save the P2M map, so those mappings
> are saved in the guest. But for the MFNs that have been exchanged - I don't
> believe the hypervior is notified about which MFNs have been transfered
> to another domain? Or these MFNs extracted during save/resume

On save the toolstack will rewrite all the mfns in the pagetables and
the p2m map to canonical pfn form, then rewrite them back to mfns on
resume when it has allocated pages for them all.  I don't believe
there's any mechanism to remember what regions were mfn-contiguous and
restore that on resume.

    J

RE: Assigning contiguous memory to a driver domain

[Dan Magenheimer]

>  On 09/20/2010 02:41 PM, Konrad Rzeszutek Wilk wrote:
> > On Mon, Sep 20, 2010 at 01:27:24PM -0700, Jeremy Fitzhardinge wrote:
> >>  On 09/20/2010 12:48 PM, Konrad Rzeszutek Wilk wrote:
> >>> Let me expand this. During bootup Xen-SWIOTLB  (which for DomU you
> have
> >>> to enable via the 'iommu=soft'), allocated 32 2MB chunks of
> contingous
> >>> memory under the 4GB limit. Those chunks stay in DomU and are used
> >>> during the the runtime of the DomU. They don't go back to Xen
> unless the
> >>> domain has been terminated. Any of the DMA operations that any
> driver
> >>> does go through the SWIOTLB bufer if the physical (mfn) for the DMA
> is
> >>> outside the driver capabilities (say, your ping buffer is allocated
> above
> >>> the 4GB, and your r8169 can only do 32-bit, then SWIOTLB would be
> utilized
> >>> to "bounce" the memory).
> >> BTW, are there any hooks to make sure these pages are still
> contiguous
> >> after migration/restore?
> > <shudders>I don't really know. We do save the P2M map, so those
> mappings
> > are saved in the guest. But for the MFNs that have been exchanged - I
> don't
> > believe the hypervior is notified about which MFNs have been
> transfered
> > to another domain? Or these MFNs extracted during save/resume
> 
> On save the toolstack will rewrite all the mfns in the pagetables and
> the p2m map to canonical pfn form, then rewrite them back to mfns on
> resume when it has allocated pages for them all.  I don't believe
> there's any mechanism to remember what regions were mfn-contiguous and
> restore that on resume.

I may be too far behind on this thread, but aren't we only talking about
driver domains with passthrough?   Is it even possible to migrate
driver domains yet?

Re: Assigning contiguous memory to a driver domain

[Jeremy Fitzhardinge]

 On 09/20/2010 04:42 PM, Dan Magenheimer wrote:
>>  On 09/20/2010 02:41 PM, Konrad Rzeszutek Wilk wrote:
>>> On Mon, Sep 20, 2010 at 01:27:24PM -0700, Jeremy Fitzhardinge wrote:
>>>>  On 09/20/2010 12:48 PM, Konrad Rzeszutek Wilk wrote:
>>>>> Let me expand this. During bootup Xen-SWIOTLB  (which for DomU you
>> have
>>>>> to enable via the 'iommu=soft'), allocated 32 2MB chunks of
>> contingous
>>>>> memory under the 4GB limit. Those chunks stay in DomU and are used
>>>>> during the the runtime of the DomU. They don't go back to Xen
>> unless the
>>>>> domain has been terminated. Any of the DMA operations that any
>> driver
>>>>> does go through the SWIOTLB bufer if the physical (mfn) for the DMA
>> is
>>>>> outside the driver capabilities (say, your ping buffer is allocated
>> above
>>>>> the 4GB, and your r8169 can only do 32-bit, then SWIOTLB would be
>> utilized
>>>>> to "bounce" the memory).
>>>> BTW, are there any hooks to make sure these pages are still
>> contiguous
>>>> after migration/restore?
>>> <shudders>I don't really know. We do save the P2M map, so those
>> mappings
>>> are saved in the guest. But for the MFNs that have been exchanged - I
>> don't
>>> believe the hypervior is notified about which MFNs have been
>> transfered
>>> to another domain? Or these MFNs extracted during save/resume
>> On save the toolstack will rewrite all the mfns in the pagetables and
>> the p2m map to canonical pfn form, then rewrite them back to mfns on
>> resume when it has allocated pages for them all.  I don't believe
>> there's any mechanism to remember what regions were mfn-contiguous and
>> restore that on resume.
> I may be too far behind on this thread, but aren't we only talking about
> driver domains with passthrough?   Is it even possible to migrate
> driver domains yet?

Not with devices attached.  But there should be no problem in principle
with detaching all the devices, migrating, the reattaching everything.

    J

Re: Assigning contiguous memory to a driver domain

[Tim Deegan]

At 01:45 +0100 on 21 Sep (1285033549), Jeremy Fitzhardinge wrote:
> > I may be too far behind on this thread, but aren't we only talking about
> > driver domains with passthrough?   Is it even possible to migrate
> > driver domains yet?
> 
> Not with devices attached.  But there should be no problem in principle
> with detaching all the devices, migrating, the reattaching everything.

But would you want to?  :)

If you did, you'd have to have a hook post-migrate for the kernel to try
and reconstitute contiguous areas that it cares about.  At the moment
there's no way for the tools to know whether PFNs are contiguous by
chance or on purpose.

Tim.

-- 
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, XenServer Engineering
Citrix Systems UK Ltd.  (Company #02937203, SL9 0BG)

Re: Assigning contiguous memory to a driver domain

[Joanna Rutkowska]

[-- Attachment #1.1: Type: text/plain, Size: 2291 bytes --]

On 09/20/10 21:48, Konrad Rzeszutek Wilk wrote:

>>> If so, it requires nonzero Xen free memory ? And that is why when I do
>>> "ifconfig eth0 down; ifconfig eth0 up" in the driver domain the second one
>>> fails ?
> 
> There are couple of things happening when you do ifconfig eth0 up. The 
> PTE used for virtual address for the BARs are updated with the _PAGE_IOMAP
> which means that the GMFN->PFN->MFN is shortcircuited to be GMFN->MFN.
> Obviously that doesn' use any Xen heap memory. The next thing is
> that the the driver might allocate coherent DMA mappings. Those are the
> ones I think Jan is referring to. For coherent DMA mappings we just
> do page_alloc and then we swap the memory behind those pages with Xen
> to be under the 32-bit limit (xen_create_contiguous_region).
> Naturally when the driver is unloaded the de-allocation will call
> xen_destroy_contiguous_region. Loking at the code I think it swaps with
> the highest bit order (so with memory close to the end of physical space).
> 
> 

A coherent DMA mapping == continues mfns, right?

So, is there a way to assure that this page_alloc for coherent DMA
mapping *always* succeeds for a given domain, assuming it succeeded at
least once (at its startup)?

>>
>> Generally the second "up" shouldn't fail as long as the prior "down"
>> properly returned all resources. See the restrictions above.
> 
> Yeah, it might be worth looking at what it is doing to cause this. The e1000/igb
> are pretty good at cleaning everying so you can do ifup;ifdown indefinitly.
> 

But if they are so good at cleaning everything as you say, then wouldn't
that mean they are giving back the continues mfns back to Xen, which
makes it possible that they will no longer be available when we do ifup
next time (because e.g. some other drv domain will use them this time)?

> In reference to the Xen-SWIOTLB for other versions that upstream, there are a couple
> of implementations at:
> git://git.kernel.org/pub/scm/linux/kernel/git/konrad/swiotlb-2.6.git
> 
> for different Linux versions. Which version of kernel do you guys use?

We use 2.6.34.1 with OpenSuse xenlinux pacthes (sorry guys, we decided
to switch to xenlinux some time ago for better suspend and DRM).

Thanks,
joanna.


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 518 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Assigning contiguous memory to a driver domain

[Konrad Rzeszutek Wilk]

On Tue, Sep 21, 2010 at 01:28:30PM +0200, Joanna Rutkowska wrote:
> On 09/20/10 21:48, Konrad Rzeszutek Wilk wrote:
> 
> >>> If so, it requires nonzero Xen free memory ? And that is why when I do
> >>> "ifconfig eth0 down; ifconfig eth0 up" in the driver domain the second one
> >>> fails ?
> > 
> > There are couple of things happening when you do ifconfig eth0 up. The 
> > PTE used for virtual address for the BARs are updated with the _PAGE_IOMAP
> > which means that the GMFN->PFN->MFN is shortcircuited to be GMFN->MFN.
> > Obviously that doesn' use any Xen heap memory. The next thing is
> > that the the driver might allocate coherent DMA mappings. Those are the
> > ones I think Jan is referring to. For coherent DMA mappings we just
> > do page_alloc and then we swap the memory behind those pages with Xen
> > to be under the 32-bit limit (xen_create_contiguous_region).
> > Naturally when the driver is unloaded the de-allocation will call
> > xen_destroy_contiguous_region. Loking at the code I think it swaps with
> > the highest bit order (so with memory close to the end of physical space).
> > 
> > 
> 
> A coherent DMA mapping == continues mfns, right?

>From a simple standpoint - yes. If you dig in deeper it is important on Alpha
platforms due to aliasing, but on X86 it really does not matter.

> 
> So, is there a way to assure that this page_alloc for coherent DMA
> mapping *always* succeeds for a given domain, assuming it succeeded at
> least once (at its startup)?

No. But the driver need not have to use the coherent DMA API calls.
As a matter of fact, I am not sure if doing 'ifdown' would release
all of the coherent DMA APIs. It really depends on how the driver
was written.

> 
> >>
> >> Generally the second "up" shouldn't fail as long as the prior "down"
> >> properly returned all resources. See the restrictions above.
> > 
> > Yeah, it might be worth looking at what it is doing to cause this. The e1000/igb
> > are pretty good at cleaning everying so you can do ifup;ifdown indefinitly.
> > 
> 
> But if they are so good at cleaning everything as you say, then wouldn't
> that mean they are giving back the continues mfns back to Xen, which
> makes it possible that they will no longer be available when we do ifup
> next time (because e.g. some other drv domain will use them this time)?

No. They might have no need for coherent DMA mapping and instead just
call pci_map_page whenever they need. The pci_map_page takes care of
all of the intricate details of assuring that the page is visible by
PCI bus and that it can do DMA operations on it.

There are two types of operations here. One is the pci_alloc_coherent
where when you are done you return back the buffers (which makes a hypercall).
The other is to use pci_map_page, which can use the pool of contingous MFNs
that SWIOTLB has allocated - that pool is not returned to Xen unless the domain
is terminated.

That SWIOTLB buffer is 64MB and is static - it does not grow nor shrink during
the lifetime of the guest.
> 
> > In reference to the Xen-SWIOTLB for other versions that upstream, there are a couple
> > of implementations at:
> > git://git.kernel.org/pub/scm/linux/kernel/git/konrad/swiotlb-2.6.git
> > 
> > for different Linux versions. Which version of kernel do you guys use?
> 
> We use 2.6.34.1 with OpenSuse xenlinux pacthes (sorry guys, we decided
> to switch to xenlinux some time ago for better suspend and DRM).

Oh, sad to hear that as most of the DRM stuff has/is fixed now. This is the list
of hardware I've had a chance to test:

http://wiki.xensource.com/xenwiki/XenPVOPSDRM

Thought the suspend path needs a bit of loving (I've had problems on AMD 
but not that much on Intel). Which hardware did you have trouble with?