dma_mapping_ops for i386

dma_mapping_ops for i386

[Jeremy Fitzhardinge]

I'm looking at adding dom0 support to the pv-ops kernel.  One of the 
obvious things we need is to support real device drivers, and the 
associated p->m translations for devices.

I'm thinking the cleanest thing to do is make x86-64's dma-mapping.h 
with its dma_mapping_ops common to i386 and x86-64, so we can hook the 
Xen translations in there.  Presumably we'll need to do this anyway to 
support VTd for 32-bit (but I don't know if that's a reasonable thing to 
do anyway).

What do you think?

    J

Re: dma_mapping_ops for i386

[Andi Kleen]

On Tuesday 26 June 2007 21:59, Jeremy Fitzhardinge wrote:
> I'm looking at adding dom0 support to the pv-ops kernel.  One of the
> obvious things we need is to support real device drivers, and the
> associated p->m translations for devices.
>
> I'm thinking the cleanest thing to do is make x86-64's dma-mapping.h
> with its dma_mapping_ops common to i386 and x86-64, so we can hook the
> Xen translations in there.  Presumably we'll need to do this anyway to
> support VTd for 32-bit (but I don't know if that's a reasonable thing to
> do anyway).
>
> What do you think?

Ok, if you can do it without ifdefs. 

And no swiotlb on i386; that is something that is completely broken
in upstream Xen and needs to be fixed properly anyways.

-Andi

Re: dma_mapping_ops for i386

[Jeremy Fitzhardinge]

Andi Kleen wrote:
> Ok, if you can do it without ifdefs. 
>   

That should be OK.  All the existing i386 mapping operations would just 
have their own ops structure, right?

> And no swiotlb on i386; that is something that is completely broken
> in upstream Xen and needs to be fixed properly anyways.
>   

Hm, OK.  I'm not really familiar with the issues here.  What are they?  
Looks like Jan has made a number of Xen-ish changes to lib/swiotlb.c; 
are more changes be needed?

    J

Re: dma_mapping_ops for i386

[Andi Kleen]

On Wednesday 27 June 2007 16:15:17 Jeremy Fitzhardinge wrote:
> Andi Kleen wrote:
> > Ok, if you can do it without ifdefs. 
> >   
> 
> That should be OK.  All the existing i386 mapping operations would just 
> have their own ops structure, right?

I just mention it because many people's ideas of merging files
seem to add lots of ifdefs which is imho the totally wrong thing
to do.

> > And no swiotlb on i386; that is something that is completely broken
> > in upstream Xen and needs to be fixed properly anyways.
> >   
> 
> Hm, OK.  I'm not really familiar with the issues here.  What are they?  
> Looks like Jan has made a number of Xen-ish changes to lib/swiotlb.c; 
> are more changes be needed?

See the recent "quiet down swiotlb warnings" thread which uncovered
quite some corpses in Xen's current IO setup.

Xen apparently bounces for multi page IOs which get merged from block
lists because the block layer doesn't know they are not really 
continuous in machine memory.

Proper fix is to tell the block layer to not merge in the first
place instead.

And probably some similar mechanism for network drivers that limits
MTUs.

Maybe we'll still need a simple bouncing mechanism for other obscure
devices with large IOs then, but I would very much prefer if it wasn't 
swiotlb and could be solved some other way.

-Andi

Re: dma_mapping_ops for i386

[Jeremy Fitzhardinge]

Andi Kleen wrote:
> On Wednesday 27 June 2007 16:15:17 Jeremy Fitzhardinge wrote:
>   
>> Andi Kleen wrote:
>>     
>>> Ok, if you can do it without ifdefs. 
>>>   
>>>       
>> That should be OK.  All the existing i386 mapping operations would just 
>> have their own ops structure, right?
>>     
>
> I just mention it because many people's ideas of merging files
> seem to add lots of ifdefs which is imho the totally wrong thing
> to do.
>
>   
>>> And no swiotlb on i386; that is something that is completely broken
>>> in upstream Xen and needs to be fixed properly anyways.
>>>   
>>>       
>> Hm, OK.  I'm not really familiar with the issues here.  What are they?  
>> Looks like Jan has made a number of Xen-ish changes to lib/swiotlb.c; 
>> are more changes be needed?
>>     
>
> See the recent "quiet down swiotlb warnings" thread which uncovered
> quite some corpses in Xen's current IO setup.
>
> Xen apparently bounces for multi page IOs which get merged from block
> lists because the block layer doesn't know they are not really 
> continuous in machine memory.
>
> Proper fix is to tell the block layer to not merge in the first
> place instead.
>
> And probably some similar mechanism for network drivers that limits
> MTUs.
>   

Well, I think there are two issues here.  One is that two 
pseudo-physical pages won't necessarily be contigious in bus space, 
because of the pseudo-phys to machine mapping.

The second problem is that devices which can't address all machine 
physical memory (ie, 32-bit PCI devices on machines with >4G memory) 
will need to have bouncebuffers established for them.  Device drivers 
won't necessarily be able to do it because they're not really aware of 
machine addresses.

> Maybe we'll still need a simple bouncing mechanism for other obscure
> devices with large IOs then, but I would very much prefer if it wasn't 
> swiotlb and could be solved some other way.
>   

I think 32-bit-only devices are a bigger concern, no?

    J

Re: dma_mapping_ops for i386

[Muli Ben-Yehuda]

On Wed, Jun 27, 2007 at 05:26:50PM +0200, Andi Kleen wrote:

> See the recent "quiet down swiotlb warnings" thread which uncovered
> quite some corpses in Xen's current IO setup.
> 
> Xen apparently bounces for multi page IOs which get merged from block
> lists because the block layer doesn't know they are not really 
> continuous in machine memory.
> 
> Proper fix is to tell the block layer to not merge in the first
> place instead.
>
> And probably some similar mechanism for network drivers that limits
> MTUs.

Will that guarantee that block and net IOs will not straddle a page
boundary?

Cheers,
Muli

Re: dma_mapping_ops for i386

[Herbert Xu]

Muli Ben-Yehuda <muli@il.ibm.com> wrote:
>
>> And probably some similar mechanism for network drivers that limits
>> MTUs.
> 
> Will that guarantee that block and net IOs will not straddle a page
> boundary?

Mostly.  There is the thorny case of slab debugging that breaks
these nice assumptions.

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt